SciCrafter: Can Current Language Models
Close the Discovery-to-Application Loop?

Abstract

Overview: Decomposing the Discovery-to-Application Loop

We evaluate state-of-the-art language models on tasks requiring both scientific discovery and engineering application. By providing targeted interventions, we isolate where models fail in the loop.

Figure 1. Decomposing performance gaps in the Discovery-to-Application loop (Gemini 3 Pro as example). The best current model achieves only 25.0% success rate. We isolate four bottlenecks: providing oracle guidance on what to discover boosts success to 58% (Knowledge Identification, +33%); deploying a scientist sub-agent further reaches 69% (Experimental Discovery, +11%); structured knowledge templates outperform model-directed synthesis (Knowledge Consolidation, +9%); the remaining 31% gap represents the ceiling for knowledge application capacity.

Task Design

Task Schema

SciCrafter adopts a simple yet expressive task schema: ignite $N$ lamps in specified patterns (e.g., simultaneously, or following a delay sequence $[t_1, t_2, \ldots, t_n]$) within a fixed area. This design ensures evaluation remains invariant across difficulty levels while the knowledge gap to meet requirements consistently grows—forcing agents to discover new environmental mechanics rather than memorize solutions.

Figure 2. SciCrafter Task Design Illustration. Top: The model constructs a functional device within a constrained area based on instructions, iteratively interacting with the device (e.g., pressing a button) and observing behavior. The device is then evaluated by an automated script verifying output lighting patterns. Bottom: Task complexity scales parametrically by adjusting the number of lights ($N$). For some temporal tasks, difficulty further increases with sequential pattern parameters.

Five Task Families

Each task family probes distinct spatial and temporal constraints, with five difficulty levels (L1–L5) that cross discrete mechanism thresholds:

Knowledge Dimensions

Higher difficulty levels cross discrete mechanism thresholds requiring discovery across three core knowledge dimensions:

Task Family Parameters

Table 1. Task family parameters. All families scale the number of output lamps as $N \in \{4, 8, 16, 32, 64\}$ for L1–L5, while introducing family-specific spatial/temporal constraints.

Task Examples

Demonstrations of an agent constructing functional redstone devices and the automated evaluation verifying the output lighting patterns.

Main Results

Baseline Performance

We evaluate 8 state-of-the-art models using Claude Code as the agent framework with a budget of 50 verification trials per task. Results are averaged over 8 runs.

Baseline success rates (%, Curriculum setting). Even the best model achieves only 26% — despite parameter counts ranging from 32B to an estimated 1.7T.

Capacity Gap Decomposition

We isolate each capacity gap through sequential ablations. The table below shows how each intervention progressively improves performance, and the corresponding gap magnitudes:

Table 2. Model Performance and Capacity Gap Decomposition (Curriculum Setting). Gap columns (gray) show marginal gains ($\delta$) and relative ratios ($r_\delta$). For frontier models, the knowledge identification gap ($\sim$1.0×) nearly equals baseline; for weaker models, the application gap balloons (up to 5.10×).

Capacity Gap Visualization

Capacity gap decomposition across models (Curriculum setting). For frontier models, the identification gap (blue) roughly equals baseline performance. For weaker models, application capacity dominates the failure.

Knowledge Consolidation Ablation

Table 3. Knowledge Consolidation Structure Ablation (Gemini 3 Pro, 8 runs). Unstructured summarization reaches only 58.0%, above hints alone (52.5%). The Claim-Proof-Constraints-Example format performs best at 64.0%.

Per-Level Breakdown (Gemini-3-Pro)

Table 4. Per-level performance breakdown (Gemini-3-Pro, Curriculum setting, 8 runs). Baseline performance drops sharply from L3 onward. Even with full support (Hint + Scientist), no model succeeds at Level 5, leaving it open for future progress.

Failure Analysis

To ground our quantitative decomposition in concrete agent behavior, we constructed 12 device variants of the 32-lamp broadcast task, each exhibiting a distinct failure mode. These failures cluster into three categories that align with our capacity decomposition. Below are five representative cases—press play to see the button activation and which lamps light up (or fail to).

Working Reference vs. Failure Cases

Failure Taxonomy

The 12 failure cases span a progression from obvious structural errors to subtle semantic misunderstandings, mirroring the capacity gap hierarchy from our quantitative analysis:

Key insight: Category 3 (wire semantics) failures are the hardest to diagnose—circuits carry full signal power yet deliver zero to lamps. Even frontier models (Gemini-3-Pro, GPT-5.2) produce these failures, as they require recognizing that wire connection direction determines which blocks receive power—a mechanic not reliably captured in training data.

Citation

If you find SciCrafter useful in your research, please consider citing:

@article{scicrafter2025,
  title   = {SciCrafter: Can Current Language Models Close
             the Discovery-to-Application Loop?},
  author  = {Anonymous},
  journal = {Under Review},
  year    = {2025}
}