Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3

Abstract: A generally intelligent system acquires new skills efficiently without requiring humans to redesign its software scaffold for every new task. Yet LLMs rarely act alone. A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf. ARC-AGI-3 is an interactive benchmark in which agents must explore unfamiliar environments, figure out their goals on the fly, and keep learning as they go. On it, human-steered systems already report ceiling scores across the public games, so we know effective designs exist. We ask whether frontier models can build and improve such a harness on their own. Through a fixed coding bootstrap, each model constructs a complete task harness, acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation. We measure task performance with Relative Human Action Efficiency (RHAE), which combines level completion with action efficiency relative to human baselines, and record dollar-denominated inference cost as a practical resource measure. Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]). Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance. Human-steered systems demonstrate further headroom. These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol: even strong task performance does not ensure that subsequent revisions generalize or preserve existing capabilities.

• The paper evaluates whether frontline language, models can autonomously construct and improve a software harness to enhance performance.
• An autonomous process may produce functioning software without preserving abilities or generalizing improvements beyond feedback tasks.
• Revisions may increase development performance on tasks, while cost and performance effects vary significantly and must be jointly analyzed.
• Where are follow-up questions?

Research question and conceptual framework

The paper investigates whether a frontier LLM can autonomously construct and improve the software harness through which it acts. This question is distinct from ordinary self-refinement, prompt optimization, or meta-learning. The model weights remain fixed; the mutable object is an executable program that controls perception, memory, planning, tool use, model-call allocation, recovery, and action execution. The study therefore treats the deployed agent as a coupled system consisting of a fixed model and an evolving harness.

The central distinction is between four capabilities: constructing an executable harness, repairing operational failures, translating execution evidence into source-code changes, and selecting revisions that improve performance on tasks excluded from adaptation. The paper’s principal claim is that the first three capabilities do not entail the fourth. A model may produce functioning and substantially revised software while failing to preserve existing capabilities or generalize improvements beyond the tasks that generated the feedback.

This framing is important because many existing approaches optimize a constrained artifact, such as a prompt, workflow, module, or predefined agent graph. In contrast, the present protocol allows the model to create and revise a complete task-specific repository. The study consequently evaluates autonomous harness engineering as a capability in its own right rather than as a by-product of model training or human-designed architecture.

ARC-AGI-3 as an evaluation environment

ARC-AGI-3 provides an appropriate testbed because its games require sequential interaction, exploration, latent-rule induction, and adaptation to unfamiliar mechanics. The agent receives grid observations, game state, progress information, and an action interface, but it is not given the semantics of visible objects or the conditions for completing levels. It must infer these through interaction.

Figure 1: FT09 states illustrating an interactive rule-discovery problem in which the agent must infer tile relations, color-cycling actions, and later multi-tile controls.

The benchmark leaves substantial design freedom to the harness. A generated system may expose raw images, symbolic grids, object summaries, transition differences, or combinations thereof. It may use persistent memory, executable world models, replay, deterministic planning, batched model calls, or adaptive inference budgets. Thus, performance reflects the interaction between the runtime model and the orchestration program rather than the model alone.

Performance is measured using Relative Human Action Efficiency (RHAE), which rewards completed levels and discounts solutions that use more actions than hidden human baselines. Later levels receive greater weight, and incomplete level prefixes are capped. The metric therefore combines task completion, long-horizon reliability, and action efficiency. The paper also records dollar-denominated inference cost per game, treating computation as a resource that must be optimized jointly with task performance.

Controlled construction-and-revision protocol

The experiment uses nine model–effort conditions spanning seven model families, with three independent lineages per condition. Each lineage follows the same sequence:

1. A model receives a minimal repository and a fixed general-purpose coding bootstrap.
2. It constructs an ARC-AGI-3 harness without human edits or architectural guidance.
3. The candidate must pass smoke qualification on two development games.
4. The qualified harness is frozen as $H_0$ and evaluated on five development games.
5. A first revision produces $H_1$ using source code, traces, actions, scores, and costs from the development games.
6. A second revision produces $H_2$ from the corresponding evidence generated by $H_1$.
7. Every frozen snapshot is evaluated on the same five development games and twenty held-out games.

The held-out games never influence construction, revision, or selection. This design is stricter than procedures that search over many candidates and report the best validation-selected system. It also makes regressions visible: each prescribed descendant is evaluated, rather than only the strongest surviving candidate.

The final dataset contains 27 lineages, 81 frozen snapshots, and 2,025 game evaluations. All planned lineages complete construction and both revisions. Of the 81 scored stages, 46 candidates qualify without repair and 35 require smoke repair. The initial harnesses already vary considerably, with held-out RHAE ranging from 0 to 97.14 and a mean of 24.54. This dispersion establishes that the models can produce materially different interaction architectures under an ostensibly common protocol.

Aggregate adaptation results

The headline result is modest and statistically inconclusive improvement on held-out games. Revision 2 increases mean development RHAE by 8.42 points but increases mean held-out RHAE by only 2.80 points. The 95% bootstrap interval for the held-out change is $[-1.44, 7.35]$, which includes no improvement. Across the 27 lineages, 15 show strict held-out gains and 12 show losses; a two-sided sign test gives $p=.70$.

The magnitude of the average effect conceals substantial heterogeneity. Across held-out games, performance improves on 13 games and declines on seven. The largest mean improvement is 20.44 RHAE points, whereas the largest decline is 15.03 points. Consequently, the pooled gain cannot be interpreted as uniform capability improvement: revisions often transfer selectively across game mechanics and can damage performance on particular tasks.

Figure 2: Initial construction quality varies widely, while development gains transfer unreliably to held-out games across model–effort conditions.

The paper formalizes this discrepancy as the harness adaptation gap: the difference between apparent improvement on feedback-generating tasks and improvement on held-out tasks. The mean gap after two revisions is 5.62 RHAE points, with a 95% interval of $[1.78, 9.77]$. Development performance therefore overstates the benefit of adaptation by a substantial margin under this protocol.

This result has a direct methodological implication: development-set improvement is not a sufficient criterion for autonomous harness evolution. A revision process must evaluate transfer explicitly, because an updater can correctly inspect traces, identify a real development failure, and implement a functioning repair while still overfitting to the feedback-generating games.

A post hoc checkpoint-selection analysis reinforces this point. Selecting the snapshot with the highest development RHAE yields a mean held-out gain of 4.53 points, compared with 2.80 points when always deploying $H_2$. However, this rule raises mean inference cost from $7.57 to $9.15 per game and was not specified prospectively. It is therefore diagnostic rather than confirmatory. The result nevertheless suggests that revision and deployment selection are distinct capabilities: generating a candidate does not imply that the candidate should replace its parent.

Joint performance and inference cost

The study treats performance and inference expenditure as a joint deployment objective. Across 27 final descendants, nine improve both held-out RHAE and cost per game, six improve RHAE while becoming more expensive, nine become cheaper while losing RHAE, and three worsen on both dimensions. Mean cost falls by $2.72 per game, but the median reduction is only $0.54, indicating that the mean is influenced by a subset of substantial cost contractions.

Figure 3: Revision outcomes partition into joint performance–cost improvements, performance gains at higher cost, cheaper but weaker systems, and regressions on both dimensions.

The most important contradiction is that lower inference cost frequently reflects reduced capability rather than improved resource rationality. In one GPT-5.6 Sol xhigh lineage, the final harness reduces held-out cost from $H_1$07.18 per game while RHAE falls from 43.95 to 21.99. The revision replaces response chaining and extensive history handling with independent calls, a bounded notebook, and more limited context. The correlation between lower cost and lower performance is not causally isolated, but the example demonstrates why cost reductions cannot be reported as unqualified improvements.</p> <p>The converse also occurs. In a Sol-high lineage, held-out RHAE increases from 29.24 to 53.56 while cost remains essentially unchanged, moving from $H_1$18.15 per game. Across the full study, the correct interpretation is therefore Pareto-based rather than scalar: autonomous revision sometimes discovers more efficient policies, sometimes purchases performance with additional computation, and sometimes saves money by sacrificing task competence.</p> <p>The full campaign costs at least $H_1$216,130.09 is fresh selected-game inference. These figures exclude infrastructure and human labor and are lower bounds because unresolved requests and some unselected recovery attempts are not imputed. The accounting makes the evaluation resource-intensive relative to the small aggregate performance effect, although its purpose is measurement rather than cost-efficient optimization.

What the models construct and revise

The generated harnesses exhibit substantial architectural diversity. They include direct multimodal controllers, heuristic-first systems, strategist–executor hybrids, persistent manuals, explicit world models, replay mechanisms, deterministic planners, and inference-budget managers. Replicates from the same model–effort condition can produce different initial designs, showing that construction is not a deterministic mapping from model profile to architecture.

Revision is also substantial. Twenty-two of 27 first revisions reorient the harness, four extend it within the existing decision loop, and one primarily preserves or restores the incumbent. Second revisions include 12 reorientations, 12 extensions, and three preservation-oriented changes. No lineage replaces its broader harness family, but many revisions reorganize memory, perception, replay, context continuity, model-call scheduling, or action budgeting.

Figure 4: Generated harnesses grow substantially in source size and undergo extensive code churn, but neither growth nor edit volume reliably predicts held-out improvement.

Every lineage grows between the initial and second-revision snapshots. Median generated Python size increases from 2,161 lines to 2,604 and then 2,912, with a mean endpoint increase of 1,052 lines. Yet both large and small diffs occur among improving and regressing revisions. Code growth and revision activity therefore provide evidence of engineering effort, not evidence of capability gain.

The paper’s case studies illustrate both successful transfer and destructive revision. In the successful Sol-high lineage, the model responds to decision-limit exhaustion and ineffective cache usage by increasing the internal decision limit, removing explicit cache writes, and adding animation-aware observations, geometric object descriptions, and completed-level memory. The change passes 17 local tests and produces the large held-out improvement noted above. Because several components change simultaneously, the case supports an evidence-linked explanation but does not identify a causal mechanism.

The negative case shows the opposite trade-off. Removing response-ID chaining and replacing it with independent calls and bounded memory substantially reduces inference cost but also degrades held-out RHAE. The paper does not establish that context loss caused the regression, since multiple components changed together. It does establish, however, that coherent source-level revision can move the system sharply along an undesirable performance–cost frontier.

Process audit and evidence use

To examine how revision decisions are made, the authors audit all 54 parent-to-descendant episodes. Outcome-blind packets contain development feedback, traces, tool events, source diffs, tests, and qualification results. Two independent LLM annotation passes apply a fixed codebook covering evidence access, diagnosis, probing, planning, change scope, incumbent protection, verification, post-result adaptation, and resource allocation.

Reliability is uneven. Scope and post-result adaptation achieve strong weighted agreement, with $H_1$3 and $H_1$4, respectively. Planning reaches $H_1$5, but half of the packets lack an observable planning interval and the missingness is concentrated in particular conditions and larger edits. Diagnostic probing is near the threshold at $H_1$6, while diagnosis, incumbent protection, and resource metareasoning are substantially less reliable. The authors appropriately exclude unreliable labels from outcome interpretation.

All 54 episodes inspect item-specific development evidence, target at least one evidence-backed issue, and pass proximal operational checks. Nevertheless, held-out outcomes vary widely. Broad rewrites have a mean immediate held-out change of +3.75 RHAE points, compared with -0.20 for contained changes, but the interval includes zero. Episodes involving further result-driven edits have a mean change of -0.76, compared with +2.48 for episodes that hand off after the initial revision; the -3.24-point contrast is also uncertain and confounded by starting performance, repair effort, and problem difficulty.

These analyses undermine a simple process account in which evidence inspection, testing, or extensive revision guarantees improvement. They show that the models frequently perform recognizable engineering actions, but not that these actions reliably produce generalizable changes. The annotation analysis is explicitly exploratory, uses LLM coders without independent human-validation estimates, and cannot establish causal effects.

Relation to human-steered systems

The gap between autonomous revision and attainable harness performance is large. At the same Opus 5 xhigh setting, the study’s revision-2 systems average 39.95 RHAE on the twenty held-out games, whereas the human-steered Tycho system reports 100 RHAE on the same games (Lehmann et al., 30 Jul 2026). Other human-steered systems also report ceiling-level performance with comparable frontier models.

This comparison must be interpreted carefully. The human-steered systems use different development and selection procedures, and Tycho selects among policies using broader information than the strict held-out protocol here. The comparison therefore does not isolate the causal effect of human intervention. It does establish that the observed autonomous systems are not constrained by an intrinsic inability of the underlying models to support strong ARC-AGI-3 performance. Rather, effective harness design and reliable selection remain difficult for the autonomous construction-and-revision procedure.

Limitations and open questions

The evidence is limited to one benchmark, one five-game development split, and exactly two revisions. The result may not extend to other interactive environments, longer revision sequences, or updaters that can modify their own coding bootstrap. Because ARC-AGI-3 games are public, pretraining contamination cannot be excluded.

Only three lineages are run per condition, so small model-level differences are poorly resolved. A fixed-harness diagnostic also reveals meaningful execution variance: ten evaluations of one unchanged final harness have a standard deviation of 3.99 RHAE points and range from 43.38 to 55.37. This makes small lineage-level changes difficult to interpret, although every repeated evaluation of that harness remains above its initial score.

The cost analysis does not independently identify model capability, harness architecture, and inference allocation. The reported dollar costs measure model inference under a fixed pricing schedule, not latency, energy, infrastructure, or human labor. Finally, the process audit relies on LLM annotations, includes missing intervals, and reports exploratory associations without causal identification.

The central open question is consequently specific: under a longer protocol with prospective checkpoint acceptance, rollback, preserved baselines, and broader task distributions, can autonomous harness revision produce cumulative held-out gains while maintaining prior capabilities and controlling inference cost?

Conclusion

The study demonstrates that frontier models can autonomously construct functioning ARC-AGI-3 harnesses with widely varying capabilities and can make substantial source-level revisions from execution evidence. It does not demonstrate reliable autonomous self-improvement under the tested protocol. Two revisions produce only a 2.80-point mean held-out gain with an interval compatible with no benefit, while development gains exceed held-out gains by 5.62 points. Cost reductions are similarly ambiguous: some revisions improve the performance–cost frontier, whereas others become cheaper by losing task competence.

The paper’s main contribution is therefore evaluative rather than architectural. It separates harness construction, operational repair, evidence-guided modification, and generalizable capability improvement. The results show that strong initial performance and extensive revision are insufficient indicators of reliable adaptation. Held-out transfer, incumbent preservation, explicit acceptance decisions, and joint performance–cost measurement are necessary for assessing whether a model has improved the software through which it acts.