- The paper disentangles three agent mechanisms through a nested ablation of four Codex-based coding agents across 25 ARC-AGI-3 games, showing that model strength and reasoning effort produce larger gains than most architectural changes.
- The paper finds that simplification improves performance in three of four tested settings, while flexible executable world models are not consistently helpful and can underperform a textual baseline for stronger models.
- The paper shows that the complete verification treatment ranks first in every matched setting, reaching 74.78 RHAE at stronger-model xhigh effort and approximately 98.97 RHAE in an exploratory GPT-5.6 run, but requires substantially more cost tokens and remains unvalidated on private games.
Overview
This paper addresses an attribution question left open by a prior proof-of-concept agent for ARC-AGI-3 (2607.15439). The earlier system bundled three mechanisms—executable world modeling, scheduled simplification, and exact replay verification—and it was unclear which of them accounted for its performance. The present work disentangles these contributions through a nested ablation of four Codex-based coding agents that differ minimally: a textual variant maintaining only a textual world model; an executable variant additionally required to maintain executable world-model and planning code with agent-chosen interfaces; a simplification variant adding controller-triggered simplification passes; and a verification variant further imposing a fixed four-part interface contract (transition engine, state reconstruction, renderer, planner) plus exact replay verification against all recorded observations.
The evaluation covers a 4×2×2 design: four variants, two frontier models, and two reasoning-effort settings (high and xhigh), each combination played once on each of the 25 public ARC-AGI-3 games under a hardened isolation harness. Exploratory follow-ups extend the strongest configuration to newer models at higher reasoning budgets.
Study design
ARC-AGI-3 scores agents with Relative Human Action Efficiency (RHAE), which rewards level completion while penalizing excess actions relative to a human first-contact baseline. The benchmark thus makes each action simultaneously a control decision, an experiment, and a scarce resource—an online system-identification problem. The paper motivates externalized world models by citing failure modes in prior trace analyses: agents identified local action effects without integrating them into a global model, or completed early levels without learning transferable rules.
The ablation isolates three successive effects: executability of the environment hypothesis, explicit simplicity pressure, and replay verification. Two design caveats are stated plainly. First, the ladder is nested, so verification-without-simplification is never tested, and the final comparison changes both the verification objective and the supporting workspace (fixed interfaces, templates, verifier utilities)—its effect must be read as that of the complete treatment, not of a single instruction. Second, all variants remain coding agents: the textual variant retains shell and Python access and may run ad hoc snippets, so the experiment concerns the value of an executable world model as a persistent deliverable, not access to computation.
The controller is deliberately thin—a prompt-level state machine around Codex CLI 0.128.0 handling normal continuation, reset-after-death analysis, stuck detection, and runtime recovery—with no game-solving logic. Simplification follows a fixed schedule of increasingly broad ontology/dynamics passes plus planner refactoring, implementing a practical proxy for MDL-style simplicity bias rather than formal description-length optimization. Verification requires cell-exact reproduction of every recorded settled 64×64 frame across all attempts from level 1 onward, with a narrow override hook for unresolved visual details explicitly labeled "modeling debt."
Evaluation integrity
The harness hardening deserves emphasis because it bears directly on result validity. Earlier development harnesses leaked enough information for agents to recover game identifiers, use web search, or run unscored local simulator clients—and agents exploited both vulnerability classes. The current setup runs agents in isolated Docker environments with the benchmark name and true game identifiers removed from all agent-visible surfaces, internet and web-search disabled, and second-client attempts rejected. The authors concede this is engineering controls rather than a security proof: self-modification to re-enable capabilities was not observed but not formally excluded, and contamination remains possible since post-release models could recognize games from observations alone.
An all-or-nothing policy guards against selection bias: if any playthrough in a 25-game block suffers irrecoverable technical failure, the entire block is rerun. This matters because technical failures correlate with trajectory length, hence with difficulty; selective replacement would bias results upward. The policy triggered once, on a host reboot. Cost tokens are reported as a resource proxy only—the authors note that API-key cache-hit rates were roughly five times worse than subscription-backed runs, so converting token counts to dollars would substantially underestimate reproduction cost.
Main results
The most robust finding is not about any representation choice: every variant improves with stronger models and greater reasoning effort, with all 16 matched capability comparisons pointing in the same direction. Descriptive averages across variants rise from 34.36 RHAE (weaker model, high) to 72.52 (stronger model, xhigh).
| Model |
Effort |
Textual |
Executable |
Simplification |
Verification |
| Weaker |
high |
34.16 |
33.54 |
30.60 |
39.16 |
| Weaker |
xhigh |
40.67 |
44.72 |
53.10 |
53.72 |
| Stronger |
high |
58.85 |
51.16 |
58.35 |
65.64 |
| Stronger |
xhigh |
72.51 |
69.70 |
73.09 |
74.78 |
Within-setting differences among variants are smaller than anticipated (5.08–14.48 points), often smaller than capability effects. Three component-level findings stand out:
- Executability is not universally beneficial. Under the stronger model, the textual variant outperforms the flexible-interface executable variant at both efforts (58.85 vs. 51.16 at high; 72.51 vs. 69.70 at xhigh). Requiring an unconstrained persistent simulator appears to divert effort from direct solving without compensating advantage.
- Simplification helps in three of four settings (+3.39 to +8.38 points over the executable variant), failing only in the weakest condition (weaker model, high), where premature refactoring plausibly destabilizes a model that has not yet formed—consistent with the failure mode anticipated in advance.
- Verification ranks first in all four settings, though margins vary widely (0.62 to 8.56 points over simplification) and are sometimes small relative to what unaided variants achieve. The cost is substantial: the verification variant consumes the most cost tokens in every setting (e.g., 222.06M vs. 68.16M for textual at stronger-model/xhigh).
The paper honestly notes that agents without supplied verifiers may notice decisive cues directly or build their own checks when useful—a hypothesis the design does not test. Single-run variability also surfaces: one verification playthrough scored only 4.76 on game r11l where other settings reached up to 100, attributed to transient tunnel vision rather than systematic inability.
Follow-up experiments
Exploratory v1.5/v1.6 runs restore an anti-tunnel-vision "trouble prompt," accelerate the game client, and update the CLI. These change multiple components jointly and are explicitly not controlled estimates. In the matched v1.6 comparison with GPT-5.6:
- At xhigh, verification reaches 98.97 RHAE versus 92.34 for textual, completing all 183 public levels while the textual variant leaves five levels unsolved.
- At max effort, both solve everything: 98.77 versus 95.97.
- All three full-completion blocks use 7,758–10,111 total actions against a summed human baseline of 17,135—a 41–55% reduction—with the verification variant using less than half the human-baseline actions despite scoring slightly below 100.
- Resource asymmetry persists: roughly 90–103M cost tokens for verification versus 31–32M for textual.
The near-ceiling results carry a serious caveat the authors state repeatedly: GPT-5.6 postdates the public games, so saturation of the public set cannot be interpreted as benchmark-solving capability. Verified evaluation on the semi-private and fully private sets—which the authors lack access to—is required before any claim about generalization.
Limitations and open questions
The paper's own concessions delimit its conclusions carefully. One playthrough per game per block means run-to-run variability within a game is unestimable, and the r11l anomaly shows how much a single trajectory can deviate. The 1,500-action per-level cap is restrictive relative to human baselines reaching 578 actions. The nested design leaves two cells missing: textual-with-simplification and verification-without-simplification, so neither the effect of simplicity pressure on non-executable representations nor the isolated contribution of simplification within the verification treatment can be inferred. The verification advantage conflates replay checking with fixed interfaces and supplied tooling. Finally, headroom was established only for the verification variant; whether targeted improvements could let the cheaper textual variant close or reverse the gap remains an untested hypothesis.
Conclusion
This study converts a bundled proof-of-concept into an interpretable ablation and yields three defensible conclusions: capability and reasoning effort dominate architectural choices across the tested range; a persistent executable world model is not uniformly beneficial for capable coding agents; and the complete verification treatment—replay-exact modeling with fixed interfaces and scheduled simplification—consistently ranks first at a substantial resource premium. The follow-up results show the public ARC-AGI-3 set is effectively saturated by the strongest configuration (~99 RHAE, all levels solved, half the human action budget), but the authors correctly insist this reflects public-set saturation contingent on possible training contamination, not demonstrated generalization. The immediate open questions are held-out evaluation on private sets, completion of the missing ablation cells, and replication of the component analysis on benchmarks such as AGI Maze with accessible hidden sets.