- The paper introduces Twin, a coding-agent system that builds and repairs a test-time executable world model, validates every predicted transition, and plans actions through it.
- Twin scores 93.3 out of 100 on ARC-AGI-3, clears 23 of 25 games and 179 of 183 levels, and uses 0.61 times the human action count on comparable games.
- The results show that goal inference is the main bottleneck: the system’s first goal hypothesis is correct on 87.2% of completed levels, but action-level goal precision is only 0.214 versus 0.771 recall.
Overview
Twin is a system in which a frontier coding agent writes, validates, and repairs an executable world model of an unknown environment at test time, then plans and acts inside that model. The evaluation domain is ARC-AGI-3: 25 public grid-world games on 64×64 grids of colored cells, where the controls, transition rules, and win condition are all hidden and must be inferred from interaction alone. Each level defines an episodic MDP with a hidden transition function and a hidden completion predicate; the agent observes only the current grid, the action menu (four to six actions including coordinate clicks), and a completion signal after each move.
The central claim is that building a usable world model of an unknown game is tractable, while inferring what counts as winning is the harder half of the problem. Twin operationalizes this asymmetry directly: consistency with every observed transition is enforced as a hard precondition for any scored action, whereas goal reachability is pursued only through search and hypothesis testing. On the benchmark's action-efficiency metric, Twin scores 93.3 out of 100, clearing 23 of 25 games and 179 of 183 levels, versus 61.1 for the same base model in an off-the-shelf harness and 7.8 for direct play.
Method
The twin is a Python program implementing two functions under a fixed contract: step(grid, action) -> grid and goal_reached(grid) -> bool. It begins as an identity stub, so every nontrivial rule comes from interaction. The control loop has three harness routines plus a checked executor:
- Validate: replays the entire interaction log D of triples (s,a,s′) through the twin; the harness blocks any scored action until every logged transition is reproduced exactly. A failure returns the mismatched transitions and cells as a counterexample report.
- Explore: dispatches on which "wall" is live. At the dynamics wall (validation fails), it compiles failing transitions into repair targets grouped by action–object context and ranked by misprediction rate, effect variety, and support count. At the goal wall (validation passes but no plan exists), it searches the twin without R^ under an enlarged budget (depth 14, 30,000 nodes) and ranks reachable states by five progress signals — color disappearance, color appearance, local bursts, global scene change, and frontier novelty.
- Plan: breadth-first search inside the twin with T^ as successor function and R^ as goal test (depth 8, 20,000 nodes), returning a shortest route within budget.
- ExecuteChecked: submits planned actions one at a time against the real simulator, halting at the first cell-level mismatch between prediction and outcome. Every committed action therefore yields either verified progress or one localized counterexample.
Goal discovery handles the bootstrapping problem that R^ receives positive supervision only at level boundaries. When no route to a goal exists, the coding agent writes a tentative goal predicate over a top-ranked candidate state, subject to a hard consistency filter: the predicate must evaluate false on every logged frame (sound because ARC-AGI-3 replaces the winning frame upon completion). Three outcomes follow: a level boundary confirms the candidate; reaching the predicted state without a boundary permanently excludes it; a dynamics mismatch repairs T^ and leaves the candidate tentative. This is a Go-Explore-style novelty search serving WorldCoder's optimism constraint, but split from the consistency requirement rather than imposed jointly during synthesis.
The paper emphasizes contamination controls: scored actions pass only through the live engine interface, web search is disabled, an integrity audit scans transcripts, and the public games postdate the base model's training cutoff.
Main results
| System |
Score |
Games won |
Levels cleared |
| Human reference |
100.0 |
25 |
183 |
| OPINE-World |
78.4 |
20 |
160 |
| Prime Agent |
78.3 |
– |
164 |
| EWM |
63.8 |
14 |
146 |
| Codex (no-twin ablation) |
61.1 |
13 |
148 |
| Twin |
93.3 |
23 |
179 |
All systems are evaluated against the same published human baselines; the human row is the normalization reference. Twin reaches the 100.0 ceiling on 18 games and leads or ties OPINE-World on 22 of 25 games. Its largest gains appear on long multilevel games: it is the only system to clear bp35 (9 levels), lf52 (10), and sk48 (8), where OPINE-World stalls by level three or four. On dc22, Twin clears all six levels in 1,219 actions versus EWM's 1,842 actions clearing four.
On the thirteen games that Twin, EWM, and OPINE-World all fully clear — where endpoint parity makes action counts directly comparable — Twin uses 3,357 scored actions against 5,367 (OPINE-World), 5,381 (EWM), and 7,485 (human). Twin matches or beats the human count on 21 of its 23 cleared games, averaging 0.61× human actions.
The ablation isolates the harness with base model, file bridge, and sandbox held fixed: disabling the validate–explore–plan loop drops the score from 93.3 to 61.1 and cleared games from 23 to 13. The sole exception is sc25, where trial-and-error play outscores Twin (44.8 vs. 32.7) because uncleared levels receive zero credit and a hidden countdown charges real tests per hypothesis.
Goal inference is the binding constraint
The paper's most consequential empirical claim concerns goal inference. The first committed goal hypothesis is correct on 156 of 179 completed levels (87.2%), meaning most levels are played goal-directed from the first action. Yet Appendix J grades every goal claim at action granularity and finds a sharply different picture: over 11,557 graded actions, the predicates achieve recall of 0.771 but precision of only 0.214 — 138 correct claims against 508 false ones. The paper attributes this asymmetry to supervision imbalance: 11,557 dynamics labels versus 179 goal labels, a 65-to-1 ratio. Goal precision also predicts endgame plan failures (r=0.77 across games): a false accept truncates BFS early, so low-precision twins produce plans that stop short. sp80 and sc25, Twin's two unfinished games, are precisely the goal-limited cases — sp80 holds 92.3% dynamics accuracy yet never finishes its sixth level.
World-model generalization beyond replay
Replay validation certifies only logged transitions, so the appendix evaluates final twins on previously unobserved state–action pairs (20,790 evaluations against a rerun engine). Final twins predict complete next frames exactly on 70.1% of unobserved pairs, versus 99.4% on replayed pairs — evidence of reusable rules rather than memorization, though with wide per-game spread (9.9% on wa30 to 98.8% on tu93). wa30 is the cautionary case: near-perfect replay with near-total failure on unseen pairs. Errors are spatially concentrated — in the median game, 561 of 4,096 cells hold 90% of disagreement mass.
A controlled comparison runs identical bounded BFS inside the twin and inside the real engine. Six actions before each win, both searches broadly agree (both finish on 102 of 174 comparable levels). From level starts, the engine finds 61 finishes and the twin 47, but 31 of the twin's 32 wins are certified shortest, and on two levels the twin succeeds where the engine's frame-merging search fails — when hidden state exceeds what pixels show, the twin is the better search space.
Compute accounting
Twin spends 2.60B processed tokens and 91.4 wall-clock hours across 25 runs, roughly 224k tokens per scored action versus 48k for the no-twin ablation and 715k for EWM. Against EWM, Twin improves score by 29.5 points on D0 fewer tokens; against the ablation, it uses D1 total tokens to gain 32 points. The paper is explicit that the twin does not save compute relative to cheap acting — it front-loads compute offline so submitted actions are few and well-founded. Over 98% of processed tokens run on cached context.
Limitations
The paper concedes several scope restrictions plainly. Replay validation assumes deterministic, program-compressible dynamics and certifies only logged transitions; probabilistic twins remain future work. Exact replay presumes small discrete states — continuous observations would make every replay an approximate comparison requiring a tuned threshold. step reads one frame, so mechanics driven by temporal context invisible in any single frame are out of scope, though bp35 shows that camera-hidden map (a static piece of the transition function) can be handled by paying one probe per unseen place. Truly latent variables no frame ever shows stay unsolved. Fixed search budgets mean goals beyond the horizon go unfound. The generalization tests measure only behavior near visited states and observed action types, not arbitrary states. Finally, the five progress signals used for candidate ranking are fixed heuristics, and goal discovery offers no correctness guarantee — a heuristic propose-plan-test cycle rather than a procedure with convergence properties.
Conclusion
Twin demonstrates that a test-time-written, replay-validated executable world model suffices to play unknown games at better-than-human action efficiency, and that enforcing validation mechanically in the harness — rather than by prompt — is what carries performance on long multilevel games. The residue the paper itself measures is goal precision: twins learn how the world moves far more reliably than they learn what winning means, and the unfinished games are exactly those with accurate dynamics and unresolved goals. The open question the results pose is how to supply denser supervision for goal inference commensurate with the per-action supervision dynamics already receive.
