Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks

Abstract: Learning to act in unfamiliar environments requires agents to infer how the world works and revise that understanding as new evidence arrives. Yet limited observations can support multiple world models that explain past interactions but predict different outcomes in unseen states. We introduce Memento 3, building on the Memento series to enable frozen LLM agents to continually learn explicit world models through external memory. The agent maintains a natural-language rulebook as persistent semantic memory, recording revisable hypotheses about environment dynamics while leaving unknown aspects underspecified. It compiles this rulebook into executable code for prediction and planning. Through a continual loop of observation, reflection, rule revision, compilation, and verification, the agent uses prediction errors to refine both the rulebook and its code. Updated code is accepted only when the LLM judges it faithful to the rulebook and cell-exact replay reproduces the observed transitions. We investigate this process as a model-based route to recursive self-improvement (RSI): the agent autonomously explores the environment, revises its world model, and uses verified updates to guide subsequent interaction and learning, while the underlying LLM remains fixed. A population extension maintains multiple world models in parallel, sharing interaction evidence and using their predictions to guide exploration. On ARC-AGI-3, the single-model agent clears every level of all 25 public games, achieves a mean Relative Human Action Efficiency (RHAE) of 100.0, and uses 44% of the human action count. In an Atari Pong case study, a learned feedback controller wins 21:0 in each of three evaluated episodes with different openings, without further LLM calls.

• The paper introduces Code as Model, pairing a semantic rulebook with executable code so a frozen LLM can revise world-model hypotheses, replay all past interactions, and plan under updated dynamics.
• The system achieved 100.0 mean RHAE across all 25 ARC-AGI-3 games, clearing 183 of 183 levels with 7,518 actions—about 44% of the human action count.
• A population of alternative models reduced actions by 33.6% on one ARC-AGI-3 game, while a Pong controller won three 21–0 episodes after learning from only 9,504 emulator frames.

Problem setting and central thesis

“Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks” (2610.11794) addresses a specific failure mode of model-based agents operating in unfamiliar environments: finite interaction histories generally underdetermine the environment dynamics. Multiple programs may reproduce every observed transition while making incompatible predictions in unvisited states. Exact replay can therefore establish consistency with the past, but not correctness of generalization.

The paper’s central proposal, Code as Model, separates an evolving world-model hypothesis into two coupled artefacts. A natural-language rulebook records semantic claims about objects, action semantics, dynamics, goals, and unresolved alternatives. An executable program implements those claims as a transition simulator, observation predictor, and goal recognizer. The rulebook is intended to support interpretation and revision, whereas the executable provides a precise substrate for replay, simulation, and planning.

The system keeps the underlying LLM frozen. Learning occurs through external state comprising the interaction history, rulebook, and executable. After every real action, the agent observes the resulting transition, reflects on whether the discrepancy reflects an incorrect rule or an implementation error, revises the rulebook or repairs the code, recompiles when necessary, and verifies the candidate against the full history. The authors characterize this process as model-based recursive self-improvement because the current world model both guides subsequent action selection and determines which evidence becomes available for its own revision.

The paper makes a deliberately strong claim: recursive self-improvement need not require parameter updates or autonomous modification of the base model if a persistent, executable world model can be revised and reused. In the proposed formulation, improvement is instead located in the external learning state and its interaction with the environment.

Code as Model

The formal environment is a deterministic, goal-directed POMDP with hidden state, unknown transition function, partial observations, and an initially unknown goal set. The agent receives only the action interface, observations, interaction feedback, and completion signals. Its learning state is

$Z_t=(H_t,\psi_t,\theta_t),$

where $H_t$ is the complete interaction history, $\psi_t$ is the rulebook, and $\theta_t$ is the executable realization. The LLM parameters remain fixed throughout interaction.

The executable maintains an internal state that need not coincide with the environment’s observable state. It contains an initializer from the initial observation, a transition function, an observation renderer, and a goal predicate. Replaying a trajectory through the executable reconstructs the model’s current internal state. This is important for partially observable environments: the executable can retain latent variables inferred from earlier observations even when those variables are not visible in the current frame.

The rulebook-executable distinction also permits two qualitatively different updates. If the agent has misunderstood the environment, the semantic rulebook must change before a new implementation is compiled. If the rulebook is correct but the program has an implementation defect, the code may be repaired while preserving the semantic hypothesis. This distinction is operationalized through a combined acceptance criterion:

1. the executable must faithfully implement the rulebook, as assessed by the LLM; and
2. exact replay must reproduce every recorded observation.

The second condition is substantially stronger than testing only the latest transition. Every proposed executable is replayed from the relevant initial observation through all previously recorded actions, and every predicted frame must agree with the recorded frame. In the ARC-AGI-3 implementation, this comparison is cell-exact over the full $64\times64$ grid.

Figure 1: A natural-language rulebook is compiled into executable transition branches that support simulated planning toward a goal.

The authors correctly emphasize that replay is a falsification mechanism rather than an identification mechanism. A finite record defines a version space of surviving hypotheses. If two hypotheses produce identical predictions on every observed trajectory, replay cannot select between them. The paper therefore introduces simplicity as a selection bias: among admissible rulebook-executable pairs, the LLM is encouraged to prefer a concise semantic and programmatic description. This is presented as an approximate minimum-description-length or MAP procedure, not as exact Bayesian inference.

The distinction matters because the selected hypothesis is represented as a point belief. In the single-model setting, all probability mass is assigned to one rulebook, one executable, and one reconstructed internal state. The resulting planner is certainty-equivalent: it searches for a shortest path under the selected model rather than explicitly propagating uncertainty over all consistent models.

Reflective revision and verification loop

The continual update mechanism has five stages: observation, reflection, rule revision, compilation, and verification. After an action extends the interaction record, the LLM chooses among retaining the current model, revising the rulebook, and repairing the code. A rulebook revision is compiled into a new executable; a code repair can retain the existing rulebook. The candidate is then checked against the entire history before it is made available to the planner.

Figure 2: The frozen LLM repeatedly updates semantic rules and executable code, with exact replay serving as the acceptance test.

This architecture makes prediction error a structured diagnostic signal. A mismatch is not treated merely as a failed action. It is evidence that can be localized to an erroneous environmental claim, an incomplete state representation, a faulty transition implementation, or an incorrect goal condition. The Git-based workspace preserves every rulebook and executable version, allowing the agent to inspect diffs, recover earlier hypotheses, and identify which observation prompted a revision.

The resulting process is more constrained than unconstrained textual reflection. Natural-language memory alone can record advice or post hoc commentary, but it does not force a model to generate precise counterfactual predictions. Conversely, executable code alone can reproduce observed trajectories without exposing the semantic basis of its generalizations. Code as Model uses the rulebook to make claims inspectable and the executable to make them operationally testable.

Figure 3: An ARC-AGI-3 rulebook and its Python executable are linked by numbered semantic rules and checked through cell-exact replay.

The paper’s examples illustrate why this separation is useful. In one ARC-AGI-3 run, an initial hypothesis about glyph orientation is replaced after interaction establishes rotation-invariant identity. A coordinate-specific rule is subsequently generalized into a glyph-box rule, and a cursor rule is later reversed when evidence from additional levels contradicts it. The revisions are not merely append-only memories: earlier claims are explicitly retracted or superseded.

Figure 4: Successive rulebook versions show competing hypotheses, evidence-driven generalization, and later retraction of an incorrect cursor rule.

Model uncertainty and the population extension

The authors identify a fundamental weakness of the single-model setting: a model can remain self-confirming if its policy never visits states on which it is wrong. Formally, if a model’s prediction-error set does not intersect the histories induced by its own policy, then the model will not be falsified by following that policy. This is an important qualification to the claim that prediction-error-driven revision produces reliable self-improvement. The agent’s policy determines the evidence it receives, so model learning and exploration are coupled.

To mitigate this issue, Memento 3 maintains a population of $N$ rulebook-executable pairs. All members share the same real interaction history, but they may encode different hypotheses and produce different plans. A behavioral equivalence criterion removes syntactic duplicates that produce the same current action sequence. Planning episodes sample among distinct nonempty plans, allowing surviving hypotheses to induce different exploratory trajectories. Every new transition is then used to test all population members.

Figure 5: The external learning state couples environment observations, shared history, rulebook-executable hypotheses, action selection, and subsequent model revision.

The population is not a calibrated posterior. Equal weighting is used for retained members, and the initial hypotheses depend on LLM proposals. Its role is closer to posterior sampling or ensemble-based deep exploration: preserve behavioral alternatives and prevent one early hypothesis from determining the full trajectory. This is a principled response to version-space ambiguity, although it does not eliminate the possibility that all retained models share a common systematic error.

On the selected ARC-AGI-3 game wa30, increasing the population from $N=1$ to $N=2$ reduced scored actions from 899 to 597 while preserving complete task success and RHAE of 100. The reduction is 33.6%, and the population matched or improved the single-model action count on eight of nine levels. Because this comparison covers only one game and one population size, it supports the proposed mechanism but does not establish a general scaling law.

ARC-AGI-3 evaluation

The principal evaluation uses all 25 public ARC-AGI-3 games (Foundation, 24 Mar 2026). These games require agents to infer unfamiliar visual dynamics and goal conditions under a human-derived action budget. The implementation stores the rulebook as world_model.md and the executable as world_model_engine.py. The planner searches the executable’s predicted state space for a route to the inferred goal and executes it one action at a time. Execution stops at the first prediction mismatch, allowing the resulting counterexample to trigger reflection.

The reported aggregate results are unusually strong:

System	Mean RHAE	Games cleared	Levels cleared
Code as Model	100.0	25/25	183/183
baseline1	99.0	25/25	183/183
NOOA	85.1	19/25	170/183
OPINE-World	78.4	20/25	160/183
DreamTeam	38.1	6/25	98/183
Continual Harness	20.5	3/25	64/183

The agent used 7,518 environment actions against a human baseline of 17,135, corresponding to 43.9%, or approximately 44% of the human action count. Per-game action ratios ranged from 0.20 to 0.67, with a median of 0.42. The reported 100.0 RHAE requires both complete level coverage and sufficient action efficiency; it is therefore not merely a completion score.

The comparison with baseline1 should be interpreted carefully. The performance gap in mean RHAE is only one point because baseline1 also clears all 25 games. The more substantive comparison is action efficiency: Code as Model uses fewer total actions than baseline1, 7,518 versus 8,347, despite both systems completing every level. However, the experiments use Claude Opus 5 with extra-high reasoning effort, and the paper does not provide a broad factorial study separating backbone capability, prompting, harness design, and rulebook effects. The public benchmark is also close to saturation for the strongest systems, limiting its ability to distinguish architectural contributions.

Rulebook ablation

The rulebook ablation removes the natural-language semantic memory while leaving the executable world-model harness, backbone, reasoning effort, and replay verification unchanged. Three games are selected by difficulty band: ft09, ka59, and cn04. Both variants clear every level and achieve RHAE of 100, so the ablation does not show a difference in final success.

It does, however, show lower convergence cost with the rulebook:

Configuration	Actions	Agent turns
With rulebook	617	678
Without rulebook	677	830
Relative reduction	9%	18%

The rulebook variant uses fewer actions and fewer agent turns on all three games. This supports the narrower claim that persistent semantic memory improves convergence to an effective executable model. It does not demonstrate that the rulebook is necessary for solving ARC-AGI-3, since the code-only variant also reaches the task ceiling on the selected games. Nor does the ablation isolate all possible effects of textual memory from differences in LLM generation caused by the altered prompt context.

Atari Pong case study

The Pong experiment tests whether the learned executable supports feedback control rather than only finite-horizon discrete planning. The agent learns a model of ball motion, paddle interactions, opponent behavior, and bounce dynamics in the Arcade Learning Environment. During evaluation, the learned controller is fixed and operates without further LLM calls.

The controller wins three episodes with different openings by a score of 21–0 in every episode. The respective episodes require 3,358, 4,014, and 5,039 actions. The learning cost is 9,504 emulator frames.

Method	Score	Learning cost
Code as Model	21.0	9,504 frames
EfficientZero V2	20.8	400,000 frames
EfficientZero	20.1	400,000 frames
GDI	21.0	200,000,000 frames
LASER	21.0	200,000,000 frames
Same LLM without world model	-19.0	0
Random	-20.7	—

The comparison suggests a large interaction-efficiency advantage in this particular setup: the reported Code as Model learning cost is approximately 42 times lower than that of EfficientZero and roughly 21,000 times lower than the model-free baselines. These comparisons are not fully controlled because the methods differ in architecture, evaluation protocol, action/frame accounting, and implementation. The direct LLM baseline is nevertheless informative: without persistent state estimation and executable dynamics, the same backbone performs very poorly.

Figure 6: Pong rulebook revisions progressively replace an incorrect four-band bounce model with a centered-deflection rule and then an asymmetric velocity map.

The Pong revisions provide a concrete demonstration of model-based learning. An initial four-band hypothesis excludes horizontal returns. A center hit produces a horizontal return, refuting that hypothesis and motivating a centered-deflection formula that permits zero vertical velocity. Later observations show that this formula overestimates return speeds, leading to an asymmetric band map with bounded vertical velocity. Each revision changes predictions over unobserved contact offsets, which is the relevant generalization behavior for feedback control.

The result is also bounded by the case-study design. Only one game, one controller formulation, three evaluation episodes, and one stopping criterion are reported. The controller need not model every possible Pong state; it needs only sufficient accuracy for successful returns under the tested openings. The result therefore demonstrates feasibility of executable reflective modeling for control, not broad superiority over model-based RL.

Limitations and open questions

The paper’s strongest results are empirical demonstrations on public ARC-AGI-3 games and a narrowly specified Pong case study. Several limitations bear directly on their interpretation.

First, the semantic acceptance test for rulebook fidelity is itself performed by the frozen LLM. Exact replay is deterministic and auditable, but fidelity between prose and code is not mechanically established. A code implementation can pass every observed transition while omitting or misrepresenting unobserved semantics, and the LLM evaluator may fail to detect that mismatch. The paper therefore combines a rigorous observational check with a non-rigorous semantic check rather than providing a complete formal verification procedure.

Second, the single-model planner uses a point hypothesis even though the formal treatment recognizes a potentially large version space. Its minimum-description-length bias may select a concise but incorrect generalization. The population extension addresses this problem only partially: the population is finite, equally weighted, LLM-generated, and not guaranteed to contain the true model. The open technical question is how to allocate population members, calibrate their weights, and select actions that explicitly optimize information gain under an action budget.

Third, the reported ARC-AGI-3 comparison does not fully isolate the contribution of Code as Model from backbone strength and harness design. The rulebook ablation is small, with one game per difficulty band, and all selected runs reach the performance ceiling. The full benchmark establishes end-to-end capability, but held-out environments, repeated seeds, confidence intervals, and controlled comparisons across LLM backbones would be needed to estimate robustness and variance.

Fourth, the recursive self-improvement terminology is used in a restricted external-memory sense. The agent revises its rulebook and code, but the LLM that proposes revisions remains fixed. This is a coherent operational definition, but it differs from recursive systems that construct improved successor models or modify their own learning algorithms. The paper leaves open whether the same mechanism can reliably revise the planner, verifier, exploration strategy, and compilation process rather than only the environment model.

Finally, the experiments assume deterministic environments or sufficiently stable dynamics. The formalism includes a POMDP representation, but the implemented acceptance criterion is based on exact replay, which is naturally suited to deterministic transitions. Extending the verifier to stochastic or nonstationary environments requires likelihood-based validation, uncertainty calibration, and criteria for distinguishing stochastic variation from model error.

Conclusion

Memento 3 (2610.11794) presents a technically coherent architecture for external-state model learning by a frozen LLM. Its distinctive contribution is the joint maintenance of a semantic rulebook and executable world model, coupled through reflection, compilation, exact replay, and planning. The system achieves complete coverage of the 25 public ARC-AGI-3 games with mean RHAE 100.0 and 7,518 actions, while the Pong case study demonstrates a fixed post-learning controller that wins three 21–0 episodes without further LLM calls.

The evidence supports the claim that editable semantic memory can improve the convergence and reuse of executable world models. It does not yet establish calibrated uncertainty, robust model identification, or general recursive modification of the agent beyond its environment model. The most consequential open question is therefore whether rulebook-guided executable learning can retain its efficiency when environments are stochastic, held out, adversarially deceptive, or sufficiently broad that a finite LLM-generated population cannot cover the relevant version space.