AgentGarten: Code Worlds for Evolving Agents

Abstract: Interactive virtual worlds allow agents to learn through exploration and interaction. What agents can learn is bounded by the environments they practice in, which must be faithful, with consistent state, rules, and dynamics, and realistic, with observations that follow the real-world visual distributions. Achieving both across diverse worlds remains a bottleneck. We introduce AgentGarten, a framework that couples simulators and game engines with a shared neural renderer to build real-time interactive environments. Its simulation backends maintain persistent world state and execute program-defined interaction rules, while the renderer generates visual observations from structured conditions exported through a common interface. To build the neural renderer, we adapt a pretrained video model to geometry conditions, distill it with our proposed Adversarial Forcing, and optimize inference for real-time interaction. Adversarial Forcing makes history prefilling differentiable through exact replay, so that losses on later predictions update how the renderer encodes prior observations, and adds real-data adversarial supervision to improve its visual quality. In AgentGarten, agents perceive the world through visual observations, interact with it in real time, and improve by distilling each round of experience into playbooks that subsequent agents inherit and refine. Our empirical study demonstrates a substantial gain in learning efficiency, with agents learning from just 4 rounds compared with millions for a conventional reinforcement learning counterpart. As new worlds can be written as code and rendered through the same interface, environments can scale in both number and difficulty alongside their agents, a step toward agents that keep evolving through interactive experience.

• The paper introduces AgentGarten, a framework separating the evolution of world state from visual representation in interactive agent environments.
• It uses a shared geometry-conditioned neural renderer and authorative simulator engine to ensure causal consistency, making environments lightweight to construct and edit.
• Agent experiments in various tasks ( navigation, manipulation, tool use, multi-agent interaction) show pretrained agents effectively grounded in closed-loop visual action and improve through written playbooks.

Problem setting and central architecture

AgentGarten addresses a persistent incompatibility in interactive agent environments: simulators and game engines provide explicit state, programmable transitions, and inspectable rules, whereas generative video models provide visual diversity but generally encode dynamics implicitly in their generated histories. The paper proposes a factorized alternative in which world evolution and visual realization are separated. A simulator or game engine maintains the authoritative state and executes the scene program; a shared neural renderer converts structured geometric conditions into visual observations. This design is intended to preserve causal consistency without requiring bespoke high-fidelity rendering assets for every environment (2610.12374).

A code world is defined by a scene program, an engine, and a neural renderer. The program determines the transition function, including object poses, articulations, interaction rules, and task variables. Given state $s_t$, camera pose $\pi_t$, and action $a_t$, the engine produces the next state and an exported condition such as depth or surface normals. The renderer then generates the observation from the condition sequence, an appearance reference, text, and prior rendered history.

Figure 1: The interaction loop separates executable state transitions from neural rendering of observations.

The separation has an important causal consequence: renderer errors do not directly modify world state. The renderer affects subsequent state only indirectly through the actions selected by the agent after observing generated frames. Consequently, an event trajectory can be replayed under a different appearance reference or camera without changing the underlying physical sequence. This property distinguishes AgentGarten from video world models in which the generated visual history is also the effective state representation.

The framework also makes environment construction comparatively lightweight. A coding agent can generate a scene program from text or a single image, using perception models to estimate object masks, depth, camera parameters, and coarse 3D structure. The resulting program needs to encode layout, silhouettes, occlusion, and interaction dynamics, while appearance and material detail are delegated to the renderer.

Figure 2: A coding agent reconstructs an executable code world from a single image and iteratively repairs its geometry and interaction rules.

The image-to-world pipeline is not fully automatic in the strong sense. The coding agent must inspect rendered views, probe cameras, collision checks, contact queries, and occlusion behavior, then revise faulty poses or unsupported structures. Nevertheless, delegating appearance to a shared model substantially reduces the need for scene-specific texture and asset authoring.

Geometry-conditioned neural rendering

The renderer is adapted from a pretrained bidirectional video model and converted into a block-causal autoregressive generator. It receives depth or surface-normal videos as structured conditions, an appearance reference $x_0$, text $y$, and its own visual history. The choice of depth and normals is pragmatic: both are available from conventional engines, can be estimated from real video, and fit the pretrained model's three-channel video interface.

The model uses joint attention over geometry and RGB latent tokens rather than concatenating the modalities at the pixel level or introducing a separate control branch. Geometry tokens and RGB tokens share temporal and spatial coordinate systems, allowing cross-token interaction while preserving alignment. Geometry is aggressively compressed: at $480 \times 832$ resolution, a condition frame produces 28 tokens compared with 390 RGB latent tokens. Training applies modality dropout, texture suppression, spatial warps, and latent noise to reduce dependence on estimator-specific artifacts and improve robustness to the discrepancy between real-video geometry estimates and engine-rendered conditions.

The renderer is trained in three stages. First, the pretrained model is adapted to geometry conditioning. Second, it is converted to blockwise teacher-forced generation. Third, the teacher-forced model is distilled into a few-step autoregressive renderer using Adversarial Forcing.

Block-causal teacher forcing

The causal adaptation partitions the video after the reference frame into fixed-length blocks. For each target block, a clean copy represents the history and a noisy copy is denoised. The noisy copy can attend to the reference, prior clean blocks, and itself, but not to its own clean target or any future block. This mask permits parallel training of all target blocks while retaining the visibility pattern required at inference.

Figure 3: The block-causal attention mask exposes each noisy target block to earlier clean history but not to its own clean target or future blocks.

This construction addresses a standard mismatch in video generation: ordinary bidirectional training permits access to future information that is unavailable during autoregressive rollout, while fully serial training is computationally expensive. The block-level formulation preserves causal structure and permits independently sampled flow times for the target blocks.

Adversarial Forcing

Adversarial Forcing combines three objectives:

1. Distribution matching between student rollouts and a frozen bidirectional teacher.
2. Differentiable history replay so later losses can update the computation that writes prior observations into the cache.
3. A real-data adversarial objective intended to suppress long-horizon visual degradation.

The distribution-matching component follows the DMD and Self Forcing family. The student generates its own preceding blocks, and a fake-score model learns the student distribution. The teacher and fake-score model estimate clean latents for corrupted student predictions, and the student is updated toward the teacher–fake-score difference.

The paper's principal systems contribution is exact replay. In ordinary self-forcing, prior blocks are placed in a detached key–value cache. Later losses therefore train predictions conditioned on the cache but cannot update the history-prefilling computation that produced it. Retaining the entire rollout graph would expose a recursively connected computation and impose substantial memory costs.

AgentGarten instead performs two passes. A no-gradient rollout records each block's input and final clean output. A second differentiable pass recomputes the history encodings and predictions using the same block-level visibility pattern. The recorded latents remain detached, but the keys and values that represent history are recomputed inside the gradient graph.

Figure 4: Exact replay reproduces the cached rollout while restoring gradients through history encoding.

The implementation detail is central. A full-sequence replay with FlexAttention has the correct attention mask but different tensor shapes, call structure, reduction order, and floating-point execution from cached generation. In BF16, these differences accumulate. AgentGarten instead executes replay block by block with the same scaled dot-product attention calls, key–value order, grouping, and tensor shapes used in the rollout.

The numerical comparison is strong:

Replay method	Relative $L_2$ error	Forward time	Peak memory
Full-sequence FlexAttention	3.99%	2.95 s	50.83 GiB
Blockwise SDPA exact replay	0, bitwise identical	2.79 s	50.92 GiB

The proposed replay is therefore not merely more accurate; it is also 5.4% faster in the reported forward-pass measurement, with only a 0.2% increase in peak memory. The implication is that bitwise agreement between rollout and replay is achievable without retaining the serial sampling graph, provided the recomputation schedule matches the original execution structure.

Adversarial supervision and exact R1/R2

Score distillation alone produces long-horizon degradation. In the reported qualitative comparison, Self Forcing develops repetitive surface patterns and loses scene detail over 30-second rollouts, whereas Adversarial Forcing retains more natural textures and fine structure.

Figure 5: The discriminator uses a frozen teacher backbone and a trainable head to provide adversarial supervision for generated videos.

The discriminator consists of the frozen teacher backbone and a trainable aggregation head. It receives geometry, reference, and video information, allowing it to assess both appearance and geometric consistency. The adversarial objective uses relativistic pairing between real and generated clips corrupted with matched noise and diffusion timestep.

The paper also introduces an implementation of exact R1/R2 regularization that avoids double backward through fused attention kernels. Let the discriminator be $D_\phi(x)=h_\phi(B(x))$, where $B$ is frozen and $h_\phi$ is trainable. The input gradient is obtained with a vector–Jacobian product through the head and backbone. A Jacobian–vector product through the frozen backbone then supplies the direction needed to differentiate the squared input-gradient norm with respect to the head parameters. The resulting surrogate has the exact R1/R2 value and exact first derivative with respect to $\pi_t$0, while requiring no backward-over-backward operation through the fused backbone.

Figure 6: Exact R1/R2 regularization uses VJPs and JVPs through the frozen backbone and an explicit directional derivative in the trainable head.

This result depends critically on freezing the discriminator backbone. If the backbone were trainable, its features and Jacobian would depend on $\pi_t$1, and the proposed detached construction would no longer provide the full parameter derivative. The method is therefore an exact regularizer for the stated frozen-backbone architecture, not a general solution to higher-order differentiation in arbitrary discriminators.

Streaming inference and deployment

The renderer generates four latent frames per block using four denoising steps. A bounded KV cache retains a permanent sink prefix, including the appearance reference, and a sliding window of recent history. Older frames are evicted. For rollouts beyond the training horizon, top-aligned rotary-position remapping keeps active positions within the trained range while preserving recent relative temporal structure.

Figure 6: The bounded inference cache retains the reference as a sink prefix and a sliding window of recent visual history.

The deployment stack is engineered around the latency profile of short-block autoregressive generation. Custom Triton kernels fuse RMS normalization, rotary embeddings, and gated activations; CUDA graph capture reduces host launch overhead; and a distilled tiny decoder reduces pixel reconstruction time. On one NVIDIA H100 at $\pi_t$2 resolution, the reported steady-state performance is:

Component	Time per 16-frame block
Condition encoding	10.6 ms
Transformer, four denoising steps and cache publication	414.4 ms
Tiny decoder and host transfer	8.8 ms
Total wall time	438.8 ms
Throughput	36.5 frames/s

The renderer therefore exceeds 35 frames/s under the specified configuration, including condition encoding, denoising, decoding, and host transfer. The Transformer remains the dominant cost, accounting for approximately 94% of the listed stage time. The result demonstrates real-time service on a high-end accelerator, although it should not be interpreted as low-cost deployment independent of hardware or resolution.

Code-world library and agent interaction

AgentGarten supports multiple scene types through a shared renderer and compatible engines. The examples span navigation, manipulation, tool use, multi-agent interaction, and browser-based games. Since the executable state is separate from appearance, the same trajectory can be replayed with different cameras or visual styles.

Figure 7: Five code worlds demonstrate that a common renderer can support different geometries, tasks, and multi-agent trajectories.

The agent experiments use pretrained foundation-model agents that receive visual observations rather than simulator coordinates or raw geometry buffers. Agents act through short Python programs and maintain written playbooks containing reusable skills, failure analyses, and proposed tests. Each round begins with a task specification describing the goal, available actions, and constraints but not a solution. After acting, the agents review their permitted observations and write playbooks that are passed to fresh agents in subsequent rounds.

This protocol differs materially from conventional RL. The agents are not trained from random initialization, and the paper does not claim a direct sample-efficiency comparison with tabula rasa policies. They begin with pretrained semantic knowledge about tools, objects, and spatial relations. The measured adaptation concerns grounding that prior knowledge into closed-loop visual action, diagnosing failures, and refining execution through written experience.

Hide-and-seek results

The most prominent experiment revisits the hide-and-seek environment studied by Baker et al. The original self-play RL agents learned shelter construction after approximately 25 million episodes and ramp use after approximately 100 million episodes, while observing privileged object state. AgentGarten uses a sequential one-hider, one-seeker setting in which each agent sees only first-person neural-rendered observations, receives no coordinates or hidden state, and acts through short Python programs.

Figure 8: Hide-and-seek behaviors include barrier-based shelter construction, ramp transport, and repositioning after a failed crossing attempt.

The reported milestones occur much earlier in terms of interaction rounds: hiders construct shelters by round 4, and seekers use ramps to cross walls by round 10. These figures are striking but are not a direct ratio against the original episode counts. The two systems differ in initialization, observation modality, objective, and learning algorithm. The earlier result indicates that a pretrained agent can rapidly convert abstract knowledge of barriers, ramps, and spatial obstruction into effective closed-loop behavior; it does not establish that playbook-based adaptation is more sample-efficient for learning physical dynamics from scratch.

The qualitative behaviors are nevertheless significant. The hider moves a panel to obstruct an entrance, the seeker carries a ramp to an interior wall, and a seeker that initially fails to cross repositions the ramp and retries. These sequences require object selection, transport, placement, geometric reasoning, and recovery from failed execution, all based on rendered observations rather than explicit state.

Cross-world round-based learning

The same practice loop is evaluated in four additional environments: companion-dog engagement, a one-lane bridge, herding, and a quarry loader.

Figure 9: Across four additional worlds, agents write playbooks after each round and improve or stabilize task performance by round 4.

The results show heterogeneous but generally positive progression:

World	Round 1	Round 4	Interpretation
Companion dog	Score 13	Score 19	Improved engagement
One-lane bridge	71 s	41 s	Faster coordinated passage
Herding	3 of 4 sheep	4 of 4 sheep	Complete herding after round 1
Quarry loader	Score 30	Score 90.9	Rock delivery and parking achieved

The bridge time decreases from 71 to 41 seconds while both cars arrive in every round. Herding improves from three sheep penned to all four in later rounds, although the numerical score fluctuates between rounds 2–4. The quarry loader exhibits non-monotonic behavior: it scores 30 in round 1, fails in round 2, returns to 30 in round 3, and reaches 90.9 in round 4 by clearing the rocks, delivering one to the bunker, and parking within 329 of the available 360 seconds. This variability is important: the playbook mechanism supports improvement but does not guarantee monotonic learning or reliable transfer across episodes.

The strongest interpretation is that written experience can serve as an inter-round adaptation substrate for pretrained agents acting through visual interfaces. The experiments do not isolate the contribution of the renderer from that of the foundation model, task prompting, playbook quality, or manual environment design. They also evaluate one episode per round in the four auxiliary worlds, limiting the statistical strength of the reported trajectories.

Limitations and open questions

The renderer's conditioning interface is both its main simplifying assumption and a significant limitation. Depth and normals represent geometry but not all task-relevant state. A spinning object with rotationally symmetric geometry may produce unchanged depth and normals even though its painted markings or angular velocity matter. Material, color, object identity, and some interaction states are likewise underdetermined. Visual history can preserve such attributes, but the bounded cache may lose them after long occlusions or revisits.

The renderer can also fail to realize the supplied state perfectly. The executable engine remains authoritative, but the agent acts on generated frames; perceptual artifacts can therefore cause incorrect actions even when the underlying transition dynamics are exact. The paper does not report systematic task success under controlled renderer corruption, nor does it quantify how visual inconsistencies affect policy decisions.

Scaling the environment library remains unresolved. The reported worlds were authored individually, and the paper explicitly identifies automatic evaluation of candidate worlds as an open problem. A generated world must be visually observable, dynamically nontrivial, solvable under the available action interface, and useful for learning. AgentGarten supplies a shared rendering interface but does not yet provide a complete mechanism for automatically generating and validating such curricula.

The agent results also require cautious interpretation. They compare pretrained reflective agents with tabula rasa RL only as qualitative reference points, not under matched compute, interaction budgets, observations, or objectives. The four-world evaluation has few rounds and limited replication. The paper therefore establishes feasibility and strong proof-of-concept trajectories rather than a controlled estimate of general learning efficiency.

Finally, the proposed exact R1/R2 method assumes a frozen discriminator backbone and a trainable head. Its correctness does not extend unchanged to joint backbone optimization. Similarly, bitwise replay is achieved by matching the execution schedule and kernels, but this condition is implementation-specific and may be difficult to preserve across hardware, compiler, precision, or kernel changes.

Conclusion

AgentGarten presents a coherent separation between executable world state and neural visual realization. Code worlds provide persistent, editable dynamics, while a shared geometry-conditioned renderer supplies visually diverse observations without requiring conventional high-fidelity assets for every environment. Adversarial Forcing combines self-rollout distribution matching, bitwise exact replay, and real-data adversarial supervision; its reported replay experiment achieves zero relative error under blockwise recomputation and its deployment reaches 36.5 frames/s on one H100.

The agent experiments show that pretrained visual agents can ground prior knowledge into multi-step physical behavior and improve through written playbooks. Shelter construction appears by round 4, ramp-based wall crossing by round 10, and performance improves across several additional worlds. The central unresolved issues concern richer state conditioning, long-horizon visual identity, automated world authoring and validation, and controlled evaluation of playbook-based adaptation. Within those limits, the paper establishes a technically specific framework for coupling programmable environments with real-time neural rendering and inter-round agent improvement (2610.12374).