AgentGarten Framework Overview

• AgentGarten is a framework for constructing real-time interactive 'code worlds' that uses a combination of physics engines for state management and neural rendering for visually rich observations.
• The framework separates simulation from appearance generation, enabling precise state management through executable programs, and generating realistic visuals through neural networks
• Key use cases include developing agents for complex tasks like hide-and-seek and dog herding, demonstrating real-time learning and innovation through playbook inheritance.

AgentGarten is a framework for constructing real-time interactive “code worlds” in which executable simulators or game engines maintain authoritative world state while a shared neural renderer generates realistic visual observations. Agents interact through rendered images and actions, and transfer experience across successive rounds by distilling successful procedures, failures, visual cues, and unresolved questions into reusable playbooks. The framework is designed to reconcile the persistent, inspectable dynamics of conventional simulation with the visual richness and temporal coherence of learned video generation (Chi et al., 8 Oct 2026).

1. Motivation and design principles

AgentGarten addresses a fundamental tension in interactive learning environments. Faithful environments require explicit state, programmable transition rules, reproducible consequences, and inspectable long-horizon behavior. Realistic environments require visually rich observations, diverse appearances, coherent motion, and camera-dependent rendering. Conventional robotics simulators, physics engines, and procedurally generated 3D environments provide the former but make visual diversity expensive. Video world models provide the latter but generally encode state and dynamics implicitly in generated histories, making exact replay, rule inspection, and persistent interaction consequences difficult.

AgentGarten separates simulation from appearance generation. A simulator or game engine maintains the authoritative state and executes program-defined rules. A shared neural renderer converts structured geometric conditions exported by the engine into RGB observations. Agents receive those observations, act in the environment, observe consequences, and refine their behavior through playbook inheritance. Rendering errors therefore do not directly modify simulator state, and a recorded state trajectory can be rendered again with a different appearance reference or camera.

The framework’s central division of labor is:

1. Code-defined dynamics: executable programs specify geometry, interaction rules, physical behavior, task variables, terminal conditions, camera behavior, and available actions.
2. Structured conditions: the engine exports geometric information, primarily colorized depth and camera-space surface normals.
3. Neural rendering: a pretrained video model generates visual observations conditioned on appearance, text, geometry, and temporal history.
4. Visual interaction: agents act using rendered observations rather than privileged coordinates or hidden simulator state.
5. Playbook-based learning: agents review experience after each round and communicate reusable knowledge to subsequent agents.

This architecture supports a form of continual improvement through readable external memory rather than relying exclusively on weight updates. New environments can be written as code and rendered through a common interface, while agents can inherit and refine knowledge across repeated interactions.

2. Code-world formulation and interaction loop

A code world consists of a scene program $p$, an engine, persistent scene state, camera pose, structured visual conditions, and the shared renderer $R_\theta$. The state transition is defined by the program:

$(s_{t+1},\pi_{t+1})=f_p(s_t,\pi_t,a_t),$

where $s_t$ is the world state, $\pi_t$ is the observing camera pose, and $a_t$ is an action from one or more agents. The engine extracts structured conditions such as depth or surface normals:

$c_t=h_p(s_t,\pi_t).$

The neural renderer produces an RGB observation using visual history, structured conditions, an appearance reference, and a textual scene description:

$x_t=R_\theta(x_{<t},c_{\leq t},x_0,y).$

Here, $x_{<t}$ denotes previously rendered frames, $c_{\leq t}$ the current and previous structured conditions, $R_\theta$0 an appearance reference image, $R_\theta$1 a text description or semantic scene prompt, and $R_\theta$2 the observation returned to the agent.

The complete interaction loop is:

1. The agent receives rendered frames.
2. It selects an action, possibly expressed as a short Python program.
3. The engine applies the action through $R_\theta$3 and updates the authoritative state.
4. The engine updates the camera pose and exports new structured conditions.
5. The renderer combines the new conditions with the appearance reference, text, and cached visual history.
6. The renderer produces the next visual observation.
7. The agent evaluates the consequence and acts again.

Rendered pixels do not directly update simulator state. This prevents visual-generation errors from accumulating as physical or logical state errors. Conversely, the renderer can only represent state information encoded in its conditions, appearance reference, text, and visual history. Exact material identity, color, and some rotations may therefore need to be inferred from appearance and temporal context rather than depth or normals alone.

A new world can be authored by specifying geometry, spatial layout, object poses, articulations, collision and contact behavior, action spaces, task variables, rewards or terminal conditions, multi-agent interaction, camera locations, and observation rules. The program does not need to provide highly detailed materials for every scene; it primarily needs to express layout, silhouettes, occlusion, motion, contact relationships, and interaction dynamics.

AgentGarten also supports image-based and text-based construction. From a single image, perception and generation models can estimate instance masks, monocular depth, camera intrinsics, 3D bounding boxes, and object meshes. A coding agent can fit these estimates into an executable scene, extend unseen regions plausibly, and attach interaction rules. In text-based construction, a coding agent writes a scene from geometric primitives and engine assets, after which an image-editing model can produce an appearance reference while preserving layout.

Generated worlds are validated with multiple probe cameras, test rollouts, contact queries, collision-clearance checks, occlusion checks, and tests for unsupported structures or ambiguous motion. Fully automated generation, validation, and selection of useful worlds remains an unresolved problem.

3. Structured conditioning and neural rendering

AgentGarten primarily uses colorized depth and camera-space surface normals as structured conditions. Both are three-channel signals, allowing reuse of pretrained RGB-video encoders and Transformer architectures. Depth is mapped using:

$R_\theta$4

where $R_\theta$5 is depth and $R_\theta$6 is a shared scale. For metric depth, $R_\theta$7; for unknown-scale depth, the median valid depth is normalized to approximately $R_\theta$8. Surface normals $R_\theta$9 are mapped to RGB-like values through $(s_{t+1},\pi_{t+1})=f_p(s_t,\pi_t,a_t),$0.

Condition features are spatially pooled and embedded as tokens:

$(s_{t+1},\pi_{t+1})=f_p(s_t,\pi_t,a_t),$1

where $(s_{t+1},\pi_{t+1})=f_p(s_t,\pi_t,a_t),$2 is spatial pooling, $(s_{t+1},\pi_{t+1})=f_p(s_t,\pi_t,a_t),$3 is a frozen video encoder, $(s_{t+1},\pi_{t+1})=f_p(s_t,\pi_t,a_t),$4 groups encoded patches, $(s_{t+1},\pi_{t+1})=f_p(s_t,\pi_t,a_t),$5 reuses the pretrained RGB input projection, and $(s_{t+1},\pi_{t+1})=f_p(s_t,\pi_t,a_t),$6 is a zero-initialized geometry-modality embedding. Geometry and RGB tokens are processed jointly with shared temporal rotary coordinates and spatial alignment. Only RGB latent tokens produce velocity predictions.

To reduce overfitting to artifacts from a particular depth or normal estimator, training randomly selects depth or normals, occasionally drops both modalities, suppresses high-frequency texture, applies smooth spatial warps, and adds noise to condition latents. These procedures are intended to support operation with both estimated conditions from real videos and exact conditions exported from engines.

The renderer begins with the video component of Cosmos 3-Nano, consisting of 36 Transformer layers, hidden width 4096, 32 query heads, and 8 key-value heads. The text and understanding tower and the Wan video autoencoder remain frozen. The autoencoder compresses time by four and space by sixteen. At $(s_{t+1},\pi_{t+1})=f_p(s_t,\pi_t,a_t),$7 resolution, a latent frame contains 390 RGB tokens after $(s_{t+1},\pi_{t+1})=f_p(s_t,\pi_t,a_t),$8 patch embedding.

Training proceeds in three stages:

1. Geometry conditioning: the pretrained video model is adapted to accept depth and normal conditions.
2. Causal adaptation: the bidirectional model is converted to block-causal generation.
3. Adversarial Forcing: the renderer is distilled into a few-step model that conditions on its own generated history and receives real-data adversarial supervision.

During causal adaptation, the reference frame is a clean prefix and later frames are divided into blocks of four latent frames. For block $(s_{t+1},\pi_{t+1})=f_p(s_t,\pi_t,a_t),$9, a clean history copy $s_t$0 and noisy target copy $s_t$1 are constructed. The attention pattern permits:

$s_t$2

With clean latent block $s_t$3, noise $s_t$4, and flow time $s_t$5:

$s_t$6

The teacher-forcing objective is:

$s_t$7

where $s_t$8 contains appearance, text, and structured conditions.

4. Adversarial Forcing and exact replay

Adversarial Forcing addresses degradation during long autoregressive rollouts. Ordinary self-forcing can produce repetitive surface patterns and loss of fine detail because generated blocks are inserted into a detached key-value cache. Later losses depend on that cache but cannot update the computation that created the history keys and values. Fully differentiable serial rollout preserves those dependencies but has high memory cost because each block retains a recursively connected computation graph.

AgentGarten uses a two-pass replay procedure. During the first pass, a no-gradient rollout generates the sequence and records, for each block, its input $s_t$9 at the final denoising time $\pi_t$0 and its clean output $\pi_t$1. During the second pass, the rollout is recomputed with gradients through history encoding:

$\pi_t$2

The recorded latent values remain detached, so the sampling trajectory is not differentiated. However, the model recomputes the keys and values that encode earlier outputs inside the computational graph. A loss on a later block can therefore update how earlier visual observations are written into the history cache.

Replay must reproduce the original rollout execution rather than merely its abstract attention mask. A full-sequence FlexAttention replay has different tensor shapes, reduction orders, and kernels from cached blockwise execution, producing numerical divergence in BF16. AgentGarten therefore preserves the original blockwise attention structure, tensor shapes, projection grouping, feed-forward grouping, and key-value ordering.

The reported comparison used one H100, BF16 precision, 36 layers, $\pi_t$3 resolution, and 61 latent frames:

Replay method	Relative $\pi_t$4 replay error	Forward time	Peak memory
FlexAttention replay	3.99%	2.95 s	50.83 GiB
Exact blockwise SDPA replay	0, bitwise identical	2.79 s	50.92 GiB

The exact blockwise method therefore achieves bitwise-identical replay in the reported setting while avoiding the numerical divergence observed with FlexAttention replay.

Adversarial Forcing additionally introduces distribution matching and real-data adversarial supervision. The student prediction $\pi_t$5 is corrupted as:

$\pi_t$6

A frozen teacher and trainable fake-score model produce clean estimates whose normalized difference is used in the DMD objective:

$\pi_t$7

The method also uses a discriminator conditioned on geometry and appearance. With real logit $\pi_t$8 and generated logit $\pi_t$9, the relativistic losses are:

$a_t$0

The student minimizes:

$a_t$1

The discriminator uses exact $a_t$2 gradient regularization through vector-Jacobian and Jacobian-vector products. Because the backbone is frozen, the method constructs a surrogate with the exact penalty value and exact first derivative with respect to the discriminator head parameters, avoiding finite differences and double backward through fused attention kernels.

Qualitative 30-second rollouts compare Self Forcing with Adversarial Forcing. Self Forcing develops repetitive surface patterns and loses detail over time, whereas Adversarial Forcing retains natural textures and fine detail. The reported evidence supports the interpretation that adversarial supervision improves long-horizon visual quality, while exact replay enables future losses to influence history encoding.

5. Real-time deployment and world construction

At inference time, the renderer prefills an appearance reference and text, receives a new condition block, denoises it in four steps, commits it to a bounded key-value cache, decodes pixels, and streams them to the agent. The cache contains a permanent sink prefix, including the reference, and a sliding window of recent latent frames. Top-aligned rotary remapping keeps active positions within the training horizon while preserving relative temporal distances; geometry conditions continue to follow the true simulation timeline.

Deployment uses custom Triton kernels, fused RMSNorm, rotary embedding and gated activations, CUDA graph capture and replay, and a distilled tiny decoder. On one H100 at $a_t$3 resolution, a served 16-frame block requires:

Component	Time
Condition encoding	10.6 ms
Transformer, four denoising steps	414.4 ms
Tiny decoder and host transfer	8.8 ms
Total wall time	438.8 ms
Throughput	36.5 frames/s

The reported throughput exceeds 35 frames per second including condition encoding, denoising, decoding, and host transfer. This supports the use of the renderer inside a fine-grained action–observation loop rather than solely for offline video generation.

The framework can author worlds from images or text. Image-based construction uses estimated masks, depth, camera intrinsics, 3D bounding boxes, and meshes. Text-based construction uses geometric primitives and engine assets. In both cases, coding agents can iteratively test the resulting world through probe cameras, rollouts, contact queries, collision checks, and occlusion checks.

The distinction between state and rendering also permits the same state trajectory to be re-rendered with alternative appearance references or cameras. This enables variation in visual presentation without changing the underlying interaction history. The renderer nevertheless remains limited by the information encoded in the structured conditions and cached visual history. Thin structures, narrow gaps, material identity, hidden state, and motion that does not alter geometry may be difficult to preserve.

6. Agent learning through playbooks

AgentGarten does not use conventional end-to-end reinforcement learning as its primary learning mechanism. A round begins with a task file specifying the goal, available actions, and limits, but not the solution. Agents interact for a fixed time or step budget, perceive the world only through rendered camera observations, execute actions, observe consequences, and accumulate experience.

After each round, agents review their action programs and permitted visual evidence and write playbooks containing:

• successful procedures;
• failed attempts;
• visual cues;
• unresolved doubts;
• proposed tests;
• reusable code or skills.

Fresh agents or fresh conversations in later rounds receive the task file and archived playbooks. They select, test, correct, and extend inherited lessons. In the hide-and-seek environment, hiders and seekers maintain separate libraries of short Markdown skill files, and both successful and failed experience is retained.

This learning mechanism differs from tabula-rasa reinforcement learning. The foundation model is pretrained, experience is transferred as readable and executable knowledge rather than only gradients, each round may use a fresh agent instance, and agents do not receive coordinates, privileged state, or opponent-private observations. The principal learning problem is therefore grounding general semantic and physical priors in closed-loop visual action.

Hide-and-seek

The hide-and-seek environment is a sequential one-hider, one-seeker physics task. The hider prepares the scene, control passes to the seeker, and both agents operate using first-person neural-rendered observations. Actions are submitted as short Python programs, and agents receive no coordinates or hidden state.

The reported milestones are:

Strategy	Prior self-play reinforcement learning	AgentGarten agents
Build shelters	Approximately 25 million episodes	Round 4
Use ramps to enter shelters	Approximately 100 million episodes	Round 10

Observed behaviors included a hider moving a barrier panel to block an entrance, a seeker carrying a ramp to a wall and climbing over it, and a seeker repositioning a ramp after an unsuccessful attempt.

The abstract describes useful behavior emerging after four rounds, but the ramp-use milestone occurs at round 10. Thus, the four-round result does not apply equally to every strategy.

Additional code worlds

Four additional environments were each run for four rounds using the task-file and playbook procedure:

1. Companion dog: maintain engagement for 60 seconds through hand offering, petting, and ball play.
2. One-lane bridge: two cars swap ends on a bridge too narrow for both while minimizing time.
3. Herding: two dogs guide four sheep into a pen and keep them there for five seconds.
4. Quarry loader: push two rocks to staging pads, deliver one behind a wall to a bunker, and park within 360 seconds.

World	Measure	Round 1	Round 2	Round 3	Round 4
Companion dog	Engagement score	13	14	19	19
One-lane bridge	Seconds until both cars arrive	71	68	45	41
Herding	Score out of 100	60	90.1	87.6	88.3
Herding	Sheep penned	3/4	4/4	4/4	4/4
Quarry loader	Score out of 100	30	0	30	90.9

These results show improvement across rounds in several tasks, but the quarry-loader trajectory is non-monotonic: performance drops in round 2 before recovering strongly in round 4. Playbook inheritance therefore improves behavior in the reported settings but does not guarantee monotonic progress.

7. Limitations and significance

AgentGarten’s simulator–renderer separation has important limitations. Depth and surface normals may not encode color, material, object identity, hidden state, angular velocity, thin structures, narrow gaps, or small contact changes. A painted wheel can rotate without changing depth or normals, even though its visible markings should rotate. Visual history may temporarily preserve such information but can lose it after long occlusions or cache eviction.

The renderer remains a learned video generator and may hallucinate appearance details, lose identity or material consistency, fail under unusual camera motion, or degrade under conditions unlike training data. Distribution shifts may arise from differences in depth scale, normal conventions, alignment, texture leakage, occlusion patterns, object distributions, and camera distributions. Condition dropout, warping, noise, and modality randomization are intended to improve robustness but do not eliminate these shifts.

Training required 32 H100 GPUs with fully sharded data parallelism. Exact replay retains approximately 51 GiB in the reported measurement, and real-time deployment requires a high-end H100 with custom kernels. The bounded visual cache also limits long-term memory. Rotary remapping supports longer rollouts but does not restore visual information that has been evicted.

Environment authoring remains a substantial bottleneck. A code world may be visually attractive but unsolvable from its observations, dynamically trivial, excessively difficult, or invalid because of collision and contact errors. The framework makes rendering reusable, but it does not eliminate the need to write, validate, and evaluate world programs. Automatic world-quality assessment is identified as an open problem.

The empirical study is also limited in scope. The supplied results do not provide a large standardized image-quality benchmark, confidence intervals, multiple-random-seed analysis, or a complete factorial ablation of simulator fidelity, renderer quality, playbook inheritance, and agent architecture. The comparisons with tabula-rasa reinforcement learning are not direct sample-efficiency comparisons because the AgentGarten agents begin with pretrained semantic and physical priors, whereas the cited reinforcement-learning systems learn dynamics and strategies from scratch, often with privileged state.

AgentGarten’s broader significance lies in its proposed ecosystem of mutually scaling agents and environments. Code-defined worlds provide persistent and inspectable dynamics; neural rendering provides realistic visual observations; agents close the loop through action and perception; and playbooks provide a readable form of cross-round memory. The framework consequently shifts part of the challenge from bespoke rendering engineering toward the authoring, validation, and selection of useful interactive worlds. Its central contribution is a systems architecture in which environment diversity and agent experience can develop together while preserving a distinction between executable world state and learned visual appearance (Chi et al., 8 Oct 2026).