Echo-Memory: A Controlled Study of Memory in Action World Models

Abstract: We present \textbf{Echo-Memory}, a controlled study of memory mechanisms in action-conditioned world models. These models generate multi-segment videos from a first frame, text prompt, and camera-action sequence, but their central failure is often memory rather than local image synthesis: after the camera leaves and returns, the scene or salient object may silently change. Existing memory designs are hard to compare because gains are entangled with backbone, training, retrieval, and evaluation differences. Echo-Memory fixes the action-to-video interface and varies only how history is stored and read by the generator. Under a shared video diffusion backbone, optimizer, camera-action representation, sampler, and evaluation pipeline, we compare raw context, compression-based memory, spatial summaries with different read-out paths, and state-space recurrence. This matched matrix separates four otherwise conflated axes: \emph{capacity}, \emph{compression}, \emph{read-out}, and \emph{recurrence}. We also evaluate memory through a three-branch protocol: replay quality, in-domain loop revisit, and open-domain return probes. The branches routinely disagree, showing that replay fidelity is not a sufficient proxy for remembering a world. Three findings follow. Raw context is a strong capacity baseline and improves open-domain return far more than it improves replay metrics. Compactness is not a free substitute for capacity: aggressive spatial and hybrid-compression memories lose the salient evidence needed for return. Finally, block-wise state-space recurrence is the strongest open-domain return mechanism in our matrix, showing that the structure of implicit memory matters as much as the decision to use it. These results provide a compact protocol for studying memory in action world models beyond isolated replay metrics.

• The paper isolates memory as the sole experimental variable, demonstrating how different architectures impact world persistence and scene consistency.
• It reveals that high replay fidelity does not guarantee semantic memory, as increasing raw context significantly improves open-domain return metrics.
• The study advocates for multi-metric benchmarks and refined memory architectures that balance storage, compression, and retrieval in action-conditioned models.

Echo-Memory: Controlled Comparison of Memory Mechanisms in Action World Models

Introduction and Motivation

The central failure mode of action-conditioned world models—those tasked with generating long-horizon, chunk-wise video rollouts from initial states, text prompts, and camera/action sequences—is drift or loss in scene or object identity after leaving and revisiting viewpoints. This is fundamentally a memory, not a generation, failure. Standard practice in the field has made the comparison of memory mechanisms ambiguous due to experimental confounds with backbone, training schedules, retrieval strategies, and metrics. This work introduces Echo-Memory, a controlled empirical study that isolates memory as the sole experimental variable within a standardized workflow, aiming to reveal the true impact of architectural choices on world persistence and consistency.

Figure 1: The Echo-Memory workflow, where text, historical context, and camera/action states constrain memory design for chunk-wise video generation with revisitation; Context, Compression, Spatial, and State-Space are contrasted as memory architectures.

Taxonomy and Methodology

Memory Mechanism Space

Echo-Memory systematically varies only the representation and retrieval of historical context used by a shared video diffusion-transformer backbone. The investigated axes are:

• Context: Increasing the number $K$ of retrieved raw visual observations.
• Compression: Reducing cost by token weighting, pooling, or hybrid approaches on latent sequences.
• Spatial Memory: Compact scene representations (e.g., grid summaries) with ablations on read/write paths.
• State-Space (Recurrent) Memory: Different variants of recurrent states, emphasizing the structural integration depth of recurrence into the model.

  Figure 2: Four representative approaches to memory in action world models, emphasizing storage and access methods under a unified backbone and interface.

Evaluation Protocol

A three-branch evaluation framework is employed:

1. Replay: Standard long-horizon image reconstruction with GT camera motion.
2. In-domain Return: Leave-and-return on a known loop within the training domain, using pixel and VLM (e.g., Qwen3-VL) alignment for semantic return.
3. Open-domain Return: Evaluation under distribution shift; identity-anchored first frames are created with generative editing, and only semantic preservation (not direct pixel alignment) is scored with a strong VLM on appearance, presence, camera view, and background.

   Figure 3: Taxonomy of the evaluation protocol: replay, in-domain, and open-domain return, each providing distinct stress axes for memory robustness.

In addition, the open-domain pool (Figure 4) systematizes object-level identity probes not tied to training distribution statistics.

Figure 4: Example first frames for open-domain revisit, with distinctive identity anchors for VLM-based evaluation.

Results and Empirical Analysis

Replay ≠ Memory: Metric Disagreement

A principal finding is pronounced metric decoupling. Spatial Memory mechanisms can attain the highest replay PSNR, while block-wise State-Space memory ranks lowest in pixel fidelity but achieves the highest open-domain semantic return ($69.0$ VLM score). Conversely, capacity in raw context (e.g., $K=20$) improves open-domain return dramatically (from $12.25$ to $58.63$) while yielding diminishing gains in replay metrics. Compression by hybrid operators, meanwhile, sharply degrades open-domain return despite shrinking context size.

Figure 5: Replay progression for various models over a fixed camera trajectory, illustrating when and where drift or loss of world structure initiates.

Figure 6: Replay quality is a necessary health check, but does not serve as a sufficient score for memory; different mechanisms invert ranking between replay and open-domain return.

These reversals empirically falsify the assumption that replay metrics are reliable proxies for memory quality in world modeling.

Storage, Compression, and Read-out Implications

Storage versus access: Ablations on spatial memory read-out clarify that improvements in replay from memory modules can result from regularization, not actual utility at revisit. The ability of the generator to access and exploit stored tokens at return is an independent failure point from capacity.

Compression: Length-based temporal compression (e.g., pooling with $r=4$) can outperform token-weighting, whereas hybrid compression destroys necessary object-level evidence, as measured by a precipitous drop in open-domain VLM consistency.

Raw Context: The improvement in semantic return is steep for low $K$ (from $K=1$ to $K=5$) and then saturates sublinearly. This holds regardless of whether models employ explicit memory—demonstrating that uncompressed, strongly supervised context provides a stringent baseline many mechanisms do not exceed.

Figure 7: Left: Scaling context length bolsters open-domain semantic return much more sharply than image-level replay. Right: Higher replay efficiency (GPU-hours per step) does not linearly predict semantic memory capacity.

State-Space (Recurrent) Memory

Block-wise recurrence, which tightly integrates state propagation with the transformer backbone, achieves the strongest open-domain return at the expense of local replay fidelity. This demonstrates that the structure, not just presence, of recurrence, is crucial for world persistence under trajectory excursion and return.

Figure 8: In open-domain return, high replay fidelity does not entail preservation of object identity after a leave-and-return excursion.

Design and Evaluation Implications

The study dictates that memory design for world models must report a bundle of orthogonal metrics, not a single scalar, and establishes that:

• Replay metrics alone will lead to misleading design incentives by favoring mechanisms with local regularity but no world persistence.
• Storage capacity, semantic compression, and read-out accessibility all independently determine return quality.
• Raw context should be the comparison baseline for compact or implicit memory proposals, not minimal anchor-only models.
• Future-efficient memory modules must combine cost reduction with explicit, return-aware supervision and object-level read-out.

The multi-branch evaluation protocol, leveraging strong VLM-based open-domain probes, is necessary to surface failures that pixel losses and replay metrics systematically mask.

Theoretical and Practical Outlook

Echo-Memory provides not a new memory module but a rigorous protocol for controlled empirical separation of memory axes in video world models, defining target properties for next-generation architectures. Research directions implied by these findings include:

• Architectures coupling explicit object-level retention to recurrent or spatial slots, with direct supervision by downstream revisit probes.
• Compression strategies incorporating saliency or attention to world-critical anchors, departing from uniform pooling or token weighting.
• Unified benchmarks reflecting distributional shifts and genuine out-of-sight/excursion recovery.

These findings and protocols will be essential for the progress of world models towards interactive, editable, and persistent scene understanding across vision, robotics, and embodied AI domains.

Conclusion

Echo-Memory advances the empirical methodology for evaluating and designing memory in action-conditioned world models. By rigorously controlling analysis confounds and separating evaluation axes, it demonstrates that the strongest replay mechanism often fails at semantic memory, and the most robust return mechanism may sacrifice reconstruction fidelity. Practically, robust memory calls for rethinking compression, read-out, and recurrent design, as well as standardized, multi-branch benchmarks. These insights recalibrate both research and deployment trajectories for long-horizon, interactive world modeling.