Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models

Abstract: Video world models have shown immense potential in simulating the physical world, yet existing memory mechanisms primarily treat environments as static canvases. When dynamic subjects hide out of sight and later re-emerge, current methods often struggle, leading to frozen, distorted, or vanishing subjects. To address this, we introduce Hybrid Memory, a novel paradigm requiring models to simultaneously act as precise archivists for static backgrounds and vigilant trackers for dynamic subjects, ensuring motion continuity during out-of-view intervals. To facilitate research in this direction, we construct HM-World, the first large-scale video dataset dedicated to hybrid memory. It features 59K high-fidelity clips with decoupled camera and subject trajectories, encompassing 17 diverse scenes, 49 distinct subjects, and meticulously designed exit-entry events to rigorously evaluate hybrid coherence. Furthermore, we propose HyDRA, a specialized memory architecture that compresses memory into tokens and utilizes a spatiotemporal relevance-driven retrieval mechanism. By selectively attending to relevant motion cues, HyDRA effectively preserves the identity and motion of hidden subjects. Extensive experiments on HM-World demonstrate that our method significantly outperforms state-of-the-art approaches in both dynamic subject consistency and overall generation quality.

• The paper introduces a hybrid memory paradigm using HyDRA and HM-World, which precisely maintains dynamic subject consistency during occlusions.
• The approach employs a 3D-convolutional memory tokenizer and dynamic retrieval attention, achieving superior metrics like PSNR and SSIM over baselines.
• Empirical evaluations demonstrate that the method enables robust dynamic recall, crucial for applications in autonomous driving and embodied intelligence.

Hybrid Memory for Dynamic Video World Models: HyDRA and HM-World

Introduction

This work addresses the fundamental limitations of current memory mechanisms in video world models, particularly their inability to maintain the motion and appearance consistency of dynamic entities when these subjects move out of the camera's field of view and later re-emerge. Traditional memory solutions treat the environment as a static scene, leading to severe artifacts—static, distorted, or missing subjects—upon re-entrance. The paper introduces the concept of Hybrid Memory, requiring simultaneous precise retention of static backgrounds and dynamic tracking of subject motion through occlusions and re-entry events. A new large-scale dataset (HM-World) and specialized model architecture (HyDRA) are presented to operationalize and study this paradigm.

Figure 1: Instances of exit-entry camera motion essential for evaluating hybrid memory capacity.

HM-World: Dataset for Hybrid Memory

High-quality, dynamic video data specifically tailored for hybrid memory evaluation are rare in existing literature. The HM-World dataset is constructed within Unreal Engine 5 to simulate complex, dynamic video environments with exhaustive annotation. It features:

• 59K high-fidelity clips across 17 scene types
• 49 distinct subjects with diverse appearance and species
• Ten subject motion types and 28 camera trajectories, deliberately decoupled to induce frequent subject exit and re-entry events

The rendering pipeline ensures that motion correlation between camera and subject is disentangled, yielding challenging out-of-view intervals and reappearance tasks.

Figure 2: Construction procedure of HM-World, highlighting scene selection, subject and trajectory sampling, and combinatorial camera motion.

Compared to prior datasets, HM-World provides far superior dynamic complexity and annotated re-entrance events, directly supporting experiments on memory that integrates static and dynamic content—central to the study of advanced world models.

HyDRA: Hybrid Dynamic Retrieval Attention

Base Architecture

HyDRA builds on a high-capacity video diffusion model (causal 3D VAE + DiT), with explicit camera pose injection for spatial conditioning. Information about past frames and camera states is compressed and injected into the generative process through a novel hybrid memory module.

Memory Tokenization

Rather than exposing the generative model to raw, unfiltered past latents—which leads to information overload and degraded consistency—HyDRA employs a 3D-convolutional memory tokenizer. This module extracts memory tokens that jointly encode spatial and temporal context, emphasizing long-duration subject motion and facilitating discriminative retrieval of the latent state.

Dynamic Retrieval Attention

A dynamic retrieval module computes fine-grained, spatiotemporal affinities between current generation queries and stored memory tokens. For each target frame, the model dynamically selects the k most relevant tokens that maximize spatiotemporal feature correlation, ensuring that appearance and motion information about dynamic subjects is adaptively recalled only when pertinent. Local attention windows are also fused, balancing long-term context and local temporal stability.

Figure 3: Overview of HyDRA, including memory tokenization and dynamic retrieval attention for identifying and tracking out-of-view subjects.

Empirical Evaluation

Comparative and Ablation Results

On the HM-World test set (1K samples, unseen scenes/subjects), HyDRA is compared to baselines (direct latent concatenation), DFoT (history-based context), Context-as-Memory (FoV-overlap retrieval), and WorldPlay (state-of-the-art commercial baseline).

Strong quantitative gains are realized across all standard and proposed metrics, including PSNR (20.36 for HyDRA vs 18.92 for Context-as-Memory), SSIM, LPIPS, and consistency metrics (Subject, Background, and the newly proposed DSC—Dynamic Subject Consistency). Notably, HyDRA achieves the highest $\text{DSC}_{GT}$, indicating it best preserves ground-truth subject identity and motion across occlusion intervals.

Ablations reveal:

• Increasing the temporal kernel size of memory tokenization is critical; spatial dimensions are less sensitive, but temporal aggregation is necessary for hybrid memory.
• Retrieval of more tokens (up to ~10) is optimal—insufficient tokens harm dynamic recall, excess tokens add noise.
• Dynamic feature-level affinity is superior to FOV geometry-based retrieval, yielding more diverse and contextually relevant memory selection.

Qualitative visualizations show HyDRA can recover correct subject re-entrance locations, appearances, and motions, unlike competing approaches, which either freeze or hallucinate subjects. Analysis of retrieval patterns further confirms that HyDRA's token selection is rich and adaptive, as opposed to the rigid, fixed retrieval of FOV-overlap mechanisms.

Figure 4: Qualitative comparison with other methods showing HyDRA’s superior handling of subject exit-entry and motion continuity.

Figure 5: Comparison between retrieval methods; HyDRA’s feature affinity retrieval enables successful recall of relevant subject memory while FoV-based retrieval does not.

Figure 6: Distribution of token selection across DiT layers—dynamic affinity retrieval has adaptive, broad receptive fields versus the static selection in FoV overlap.

Generalization

Open-domain experiments on real, internet-sourced video further indicate HyDRA's capacity to generalize dynamic recall and maintain hybrid memory, even out-of-distribution.

Figure 7: Open-domain results confirming robust hybrid memory of HyDRA in unseen environments.

Implications and Future Directions

This research decisively demonstrates that retrieval-augmented hybrid memory is critical for maintaining spatiotemporal coherence in learned video world models, especially in dynamic, agent-rich settings. The paradigm shift from static scene memory to hybrid memory is necessary for progress toward more generalizable, reliable world simulators and generative agents. The performance gap observed with HyDRA underscores the deficiency of prior approaches that only consider static or field-of-view-localized memory.

Practically, this enables better simulation of physical and social environments required in autonomous driving, embodied intelligence, and other settings where occlusions and dynamic agent re-entry are routine.

Theoretically, the work motivates new lines of inquiry around memory tokenization, adaptive retrieval, and scalable attention over memory in long-horizon video generative models. Key challenges remain concerning scaling to very complex, multi-agent occlusion scenarios, as HyDRA's consistency degrades with more than three interacting subjects or highly cluttered scenes. Further research into hierarchical and entity-centric memory representations, as well as the integration of explicit tracking or prediction modules, could help address these issues and pave the way for robust real-world applications.

Conclusion

By introducing the hybrid memory paradigm, the HM-World dataset, and the HyDRA architecture, this paper makes clear empirical and conceptual contributions to video world modeling. The results provide a compelling benchmark and method suite, setting a new standard for future research in the dynamic, memory-augmented generative modeling of physical worlds.