HM-World Dataset for Hybrid Memory Models

• HM-World is a large-scale video dataset designed to benchmark hybrid memory models that precisely reconstruct static scenes and track dynamic subjects.
• It consists of 59,225 high-fidelity video sequences from 17 diverse 3D environments with detailed exit-entry annotations for robust object permanence evaluation.
• Benchmark results show that models like HyDRA outperform baselines in dynamic subject and background consistency, advancing video prediction and simulation research.

HM-World is a large-scale video dataset introduced to support the development and evaluation of hybrid memory in dynamic video world models. Hybrid memory refers to a model’s dual requirement: (a) to reconstruct static scene elements with precision upon camera revisit, and (b) to continually track and predict the motion and identity of dynamic subjects, even while they are out of view—thus resolving long-standing failures of static memory systems in video generation and simulation tasks (Chen et al., 26 Mar 2026).

1. Motivation and Hybrid Memory Paradigm

The canonical challenge addressed by HM-World is hybrid memory: video world models must simultaneously serve as accurate archivists for background reconstruction and as robust trackers capable of hallucinating plausible offscreen dynamics for moving subjects. Existing datasets largely feature either static backgrounds or simplistic dynamic sequences, offering insufficient coverage of out-of-view subject persistence or in-scene reappearance. HM-World was specifically designed to decouple camera and subject trajectories and to ensure frequent, controlled exit-entry events. These properties enable the precise quantification of memory, tracking, and out-of-view prediction—all previously underexplored at scale.

2. Dataset Structure and Content

HM-World comprises 59,225 high-fidelity rendered video sequences, each of which contains:

• Scene diversity: 17 distinctive 3D environments, spanning urban, rural, natural, and indoor settings (e.g., streets, parks, farms, indoor halls, forests).
• Subject variety: 49 distinct animated actors, including human avatars of varied appearance and a range of animal species.
• Motion trajectories: Each subject follows one of 10 motion-path primitives. Camera motion and subject motion are sampled independently across 28 possible ego-motion routes, enforcing complex exit-entry cycles.

Each sequence consists of continuous camera and subject movement, typically sampled for up to 77 context frames (approximately 3 seconds at 24 fps) for model conditioning, with subsequent frames used for prediction. Data are rendered via Unreal Engine 5 at a default resolution of 256×256 pixels at 24 fps (modifiable on download).

3. Trajectory Design and Exit-Entry Annotations

Camera pose $P_{\mathrm{cam}}(t)$ and subject world position $S_i(t)$ are independently sampled, ensuring that dynamic subjects frequently exit and re-enter the camera’s field of view under controlled, reproducible conditions. For each subject $i$ and every clip, precise exit and entry events are annotated as:

$$t_\text{exit}^{(i)} = \min \{ t \mid \mathrm{proj}(P_\text{cam}(t), S_i(t)) \notin \textrm{image} \}$$

$$t_\text{enter}^{(i)} = \min \{ t > t_\text{exit}^{(i)} \mid \mathrm{proj}(P_\text{cam}(t), S_i(t)) \in \textrm{image} \}$$

where $\mathrm{proj}(.)$ denotes the projection operator from 3D world coordinates to image plane. Every clip guarantees at least one (exit, entry) pair per subject. Average statistics are as follows: 3.2 exit-entry cycles per subject and 1.8 subjects per sequence, with each subject and scene recurring across ~1,200 and 3,500 clips respectively.

4. Data Organization, Splits, and Annotations

The dataset is split into 58,225 training sequences, 1,000 validation sequences, and 1,000 test sequences, with the test and validation splits drawn from held-out scenes and subjects to ensure generalization.

Each sample contains:

• Raw video (MP4) and per-frame PNGs.
• Camera poses ($P_\text{cam}(t)$) as 4×4 matrices in JSON.
• Per-frame subject world positions and pose keypoints in JSON.
• Explicit exit-entry event timestamps ($t_\text{exit}^{(i)}$, $t_\text{enter}^{(i)}$) per subject.
• MiniCPM-V–generated video caption (plain text).

The directory structure is organized by split as follows: $S_i(t)$2

Total dataset size is approximately 120 GB. Licensing is CC BY-NC-4.0, permitting research and noncommercial use.

5. Supported Research Tasks and Metrics

HM-World is intended for three primary research tasks:

• Video Prediction & World Modeling: Model must generate plausible future frames under novel camera and subject trajectories.
• Object Permanence / Long-term Tracking: Model is tested on its ability to extrapolate hidden subject motion and maintain identity coherence during out-of-view intervals.
• Long-Horizon Consistency: Simultaneous evaluation of background reconstruction and subject dynamics over extended sequences.

Evaluation is conducted with the following metrics:

• Standard Video Quality: PSNR, SSIM, LPIPS.
• Frame-Level Consistency: Subject Consistency and Background Consistency as per the Vbench protocol.
• Dynamic Subject Consistency (DSC): Given CLIP feature maps $$F^{\mathrm{pred}}, F^{\mathrm{gt}}, F^{\mathrm{ctx}} \in \mathbb{R}^{d \times H' \times W'}$$ for the predicted, ground-truth, and previous context subject regions, DSC is defined as:

$S_i(t)$0

with

$S_i(t)$1

6. Baselines and Experimental Performance

Empirical results on the test set (1,000 clips) provide reference values for several baseline and state-of-the-art models including Baseline (no memory), DFoT, Context-as-Memory, WorldPlay, and HyDRA (the architecture proposed in conjunction with HM-World). The HyDRA model demonstrates superior performance in dynamic subject consistency and background coherence. The following table summarizes benchmark results for key metrics:

Method	PSNR	SSIM	LPIPS	DSC_ctx	DSC_GT	Subj. Cons.	Bg. Cons.
Baseline (no memory)	18.696	0.517	0.356	0.812	0.837	0.903	0.925
DFoT (ICML ’25)	17.693	0.482	0.410	0.803	0.826	0.893	0.913
Context-as-Memory	18.921	0.530	0.342	0.816	0.839	0.911	0.922
WorldPlay (zero-shot)	14.855	0.355	0.500	0.822	0.832	0.910	0.925
HyDRA (ours)	20.357	0.606	0.289	0.827	0.849	0.926	0.932

HyDRA’s improvements are particularly marked in SSIM, LPIPS, subject, and background consistency, reflecting advances in hybrid memory representation and retrieval (Chen et al., 26 Mar 2026).

7. Access and Relevance

HM-World is distributed at https://kj-chen666.github.io/Hybrid-Memory-in-Video-World-Models/ with comprehensive documentation and per-split downloads. Its introduction addresses a significant gap for hybrid-memory video world model research, enabling systematic study and benchmarking of both in-view and out-of-view dynamic tracking, robust object permanence, and long-range spatial-temporal consistency in generative video models (Chen et al., 26 Mar 2026).

A plausible implication is that HM-World’s release and adoption will accelerate progress in physically plausible, memory-augmented video world models with applications spanning robotics, autonomous agents, and embodied AI.