HM-World: Hybrid Memory & Dynamic Models

• HM-World is a framework that combines world modeling, hybrid memory, and embodied perception to simulate coherent 3D environments.
• It employs innovative techniques like memory tokenization and spatiotemporal retrieval to enhance dynamic subject continuity and background consistency.
• Applications in human motion recovery, camera trajectory estimation, and robotic planning demonstrate HM-World’s impact on improving real-time control and perception.

HM-World defines a class of datasets, benchmarks, and modeling paradigms at the intersection of world modeling, memory, embodied perception, and planning, unified by the requirement to model and reason about world-coherent phenomena such as 3D spatial structure, dynamic object continuity, and agent–environment interaction under uncertainty. The term encompasses several major instantiations, including large-scale video datasets for hybrid memory in dynamic scene generation, multimodal benchmarks for world-grounded human motion and camera trajectory estimation, and data-driven world models supporting embodied robotic planning. Across these lines, HM-World advances general-purpose agents and simulation frameworks that must simultaneously track, memorize, and predict both static components and dynamic actors within complex, three-dimensional worlds. Below, core HM-World contributions, mechanisms, and applications are articulated.

1. HM-World Datasets: Motivation and Composition

HM-World datasets are constructed to address critical shortcomings of prior video world models, particularly with respect to hybrid memory and decoupled object–scene dynamics (Chen et al., 26 Mar 2026). Conventional models, which operate on a “static canvas” paradigm, fail to maintain appearance and motion continuity for dynamic subjects that move out-of-view and later re-enter, resulting in frozen, corrupted, or missing agents.

The canonical HM-World dataset consists of 59,225 video clips, each containing at least one subject exit–entry event. Clips span 17 scenes (urban, forest, indoor) and feature 49 distinct dynamic subjects (human and animal avatars), with motion and camera trajectories parametrically decoupled during simulation:

• Camera pose: sequence $P = \{P_i\}$, $P_i = (R_i, t_i)$ ($R_i \in SO(3), t_i \in \mathbb{R}^3$), recorded per frame.
• Subject trajectory: $S_j = \{s_{j, k}\}$, $s_{j, k} \in \mathbb{R}^3$, for each subject $j$.
• Statistics: average 1.8 exit–entry events per clip (mean occlusion 40 frames), mean length 150 frames, per-scene mean 3,484 clips.

Rendering is performed in Unreal Engine 5 with full annotation of RGB, depth, bounding boxes, 3D poses, and trajectory metadata, guaranteeing hybrid memory stress events by design (Chen et al., 26 Mar 2026).

2. Hybrid Memory Paradigm and the HyDRA Model

Hybrid memory, as operationalized in HM-World, is the requirement that a model be both (a) a static archivist for unchanging backgrounds, and (b) a dynamic, identity-consistent tracker for subjects as they undergo occlusions.

To support this paradigm, the HyDRA architecture employs:

• Memory tokenization: latent histories $Z_{mem}$ are compressed to spatiotemporal tokens $M = T_{mem}(Z_{mem})$ via 3D convolutions, enabling disentanglement of dynamic and static cues.
• Spatiotemporal relevance-driven retrieval: for each future query $q_i$, similarity scores

$$S_{i,j} = \frac{1}{\sqrt{d}} \sum_{x,y} \langle \tilde{q}_i(x, y), k_{mem,j}(x, y) \rangle$$

select the top-K tokens for attention, ensuring dynamic subjects can be tracked even during off-screen intervals.

By leveraging decoupled $P_i = (R_i, t_i)$0 and $P_i = (R_i, t_i)$1 annotations, HyDRA achieves markedly superior dynamic subject consistency (DSC) and background consistency (BC) compared to prior video generation models (Chen et al., 26 Mar 2026).

3. World-Grounded Human Motion and Camera Estimation

World-coordinate recovery of human and camera trajectories from monocular video is a major HM-World application, exemplified by HumanMM (Zhang et al., 10 Mar 2025) and WHAC (Yin et al., 2024). The methodological core is the joint estimation of expressive body pose/shape (e.g., SMPL-X models), global human trajectory, and camera extrinsics within a shared metric world frame. Both approaches address scale ambiguity and discontinuities induced by multi-shot transitions or occlusions:

• HumanMM: Introduces a robust alignment module to stitch together per-shot pose and orientation across video cuts, employing epipolar geometry and learned pose refinement. Temporal consistency and foot-ground contact are enforced via custom integrators, yielding low root-translation (RTE) and orientation error (ROE) in evaluation (Zhang et al., 10 Mar 2025).
• WHAC: Fuses (i) monocular human depth from weak-perspective projection and known intrinsics, (ii) relative camera trajectories from visual odometry (VO), and (iii) human motion priors to reconstruct world-metric human and camera trajectories. Canonicalization and rigid Umeyama alignment recover true scale, enabling real-time performance and resilience to scale ambiguity (Yin et al., 2024).

The WHAC-A-Mole dataset provides synthetic benchmarks for such joint estimation, with 1.46M annotated SMPL-X frames and ground-truth camera–human transforms.

4. Datasets for Embodied Perception and Semantic Navigation

Beyond synthetic or rendered environments, the HM3D(HM-World) and Habitat-Matterport 3D Semantics (HM3DSem) datasets provide high-fidelity, densely annotated real-world spaces. These encompass:

• 216 3D scanned scenes, spanning 3,100 rooms, with 142,646 object-instance (≈108k non-architectural) annotations assigned as textures for pixel-accurate object boundaries.
• 40 canonical object categories, mapped from 1,625 raw text strings.

Annotation methodology is based on artist-painted ID textures, aligning perfectly with RGB, facilitating semantic simulator tasks (e.g., ObjectNav) with unprecedented scale and object diversity (Yadav et al., 2022).

Empirical results show that reinforcement- and imitation-learning policies trained on HM3DSem outperform those trained on Replica, Gibson, ScanNet, or MP3D for navigation and generalization across evaluation splits. Metrics such as Success Rate (SR) and SPL (“Success weighted by Path Length”) provide quantitative measures.

5. World Models for Humanoid Robotics and Planning

Humanoid World Models (HWM) (Ali et al., 1 Jun 2025) exemplify video-based, action-conditional world models for real-time planning and RL in open-world humanoid robotics. They introduce a family of lightweight models, efficiently trainable on 1–2 GPUs using 100 hours of egocentric humanoid demonstration video.

Two primary architectures are provided:

• Masked-Transformer HWM: Discrete latent sequences (via VQ-VAE), transformer blocks with modular attention and iterative denoising. Multiple attention and parameter-sharing strategies trade off memory use and fidelity.
• Flow-Matching HWM: Continuous VAE latents mapped by learned velocity fields via ODE integration, sharing architectural motifs with Masked-HWM but operating in continuous space.

Empirical evaluation on standard video metrics (FID, PSNR) shows that parameter sharing variants can reduce model size by 33–53% with minimal reduction in visual fidelity. These models are readily integrated into model-based planning loops, e.g., via latent-space rollouts, CEM-based action search, and “what if” counterfactual prediction.

6. Structured World Models and Soft-Hamiltonian Latent Dynamics

HaM-World (Tang et al., 7 May 2026) introduces a planner-facing latent dynamics interface with explicit Hamiltonian structure and Markovian memory. The latent is decomposed as:

• Canonical subspace $P_i = (R_i, t_i)$2 (configuration, momentum; geometric organization)
• Context subspace $P_i = (R_i, t_i)$3 (semantic, dissipative, non-conservative effects)
• Mamba selective memory state $P_i = (R_i, t_i)$4: enables history-conditioned memory for approximate Markov completeness

The evolution of $P_i = (R_i, t_i)$5 follows a Soft-Hamiltonian vector field with learnable residual (control) dynamics. The Mamba memory stack handles partial observability and action delay, feeding all history into the shared transition core:

$P_i = (R_i, t_i)$6

The unified interface serves all planner tasks: rollouts, reward/value heads, and CEM-based action plan search. Training objectives include JEPA-style representation loss, long-horizon rollout consistency, symlog/two-hot reward and value heads, and multiple geometric regularizers (Hamiltonian alignment, no-action energy drift).

On DeepMind Control Suite, HaM-World sets the highest AUC (+9.5%), halves multi-step latent rollout error, and demonstrates OOD robustness (e.g., +10.2% on Finger Spin under perturbations), validating the utility of geometrically structured, selectively memorized latent spaces for stable planning (Tang et al., 7 May 2026).

7. Evaluation Protocols and Benchmark Metrics

A unifying feature across HM-World instantiations is rigorous, world-coherent evaluation. Metrics are designed to probe both static and dynamic memory, spatial consistency, and prediction quality:

• Static coherence: Background Consistency (BC), PSNR, SSIM, LPIPS over static regions.
• Dynamic coherence: Subject Consistency (SC), Dynamic Subject Consistency (DSC), based on CLIP feature similarity during exit–entry cycles (Chen et al., 26 Mar 2026).
• Navigation/perception: Success Rate, SPL computed over thousands of evaluation episodes (Yadav et al., 2022).
• Embodied prediction/planning: FID, PSNR in video synthesis; PA-MPJPE, WA-MPJPE, RTE, ROE for 3D trajectory estimation (Zhang et al., 10 Mar 2025, Yin et al., 2024).
• Control/model-based planning: Area-under-curve of learning curves, multi-step rollout error, and OOD returns under perturbations (Tang et al., 7 May 2026).

These protocols ensure that progress in HM-World is understood both in terms of memory, continuity, and high-level agent–world performance.

---

References: (Chen et al., 26 Mar 2026, Zhang et al., 10 Mar 2025, Yin et al., 2024, Yadav et al., 2022, Ali et al., 1 Jun 2025, Tang et al., 7 May 2026)