AnchorWorld: Unified Egocentric Simulation

• AnchorWorld is a unified framework for embodied egocentric simulation that integrates explicit human motion input with localized anchor-view customization to generate controlled video sequences.
• It employs hybrid-view supervision, pose-anchored injections, and prompt-driven evolution within a diffusion backbone to achieve high scene consistency, trajectory accuracy, and visual quality.
• Its unified coordinate system and dynamic text prompts allow for seamless spatio-temporal evolution, enabling refined simulation in complex, interactive environments.

AnchorWorld is a unified framework for embodied egocentric world simulation that enables highly controllable video generation via explicit human motion input and localized world customization. The approach addresses longstanding limitations in egocentric simulation—namely, insufficient interaction integrity stemming from occluded or truncated first-person views, and the lack of flexible mechanisms for spatio-temporal scene evolution in complex environments. AnchorWorld achieves these capabilities by weaving together hybrid-view supervision, pose-anchored anchor injections, and compositional, prompt-driven world evolution within a diffusion backbone. Empirical results demonstrate that AnchorWorld consistently surpasses prior baselines in scene consistency, trajectory accuracy, and prompt-alignment, while maintaining or exceeding visual quality across benchmarks (Li et al., 5 Jun 2026).

1. System Architecture and Module Overview

AnchorWorld integrates three tightly coupled modules, each addressing a fundamental obstacle in egocentric world modeling:

1. Hybrid‐View Egocentric Renderer: The core video synthesis engine is a flow-matching DiT backbone (Wan2.2 TI2V 5B) generating 77-frame 480p egocentric videos. Conditioning is provided by both 3D human-motion sequences and a set of anchor-view priors. Input video latents $z_t$ are iteratively denoised via a U-Net with interleaved self- and cross-attention, yielding $\hat z_{t-1}$ at every diffusion step.
2. Exogenous-View Auxiliary Supervision: To overcome egocentric data’s inherent occlusion, pre-training leverages large-scale exogenous (third-person) datasets, such as MultiCamVideo and internal captures, providing full-body supervision and spatial grounding. Exocentric and egocentric camera parameters share a common projection-based action control framework, enabling seamless transfer and fusion.
3. Anchor-View Customization: Users define sets $\mathcal{A} = \{(I_i, P_i, t_i)\}_{i=1}^n$ where each anchor comprises an RGB image $I_i$, a 6-DoF pose $P_i = [R_i | t_i]\in\mathbb R^{3\times 4}$ in a global coordinate system, and an evolution prompt $t_i$ dictating localized scene dynamics. These anchors are injected into the model via frame-axis latent concatenation, 3D RoPE positional embeddings, and masked cross-attention conditioned on textual semantics.

2. Mathematical Foundations and Conditioning Mechanisms

AnchorWorld’s architectural pipeline leverages several technical innovations in conditioning and representation:

• Motion and Camera Encoding: Human motion $M \in \mathbb R^{f\times k\times 6}$, comprising 3D joint position and axis-angle orientation, is downsampled temporally and processed into tokens $E_m\in\mathbb R^{f'\times k\times d}$. Camera trajectories $C\in\mathbb R^{f\times 3\times 4}$ are similarly encoded as $E_c$.
• Spatial Pose Attention: At each self-attention block, video latents $\hat z_{t-1}$0 are concatenated with $\hat z_{t-1}$1 and $\hat z_{t-1}$2:

$\hat z_{t-1}$3

Multi-head attention updates the latents, after which auxiliary pose tokens are truncated, preserving only the transformed video representation.

• In-Context Anchor Injection: Anchor images $\hat z_{t-1}$4 are encoded by a VAE to latents $\hat z_{t-1}$5, then stacked:

$\hat z_{t-1}$6

All pose embeddings $\hat z_{t-1}$7 are broadcasted and spatially added, grounding tokens in the global 3D frame.

• Text-Driven Local Evolution: Evolution prompts $\hat z_{t-1}$8 are encoded into keys/values $\hat z_{t-1}$9 for cross-attention, restricted by a locality-preserving mask:

$\mathcal{A} = \{(I_i, P_i, t_i)\}_{i=1}^n$0

This mechanism enforces non-interference between anchors and maintains localized evolution trajectories per instruction.

• Auxiliary Spatial Grounding Loss: Supervision combines standard diffusion $\mathcal{A} = \{(I_i, P_i, t_i)\}_{i=1}^n$1 loss,

$\mathcal{A} = \{(I_i, P_i, t_i)\}_{i=1}^n$2

and a reprojection penalty,

$\mathcal{A} = \{(I_i, P_i, t_i)\}_{i=1}^n$3

balancing reconstruction with spatial grounding via $\mathcal{A} = \{(I_i, P_i, t_i)\}_{i=1}^n$4.

3. Unified World Coordinate System and Anchor Poses

AnchorWorld employs a single global (“world”) coordinate frame to ensure consistent spatial referencing for all entities:

• World Frame: 3D poses for the human skeleton, camera trajectories, and anchor views are defined in this space.
• Anchor Pose: Each anchor’s camera pose $\mathcal{A} = \{(I_i, P_i, t_i)\}_{i=1}^n$5 serves as a rigid body transformation mapping $\mathcal{A} = \{(I_i, P_i, t_i)\}_{i=1}^n$6 (world points) to $\mathcal{A} = \{(I_i, P_i, t_i)\}_{i=1}^n$7 (camera frame): $\mathcal{A} = \{(I_i, P_i, t_i)\}_{i=1}^n$8.
• Egocentric Head Pose: The agent’s head camera pose $\mathcal{A} = \{(I_i, P_i, t_i)\}_{i=1}^n$9 is handled equivalently, differing only by which transformation is active (by swapping $I_i$0 for exo/ego adaptation).

Coordinate transforms are:

$I_i$1

where $I_i$2 applies camera intrinsics for projection to image space. This unified system allows for seamless injection of view anchors and synchronizes egocentric and exocentric information.

4. Dynamic Evolution via Textual Prompts

AnchorWorld enables prescribed local scene evolution by using natural-language prompts at anchor points:

• Each anchor’s prompt $I_i$3 is tokenized and encoded by a frozen text encoder (e.g., Qwen3-VL).
• Embeddings enter the U-Net via masked cross-attention focused on the spatial scope of the anchor and temporally adjacent frames.
• The model learns to associate local scene semantics to transformations (e.g., “the mug tips over,” “the sofa occupant stands up”), enabling the generation of temporally coherent egocentric sequences that reflect the evolution prescribed by user instructions.
• At inference, varying prompts $I_i$4 for fixed $I_i$5 and $I_i$6 results in distinct localized evolutions, confirming prompt adherence and composability.

5. Training Regime, Evaluation Metrics, and Empirical Results

AnchorWorld is trained and validated on a multi-stage progression (exocentric → egocentric → static anchors → dynamic anchors) to build robust spatial priors and evolution capabilities.

Key Evaluation Metrics:

• Camera Accuracy: Absolute Translation Error (ATE), Relative Translation Error (RTE), Relative Rotation Error (RRE).
• Scene Consistency: Matched Pixels (MatPix), CLIP‐V similarity, PSNR, SSIM, LPIPS.
• Dynamic Evolution: VideoAlign-TA (semantic text–frame agreement).
• Video Quality: VBench (composite, higher is better).

Summary of Results:

Method	MatPix ↑	CLIP-V ↑	PSNR ↑	SSIM ↑	LPIPS ↓	ATE ↓	RTE ↓	RRE ↓	TA ↑	VBench ↑
PlayerOne	3962	0.845	13.26	0.459	0.596	0.131	0.037	3.741	–	0.734
CaM-Ego	4380	0.872	15.16	0.554	0.515	0.125	0.032	3.207	–	0.748
AnchorWorld	4493	0.885	16.06	0.578	0.470	0.112	0.029	3.145	–	0.748

In dynamic scene benchmarks, AnchorWorld yields a further increase in consistency, accuracy, and semantic alignment. Across all scenarios, it achieves the highest MatPix, CLIP-V, PSNR, SSIM, lowest LPIPS, and superior trajectory metrics.

6. Ablation Insights and Methodological Findings

Ablation studies highlight the necessity of each design component:

• Hybrid-view (Stage I exocentric pretraining) is indispensable; absence results in a Relative Rotation Error increase from $I_i$7 to $I_i$8.
• Spatial Pose Attention outperforms naïve fusion, improving WA/PA-MPJPE scores by 20–30%.
• Anchor-view pose injection and RoPE positional encoding are both critical; omitting either decreases MatPix by $I_i$9100K ($P_i = [R_i | t_i]\in\mathbb R^{3\times 4}$02.2%) and reduces CLIP-V by 0.6–0.7.
• Progressive four-stage training (exocentric to egocentric to static to dynamic anchors) provides 1–2 points of improvement in scene consistency and 1–2% in TextAlign, outperforming joint training approaches.

A plausible implication is that robust egocentric world simulation requires both multi-view spatial grounding and locality-preserving, prompt-driven dynamic mechanisms for customizable and temporally consistent outcomes.

7. Context and Research Significance

AnchorWorld constitutes the first framework for embodied egocentric simulation that is both action-accurate and spatially/temporally customizable. The system demonstrates the feasibility of explicit, user-driven world evolution in a unified coordinate frame, leveraging hybrid-view supervision and prompt-centric anchor customization. This enables practical and versatile simulation in research and application settings where interactive, evolving environments are required, notably surpassing existing state-of-the-art methods in multiple quantitative metrics (Li et al., 5 Jun 2026).