Captain Safari: Pose-Aware World Engine

• Captain Safari is a world engine that uses pose-conditioned retrieval from a persistent memory to maintain stable 3D geometry over long, aggressive 6-DoF trajectories.
• It employs a diffusion-transformer that integrates pose-aligned world tokens to ensure high-fidelity, coherent video synthesis in complex outdoor FPV scenarios.
• The system features a novel memory extraction and retrieval architecture validated on the OpenSafari dataset for robust trajectory following and enhanced video quality.

Captain Safari is a pose-aware world engine for long-horizon, camera-controlled video synthesis. In the formulation introduced in "Captain Safari: A World Engine" (Chou et al., 28 Nov 2025), a world engine is a video generator with an explicit, persistent world memory that supports long-horizon, 3D-consistent synthesis under user-specified 6-DoF camera motion. The system is designed for in-the-wild FPV video, where strong parallax, sharp turns, and complex outdoor clutter expose limitations of memory-less or short-context camera-controlled generators. Its central mechanism is pose-conditioned retrieval from a persistent world memory: at each step, the generator retrieves pose-aligned world tokens and injects them into a diffusion-transformer denoiser so that geometry remains coherent while the camera follows aggressive trajectories (Chou et al., 28 Nov 2025).

1. Problem setting and conceptual definition

Captain Safari addresses a specific failure regime of camera-controllable video generation: long videos under aggressive 6-DoF motion. In this regime, prior methods are described as losing long-range geometric coherence, deviating from the target path, or collapsing into overly conservative motion. The target setting is not merely view synthesis from a static scene representation, but interactive scene exploration in which the camera pose evolves along an arbitrary path and the generated video must preserve stable 3D structure throughout (Chou et al., 28 Nov 2025).

The paper defines the core problem through three requirements. First, the model must preserve long-range geometry across large viewpoint changes and parallax. Second, it must adhere to aggressive 6-DoF paths without damping motion into near-forward trajectories. Third, it must remain visually high-fidelity. The stated motivation is that clip-wise generators, and more generally memory-less or short-context approaches, forget long-range scene structure and fail on sharp turns or large rotations, especially in complex outdoor FPV scenarios.

Within this framing, Captain Safari differs from time-indexed conditioning. Its conditioning signal is pose-indexed: the model retrieves a pose-aligned snapshot of the local world at each step rather than relying on nearby frames in time. A persistent memory therefore functions as an explicit world prior. This is the sense in which the system is called a world engine rather than only a camera-conditioned generator.

A common misconception is to treat a world engine as equivalent to a generator that receives camera parameters as auxiliary input. In the terminology of the paper, that is insufficient. The defining property here is the explicit, persistent world memory that is maintained across video history and consulted during generation. Another misconception is to equate better perceptual quality metrics with better control; the reported ablations show that a memory-removed variant attains slightly better FVD yet weaker 3D consistency and trajectory metrics, indicating that perceptual quality alone does not characterize controllable world modeling.

2. Representation, memory, and retrieval architecture

The video is represented as $V = \{I_t\}_{t=0}^T$. Camera poses are represented as $C = \{(R_t, T_t)\}$ where $R_t \in SO(3)$ and $T_t \in \mathbb{R}^3$; the paper also uses the standard $SE(3)$ form $T_t \in SE(3)$ with $T_t = [R_t \mid t_t]$. Relative pose follows

$T_{\mathrm{rel}}(t \leftarrow s) = T_t T_s^{-1}.$

The persistent memory stores, for each past time step $\tau$, a pose token $p_\tau$ derived from camera extrinsics and a set of memory tokens $C = \{(R_t, T_t)\}$0 extracted by a geometry encoder. The collection $C = \{(R_t, T_t)\}$1 forms what the paper describes as an implicit world table. In practice, the geometry encoder is a pretrained StreamVGGT. Features are extracted from layers $C = \{(R_t, T_t)\}$2, each contributing $C = \{(R_t, T_t)\}$3 tokens, giving $C = \{(R_t, T_t)\}$4 tokens per frame with token dimension $C = \{(R_t, T_t)\}$5 (Chou et al., 28 Nov 2025).

The architecture distinguishes between a global memory and a dynamic local memory. The global bank $C = \{(R_t, T_t)\}$6 stores features over the whole video history, but it is too large for direct use. For a target clip interval $C = \{(R_t, T_t)\}$7, the model samples a bounded local memory window

$C = \{(R_t, T_t)\}$8

subject to

$C = \{(R_t, T_t)\}$9

with fixed bound $R_t \in SO(3)$0. These constraints enforce locality to the clip entrance, bounded duration, and overlap with the clip.

Memory encoding proceeds by embedding each pose token with a learnable function $R_t \in SO(3)$1 and each memory token with $R_t \in SO(3)$2. For each $R_t \in SO(3)$3,

$R_t \in SO(3)$4

and a transformer $R_t \in SO(3)$5 with 3D-aware positional encodings produces

$R_t \in SO(3)$6

The local memory sequence is then concatenated as

$R_t \in SO(3)$7

optionally masked for padding.

Retrieval is pose-conditioned. At time $R_t \in SO(3)$8, the current pose token is embedded as a query $R_t \in SO(3)$9 and concatenated with $T_t \in \mathbb{R}^3$0 learnable tokens $T_t \in \mathbb{R}^3$1:

$T_t \in \mathbb{R}^3$2

Cross-attention reads the encoded memory:

$T_t \in \mathbb{R}^3$3

and the slots of $T_t \in \mathbb{R}^3$4 corresponding to $T_t \in \mathbb{R}^3$5 define the retrieved world tokens

$T_t \in \mathbb{R}^3$6

The attention itself follows the standard formulation

$T_t \in \mathbb{R}^3$7

$T_t \in \mathbb{R}^3$8

A stack of retrieval blocks can iteratively refine queries and retrieved tokens, which the paper describes as softly routing the query pose to the most relevant past observations.

These retrieved tokens condition a DiT video generator. The base model is Wan2.2-Fun-5B-Control-Camera with hidden dimension $T_t \in \mathbb{R}^3$9. The retrieved world tokens are mapped into the DiT hidden space by an MLP $SE(3)$0 to form

$SE(3)$1

For clip latents $SE(3)$2, each layer updates

$SE(3)$3

Because the same clip-level world tokens are reused as keys and values across all layers, they act as a persistent, pose-aligned 3D prior throughout denoising. This is the architectural basis for the claim that pose-conditioned world memory stabilizes structure beyond ordinary clip context.

3. OpenSafari dataset and trajectory verification

OpenSafari is the dataset introduced for this setting. It is described as a new in-the-wild FPV corpus built for aggressive 6-DoF camera control, with FPV flights containing large parallax, rapid 6-DoF turns, and complex outdoor clutter (Chou et al., 28 Nov 2025). The dataset is collected from AirVuz and YouTube. Videos are downloaded at highest resolution, normalized to $SE(3)$4p and $SE(3)$5 fps, and center-cropped to $SE(3)$6. Scene detection yields single-shot segments, which are then uniformly sliced into fixed-length sequences. RAFT flow magnitudes are used to filter out low-motion clips, emphasizing parallax-rich content.

Camera trajectory reconstruction is performed at $SE(3)$7 fps. The paper describes a Hierarchical Localization pipeline involving local features, exhaustive pair matching per video, SfM with COLMAP-style outputs, and export of per-frame camera intrinsics and extrinsics. This is then followed by a multi-stage validation pipeline.

The first validation stage is a database check using SfM inlier counts and ratios to flag unreliable transitions. The second is a geometric check: for suspicious pairs, the essential matrix $SE(3)$8 is recomputed from stored keypoints and matches, and the method thresholds symmetric epipolar error. The constraint is written as

$SE(3)$9

A typical symmetric error is

$T_t \in SE(3)$0

where $T_t \in SE(3)$1 is the fundamental matrix and $T_t \in SE(3)$2 is point-to-line distance. The exact threshold is not given.

The third validation stage is a kinematics check on pose sequences, analyzing translation spikes, rotation jumps, forward-direction flips, and smoothness violations. Robust detection uses MAD-based scores; the standard robust $T_t \in SE(3)$3-score is

$T_t \in SE(3)$4

with suspicious cases flagged when $T_t \in SE(3)$5 exceeds a threshold, again without the exact threshold being reported.

When bad transitions are sparse, the fix policy is to linearly interpolate camera centers and apply SLERP for rotations with a cap on interpolation angle; boundary cases may be extrapolated. The repaired trajectory is then re-validated. Otherwise, the segment is discarded. This pipeline is central to OpenSafari’s positioning as a benchmark with verified camera trajectories rather than only approximate camera metadata.

The reported dataset statistics are: $T_t \in SE(3)$6 training candidates before filtering, reduced to $T_t \in SE(3)$7 training clips after motion and diversity filtering, and a test set of $T_t \in SE(3)$8 non-overlapping clips. Videos are normalized to $T_t \in SE(3)$9p at $T_t = [R_t \mid t_t]$0 fps, while verified camera trajectories and memory features are sampled at $T_t = [R_t \mid t_t]$1 fps. The paper explicitly states that the dataset emphasizes aggressive trajectories and strong parallax rarely covered by RealEstate10K or synthetic worlds.

4. Training procedure and rollout inference

Training uses OpenSafari videos normalized to $T_t = [R_t \mid t_t]$2p and $T_t = [R_t \mid t_t]$3 fps, center-cropped to $T_t = [R_t \mid t_t]$4, with camera poses and memory features sampled at $T_t = [R_t \mid t_t]$5 fps. The paper generates $T_t = [R_t \mid t_t]$6 s clips at $T_t = [R_t \mid t_t]$7 fps from $T_t = [R_t \mid t_t]$8 s videos with a $T_t = [R_t \mid t_t]$9 s stride, producing the $T_{\mathrm{rel}}(t \leftarrow s) = T_t T_s^{-1}.$0 initial candidates already noted. For each clip, a single descriptive caption is generated using Qwen2.5-VL-7B and used as the text condition.

Memory extraction is performed once per video using StreamVGGT features from layers $T_{\mathrm{rel}}(t \leftarrow s) = T_t T_s^{-1}.$1, yielding $T_{\mathrm{rel}}(t \leftarrow s) = T_t T_s^{-1}.$2 tokens per frame with $T_{\mathrm{rel}}(t \leftarrow s) = T_t T_s^{-1}.$3. Optimization then proceeds in two stages. The first stage is retriever warm-up with pose-aligned memory regression. Given $T_{\mathrm{rel}}(t \leftarrow s) = T_t T_s^{-1}.$4 and the query pose, the system retrieves $T_{\mathrm{rel}}(t \leftarrow s) = T_t T_s^{-1}.$5 and uses a linear head to map $T_{\mathrm{rel}}(t \leftarrow s) = T_t T_s^{-1}.$6 back to the original memory space so as to reconstruct the target memory tokens at the query pose. The loss is an $T_{\mathrm{rel}}(t \leftarrow s) = T_t T_s^{-1}.$7 reconstruction loss over tokens at the query pose.

The second stage jointly trains the retriever and the DiT backbone end-to-end. The DiT is updated with LoRA. Memory cross-attention is initialized from the corresponding context cross-attention weights, while new layers use standard initialization. The diffusion model is then trained with the standard denoising objective on latent clips conditioned on $T_{\mathrm{rel}}(t \leftarrow s) = T_t T_s^{-1}.$8.

Inference takes as input a text prompt $T_{\mathrm{rel}}(t \leftarrow s) = T_t T_s^{-1}.$9 and a user-specified 6-DoF camera trajectory $\tau$0 and generates a $\tau$1 s video by rolling out overlapping $\tau$2 s clips while carrying world memory forward (Chou et al., 28 Nov 2025). For each clip $\tau$3, the system first builds a local memory window $\tau$4 satisfying the window constraints with $\tau$5. If $\tau$6, the local memory is initially empty and is seeded online.

Second, the system performs pose-aligned retrieval. It forms $\tau$7 from $\tau$8; the paper specifies that the terminal pose $\tau$9 is used as the query for the $p_\tau$0 s clip. The query sequence $p_\tau$1 is processed by $p_\tau$2 and cross-attended to encoded local memory to obtain $p_\tau$3, optionally through stacked retrieval blocks. Third, the retrieved tokens are embedded to $p_\tau$4 and injected into the DiT, which denoises the latent clip tokens to synthesize $p_\tau$5. Fourth, the generated frames are passed through the pretrained geometry encoder, new memory features $p_\tau$6 are appended to the global bank, and the local window slides for the next clip.

The paper also emphasizes several efficiency strategies. Local memory windowing with $p_\tau$7 s bounds the number of memory tokens. The world tokens $p_\tau$8 are reused as keys and values across all DiT layers, allowing caching per clip. Sampling geometry and memory at $p_\tau$9 fps reduces memory bandwidth. LoRA reduces training-time memory and compute relative to full fine-tuning. Per-layer memory cross-attention scales as $C = \{(R_t, T_t)\}$00, where $C = \{(R_t, T_t)\}$01 is the spatio-temporal latent token length; inference overhead arises from repeated retrieval and feature extraction for memory updates.

5. Quantitative evaluation and ablation evidence

Evaluation is organized along three axes: video quality, 3D consistency, and trajectory following (Chou et al., 28 Nov 2025). The reported video-quality metrics are FVD and LPIPS. The FVD used in the paper is the standard Fréchet Video Distance

$C = \{(R_t, T_t)\}$02

computed between reference and generated features. For 3D consistency, the paper reports MEt3R and reconstruction rate using a reconstruction-based protocol at matched time steps. For trajectory following, it reports AUC@30, AUC@15, and cosine similarity of flattened pose sequences, where AUC@30 and AUC@15 are relocation accuracy curves from VGGT-style camera localization.

On OpenSafari, Captain Safari reports FVD $C = \{(R_t, T_t)\}$03 and LPIPS $C = \{(R_t, T_t)\}$04. The listed baselines are Wan2.2-5B-Control-Camera with FVD $C = \{(R_t, T_t)\}$05 and LPIPS $C = \{(R_t, T_t)\}$06, Real-CamI2V with FVD $C = \{(R_t, T_t)\}$07 and LPIPS $C = \{(R_t, T_t)\}$08, and Geometry Forcing with FVD $C = \{(R_t, T_t)\}$09 and LPIPS $C = \{(R_t, T_t)\}$10. An ablated Captain Safari without memory attains FVD $C = \{(R_t, T_t)\}$11 and LPIPS $C = \{(R_t, T_t)\}$12. The paper interprets this as showing that Captain Safari maintains competitive FVD while delivering stronger 3D control.

For 3D consistency, Captain Safari reports MEt3R $C = \{(R_t, T_t)\}$13 versus $C = \{(R_t, T_t)\}$14 for the strongest baseline, Real-CamI2V, and a reconstruction rate of $C = \{(R_t, T_t)\}$15 versus $C = \{(R_t, T_t)\}$16 for Real-CamI2V. Wan2.2 and Geometry Forcing obtain reconstruction rates of $C = \{(R_t, T_t)\}$17 and $C = \{(R_t, T_t)\}$18, respectively. The memory-removed variant reports reconstruction rate $C = \{(R_t, T_t)\}$19. In the abstract, the paper summarizes this as reducing MEt3R from $C = \{(R_t, T_t)\}$20 to $C = \{(R_t, T_t)\}$21.

For trajectory following, Captain Safari reports AUC@30 $C = \{(R_t, T_t)\}$22, compared with $C = \{(R_t, T_t)\}$23 for Wan2.2, $C = \{(R_t, T_t)\}$24 for Real-CamI2V, and $C = \{(R_t, T_t)\}$25 for Geometry Forcing. AUC@15 is $C = \{(R_t, T_t)\}$26, equal to the memory-removed variant and above Real-CamI2V at $C = \{(R_t, T_t)\}$27 and Wan2.2 at $C = \{(R_t, T_t)\}$28. Cosine similarity of flattened pose is $C = \{(R_t, T_t)\}$29 for Captain Safari, versus $C = \{(R_t, T_t)\}$30 for the memory-removed variant, $C = \{(R_t, T_t)\}$31 for Wan2.2, $C = \{(R_t, T_t)\}$32 for Real-CamI2V, and $C = \{(R_t, T_t)\}$33 for Geometry Forcing. In the abstract, the trajectory result is summarized as improving AUC@30 from $C = \{(R_t, T_t)\}$34 to $C = \{(R_t, T_t)\}$35.

The ablation with and without pose-conditioned memory is especially diagnostic. Adding memory improves MEt3R from $C = \{(R_t, T_t)\}$36 to $C = \{(R_t, T_t)\}$37, reconstruction from $C = \{(R_t, T_t)\}$38 to $C = \{(R_t, T_t)\}$39, AUC@30 from $C = \{(R_t, T_t)\}$40 to $C = \{(R_t, T_t)\}$41, and cosine similarity from $C = \{(R_t, T_t)\}$42 to $C = \{(R_t, T_t)\}$43, at a small FVD trade-off from $C = \{(R_t, T_t)\}$44 to $C = \{(R_t, T_t)\}$45. This directly supports the paper’s claim that pose-conditioned world memory is crucial for 3D stability and path adherence, even when a perceptual metric may slightly worsen.

The human study uses $C = \{(R_t, T_t)\}$46 participants, $C = \{(R_t, T_t)\}$47 cases each, and $C = \{(R_t, T_t)\}$48-way anonymized comparisons under Video Quality, 3D Consistency, and Trajectory Following, totaling $C = \{(R_t, T_t)\}$49 votes. The detailed evaluation reports an average preference of $C = \{(R_t, T_t)\}$50 for Captain Safari, $C = \{(R_t, T_t)\}$51 for the memory-removed variant, $C = \{(R_t, T_t)\}$52 for Real-CamI2V, $C = \{(R_t, T_t)\}$53 for Wan2.2, and $C = \{(R_t, T_t)\}$54 for Geometry Forcing. Per-axis preferences for Captain Safari are $C = \{(R_t, T_t)\}$55 for quality, $C = \{(R_t, T_t)\}$56 for consistency, and $C = \{(R_t, T_t)\}$57 for trajectory. The abstract reports the closely related summary that $C = \{(R_t, T_t)\}$58 of preferences favor the method across all axes.

Qualitative analysis is aligned with the quantitative results. The paper states that Captain Safari preserves building façades, consistent field markings, and smooth object motion under fast 6-DoF turns, while baselines exhibit flicker, distortions, popping geometry, or path drift. This suggests that the gains are not confined to a single metric family but reflect the intended trade-off between geometry stabilization and trajectory execution.

6. Position in the literature, limitations, and future directions

Captain Safari is positioned at the intersection of camera-controllable diffusion models and world models with persistent memory (Chou et al., 28 Nov 2025). The stated distinction from systems that condition only on camera parameters or short-term context is the explicit pose-indexed world memory retrieved on demand. The paper further distinguishes the approach from systems that reconstruct a one-off 3D scene or rely on implicit clip-bound memories: pose-conditioned retrieval provides a persistent, geometry-aware prior shared across time, with the aim of supporting both long-horizon consistency and accurate trajectory following in open-world FPV settings.

The practical limitations reported are concrete. Inference overhead is nontrivial because retrieval and cross-attention operate on large memory sets and because the model repeatedly extracts geometry features to update memory online. The method may be brittle in rare outdoor layouts, highly dynamic scenes, or under drastic illumination and weather changes. It also relies on a pretrained geometry encoder for memory features, and generated frames used for online memory updates can accumulate memory noise. These are not framed as failures of the world-memory idea itself, but as operational constraints of the current implementation.

The paper also identifies ethical considerations. OpenSafari is collected from online sources with normalization and curation, and use is stated to require respect for original content licensing. Generated FPV content could be misused for deceptive media if not watermarked. These concerns are presented as part of deployment context rather than as a separate normative framework.

Future work is outlined in terms that follow directly from the architecture. The paper points toward real-time world engines through memory compression, more efficient backbones, and better caching; tighter coupling with explicit geometry such as renderable 3D or learned SLAM-like maps; and improved handling of dynamic objects and semantics-aware memory. A plausible implication is that the present system should be understood as a memory-centric design point rather than a complete solution to world modeling. Its main research significance lies in showing that retrieval in pose space, rather than time-indexed conditioning alone, can anchor long-horizon video generation under aggressive 6-DoF motion while preserving stable 3D structure.