WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory

Abstract: Video world models enable interactive exploration of dynamic environments, yet struggle to respect prior observations over long horizons and across viewpoints. We present WorldCrafter, a video world model that learns a camera-queryable implicit 3D-aware memory for this purpose. The key insight is to let the requested viewpoint shape how multi-view evidence is compressed into the video generator's limited token budget. Trained jointly with the video generator, a memory encoder and pose-conditioned readout module integrate historical observations into a fixed set of target view-specific tokens before denoising, without explicit depth-based correspondences. By combining this memory with recent temporal context and few-step distillation, WorldCrafter enables streaming scene exploration from a single input image or text prompt. Experiments across static and dynamic scenes show substantial gains in long-horizon consistency and camera-control accuracy while preserving visual quality during minute-scale exploration.

• The paper presents WorldCrafter, a memory mechanism that uses a 3D-aware, implicit latent scene representation, significantly improving video world consistency on revisit metrics.
• WorldCrafter incorporates max-coverage retrieval and pose-guided readout to selectively retain historical information, reducing memory-processing costs by 21.7 times compared to traditional depth-based approaches.
• The model achieves superior performance on camera consistency metrics, generating at 16 frames per second on a four-GPU machine, thus being practical for real-time applications.

Problem formulation and central contribution

WorldCrafter addresses a specific failure mode of interactive video world models: camera-conditioned autoregressive generation can follow a prescribed trajectory locally while losing previously observed appearance and geometry over long rollouts. Recent systems mitigate this problem through retrieved context, explicit spatial representations, or recurrent implicit states, but each approach imposes a limitation. Full-history attention is computationally expensive; sparse frame retrieval can provide incomplete view coverage; explicit depth-based memories depend on geometric estimation and warping; and geometry-oriented representations may preserve structure without retaining the appearance required for faithful visual revisits.

"WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory" (2609.24984) proposes an implicit memory mechanism that compresses historical latent frames and camera poses into a learned, 3D-aware representation. The representation is not an explicit point cloud, depth map, or target-view image. Instead, it is a latent scene memory produced by a multi-view encoder and converted into a fixed number of tokens through a camera-queryable readout module. The key design claim is that the target camera trajectory should determine which historical information is retained in the generator’s constrained memory-token budget.

WorldCrafter combines four components: a camera-conditioned video DiT, recent temporal context, a memory encoder initialized from a pretrained novel-view-synthesis model, and a pose-guided memory readout. At each autoregressive rollout step, the system retrieves a complementary subset of historical latent frames, encodes them into an implicit scene representation, queries that representation using poses from the upcoming trajectory, and injects the resulting fixed-size tokens into the DiT. Generated chunks are appended to the history archive and can subsequently contribute to memory formation.

Architecture and memory processing

The model generates video in fixed-length latent chunks using conditional flow matching. Let the current chunk be represented by a noisy latent and let $\mathbf{z}^{\mathrm r}$ denote a short recent-history window. In addition to recent context, the denoiser receives memory tokens $\mathbf M$, camera poses and intrinsics $\mathbf C$, and the text condition. The resulting conditioning separates short-term temporal continuation from long-term scene recall: recent context supports local motion continuity, while the memory stream provides information from observations that may have fallen outside the recent window.

Camera conditioning is implemented using the UCPE camera-attention branch. Relative camera geometry is injected into the noisy current-chunk tokens, whereas memory tokens and recent-history tokens are not directly subjected to camera injection. This distinction is important: the camera trajectory controls the generation query, while the memory representation remains a reusable latent scene state.

The memory encoder $\Phi$ maps selected history latents and their associated camera parameters to a representation containing a fixed number of tokens per input latent frame. Its architecture is initialized from LagerNVS, a latent geometry model for neural novel-view synthesis. The authors discard LagerNVS’s shallow image-processing layers and replace them with a patch embedding layer operating directly on VAE latents. Camera poses are expressed relative to the latest history frame and supplied as camera tokens during encoding. Consequently, the memory encoder inherits a multi-view inductive bias without explicitly constructing geometry.

The encoder does not consume the entire history. The latest latent frame is always retained, and additional frames are selected by maximizing joint field-of-view coverage of the upcoming camera trajectory. This differs from pairwise similarity retrieval, which independently ranks frames by their overlap with target views. Max-coverage retrieval instead seeks complementary observations. The distinction is consequential because a set of individually similar frames can undersample the target region, whereas a jointly optimized subset can provide broader spatial evidence under the same input budget.

The encoded representation is reduced to a fixed number of DiT-compatible memory tokens in one of two ways. Pose-free readout compresses the representation independently of the upcoming trajectory. Pose-guided readout uses query poses sampled from the target camera path. The latter is adopted in the final model because it allocates the token budget to information relevant to the immediate generation request.

Figure 1: WorldCrafter retrieves complementary history, encodes it into an implicit 3D-aware representation, and performs pose-conditioned memory readout before chunk denoising.

The architecture therefore contains an implicit form of view-dependent information selection. It does not synthesize target-view images before generation, as spatial-memory methods commonly do. Instead, it directly produces latent tokens that the DiT can attend to while generating the target chunk. This avoids an intermediate RGB reconstruction and allows the memory representation and generator token space to co-adapt during training.

Training and real-time inference

Training proceeds in four stages. First, the Helios-base video DiT is adapted to replace a compressed history segment with memory slots corresponding to four uncompressed latent frames. Second, camera control is introduced through the UCPE branch while the DiT backbone remains frozen. Third, the pretrained LagerNVS-derived memory encoder is adapted to VAE latent inputs. Finally, the memory encoder, readout module, DiT, and camera-conditioning branch are jointly optimized.

The joint-training stage is central to the method’s premise. The memory representation is not treated as a fixed feature space that the video generator must passively consume. Instead, the memory encoder learns to produce features that are useful for the DiT’s denoising computation, while the DiT learns how to interpret those features. The ablation results show that freezing the memory encoder degrades both revisit consistency and camera control, supporting the claim that the representation must co-adapt with the generator.

WorldCrafter-fast applies pyramid distribution-matching distillation with three spatial resolutions and two denoising steps per resolution. The authors use a hybrid distilled model because synthetic data improve subject-following behavior but can introduce smeared textures. A high-noise model trained with synthetic data handles early denoising, while a low-noise model trained primarily on natural video data performs the final denoising step. The resulting system generates at 16 frames per second on a four-GPU machine, establishing the practical relevance of the memory design for streaming interaction. This speed result is conditional on the stated hardware and does not imply equivalent performance on a single commodity GPU.

Benchmark and quantitative results

The evaluation benchmark contains 145 initial images: 62 static scenes and 83 dynamic object-centric scenes. Each image is paired with five metric camera trajectories, producing 725 videos per method. Trajectories range from 528 to 1,648 frames and include closed-loop revisits. This setup evaluates long-horizon consistency under camera paths that deliberately return to previously observed regions rather than measuring only local frame quality.

WorldCrafter and WorldCrafter-fast achieve the two best results on all four revisit-consistency metrics. The full model reduces LPIPS from the strongest baseline’s 0.487 to 0.255 and improves PSNR from 14.050 to 18.016 dB. It also obtains an SSIM of 0.517 and an MEt3R score of 0.166. The distilled variant is stronger on these memory metrics, reaching LPIPS 0.186, PSNR 20.868 dB, SSIM 0.616, and MEt3R 0.129.

Method	MEt3R $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
Lyra 2.0	0.334	0.487	14.050	0.390
WorldCrafter	0.166	0.255	18.016	0.517
WorldCrafter-fast	0.129	0.186	20.868	0.616

The paper reports a 47.6% improvement in revisit consistency relative to the strongest baseline, although the precise metric and aggregation underlying this headline comparison should be interpreted alongside the individual metrics. The agreement across MEt3R, LPIPS, PSNR, and SSIM is more informative than any single score: the model improves both perceptual similarity and pixel- and structure-based correspondence.

Qualitative comparisons show that the model preserves scene appearance after extended exploration and re-entry. In static scenes, reconstructed point clouds exhibit more coherent spatial alignment between first visits and revisits. In dynamic scenes, the model maintains moving subjects more effectively while retaining broader scene consistency. These results support the intended division between memory and recent context: historical memory supplies stable scene information, while recent frames accommodate temporal evolution.

Figure 2: Static-scene revisits show the preservation of previously observed appearance and structure after long camera trajectories.

Figure 3: Dynamic-scene revisits evaluate whether scene memory remains compatible with moving subjects and nonstationary content.

Camera control is evaluated by estimating trajectories with VGGT-$\Omega$, normalizing them relative to the first pose, and applying Sim(3) alignment. WorldCrafter achieves the lowest error on all three metrics: RotErr of 13.536, TransErr of 1.475, and CamMC of 1.546. The distilled model ranks third on every camera metric, with RotErr 18.251, TransErr 1.638, and CamMC 1.737. Thus, distillation preserves a substantial portion of camera fidelity but introduces a measurable degradation relative to the full-step model.

Method	RotErr $\downarrow$	TransErr $\mathbf M$0	CamMC $\mathbf M$1
Lyra 2.0	16.145	1.538	1.624
WorldCrafter	13.536	1.475	1.546
WorldCrafter-fast	18.251	1.638	1.737

The camera results are particularly relevant because improved revisit consistency could otherwise be confounded by inaccurate trajectory execution. Here, the model both recalls scene content and follows the requested camera path more accurately than the evaluated baselines. However, trajectory recovery depends on VGGT-$\mathbf M$2 estimates and Sim(3) alignment, so the results measure consistency under an estimated geometric evaluation pipeline rather than direct ground-truth pose error.

On VBench, WorldCrafter obtains the highest overall score, 81.910, and leads in five of eight reported dimensions. It achieves the best subject consistency, background consistency, motion smoothness, and overall consistency, while WorldCrafter-fast achieves the best temporal-flicker score. The full model’s imaging-quality score is 69.058, below the strongest baseline’s 71.249, and its dynamic-degree score is 96.893, below several methods reaching 100.000. Therefore, the method’s principal advantage is not uniform dominance across all perceptual dimensions; it is concentrated in long-horizon consistency and camera-conditioned scene preservation.

Ablation evidence

The ablations isolate the principal design choices under matched training and inference settings and a fixed DiT memory-token budget. Replacing implicit memory with four retrieved history latent frames substantially weakens performance: MEt3R increases from 0.166 to 0.382, LPIPS from 0.255 to 0.497, and RotErr from 13.536 to 26.522. This result supports the claim that compressing multiple observations into a learned 3D-aware representation is more effective than directly supplying a similarly sized context memory.

Freezing the memory encoder also degrades performance, yielding MEt3R 0.227, LPIPS 0.305, PSNR 16.873 dB, and RotErr 15.428. The implication is that initialization from a pretrained view-synthesis encoder is insufficient by itself: the memory features must be adapted to the video DiT’s denoising objective.

Pose-guided readout improves over pose-free readout across both memory and camera metrics. LPIPS decreases from 0.333 to 0.255, while RotErr decreases from 18.307 to 13.536. This provides direct evidence for the paper’s central allocation argument: under a fixed output-token budget, target-pose queries help preserve information that is relevant to the next camera-conditioned generation.

Max-coverage retrieval also outperforms similarity-based retrieval. The latter produces MEt3R 0.213, LPIPS 0.296, PSNR 17.125 dB, and RotErr 14.657, whereas max-coverage retrieval reaches 0.166, 0.255, 18.016 dB, and 13.536. Because both variants retain the latest frame and select the same number of additional history frames, the improvement is attributable to the selection criterion rather than a larger memory input.

Figure 4: Ablations show the contributions of implicit 3D-aware memory, joint optimization, pose-guided readout, and max-coverage retrieval.

The efficiency comparison is also pronounced. At a resolution of $\mathbf M$3, depth-based spatial memory requires 0.409 seconds for depth estimation and 0.937 seconds for batched warping, for a total of 1.346 seconds per chunk. WorldCrafter requires 0.049 seconds for memory encoding and 0.013 seconds for readout, totaling 0.062 seconds. The reported 21.7-fold reduction in memory-processing cost follows directly from these component measurements. The comparison excludes shared VAE decoding and video denoising, and therefore characterizes memory overhead rather than end-to-end latency.

Limitations and open questions

The paper concedes that consistency can still fail on particularly complex or extended trajectories. This is an important qualification because the benchmark trajectories, although long, do not establish indefinite consistency. The memory encoder is also re-run over selected history frames at every chunk, introducing repeated computation despite the compact readout. The authors identify an autoregressive streaming memory encoder as a possible remedy, but do not evaluate such a mechanism.

The implicit representation also leaves unresolved how appearance, geometry, and dynamics are disentangled internally. The method performs well on the reported dynamic-scene benchmark, but its memory is derived from a multi-view representation whose pretraining objective emphasizes view synthesis. The paper does not provide a controlled analysis of failure modes involving severe occlusion, topology changes, independently moving objects, or conflicting observations across time. Similarly, the use of estimated metric poses during training and VGGT-$\mathbf M$4 during evaluation makes the system’s sensitivity to camera-pose noise an open empirical question.

Finally, the benchmark is curated from images and text descriptions, with trajectories supplied independently of interactive policy behavior. The results therefore establish controlled camera-conditioned generation rather than closed-loop decision-making in which an agent selects actions based on generated observations. Whether the same memory mechanism supports policy learning or action-conditioned dynamics is not determined by the experiments.

Conclusion

WorldCrafter presents a coherent memory architecture for long-horizon camera-controlled video generation. Its main contribution is the integration of max-coverage history selection, an implicit 3D-aware encoder, and pose-guided fixed-budget readout directly into an autoregressive video DiT. The strongest evidence comes from the combination of improved revisit metrics, superior camera-control errors, favorable VBench consistency scores, and a 21.7-fold reduction in memory-processing overhead relative to depth-based spatial memory. The results support the paper’s central claim that a learned, target-view-queryable latent memory can preserve appearance and structure more effectively than sparse context retrieval while remaining compatible with real-time streaming inference.