Beyond Pixels: From Video Priors to 4D Worlds

Abstract: 4D generation synthesizes dynamic 3D scenes from conditions such as text or images. Existing methods either reconstruct generated RGB videos with a separate 4D model or adapt a particular video generator to predict geometry directly. The former suffers from distribution mismatch and error propagation, whereas the latter ties 4D prediction to a specific generator and may require retraining when the generator or conditioning regime changes. We ask whether the final denoised latents of video models that share a variational autoencoder (VAE) can instead provide a reusable interface to explicit 4D prediction. Building on this insight, we introduce direct latent-to-4D generation and instantiate it as Latent-to-4D, which bypasses RGB by aligning a video latent with the token grid of a pretrained 4D decoder and refining it through frame-wise and global spatiotemporal attention. Trained on roughly 1K existing reconstruction clips, a single checkpoint transfers unchanged across multiple video diffusion transformers within the same VAE family. On Text4D-200 and I4D-200, Latent-to-4D surpasses matched same-latent Wan+4RC cascades in projection-based DINO-F1 by 2.88--3.45 and 5.81 points, respectively, while also being preferred by human raters for geometry, temporal stability, and overall quality.

• The paper introduces L4AR, a learned interface that aligns frozen video VAE latents with a pretrained 4D decoder using 3D convolution and alternating frame-wise and global attention.
• The method improves DINO-F1 by 2.88–3.45 points for text-conditioned generation and 5.81 points for image-conditioned generation over matched RGB reconstruction cascades.
• The results show stronger geometric completeness, temporal stability, and control transfer, but the approach currently depends on shared VAE conventions and lacks comprehensive metric 4D evaluation.

Direct Latent-to-4D Generation from Video Priors

Research Problem and Positioning

“Beyond Pixels: From Video Priors to 4D Worlds” (2608.10744) addresses the interface between video generation and explicit dynamic-scene reconstruction. The central question is whether the final denoised latent produced by a video diffusion model can serve as a reusable representation for predicting cameras and dynamic world-space geometry, without decoding the latent into RGB frames.

Existing approaches generally follow one of two designs. Generate-then-reconstruct methods first synthesize RGB videos and subsequently apply a 4D reconstruction model. This preserves modularity but introduces a representation boundary: the reconstruction network must interpret generated frames whose artifacts, appearance distribution, and temporal inconsistencies may differ from those of reconstruction data. Integrated methods instead adapt a specific video generator to predict geometry directly. Although this can reduce the RGB-induced mismatch, it couples the geometry predictor to a particular DiT architecture, generator, or conditioning modality.

The paper proposes a third alternative: direct latent-to-4D generation. The final denoised latent is consumed directly by a learned interface and converted into explicit dynamic geometry. The proposed Latent-to-4D system uses approximately 1,143 annotated reconstruction clips, freezes the video generators and VAE, and trains an alignment-and-refinement pathway connected to a pretrained 4D decoder. The resulting checkpoint is evaluated without retraining across two text-to-video DiTs and one image-to-video DiT sharing the same Wan VAE (Wan et al., 26 Mar 2025).

The conceptual contribution is therefore not a new 4D scene representation, but a transfer mechanism between two pretrained representation spaces. The paper treats the VAE latent as a common computational interface across compatible video generators, while using a 4D reconstruction hierarchy to impose camera and geometric structure.

Latent Space as a Cross-Model Interface

The proposed interface depends on a relatively strong compatibility condition. The video models must share the VAE checkpoint, latent normalization, tensor layout, compression convention, and supported latent shape. Under these constraints, their DiT backbones and conditioning mechanisms may differ. A text-to-video model, an image-to-video model, or a model with motion, pose, trajectory, manipulation, or navigation controls can produce a terminal latent that is passed through the same downstream 4D pathway.

This design separates two roles. The upstream DiT determines the semantic, appearance, and motion content induced by the condition. The downstream Latent-to-4D network maps that realized content into cameras and dynamic world-space geometry. The condition itself does not need to be exposed to the 4D decoder, because it has already been integrated into the terminal latent.

The distinction from conventional generate-then-reconstruct processing is important. In a cascaded system, the video latent is decoded into RGB and then re-encoded by an RGB-based reconstructor. This creates opportunities for information loss and error propagation through VAE decoding, RGB quantization, hallucinated texture, motion blur, and generator-specific artifacts. Latent-to-4D removes the decoding and re-encoding steps, although it replaces them with a learned cross-representation alignment problem.

Figure 1: Latent-to-4D maps video latents directly to dynamic 4D geometry, bypassing RGB decoding and reconstruction.

The method is related in spirit to efforts that repurpose video diffusion representations for depth and geometry, including “Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation” (Ke et al., 2023), “DepthCrafter” (Hu et al., 2024), “GeometryCrafter” (Xu et al., 1 Apr 2025), and “Geo4D” (Jiang et al., 10 Apr 2025). Its distinguishing claim is that a frozen, general-purpose video generator can remain unchanged while a single learned interface transfers its latent representation to a structured dynamic-scene decoder.

Latent-to-4D Alignment and Refinement

The technical core is Latent-to-4D Alignment and Refinement, or L4AR. The input is a video latent with a spatiotemporal grid and channel dimension determined by the VAE. The pretrained 4D decoder expects a different token layout, temporal resolution, spatial resolution, and feature dimension. L4AR addresses this mismatch in three stages.

First, the latent is resampled using trilinear interpolation to match the target spatiotemporal resolution. A learned 3D convolution then aggregates local neighborhoods and projects the VAE channels into the feature dimension expected by the 4D hierarchy. This operation produces an initial token grid with explicit frame and spatial-token structure.

Second, the aligned tokens undergo hierarchical spatiotemporal refinement. Frame-wise self-attention operates independently within each frame and consolidates spatial structure. Global attention then operates over all spatial tokens and frames, enabling cross-frame correspondence, viewpoint reasoning, and motion propagation. The two attention modes are alternated rather than collapsed into a single attention operation. Intermediate features from multiple depths are concatenated to provide the decoder with both local spatial detail and long-range temporal context.

Third, the refined representation is passed to a 4D decoder initialized from 4RC (Luo et al., 10 Feb 2026). The decoder predicts per-frame camera parameters and dynamic point maps in a shared world coordinate system. Geometry is represented through depth and world-space ray quantities, allowing each point to be reconstructed from its predicted ray origin, direction, and depth. This formulation explicitly couples camera estimation with dynamic geometry rather than treating the output as a sequence of independently reconstructed frames.

Figure 2: The training pathway encodes observed videos into frozen VAE latents, aligns them with the 4D token grid, applies alternating frame-wise and global attention, and predicts cameras and dynamic world-space geometry.

Only selected components are trained: the alignment module, prediction heads, and rank-16 LoRA updates within the refinement hierarchy. The VAE, DiT weights, original 4D Transformer weights, camera and temporal tokens, motion decoder, and tracking head remain frozen. Progressive activation of trainable components is used to preserve the pretrained geometric prior during optimization.

The training distribution contains observed videos encoded by the frozen VAE, whereas inference uses final denoised latents sampled from conditional DiTs. Consequently, the method requires a distribution transfer from observation-derived VAE latents to generator-derived terminal latents. The paper’s main generalization claim is that this transfer can be achieved without generator-specific fine-tuning when the VAE interface is shared.

Experimental Protocol

Evaluation is performed on Text4D-200 and I4D-200, each containing 200 locked cases for text-conditioned and image-conditioned generation. The generated latent is held fixed when comparing Latent-to-4D with matched Wan-plus-reconstruction cascades. This is a meaningful control because it isolates the effect of consuming the latent directly from differences in the upstream generated content.

The primary metrics include Text CLIP, RGB-reference CLIP-I, global DINO similarity, valid-patch DINO matching, and DINO set F1. The DINO-based measures are computed after rendering predicted geometry from off-axis cameras. They therefore assess visible semantic and structural coherence, but they are not direct measurements of metric reconstruction accuracy. The paper explicitly acknowledges this limitation.

The baselines include 4RC, $\pi^3$ (Wang et al., 17 Jul 2025), and Any4D (Karhade et al., 11 Dec 2025) applied after RGB decoding, as well as CogVideoX-based cascades (Yang et al., 2024) and the native 4DNeX model (Chen et al., 18 Aug 2025). The comparison is consequently strongest against matched generate-then-reconstruct pipelines, rather than against every possible integrated 4D generation system.

Quantitative Results

On Text4D-200, Latent-to-4D obtains DINO-F1 scores of 57.01 and 57.09 with Wan2.1-14B and Wan2.1-1.3B, respectively. The corresponding matched Wan-plus-4RC systems achieve 53.56 and 54.21. This produces gains of 3.45 and 2.88 points. The near-equivalence across the 14B and 1.3B generators supports the claim that the downstream checkpoint is not tightly dependent on the scale of the upstream DiT.

The method does not dominate every metric. CogVideoX-5B plus 4RC attains the highest RGB-reference CLIP-I among the reported text-conditioned systems, while Latent-to-4D achieves stronger structure-sensitive DINO metrics. This discrepancy is theoretically informative: improved geometric completeness and cross-view consistency need not imply maximal pixel-level similarity to decoded RGB references.

On I4D-200, Latent-to-4D achieves a DINO-F1 of 61.60, compared with 55.79 for Wan2.2-I2V-A14B plus 4RC, a gain of 5.81 points. It also ranks first on the reported text-alignment, image-alignment, global DINO, matching, and F1 measures. The larger image-conditioned improvement suggests that direct latent processing may be particularly beneficial when RGB reconstruction introduces substantial appearance or structure degradation before 4D lifting.

Figure 3: Text-conditioned examples indicate that direct latent processing better preserves subject structure and surrounding scene support than RGB reconstruction cascades.

Figure 4: Image-conditioned results show improved retention of foreground structure and scene content under direct latent-to-4D decoding.

Human evaluation reinforces the automated results. Fifty participants assessed 50 cases per benchmark using randomized pairwise comparisons, with ten ratings per case. Latent-to-4D was preferred over the baselines in every category. For text-to-4D, preferences were 59.2% for condition fidelity, 66.8% for geometry and completeness, 63.5% for temporal stability, and 65.7% for overall quality. For image-to-4D, the corresponding values were 66.4%, 72.1%, 68.3%, and 70.6%. All reported bootstrap intervals exceed 50%, with the strongest advantage in geometry and completeness.

These results support the practical claim that bypassing RGB improves perceptual 4D quality. They do not, however, establish superiority in metric camera accuracy, metric depth, dynamic reconstruction error, or physical validity of the generated scene.

Ablation and Interface Stability

The component ablations on 7-Scenes and NRGBD show that all principal elements of L4AR contribute materially. The full system achieves accuracy errors of 3.121 cm and 5.202 cm on the two datasets, compared with 3.783 cm and 5.823 cm without grid alignment. Removing the 3D convolution is more damaging, increasing the errors to 6.944 cm and 12.439 cm. Removing frame-wise or global attention causes similarly substantial degradation, particularly in completeness and normal consistency.

The pattern indicates that simple spatial resizing is insufficient. The learned 3D convolution is required for local cross-representation conversion, while both attention scopes are necessary: frame-wise attention preserves intra-frame structure, and global attention supplies temporal and spatial correspondence. The ablation therefore supports the architectural decomposition rather than merely showing that additional capacity improves performance.

The paper also introduces a controlled sensitivity experiment involving a DiT-derived near-terminal residual projected into the width-null space of the grid-alignment module. At perturbation strength $\rho=0.6$, point-map drift for Latent-to-4D is reported as 0.0053 and 0.0047 on 7-Scenes and NRGBD, compared with 0.3827 and 0.3160 for the RGB baseline. All 30 comparisons favor Latent-to-4D. This is a strong diagnostic result because it suggests that the latent interface suppresses certain residual perturbations that become amplified after RGB decoding and re-encoding.

Figure 5: Controlled DiT-derived residuals produce substantially less geometry and camera drift in the direct latent pathway than in the RGB reconstruction baseline.

Nevertheless, this experiment should not be interpreted as robustness to arbitrary generator errors. The perturbation is specifically constructed from a near-terminal residual and projected into a designated null space. It probes one structured failure mode rather than the full distribution of artifacts produced by incompatible or poorly calibrated video generators.

Control Transfer and Applications

A single L4AR checkpoint inherits several upstream controls, including motion, appearance, pose, and trajectory conditioning. This is a direct consequence of consuming the terminal latent rather than adding separate condition-specific branches. The demonstrations indicate that control information survives the latent-to-geometry transfer.

Figure 6: Direct latent consumption transfers motion, appearance, pose, and trajectory controls into the predicted 4D scene.

The paper further presents manipulation- and navigation-conditioned examples from a compatible backbone. These results suggest a possible route toward integrating video generation, dynamic-world reconstruction, and embodied-agent interfaces. However, the demonstrations establish representational compatibility only; they do not evaluate action success, collision avoidance, physical plausibility, or closed-loop control.

Figure 7: Action-conditioned latents are converted into dynamic geometry for manipulation and navigation scenarios.

Theoretical and Practical Implications

Theoretically, the work argues that the final denoised VAE latent can function as a model-agnostic semantic interface within a restricted common-VAE family. This challenges the assumption that generated RGB is the natural transfer medium between video synthesis and geometric reconstruction. RGB remains useful for visualization and evaluation, but it is not necessarily the optimal computational representation for downstream 4D inference.

The result also exposes an important distinction between generator compatibility and representation compatibility. Latent-to-4D does not require identical DiT architectures or conditioning inputs, but it does require a shared VAE convention. Thus, the method decouples 4D supervision from the upstream generator only within a carefully defined latent-space boundary. Its generality is substantial relative to generator-specific adaptation, but it is not universal across arbitrary video models.

Practically, the approach offers three advantages. It avoids a VAE decode–encode cycle, reuses pretrained video controls, and amortizes 4D supervision across multiple generators. Training on roughly 1K reconstruction clips is comparatively modest relative to video-generation datasets, and inference does not require per-scene optimization. The same pathway can therefore support text, image, camera, motion, and action-conditioned generation as long as the upstream model produces compatible latents.

The principal limitations concern dataset scale, compatibility, and evaluation. The method is trained on a limited reconstruction corpus and evaluated on generated-latent benchmarks of 200 cases each. The reported DINO metrics are projection-based proxies rather than ground-truth dynamic-scene measurements. The human study provides perceptual evidence but cannot replace metric evaluation. In addition, the shared-VAE assumption may become restrictive as video models adopt different latent spaces, compression rates, temporal strides, or VAE tokenizations.

Future Directions

Future work could learn a universal latent adapter across multiple VAE families rather than relying on exact VAE compatibility. This might involve latent-space canonicalization, contrastive alignment, optimal-transport matching, or a mixture-of-adapters conditioned on latent statistics. Another direction is joint calibration between terminal diffusion latents and geometric uncertainty, allowing the 4D decoder to quantify when a generated latent lacks sufficient evidence for reliable camera or geometry prediction.

The method could also benefit from evaluation protocols that directly measure metric geometry, dynamic scene flow, camera trajectory accuracy, multi-view consistency, and temporal topology. Integration with 4D representations beyond point maps—such as dynamic Gaussian primitives, neural fields, or explicit articulated models—would test whether the latent interface is specific to the 4RC-style decoder or reflects a broader property of video representations.

A further development would be closed-loop latent-to-world modeling. In such a system, the generated latent would not only produce geometry but also support rendering, action prediction, and recurrent world-state updates. The action-conditioned examples indicate a possible architectural foundation, but physical consistency and long-horizon temporal stability remain open problems.

Conclusion

“Beyond Pixels: From Video Priors to 4D Worlds” (2608.10744) presents direct latent-to-4D generation as an alternative to RGB-mediated reconstruction. Its L4AR module aligns video VAE latents with a pretrained 4D decoder through learned 3D convolution and alternating frame-wise and global attention. A single checkpoint transfers across three compatible video DiTs and improves DINO-F1 by 2.88–3.45 points for text-to-4D and 5.81 points for image-to-4D relative to matched Wan-plus-4RC cascades. Human raters also prefer the outputs in geometry, completeness, temporal stability, and overall quality. The claims are strongest within the shared-VAE setting; extending the interface across heterogeneous latent spaces and validating metric 4D accuracy remain central directions for subsequent research.