Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction

Abstract: Accurate future video prediction requires both high visual fidelity and consistent scene semantics, particularly in complex dynamic environments such as autonomous driving. We present Re2Pix, a hierarchical video prediction framework that decomposes forecasting into two stages: semantic representation prediction and representation-guided visual synthesis. Instead of directly predicting future RGB frames, our approach first forecasts future scene structure in the feature space of a frozen vision foundation model, and then conditions a latent diffusion model on these predicted representations to render photorealistic frames. This decomposition enables the model to focus first on scene dynamics and then on appearance generation. A key challenge arises from the train-test mismatch between ground-truth representations available during training and predicted ones used at inference. To address this, we introduce two conditioning strategies, nested dropout and mixed supervision, that improve robustness to imperfect autoregressive predictions. Experiments on challenging driving benchmarks demonstrate that the proposed semantics-first design significantly improves temporal semantic consistency, perceptual quality, and training efficiency compared to strong diffusion baselines. We provide the implementation code at https://github.com/Sta8is/Re2Pix

• The paper introduces Re2Pix, a two-stage framework that first predicts semantic features using a frozen vision encoder and then synthesizes high-fidelity video frames.
• It demonstrates significant improvements in semantic consistency, visual quality (lower FID/FVD), and training efficiency over standard diffusion approaches.
• Key innovations include hierarchical decomposition, nested dropout, and mixed supervision to robustly handle noisy predictions and bridge the train-test gap.

Semantics-Guided Hierarchical Video Prediction: A Technical Overview of Re2Pix

Introduction and Motivation

Accurate future video prediction is critical in domains such as autonomous driving, where models must anticipate the evolution of a scene under complex, dynamic conditions. The challenge lies in generating temporally consistent, photorealistic frames while preserving high-level semantics—such as object identity, scene structure, and spatial relationships—over extended sequences. Conventional approaches often predict pixel or VAE-latent representations directly, forcing the model to simultaneously learn semantic forecasting and high-fidelity synthesis in a coupled space. This leads to issues such as object identity drift, flickering artifacts, degraded temporal consistency, increased data demands, and slower training convergence.

To address these limitations, "Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction" (2604.11707) introduces Re2Pix, a two-stage, semantics-first hierarchical framework wherein future scene structure is first forecast in the feature space of a frozen vision foundation model (VFM), followed by photorealistic frame synthesis conditioned on these intermediate semantic representations.

Methodology

Hierarchical Decomposition of Video Prediction

Re2Pix explicitly separates future prediction into two interacting subproblems:

1. Semantic Representation Prediction: Given $M$ observed frames, semantic feature maps $h_{1:M}$ are extracted from each frame using a frozen VFM encoder (e.g., DINOv2 ViT-B). An autoregressive masked transformer then predicts semantic features $h_{M+1:K}$ for the future frames. This architectural separation allows the prediction model to focus exclusively on temporal scene dynamics and high-level reasoning in an abstracted, robust feature space.
2. Semantics-Guided Visual Synthesis: A latent video diffusion model conditions on both past and predicted semantic features ($h_{1:K}$) alongside VAE-compressed context latents ($z_{1:M}$) to generate future VAE latents ($z_{M+1:K}$), which are subsequently decoded to RGB frames via a 3D VAE. Early fusion merges the semantic features with VAE latents before the transformer backbone, allowing joint optimization and stable conditioning without increasing the token count.

This two-stage hierarchy decouples semantic structure prediction from photorealistic synthesis, enabling more structured and sample-efficient modeling of complex video sequences.

Bridging Train-Test Mismatch in Conditioning

A fundamental challenge arises from the discrepancy between clean, ground-truth VFM features used during training (teacher forcing) and the noisy, autoregressively predicted features encountered at inference. To address this, the framework introduces two robust conditioning strategies:

• Nested Dropout: During training, randomly truncate the semantic feature channels (PCA components), replacing fine-grained ones with zeros. This exposes the model to semantic perturbations, forcing robustness and preventing overfitting to high-dimensional, precise representations.
• Mixed Supervision: Train each batch with a 90/10 mix of ground-truth and predicted features, directly regularizing the model against over-reliance on ideal conditioning and improving generalization to noisy predictions during inference.

Architectural Details

The diffusion model backbone leverages a DiT-based design (14 layers, 16 heads, 2048 embedding dim) and inherits efficient building blocks from Cosmos-Predict, such as 3D RoPE, query/key normalization, and LoRA-based adaptive normalization. Semantic features and VAE latents are spatially aligned via patchification and independent embedding before summation at the input stage. This architecture supports direct, early conditioning while maintaining computational scalability.

The autoregressive semantic predictor is a masked transformer trained with a Smooth L1 loss on VFM features. Semantic features are projected to 1152 dimensions via PCA over the concatenated outputs of multiple transformer blocks.

Empirical Results

Quantitative Performance

Experiments were conducted on driving benchmarks (Cityscapes, nuScenes, CoVLA, KITTI), with systematic comparisons to strong diffusion baselines (standard video diffusion, REPA, VideoREPA), parameter scaling ablations, and large-scale pretrained systems such as Vista and Cosmos-Predict-2.

Main findings:

• Semantic consistency (mIoU on all/moving classes, depth metrics):

Re2Pix achieves absolute improvements in frame 19 mIoU of +2.98 (all classes) and +4.65 (moving classes) over the baseline on Cityscapes, with enhanced depth accuracy.

• Generation quality (FID/FVD):

Substantial improvement, with a reduction in FVD by -8.04 and in FID by -2.96 compared to the baseline.

• Training efficiency:

Re2Pix accelerates convergence by a factor of 7× for generation quality metrics (FID, FVD) and up to 14× for segmentation mIoU, substantially lowering computational overhead.

• Scalability and generalization:

When jointly trained on multiple datasets, Re2Pix outperforms both parameter-matched baselines and fine-tuned large-scale pretrained models (Vista, Cosmos-Predict-2) in in-domain and zero-shot settings, despite using orders-of-magnitude less data and compute.

Ablations and Robustness Analyses

• Nested Dropout Effectiveness:

Training with nested dropout across semantic feature dimensionalities significantly improves generation quality and maintains or enhances semantic fidelity. The model is robust to aggressive feature truncation at test time, retaining strong performance with as few as 128 PCA components.

• Mixed Supervision:

The 90/10 mix of ground-truth and predicted conditioning balances semantic consistency and synthesis realism, alleviating the quality degradation from the train-test gap.

• VFM Encoder Variability:

Re2Pix performs consistently well regardless of the chosen VFM encoder (DINOv2, SigLIP-2), indicating robustness to the feature backbone.

• CFG-style representation guidance (Appendix): Exploiting hierarchical semantic feature dropout during sampling enables classifier-free guidance-like mechanisms, which can further modulate the trade-off between semantic fidelity and perceptual sharpness.

Qualitative Analysis

Visual comparisons across diverse scenes demonstrate that Re2Pix preserves semantic boundaries, object identities, and scene geometry more faithfully than all baselines. Temporal consistency is maintained across challenging, cluttered, and rapidly evolving traffic scenarios.

Implications and Future Prospects

Re2Pix substantiates a key architectural claim: Explicitly modeling future semantic representations as hierarchical intermediates enhances both the semantic and perceptual quality of video prediction while achieving significant sample efficiency and convergence speed-up. This modularization supports interpretability, controllability, and decouples the learning of temporal dynamics from fine-grained synthesis.

Practical implications include more reliable world models for planning in autonomous systems, sample-efficient learning for large-scale video prediction, and user-controllable generative pipelines via explicit semantic, geometric, or multimodal guides.

Theoretical implications suggest a move toward structured, semantics-driven generative modeling, with further research directions encompassing integration of richer scene-level priors (e.g., 3D geometry, scene graphs), explicit action/trajectory conditioning, and extending the pipeline to collaborative foundation models.

Conclusion

Re2Pix presents a semantics-first, hierarchical framework for video prediction that leverages explicit future scene structure modeling in VFM feature space to guide generative video diffusion. Extensive evaluation evidences improvement in temporal semantic consistency, visual fidelity, and training efficiency over strong baselines and large-scale competitors. Architectural innovations such as nested dropout and mixed supervision resolve critical issues in conditional robustness. The approach offers a scalable and general template for structured video generation in complex dynamic environments and motivates future research into more nuanced forms of hierarchical generative modeling.