Re2Pix: Semantic Video Prediction Framework

• The paper introduces a hierarchical two-stage approach that decouples semantic dynamics from image appearance, significantly improving video prediction fidelity.
• It employs a frozen vision foundation model and an autoregressive transformer to forecast semantic features before guiding a latent diffusion model for photorealistic synthesis.
• The architecture achieves faster convergence and superior performance on autonomous driving benchmarks compared to traditional latent diffusion models.

Re2Pix (“Representations Before Pixels”) is a hierarchical video prediction framework introduced for high-fidelity, temporally consistent video generation, particularly in the domain of autonomous driving. It addresses the challenge of accurately forecasting future video frames by decomposing the task into two distinct stages: first predicting future scene structure in the semantic feature space of a frozen vision foundation model, and then synthesizing photorealistic RGB frames conditioned on these predicted semantic representations using a latent diffusion model. This design separates the modeling of scene dynamics (“what changes”) from image appearance (“how it looks”), resulting in improved semantic coherence, perceptual quality, and training efficiency compared to monolithic latent diffusion baselines (Karypidis et al., 13 Apr 2026).

1. Hierarchical Architecture and Workflow

Re2Pix implements a two-stage approach:

1. Semantic Representation Prediction: For each input frame $x_t$, a high-level feature map $h_t$ is extracted using a frozen vision foundation model encoder, specifically DINOv2-Reg ViT-B/14, such that $$h_t = E_h(x_t) \in \mathbb{R}^{H_h \times W_h \times C_h}$$. These representations abstract away low-level texture, capturing predominantly object layout and scene structure. An autoregressive predictor $G_h$ forecasts future semantic features one frame at a time via $\hat{h}_{t+1} = G_h(h_{1:t})$.
2. Representation-Guided Visual Synthesis: Future latent representations are generated using a diffusion model $G_z$ conditioned on both past encoded RGB frames, $z_{1:m}$—where $z_t = E_z(x_t)$ using a causal 3D VAE encoder—and the full sequence of semantic features $h_{1:K}$. The diffusion process denoises future latents $z_{m+1:K}$; final RGB frames are recovered by decoding denoised latents with the VAE decoder $h_t$0.

This separation of structure (semantic dynamics) and appearance enables focused learning in each stage, enhancing both the temporal semantic consistency and fidelity of the predicted video.

2. Architectural Components

Re2Pix’s components and design decisions are as follows:

• Vision Foundation Model Encoder: The backbone, DINOv2-Reg ViT-B/14, outputs patch-based feature vectors with $h_t$1 dimensions, spatially organized according to input resolution (e.g., $h_t$2 RGB frames yield a $h_t$3 feature grid at patch size 8).
• Autoregressive Predictor $h_t$4: Realized as a Masked Feature Transformer with 12 transformer layers, hidden size $h_t$5, and 16 attention heads. Training employs a regression-with-masking paradigm; autoregressive roll-out is used at inference.
• Latent Diffusion Model $h_t$6: Based on DiT/Cosmos-Predict video transformer, comprising 14 transformer layers with embedding dimension 2048, 16 heads, RMSNorm, and RoPE. Noise-level conditioning utilizes LoRA-modified AdaLN. Early fusion conditioning is employed: both VAE latents and semantic features are patchified and channel-summed at the transformer input.

3. Mathematical Formulation and Objectives

The forward path and supervisory signals are characterized as follows:

• Semantic Forecasting: Input $h_t$7 is mapped to $h_t$8 for $h_t$9. The predictor outputs $$h_t = E_h(x_t) \in \mathbb{R}^{H_h \times W_h \times C_h}$$0 and the semantic loss is $$h_t = E_h(x_t) \in \mathbb{R}^{H_h \times W_h \times C_h}$$1.
• Latent Diffusion: For each future frame $$h_t = E_h(x_t) \in \mathbb{R}^{H_h \times W_h \times C_h}$$2 ($$h_t = E_h(x_t) \in \mathbb{R}^{H_h \times W_h \times C_h}$$3), forward noising is $$h_t = E_h(x_t) \in \mathbb{R}^{H_h \times W_h \times C_h}$$4. The denoiser predicts $$h_t = E_h(x_t) \in \mathbb{R}^{H_h \times W_h \times C_h}$$5 with diffusion loss $$h_t = E_h(x_t) \in \mathbb{R}^{H_h \times W_h \times C_h}$$6. The overall objective is $$h_t = E_h(x_t) \in \mathbb{R}^{H_h \times W_h \times C_h}$$7.

4. Conditioning Strategies for Robustness

To address train-test mismatch due to reliance on ground-truth representations during training but predicted features at inference, Re2Pix introduces two mitigation strategies:

• Nested Dropout: Semantic PCA channels are sorted by explained variance. During training, a channel count $$h_t = E_h(x_t) \in \mathbb{R}^{H_h \times W_h \times C_h}$$8 is sampled from $$h_t = E_h(x_t) \in \mathbb{R}^{H_h \times W_h \times C_h}$$9; only the first $G_h$0 channels are retained in $G_h$1, with zeros for the rest: $G_h$2. This compels $G_h$3 to learn robustness to missing fine detail.
• Mixed Supervision: With $G_h$4, $G_h$5 is conditioned on predicted features $G_h$6 to simulate test conditions; otherwise, ground-truth $G_h$7 is supplied. This 90/10 ratio exposes the model to challenging, noisy autoregressively-predicted semantics during training, bridging the train-test gap.

5. Training Regimen and Performance

Datasets: Primary training leverages Cityscapes (urban driving), with extended evaluations on nuScenes, CoVLA, and zero-shot generalization on KITTI. Hyperparameters: Sequences length $G_h$8, context $G_h$9. The VAE WAN2.1 compresses spatially ($\hat{h}_{t+1} = G_h(h_{1:t})$0) and temporally ($\hat{h}_{t+1} = G_h(h_{1:t})$1) to yield 7 latent frames of $\hat{h}_{t+1} = G_h(h_{1:t})$2 tokens. Optimization uses Adam (lr $\hat{h}_{t+1} = G_h(h_{1:t})$3), linear warmup/decay, 40K iterations/dataset, batch size 8 on 8 GPUs.

Evaluation Metrics:

Metric	Purpose	Lower/Better
mIoU (All/M)	Semantic consistency	Higher
Depth AbsRel	Depth error	Lower
Threshold $\hat{h}_{t+1} = G_h(h_{1:t})$4	Depth accuracy	Higher
FID/FVD	Perceptual quality	Lower

Key Performance on Cityscapes:

Model	mIoU (A)	mIoU (M)	FID	FVD
DiT Baseline	60.55	57.64	12.86	60.70
Re2Pix	63.53	62.29	9.90	52.66

Re2Pix converges approximately 7$\hat{h}_{t+1} = G_h(h_{1:t})$5 faster on FID/FVD and 14$\hat{h}_{t+1} = G_h(h_{1:t})$6 faster on mIoU than the baseline latent diffusion model. Scalability and generalization gains persist across multi-dataset and zero-shot scenarios.

6. Component Effects and Interpretive Summary

• Semantic Forecasting (Stage 1): Enforces “understanding” of object interactions and per-frame scene layout in a semantically informed latent space, mitigating problems such as identity drift and frame-level flicker.
• Early Fusion Conditioning: Grants the diffusion model unconditional access to both semantic and latent signals, achieving pixel-structure alignment from the earliest diffusion step.
• Conditioning Robustness (Nested Dropout + Mixed Supervision): The synthesis model ($\hat{h}_{t+1} = G_h(h_{1:t})$7) gains resilience against inference-time input noise. This prevents catastrophic error propagation or generation collapse, ensuring sharp, stable results.
• Specialization of Stages: As Stage 2 no longer needs to learn both generative dynamics and appearance, its entire modeling capacity enhances photorealistic synthesis efficiency—yielding both higher perceptual metrics and faster convergence.

This suggests that multi-stage, semantics-first architectures set a new standard for video prediction fidelity, efficiency, and robustness in complex dynamic environments such as autonomous driving. For implementation details and further reproduction, the codebase is publicly available at https://github.com/Sta8is/Re2Pix (Karypidis et al., 13 Apr 2026).