V-RAE: Rethinking Video Latent Spaces for Generation

Abstract: Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. We propose V-RAE, a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features. We evaluate V-RAE with four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation. V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventional video tokenizer latents. Under matched generation settings, our best variant achieves gFVD scores of 117.86 and 19.16 on UCF101 and K600, respectively, while converging up to 6x faster}. We further show that reconstruction quality alone is insufficient to characterize generative utility and introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. Beyond video generation, V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched prediction settings. Taken together, the experiments show that frozen semantic representations can support video reconstruction, generation, and predictive modeling. The project page: https://v-rae.github.io/.

• The paper introduces V-RAE, which combines frozen vision foundation-model features, lightweight temporal attention pooling, and a spatiotemporal decoder to create semantically rich video latents.
• V-RAE achieves 117.86 gFVD on UCF101 and 19.16 on K600, outperforming evaluated VAE spaces while reaching comparable generation quality up to six times faster during training.
• The paper shows that reconstruction quality poorly predicts generation performance and proposes tFVD, a temporal smoothness diagnostic that correlates more strongly with gFVD than rFVD.

Motivation and problem statement

Latent video generation pipelines depend on a video autoencoder to define the space in which a diffusion or autoregressive generator operates. Existing video autoencoders—continuous VAEs, discrete tokenizers, and query-based designs—are overwhelmingly optimized for pixel-level reconstruction, producing latents rich in local texture but poor in semantic organization. The paper argues that a reconstruction-optimal latent space need not be favorable for generative modeling, an argument it substantiates empirically rather than merely asserting. Building on the representation autoencoder (RAE) paradigm in image generation, where frozen encoder features serve directly as the generative latent space, V-RAE extends this to video—a non-trivial extension because image encoders provide no temporal compression (and native video encoders only $2\times$), temporal coherence must survive both compression and decoding, and compression must not destroy the semantic structure that motivates the approach in the first place.

Architecture

V-RAE couples three components. First, a frozen vision foundation model encoder produces temporally dense features; four instantiations are studied spanning distinct pretraining paradigms: DINOv3-L, SigLIP2-L, and EUPE-B (frame-wise image encoders) and V-JEPA 2.1-L (a video-native encoder with tubelet size 2). Second, a learnable temporal attention pooling module compresses adjacent feature frames by a factor $r_P$, using a shared learnable query per head with learnable temporal-offset biases; identity-initialized projections make the pooler start as mean pooling and depart from it only when reconstruction evidence warrants. A non-affine LayerNorm (fixed $\gamma=1$, $\beta=0$) is applied to prevent an SNR-scaling shortcut under noise-augmented training. Third, a ViT-XL spatiotemporal Transformer decoder with 3D RoPE reconstructs pixels, using chunk-causal attention for image-encoder variants and full attention for V-JEPA 2.1, with a multi-frame unpatchify mapping each latent step to $r_{\mathrm{all}}$ consecutive frames.

Training uses L1, LPIPS, GAN (frozen VideoMAE-B discriminator), and Gram losses, with Gaussian noise augmentation on latents ($\sigma\sim\mathcal{U}(0,0.8)$). For generation, a DiT trained with rectified flow operates in the frozen latent space, using a dimension-dependent time shift extended to account for temporal latent length, clean-latent prediction parameterization, and a dual-head auxiliary objective following RAEv2.

Reconstruction results

On K600 validation, V-RAE with V-JEPA 2.1-L attains 2.13 rFVD, a 40.5% improvement over the strongest evaluated large-scale pretrained video VAE (Wan2.1 at 3.58); all four V-RAE variants beat every large-scale VAE baseline on K600. On UCF101, DINOv3-L reaches 6.12 rFVD, second only to Wan2.1's 6.05. Notably, V-RAE trails several VAEs on PSNR, SSIM, and LPIPS while winning on rFVD—the paper reads this correctly as a dissociation between frame-level fidelity and video-level distributional similarity, since the frozen semantic encoder was never optimized for pixel recovery. A supplementary diagnostic, TRED (temporal reconstruction error difference), shows reduced frame-to-frame error fluctuation relative to the frame-wise RAEv2 baseline, indicating that explicit temporal modeling suppresses flicker rather than merely improving aggregate statistics.

The pooling ablation is one of the more instructive results: mean pooling preserves semantics best (91.82% probe accuracy) but reconstructs poorly (94.25 rFVD); convolutional pooling achieves the best rFVD (5.91) at a steep semantic cost (78.26%); Q-Former pooling, despite 67M parameters, is dominated on both axes. Temporal attention pooling, with only ~3M parameters, lands on the Pareto frontier at 6.12 rFVD / 89.13%. The conclusion—that content-adaptive aggregation, not capacity, is the key variable—is well supported within this design family, though only four designs are compared.

Semantic preservation

Probing compressed generation tokens shows V-RAE latents retain far richer semantics than VAE latents: 90.92% top-1 on UCF101 (SigLIP2-L), 72.91% on SSv2 (V-JEPA 2.1), and 83.12% on K400 (DINOv3-L), versus 30.83%, 45.05%, and 53.27% for the strongest VAE baselines. Compression costs are modest: image-encoder variants stay within 3.85 points of their frozen encoders; V-JEPA 2.1 loses 6.37 points on UCF101. This establishes the central claim that temporal compression need not erase inherited semantic structure.

Generation results

Under matched DiT backbones and a fixed 1,280-token budget, all four V-RAE variants outperform all evaluated VAE latent spaces. The best variant (V-JEPA 2.1) achieves 117.86 gFVD on UCF101 and 19.16 on K600, improvements of 25.14 and 22.50 points over the strongest non-V-RAE baselines. Convergence is also markedly faster: EUPE-B matches Wan2.2 VAE's 180K-update gFVD after roughly 30K updates on K600 ($6\times$ faster), and V-JEPA 2.1 shows a comparable $\sim5\times$ speedup on UCF101. Across encoders, stronger semantic preservation generally coincides with better generation, though the authors acknowledge this association is correlational and its precise mechanism remains open.

Reconstruction does not predict generation, and tFVD

A key empirical finding is that rFVD and gFVD induce markedly different rankings of autoencoders, with Pearson correlations of only 0.200 (UCF101) and 0.473 (K600). Concrete counterexamples include HunyuanVideo VAE beating V-RAE (EUPE-B) on UCF101 rFVD (7.73 vs. 8.05) while losing badly on gFVD (211.53 vs. 125.98). To diagnose generative suitability without training a generator, the paper introduces tFVD: interior latent codes are replaced by midpoints of their temporal neighbors, decoded, and scored against real clips with FVD. This deliberately breaks the encode-decode path, probing local temporal smoothness of the latent trajectory and decoder robustness to off-trajectory codes. tFVD correlates substantially better with gFVD ($r=0.919$ on K600, $r=0.621$ on UCF101). Two caveats deserve emphasis: tFVD is a controlled stress test rather than a simulation of generation, and the UCF101 correlation, while improved, is still moderate—so tFVD should be read as a stronger but imperfect proxy.

Future prediction on Cityscapes

Extending beyond class-conditional generation, a conditional DiT predicts future latents from context in both Wan2.2 VAE and V-RAE (EUPE-B, CoVLA-fine-tuned) spaces under identical architecture and budget. V-RAE reduces gFID from 15.02 to 11.52 and gFVD from 144.47 to 111.36—despite substantially worse rFVD (29.29 vs. 7.03)—while achieving lower tFVD (224.60 vs. 319.02). This reversal directly reinforces the paper's thesis: temporal smoothness and decoder robustness matter more than reconstruction fidelity when a predictor must emit approximate future codes. Because architecture and optimization are controlled, the gap is reasonably attributable to the latent representation itself.

Limitations and open questions

The authors are explicit about scope constraints. All experiments use controlled benchmarks (UCF101, K600, Cityscapes) and moderate-scale generators; whether the advantages persist for large-scale open-domain text-to-video generation with higher resolutions, longer videos, and much larger DiTs is untested. The study does not disentangle how specific pretraining objectives, architectures, or spatiotemporal inductive biases drive reconstruction versus generation performance across encoders. And although semantic preservation correlates with generative utility, the causal relationship between latent semantic structure and generative ease remains uncharacterized. One might add that tFVD's protocol (six-code sequences, midpoint interpolation) is a narrow probe, and its generalization to other perturbation schemes is unexamined.

Conclusion

V-RAE demonstrates that frozen vision foundation model representations, equipped with lightweight content-adaptive temporal pooling and a spatiotemporal Transformer decoder, constitute competitive video reconstruction latents, semantically rich tokenizers, superior latent spaces for diffusion-based generation with up to $r_P$0 faster convergence, and effective state spaces for future prediction. Its methodological contribution—showing that rFVD poorly predicts gFVD and proposing tFVD as a better-correlated diagnostic—arguably has value independent of the specific architecture, as it challenges the standard practice of evaluating video tokenizers by reconstruction fidelity alone.