- The paper introduces VideoRAE, a novel framework that leverages frozen video foundation models for semantically aligned generative video autoencoding.
- It employs dual latent regimes with continuous and discrete representations through multi-codebook quantization and REPA alignment, achieving state-of-the-art rFVD and gFVD scores.
- Empirical results demonstrate a ~5x faster convergence and improved video synthesis quality over traditional pixel-based approaches, transforming generative video pipelines.
VideoRAE: Leveraging Video Foundation Models for Unified Video Autoencoding
Motivation and Background
Generative modeling for video has evolved from direct pixel-space synthesis to latent-space approaches, predominantly orchestrated by diffusion and autoregressive models relying on 3D-VAEs. Despite efficiency gains, traditional 3D-VAEs are fundamentally constrained by their pixel-level reconstruction objectives, which result in latent spaces impoverished in semantics and global spatio-temporal relationships. This restricts downstream generative models' ability to synthesize coherent, semantically rich video.
In parallel, Video Foundation Models (VFMs) such as V-JEPA 2 and VideoMAEv2 have demonstrated superior semantic extraction, yet their utility in generative modeling remains largely unexplored. The core research question addressed by this paper is whether frozen VFM features—optimized for understanding—can be transformed into compact, reconstruction-capable, and generation-friendly latents suitable for both diffusion and autoregressive generative paradigms.
Methodology
Semantic-Driven Encoder and Compression
VideoRAE diverges fundamentally from pixel-driven tokenizer paradigms. It utilizes a frozen VFM as an encoder, aggregating hierarchical intermediate features spanning both shallow and deep layers to capture macroscopic semantics and micro-level dynamics. To address the redundancy and sequence length bottlenecks inherent in VFM outputs, a lightweight 1D self-attention projector compresses these features into a compact base-token sequence.
Dual Latent Regimes: Continuous and Discrete
- Continuous Latents: For diffusion-based generative architectures, VideoRAE employs a simple linear projection to the latent space, fully bypassing KL-divergence regularization. The intrinsic semantic structure of VFM features, further regularized by a Representation Alignment (REPA) module, ensures the latent manifold is well-behaved and expressive.
- Discrete Latents: For autoregressive models, VideoRAE introduces a Multi-Codebook SimVQ quantization strategy. Base tokens are partitioned into sub-vectors, each quantized via a dynamically mapped codebook whose parameters are updated by learnable MLPs. This mechanism retains high-dimensionality, preventing codebook collapse and preserving the complexity and semantic richness of the VFM features.
Decoder and Representation Alignment
The decoder, a transformer-based architecture, reconstructs pixels from compressed latents. Crucially, the REPA module aligns intermediate decoder features with VFM teacher outputs both locally and globally, regularizing the latent space topology semantically. This alignment replaces the traditional KL penalty, enforcing the latent distribution's adherence to semantic structure.
Training Objectives
Loss functions comprise pixel-level reconstruction (L1 and LPIPS), adversarial high-frequency texture recovery (GAN loss), semantic alignment (REPA), and quantization commitment (for the discrete regime). Ablation studies confirm the superiority of semantic alignment and multi-scale feature aggregation for both reconstruction and generation.
Empirical Results
Reconstruction and Generation Quality
- Discrete Regime: VideoRAE achieves state-of-the-art rFVD scores on UCF-101 (e.g., 13 with V-JEPA 2, 28 on TokenBench), outperforming leading tokenizers such as LARP and SweetTok. VideoRAE models use fewer tokens and smaller generator architectures while attaining superior generative Fréchet Video Distance (gFVD) scores (e.g., 40 in class-to-video generation).
- Continuous Regime: VideoRAE matches or outperforms industry-grade VAEs—such as LTX-VAE and Lean VAE—in spatial fidelity and perceptual quality. For diffusion transformer settings, VideoRAE reduces gFVD benchmarks to 93 (V-JEPA 2), a substantial improvement over predecessors.
- Text-to-Video Generation: Replacing LTX-VAE with VideoRAE in a 2B-parameter diffusion transformer pipeline on VideoUFO yields faster convergence and higher VBench scores, confirming VideoRAE's utility as a versatile generative backbone.
Convergence Speed
VideoRAE's semantically structured latent space enables ~5x faster convergence for both autoregressive and diffusion models relative to their pixel-centric counterparts, as evidenced by empirical training curves.
Ablations
- Quantization: Multi-codebook SimVQ with high dimensionality achieves the highest PSNR and lowest gFVD, validating the necessity of high-dimensional quantization for semantic preservation.
- Representation Alignment: Incorporating REPA substantially improves generative and perceptual metrics, with KL regularization penalizing both reconstruction and generative quality.
- Feature Aggregation: Multi-scale aggregation (layers 8-24) delivers the optimal trade-off between reconstruction fidelity and semantic richness, with element-wise addition outperforming more complex fusion strategies.
Implications and Theoretical Impact
VideoRAE provides a formal demonstration that frozen VFM representations—previously thought unsuitable for pixel reconstruction—can be repurposed as robust, generative latents. The framework unifies continuous and discrete paradigms, obviates the need for KL-regularization, and establishes a foundation for semantically structured latent spaces that drastically reduce data and computational requirements for generative model training.
Practically, VideoRAE shifts the design landscape for video generative pipelines, positioning VFMs as the standard for latent encoding and enabling more efficient, higher-fidelity generative model deployment. Theoretically, the results confirm that semantic alignment and feature topology preservation are more valuable for generative performance than pixel-level fidelity, and accelerate the blurring of boundaries between representation learning for understanding and generative tasks.
Future developments may extend VideoRAE's framework to multi-modal (audio-visual), long-form, or open-domain settings, and further leverage the structured latent space for planning and prediction in embodied AI contexts.
Conclusion
VideoRAE decisively bridges visual understanding and generative modeling by leveraging frozen video foundation representations with hierarchical feature aggregation, semantic-driven compression, and topological alignment. Empirical evaluations confirm that VFM-derived latent spaces are not only reconstruction-capable but inherently suited for generative modeling, with significant gains in convergence, generative quality, and efficiency. By establishing VFMs as the latent backbone for video synthesis, VideoRAE marks a pivotal advancement in the architecture and training of generative video models (2607.14088).