Bridging Video Understanding and Generation in a Unified Framework

Abstract: Recently, unified image generation and understanding have been extensively explored. However, extending such unified modeling paradigms to the video domain remains largely underexplored. A central challenge is that video understanding favors compact, discriminative semantic representations, whereas video generation requires dense signals that preserve visual details and temporal coherence. Videos naturally capture both spatial semantics and temporal dynamics, making them a more suitable modality for unified multimodal modeling compared to static images. In this paper, we propose Vega, a unified framework that bridges video understanding and generation. Vega leverages a shared vocabulary to jointly model text and visual representations and employs a hybrid architecture combining autoregressive (AR) prediction with diffusion-based rendering. Specifically, the AR model focuses on predicting semantically meaningful visual tokens for keyframes, providing a structured representation that guides the diffusion module in rendering dense, high-resolution video frames. Extensive experiments demonstrate that Vega achieves strong performance on video generation benchmarks such as VBench and video understanding benchmarks like VideoMME.

• The paper presents Vega, a unified video framework that fuses multimodal tokens with autoregressive and diffusion models.
• It achieves competitive generative and comprehension results using a shared codebook, dual-flow frame selection, and noise-controlled conditioning.
• The framework improves temporal reasoning and parameter efficiency, enhancing both text-to-video generation and video analysis tasks.

Vega: A Unified Framework for Video Understanding and Generation

Introduction

The paper "Bridging Video Understanding and Generation in a Unified Framework" (2606.31326) introduces Vega, an end-to-end architecture that explicitly unifies video understanding and generation via multimodal token fusion and a hybrid autoregressive–diffusion modeling paradigm. Vega addresses fundamental gaps in prior approaches, which either prioritize generation (typically using diffusion models) or understanding (via vision-LLMs), but fail to bring these modalities together in a parameter-efficient unified framework. Unique to Vega is its design that leverages a shared codebook for vision and language tokens, semantic-space autoregressive modeling, and a pixel-space diffusion decoder, enabling robust multimodal representations, hierarchical temporal modeling, and competitive performance on both generative and understanding benchmarks.

Figure 1: Vega framework architecture: discrete tokens from text and vision are jointly modeled via an autoregressive transformer, followed by a diffusion decoder for dense video rendering and dual-flow visual token extraction for understanding.

Framework Architecture

Multimodal Tokenization and Autoregression

Vega concatenates both text and semantic visual data as discrete tokens, utilizing a vision tokenizer (TA-Tok) that encodes images and video frames in the semantic feature space, not raw pixels. This results in dramatically reduced token counts and improved semantic compactness compared to pixel-space quantization. The autoregressive transformer, with a unified codebook, performs next-token predictions across mixed modality streams, capturing high-level latent scene dynamics and cross-modal dependencies. This approach enables seamless integration and context-aware modeling for both text and visual signals.

Hybrid Semantics–Pixels Generation

For generative tasks, Vega predicts sparse, keyframe-level visual tokens autoregressively in the semantic space. These tokens condition a diffusion decoder, which reconstructs dense, high-res video frames. This design achieves favorable trade-offs: the autoregressive branch delivers structured temporal representations, while the diffusion decoder ensures high-fidelity spatial continuity and visual details. Crucially, Vega employs noise-controlled conditioning, utilizing masked diffusion training for selective frame noise injection, which supports seamless transitions in image-to-video (I2V) and image editing scenarios and preserves fine-grained visual consistency.

Figure 2: Noise-controlled visual conditioning augments semantic token prediction, improving cross-frame coherence and enabling image editing as two-frame video modeling.

Unified Video Understanding

Vega introduces a dual-flow frame selection mechanism for video understanding, inspired by slowfast networks. Pivot frames, sampled uniformly, capture main semantic events and are processed with minimal spatial token pooling; detail frames between pivots are pooled at a higher scale and deliver fine-grained context. This hierarchical temporal modeling balances token budget, computational efficiency, and semantic coverage, enabling effective reasoning across long-duration and multi-plot videos.

Further, Vega utilizes generative visual supervision: masked token prediction on selected visual tokens drives the model to encode robust spatio-temporal structures, complementing task-specific textual supervision.

Figure 3: Dual-flow mechanism: pivots for coarse event modeling, details for dense temporal cues, each pooled at different scales for efficient token allocation.

Experimental Results

Video Generation

Vega demonstrates competitive scores on the VBench and VBench++ benchmarks for text-to-video and I2V generation. Remarkably, despite its smaller parameter count (3B), Vega rivals or surpasses several specialized diffusion-based models and prior unified multimodal architectures. Ablation studies confirm that visual token conditioning outperforms text-only conditioning for diffusion rendering, and that increasing tokens per frame does not always result in improved quality; optimal performance is achieved with adaptive frame selection and moderate token density.

Figure 4: Qualitative samples from Vega's text-to-video generation, reflecting adherence to prompts and visual realism.

Video Understanding

On multimodal evaluation benchmarks including VideoMME, MLVU, LongVideoBench, NextQA, and Egoschema, Vega achieves strong accuracy, rivaling understanding-only models with much larger parameter scales. The dual-flow selection and auxiliary generative supervision were empirically shown to improve semantic reasoning and temporal comprehension while containing the token footprint.

Figure 5: Vega's video understanding: extraction and analysis of semantic content in diverse video scenarios, including fine-grained and causal reasoning.

Failure Analysis

Limitations persist in frame selection and spatial detail recognition in extended or detail-heavy scenarios—pooling reduces granularity, and token subsampling may miss critical frames (e.g., subtitle recognition or small objects). These gaps appear in benchmark evaluations and are substantiated by representative failure cases.

Figure 6: Failure cases: insufficient frame selection and pooling compromise semantic identification and detail recovery on video understanding tasks.

Theoretical and Practical Implications

Vega delivers several practical insights:

• Parameter efficiency: Through semantic tokenization and hierarchical frame selection, the framework achieves high performance with substantially fewer parameters and visual tokens compared to conventional pixel-space models.
• Unified modeling: Vega’s architecture supports generative (text-to-video, I2V, image editing) and understanding tasks under a shared token-centric modeling pipeline, eliminating dependence on separate frozen MLLMs or text encoders.
• Auxiliary generative supervision: Integration of masked visual token recovery as an understanding objective enhances temporal reasoning and semantic capacity, suggesting future directions for autoregressive and diffusion hybrids in multimodal learning.

Theoretically, Vega aligns with contemporary trends in foundation models favoring unified representation spaces, semantic abstraction for generative modeling, and hierarchical temporal segmentation for large-context perception. The adoption of semantic tokenization and hybrid rendering is likely extensible to broader video-centric tasks, including reinforcement learning, interactive agents, and multimodal in-context learning.

Conclusion

Vega is a unified end-to-end framework for both video generation and understanding, built on joint autoregressive semantic modeling and diffusion-based pixel rendering. Its use of a shared codebook, dual-flow frame selection, and auxiliary generative supervision achieves high accuracy and generative fidelity with strong parameter efficiency. While limitations remain in fine-grained spatial perception and long-context coverage, Vega demonstrates the viability of video-centric unified architectures for multimodal learning, with clear implications for future AI video modeling, contextual agents, and reinforcement learning.