Omni-Video: Unified Video Modeling

• Omni-Video is a unified framework for video understanding, generation, and editing that connects a pretrained MLLM to a diffusion decoder using continuous visual clues.
• Its architecture leverages a frozen MLLM, a lightweight vision head, and an adapter to bridge understanding and generation effectively through a multi-stage training scheme.
• Empirical results demonstrate competitive performance in video-QA, text-to-video synthesis, and editing tasks, achieving high frame consistency and reduced temporal flicker.

Searching arXiv for the primary paper and closely related omni-video work to ground the article in current literature. Omni-Video is a unified framework for video understanding, video generation, and instruction-based editing that connects a pretrained multimodal LLM (MLLM) to a video diffusion decoder through continuous visual clues rather than text-only conditioning (Tan et al., 8 Jul 2025). Its central design claim is that an existing MLLM can be taught to produce continuous visual tokens, which are then adapted into the conditional space of a diffusion decoder, yielding a single system that supports question answering, captioning, pure generation, and source-conditioned editing with a lightweight architecture and a multi-stage training scheme (Tan et al., 8 Jul 2025). Within the broader literature, this places Omni-Video between video-understanding unification efforts such as OmniViD and video-generation or editing unification efforts such as OmniV2V and Omni-Video 2 (Wang et al., 2024, Liang et al., 2 Jun 2025, Yang et al., 9 Feb 2026).

1. Definition and research setting

Omni-Video is introduced to address a gap left by unified image models: notable progress in unified understanding and generation had largely concentrated on images, while unified modeling for video understanding and generation remained underdeveloped (Tan et al., 8 Jul 2025). The framework is explicitly described as “an efficient and effective unified framework for video understanding, generation, as well as instruction-based editing,” and its key insight is to make MLLMs emit continuous visual clues that are consumed by a diffusion decoder (Tan et al., 8 Jul 2025).

This positioning is technically distinct from several nearby strands of work. OmniViD reformulates diverse video understanding tasks as video-grounded token generation using word, time, and box tokens in a fully shared encoder-decoder architecture, but it is a video-understanding framework rather than a unified generation-and-editing system (Wang et al., 2024). OmniV2V is a video model for diverse generation and editing scenarios, including object movement, object addition, mask-guided editing, try-on, inpainting, outpainting, human animation, and controllable character video synthesis, but it is centered on dynamic content manipulation within a DiT-based generator rather than a joint understanding-generation stack (Liang et al., 2 Jun 2025). Omni-Video 2 later scales the MLLM-conditioned diffusion paradigm for unified generation and editing, introducing an editing prompt reasoner that converts free-form instructions into explicit target captions and extending the diffusion backbone to 14B (Yang et al., 9 Feb 2026).

A plausible implication is that Omni-Video occupies an intermediate historical position in the emergence of “omni-video” systems: it directly unifies understanding, generation, and editing in one framework, but does so with a deliberately lightweight connector between a frozen MLLM and a pretrained diffusion decoder rather than by replacing either backbone wholesale (Tan et al., 8 Jul 2025).

2. Core architecture

The architecture has three principal components: a frozen MLLM, a lightweight vision head attached on top of the MLLM, and a lightweight adapter placed before the diffusion decoder (Tan et al., 8 Jul 2025). Let the frozen MLLM produce hidden states

$H \in \mathbb{R}^{T\times d},$

where $T$ is the total number of tokens and $d$ is the MLLM hidden size. The vision head maps these hidden states into $N$ continuous output tokens

$V_o \in \mathbb{R}^{N\times c}$

through a two-layer MLP with optional LayerNorm:

$Z = \mathrm{LayerNorm}(H),$

$M = \mathrm{GELU}(ZW_1^v + b_1^v),$

$V_o = MW_2^v + b_2^v.$

During the first training stage, these tokens are explicitly aligned to embeddings $E\in\mathbb{R}^{N\times c}$ from a frozen visual encoder such as SigLIP-v2 through the regression objective

$\mathcal{L}_{vision} = \|V_o - E\|_2^2.$

The adapter bridges $T$0 into the conditioning space of the text-to-video diffusion model:

$T$1

In practice, the adapter is also a two-layer MLP with LayerNorm and GELU:

$T$2

$T$3

$T$4

The resulting $T$5 is the per-frame conditional embedding sequence injected via cross-attention into every U-Net block of the diffusion decoder (Tan et al., 8 Jul 2025).

The diffusion decoder follows the standard noising and denoising construction. The forward process is

$T$6

and the reverse process is parameterized as

$T$7

with

$T$8

Training uses the simplified DDPM objective

$T$9

The authors describe the composition of the vision head, adapter, and diffusion U-Net as a plug-and-play “understand→generate” pipeline (Tan et al., 8 Jul 2025). In architectural terms, the system is designed to preserve the pretrained competence of both sides: the MLLM retains its text-side reasoning and answer generation, while the diffusion decoder retains its generative prior, and the newly trained bridge is intentionally small.

3. Multi-stage training scheme

Omni-Video is trained with a three-stage schedule intended to connect frozen MLLMs and diffusion decoders with limited data and computational resources (Tan et al., 8 Jul 2025). The stages differ in trainable modules, losses, and data mixture.

Stage	Trainable modules	Data and optimization summary
Stage-1	Vision Head, Text Head	Image Understanding (~17.2M), Video Understanding (~2.0M), Text-to-Image (~15.0M), plus small amounts of image/video editing; batch size $d$0, LR $d$1, train for $d$2 steps
Stage-2	Adapter only	Mixed T2I, T2V, image-edit, video-edit (~12M total); batch size $d$3, LR $d$4, $d$5 steps
Stage-3	Vision Head, Adapter, Diffusion Decoder	High-quality videos, 12 fps, longer motion coverage; batch size $d$6, LR $d$7, $d$8 steps

Stage-1 jointly tunes the Vision Head and Text Head with

$d$9

where

$N$0

Its purpose is to align the MLLM’s newly attached visual output space with a frozen visual encoder while preserving language-model competence (Tan et al., 8 Jul 2025).

Stage-2 freezes the diffusion decoder and trains only the adapter using the diffusion matching loss. The stage is explicitly split into two substages for speed: first T2I + T2V until basic video quality is achieved, then the addition of editing tasks for full coverage. Hybrid mixed batching keeps same-resolution frames in a single batch (Tan et al., 8 Jul 2025).

Stage-3 performs end-to-end fine-tuning over the Vision Head, Adapter, and Diffusion Decoder with the combined loss

$N$1

The reported outcome is “sharper, more temporally coherent generations” (Tan et al., 8 Jul 2025).

The training design is one of the paper’s main claims to efficiency. The summary states that the three-stage schedule allows the model to leverage frozen MLLMs and video diffusers with only a few million examples and modest GPU budgets (Tan et al., 8 Jul 2025). This suggests that the framework is intended less as a monolithic end-to-end retraining recipe than as an adapter-based integration protocol.

4. Unified understanding, generation, and editing workflows

In video understanding mode, the MLLM side of Omni-Video remains directly usable. Video frames are fed through the pre-trained visual encoder, then into the MLLM, and finally through the text head “exactly as in VILA,” supporting question answering, captioning, and related understanding tasks (Tan et al., 8 Jul 2025). Understanding is therefore not an auxiliary capability bolted onto a generator; it is a retained operating mode of the unified system.

In generation mode, the route is the reverse of understanding: the MLLM produces visual tokens around the special span <BOV>…<EOV>, the adapter maps them into the conditional space $N$2, and the diffusion decoder produces the video (Tan et al., 8 Jul 2025). The paper’s framing is that the MLLM does not output discrete symbolic plans alone; it outputs continuous visual clues that are close enough to the decoder’s expected conditioning interface to support high-quality synthesis.

In instruction-based editing, the workflow is more elaborate. The user prompt, such as “Remove tree from right side,” is first processed by the MLLM, which generates an editing plan and visual tokens around <BOV>…<EOV> (Tan et al., 8 Jul 2025). If a source video is provided, it is encoded with a 3D-causal VAE into latent embeddings $N$3, and these are appended to $N$4 in the adapter. The diffusion U-Net then denoises from the noised version of the original VAE latents, conditioned jointly on $N$5 and $N$6, thereby preserving unedited regions (Tan et al., 8 Jul 2025).

This workflow is structurally different from systems that treat editing as a separate control path. In Omni-Video, pure understanding uses the text head, pure generation uses the visual-token-to-diffusion path, and fine-grained editing augments that same path with source latent conditioning (Tan et al., 8 Jul 2025). The unification therefore resides in shared intermediate representations and shared backbones rather than merely in multi-task training.

5. Empirical results and ablation evidence

The paper reports results across text-to-image, text-to-video, editing, and understanding settings (Tan et al., 8 Jul 2025). On GenEval for text-to-image, Omni-Video achieves an overall score of 0.49 and is described as competitive with 1.3 B unified models. In qualitative text-to-video generation, the model shows finer micro-expressions, with laplacian PSNR $N$7 dB on held-out clips, and high frame consistency, with FVD decreasing from 720 to 420 after Stage-3 on the UCF-101 split (Tan et al., 8 Jul 2025).

For editing, temporal flicker is reported to decrease by 60% versus a baseline diffusion editing system. For understanding, the model reaches 92% accuracy on video-QA benchmarks and is described as comparable to VILA (Tan et al., 8 Jul 2025). These results are presented as evidence that one model can retain useful competence across all three operating modes without task-specific architectural forks.

The ablation study is central to the paper’s technical argument. Removing Stage-1, and thus removing explicit vision-head alignment, causes a 70% drop in visual fidelity with increased FVD. Removing the adapter and replacing it with a direct $N$8 projection leads to unstable training and a 30% FVD increase. Skipping Stage-3 joint fine-tuning yields sharper images but motion artifacts (Tan et al., 8 Jul 2025). Taken together, these ablations support the claim that the system’s performance depends not only on the existence of a bridge between MLLM and diffusion decoder, but on the particular decomposition into aligned vision head, lightweight adapter, and final joint tuning.

A plausible interpretation is that Omni-Video’s empirical profile reflects a deliberate trade-off. The reported metrics indicate satisfactory generalization across heterogeneous tasks, while the ablations indicate that the model is sensitive to representational mismatch between understanding-side latents and generation-side latents (Tan et al., 8 Jul 2025).

6. Place within the broader omni-video literature

The phrase “omni-video” now spans models, datasets, and benchmarks that extend beyond Omni-Video’s original generation-understanding-editing triad. VideoMind defines “omni-modal video” as a sample containing images, video, audio, text, ASR, OCR, tags, and multi-layer captions, and adds factual, abstract, and intent descriptions for 103,000 samples (Yang et al., 24 Jul 2025). StreamOV addresses streaming omni-video understanding by introducing multimodal evidence-guided long-short term memory and a hidden-state-driven response trigger under bounded memory (Xie et al., 25 May 2026). VideoOdyssey shifts the focus to ultra-long-context and omni-modal video understanding, with 100 videos of mean duration 109 minutes and average continuous certificate lengths of 16 minutes for VideoOdyssey-V and 12.8 minutes for VideoOdyssey-AV (He et al., 21 May 2026).

This wider landscape clarifies what Omni-Video does and does not attempt. It does attempt unified offline understanding, generation, and editing in one framework (Tan et al., 8 Jul 2025). It does not directly target streaming multi-turn interaction, proactive response timing, or hour-scale context retention, which become explicit concerns in StreamOV and OmniPro (Xie et al., 25 May 2026, Zhao et al., 18 May 2026). Nor does it address intent-grounded dataset design in the sense introduced by VideoMind, where the “why” dimension is annotated as part of a 6W schema (Yang et al., 24 Jul 2025).

Within generation-oriented work, Omni-Video also occupies a specific point in an evolving design space. OmniV2V unifies many editing and manipulation tasks through latent DiT token fusion and a LLaVA-based visual-text instruction module (Liang et al., 2 Jun 2025). Omni-Video 2 scales MLLM-conditioned diffusion, adds explicit target-caption reasoning, and reports 73.5 FiVE-Acc on fine-grained editing and 84.7 total score on VBench (Yang et al., 9 Feb 2026). OmniWeaving further combines multimodal composition and reasoning-informed generation through Qwen2.5-VL, HunyuanVideo-1.5, DeepStacking, and a “thinking mode” (Pan et al., 25 Mar 2026). These systems indicate that unified video generation has increasingly been coupled to stronger MLLM reasoning and larger diffusion backbones.

7. Limitations and subsequent directions

The limitations stated for Omni-Video are twofold. First, long-duration high-resolution video quality remains limited: FVD and PSNR still trail specialized text-to-video systems. Second, there is some residual mismatch between generation and understanding latents, and the summary states that addressing this may require fully end-to-end latent co-optimization in future work (Tan et al., 8 Jul 2025). These limitations align with the paper’s own empirical and ablative evidence, especially the sensitivity to Stage-1 alignment and Stage-3 joint tuning.

Subsequent work in adjacent areas points to several orthogonal directions rather than a single agreed remedy. Owl-1 introduces a persistent latent world state $N$9 for consistent long video generation, explicitly targeting long-horizon coherence beyond last-frame conditioning (Huang et al., 2024). Omni-Video 2 strengthens instruction faithfulness by having an MLLM produce an explicit target caption before diffusion-based editing or generation (Yang et al., 9 Feb 2026). OmniWeaving activates reasoning traces and multimodal composition at scale, while DreamVideo-Omni focuses on multi-subject customization with omni-motion control and latent identity reward feedback learning (Pan et al., 25 Mar 2026, Wei et al., 12 Mar 2026).

Seen in retrospect, Omni-Video establishes a specific and influential formulation of unified video modeling: an MLLM produces continuous visual clues, a lightweight adapter aligns them to a diffusion decoder, and a staged optimization scheme makes the overall system usable for understanding, generation, and editing within one architecture (Tan et al., 8 Jul 2025). The later literature broadens the scope of “omni-video” toward longer context, richer modalities, stronger reasoning, and more specialized control, but the core integration problem addressed by Omni-Video—the efficient connection of language-grounded multimodal reasoning to high-quality video synthesis—remains central across these developments.