UniVid: Unified Video-Modeling Systems

• UniVid is a research direction unifying various video-modeling systems to replace task-specific pipelines with a single, multimodal architecture.
• It employs techniques such as visual sentences, dual-stream diffusion, and stochastic condition masking to jointly handle video generation, editing, and segmentation.
• Unified design patterns enable versatile applications, improving performance in video generation, understanding, and policy-aware moderation while streamlining workflows.

Searching arXiv for the most relevant "UniVid" and closely related unified video-model papers to ground the article. UniVid is a label associated with a cluster of unified video-modeling systems rather than a single canonical architecture. In the literature represented by "UniVid: Unifying Vision Tasks with Pre-trained Video Generation Models" (Chen et al., 26 Sep 2025), "UniVid: The Open-Source Unified Video Model" (Luo et al., 29 Sep 2025), "UniVid: Pyramid Diffusion Model for High Quality Video Generation" (Xiao et al., 14 Mar 2026), and "UniVideo: Unified Understanding, Generation, and Editing for Videos" (Wei et al., 9 Oct 2025), the unifying objective is to place multiple input modalities, task types, or training paradigms under one video-native model. Closely related works extend the same orientation to video moderation in "UNIVID: Unified Vision-LLM for Video Moderation" (Yang et al., 4 Jun 2026), to universal video segmentation in "UniVS: Unified and Universal Video Segmentation with Prompts as Queries" (Li et al., 2024), to multimodal world-aware generation in "UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation" (Huang et al., 8 Dec 2025), and to omni-directional multimodal conditional generation in "UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors" (Chen et al., 1 May 2026).

1. Scope and nomenclature

In current usage, the name appears across several distinct systems. Some are centered on video generation, some on joint understanding and generation, and others on moderation or dense prediction. The shared theme is not a single implementation, but the attempt to replace task-specific pipelines with one architecture, one conditioning interface, or one training formulation.

System	Primary focus	Representative design
UniVideo	Understanding, generation, editing	Dual-stream MLLM + MMDiT
UniVid	Vision tasks via video diffusion	"Visual sentences"
UniVid	Open-source unified video model	MLLM + diffusion decoder
UNIVID	Video moderation	Policy-aware captioning
UniVS	Universal video segmentation	Prompts as queries

"UniVideo" explicitly defines "unified" in three ways: a single architecture handles diverse input modalities and output types, the same generator is trained jointly across multiple tasks rather than via task-specific adapters or pipelines, and a single instruction interface supports compositional control and multimodal prompting (Wei et al., 9 Oct 2025). "UniVid: Unifying Vision Tasks with Pre-trained Video Generation Models" instead frames unification through a "visual sentence" formulation in which image and video tasks are represented as context-target clip sequences (Chen et al., 26 Sep 2025). "UniVid: The Open-Source Unified Video Model" couples an MLLM with a diffusion decoder so that one system supports both video understanding and generation (Luo et al., 29 Sep 2025). "UNIVID" applies unification to industrial moderation by replacing fragmented policy-specific classifiers with policy-aware captioning and a single backbone (Yang et al., 4 Jun 2026). "UniVS" applies the same principle to dense prediction by converting different video segmentation tasks into prompt-guided target segmentation (Li et al., 2024).

2. Core formulations of unification

The most expansive formulation appears in UniVideo. It supports text-to-video, image-to-video, in-context video generation, in-context video editing, image editing, style transfer, task composition, and visual-prompt-based generation under one multimodal instruction paradigm, with no task-specific adapters or mask requirements during inference (Wei et al., 9 Oct 2025). This design is notable for treating composition itself as a first-class property: a single instruction can request operations such as identity replacement, deletion, and stylization simultaneously.

A different but related abstraction is the "visual sentence" used by UniVid. A training or inference instance is represented as a sequence $S = [C_1, \ldots, C_k, Y_1, \ldots, Y_m]$, operationalized as four clips $V = [A, A', B, B']$ packed along the time axis. Here $A$ is an example input, $A'$ an example output, $B$ a query input, and $B'$ the target to be generated. On this formulation, understanding and generation become sequence-order variants of the same conditional diffusion problem: reversing the order converts an understanding task into a conditional generation task with identical architecture and loss (Chen et al., 26 Sep 2025).

UniVidX unifies tasks through stochastic partitioning of modalities into clean conditions and noisy targets. Its Stochastic Condition Masking samples condition and target sets at each iteration so the model learns omni-directional conditional generation instead of a fixed mapping such as RGB $\rightarrow$ normal or blended RGB $\rightarrow$ alpha (Chen et al., 1 May 2026). UnityVideo pursues a parallel objective through dynamic noising, which routes samples among conditional generation, modality estimation, and joint generation within one Conditional Flow Matching framework (Huang et al., 8 Dec 2025). In each case, the common move is to make the task definition emerge from the conditioning pattern rather than from architectural specialization.

3. Architectural patterns

A recurrent pattern in the UniVid literature is the separation of semantic parsing from video synthesis. UniVideo uses a dual-stream system that couples qwen2.5VL-7B, kept frozen, with a modified HunyuanVideo-T2V-13B. The MLLM processes text, images, and short video clips, and its last-layer hidden states are mapped by a trainable MLP connector with 4× expansion into the MMDiT conditioning space. Visual conditions and noisy target latents are encoded by a VAE and concatenated along the temporal axis, with 3D positional embeddings that preserve spatial indices and increment only the temporal dimension when new condition sequences are appended (Wei et al., 9 Oct 2025). The paper reports that identity preservation drops sharply if visuals are fed only to the MLLM rather than also to the MMDiT, which makes cross-stream conditioning central rather than auxiliary.

The open-source UniVid uses a related hybrid design. Its MLLM follows the BAGEL framework and Qwen2 choices, uses SigLIP2-so400m/14 for semantics and FLUX/CV-VAE for pixel-level details, and conditions a Wan 2.2 DiT-based video generator through a lightweight adapter (Luo et al., 29 Sep 2025). The key addition is Temperature Modality Alignment, which modulates cross-modal attention over normalized flow progress $u \in [0,1]$: textual guidance is boosted in the first $40\%$ of the trajectory and, for reference-image conditions, visual identity refinement is boosted after $V = [A, A', B, B']$0 of the trajectory. This directly addresses the text-visual token imbalance identified in MM-DiT-style decoders.

A more classical generation-oriented architecture appears in the pyramid-diffusion UniVid. Starting from Stable Diffusion 2.1, it inserts Pyramid Spatial-Temporal Attention, Pyramid Spatial-Temporal Convolution, dual-stream cross-attention over CLIP text tokens and CLIP image patch tokens, and a temporal self-attention layer. Control weights $V = [A, A', B, B']$1 and $V = [A, A', B, B']$2 interpolate text-only, image-only, and hybrid conditioning at inference (Xiao et al., 14 Mar 2026). Here the unification problem is narrower than in UniVideo or the open-source UniVid: it concerns T2V, I2V, and $V = [A, A', B, B']$3 under one latent diffusion model.

UnityVideo and UniVidX generalize the architectural template further into multimodal latent spaces. UnityVideo uses a shared 10B DiT with a modality switcher, in-context learner, 3D RoPE, and modality-specific output heads so RGB, segmentation masks, human skeletons, DensePose, optical flow, and depth maps can all be processed by one backbone (Huang et al., 8 Dec 2025). UniVidX keeps the Wan2.1-T2V-14B backbone frozen and inserts Decoupled Gated LoRA and Cross-Modal Self-Attention, with modality latents concatenated along the batch dimension rather than the channel dimension in order to preserve the backbone’s native priors (Chen et al., 1 May 2026).

4. Task coverage and generalization

UniVideo presents one of the broadest task envelopes. Its supported capabilities include text-to-video, image-to-video, in-context video generation in single-ID and multi-ID settings, in-context video editing for insert, swap, delete, and stylize operations without masks, task composition, and visual-prompt-based generation through drawings, annotations, or diagrams (Wei et al., 9 Oct 2025). Two generalization forms are emphasized. The first is compositional generalization, such as deleting one identity while stylizing another in the same request. The second is modality and task transfer: editing control learned from large-scale image editing and limited in-context video edits transfers zero-shot to free-form video edits such as green-screening a character, changing material to chrome, or making a scene rainy.

UniVid, in the visual-sentence sense, extends a pre-trained video diffusion transformer to both generation and pixel-level understanding. Its reported task set includes scribble map transfer, style transfer, camera movement transfer, depth estimation, semantic segmentation, salient object tracking, edge map prediction, and surface normal estimation (Chen et al., 26 Sep 2025). Because images are encoded as single-frame clips and videos as multi-frame clips, mixed contexts such as image-image-video-video or image-video-image-video are handled without introducing special tokens. The paper’s central claim is that cross-modal inference and cross-source transfer can arise from few-shot supervised fine-tuning on natural-video-pretrained backbones.

UnityVideo broadens the scope from task unification to world-aware multimodality. It supports controllable generation $V = [A, A', B, B']$4, modality estimation $V = [A, A', B, B']$5, and joint generation $V = [A, A', B, B']$6 across segmentation masks, human skeletons, DensePose, optical flow, depth maps, RGB video, and text (Huang et al., 8 Dec 2025). The reported qualitative result is stronger physical reasoning, including refraction through transparent media, plausible trajectories, more faithful depth-guided generation, and improved temporal stability.

UniVidX realizes a narrower but highly flexible form of multimodal unification over pixel-aligned stacks. UniVid-Intrinsic covers RGB, albedo, irradiance, and normal, while UniVid-Alpha covers blended RGB, alpha, foreground, and background. Together, the two models support text-to-intrinsic, inverse rendering, forward rendering, text-to-RGBA, auxiliary-free video matting, layered composition, relighting, retexturing, material editing, and inpainting (Chen et al., 1 May 2026). A plausible implication is that the UniVid program increasingly treats a video model as a latent operator over heterogeneous but aligned modalities rather than only as a text-conditioned RGB synthesizer.

5. Understanding-oriented and task-specific extensions

Not all UniVid-related systems are generative in the narrow sense. UNIVID applies the unification idea to moderation by generating policy-aware captions that act as an interpretable intermediate representation. The captioner is integrated into a pipeline with a Risk Filter, a Moderation Actor, and Trend Governance; captions and embeddings are cached for reuse in risk filtering, moderation acting, trend detection, and cross-functional applications such as ads keywording and brand safety (Yang et al., 4 Jun 2026). The system is explicitly positioned against fragmented black-box classifiers and against open-source or commercial VLMs that suffer from safety-guardrail refusals or lack platform-specific policy alignment.

UniVS extends unification to dense spatiotemporal prediction. Its central move is to treat prompts as queries and to use predicted masks from previous frames as future visual prompts, thereby converting category-specified and prompt-guided video segmentation tasks into a common prompt-guided target segmentation problem (Li et al., 2024). The architecture includes a prompt encoder, target-wise prompt memory pools, a target-wise prompt cross-attention layer, separated self-attention, and explicit mask decoding. Because the same set of weights covers video instance, semantic, panoptic, object, referring, and prompt-guided video segmentation, UniVS shows that the UniVid idea is not restricted to diffusion-based generation.

An earlier precursor is UniDual, which unifies image and video understanding through shared 2D spatial convolutions followed by modality-specific point-wise operations: a 1×1 convolution for images and a temporal 1D convolution for videos (Wang et al., 2019). Repeated stacking yields two partially distinct execution pathways that are unified by spatial filters but separated where temporal structure matters. This is architecturally simpler than later MLLM-DiT systems, but it already embodies the same principle: share what is common across modalities and specialize only where modality-specific statistics become unavoidable.

6. Empirical performance, limitations, and research trajectory

The literature reports strong but heterogeneous empirical results. UniVideo reaches MMBench 83.5, MMMU 58.6, and MM-Vet 66.6 through the frozen qwen2.5VL-7B, achieves VBench T2V 82.58, and reports that multi-task learning improves prompt following by +0.16, subject consistency by +0.11, and overall score by +0.06 on average across tasks (Wei et al., 9 Oct 2025). In a decisive ablation, removing visual inputs from the generator stream reduces single-ID subject consistency from 0.88 to 0.32, indicating that unified multimodal conditioning is not reducible to MLLM-side parsing alone.

The open-source UniVid reports a VBench-Long total score of 85.27, a 2.2% improvement over EasyAnimateV5.1, and QA gains of 1.0% on MSVD-QA and 3.3% on ActivityNet-QA over the best prior 7B baselines (Luo et al., 29 Sep 2025). The visual-sentence UniVid reports improvements over LVM on style transfer, edge prediction, segmentation, depth, and surface normal estimation, despite using only 20 paired examples per task for supervised fine-tuning (Chen et al., 26 Sep 2025). UNIVID reports a 42.7% relative reduction in violation leakage, a 37.0% relative reduction in overkill rate, FP8 deployment on H100 GPUs at 5.7 QPS per device, and an inference cost of approximately \$180 per 1M videos (Yang et al., 4 Jun 2026). UniVS is reported as the only model in its comparison set to handle all six video segmentation task families with a single set of weights, with the Swin-L variant reaching 59.8 mIoU and 92.3 mVC_8 on VSPW, 49.3 VPQ and 58.2 STQ on VIPSeg, and 68.6 on VIPOSeg (Li et al., 2024).

The limitations are equally consistent across papers. UniVideo occasionally over-edits unrelated regions, struggles with complex motion and long videos, and remains an "assembled" system with a frozen MLLM plus trained generator rather than a natively end-to-end multimodal video model (Wei et al., 9 Oct 2025). The open-source UniVid notes that coupling an autoregressive MLLM with a DiT-based decoder increases compute and memory, while its understanding branch remains keyframe-driven and struggles with very long videos and fine motion (Luo et al., 29 Sep 2025). The visual-sentence UniVid identifies sensitivity to context design, sampling cost, and performance dependence on the spatiotemporal competence of the pre-trained video model (Chen et al., 26 Sep 2025). UniVidX reports practical limits of four modalities, 21 frames, and at most 480p resolution per clip under its current 14B configuration, while UnityVideo notes autoencoder artifacts and possible propagation of biases from modality extractors such as SAM, RAFT, and DensePose (Chen et al., 1 May 2026); (Huang et al., 8 Dec 2025).

This suggests that "UniVid" has become less the name of one model than the name of a research direction. Across these papers, the central hypothesis is stable: a video-native backbone can be made more useful by replacing fixed task mappings with multimodal conditioning, joint training, or interpretable intermediate representations. What varies is the operational definition of unification—instruction parsing, visual sentences, policy-aware captioning, prompt-as-query segmentation, modality routing, or dual-stream diffusion—but the research program is coherent in its attempt to turn video systems from narrow pipelines into general-purpose multimodal infrastructures.