Kling-Omni: Unified Video Generation Framework

• Kling-Omni is a multimodal generative framework that synthesizes high-quality videos from text, image, and video prompts using advanced diffusion-transformer methods.
• The system employs a specialized prompt enhancer, unified representation layer, and video decoder to encode semantic priors and support seamless editing capabilities.
• Optimized with rigorous dataset curation, reinforcement learning, and efficient infrastructure, it achieves state-of-the-art performance in video generation and editing tasks.

Kling-Omni is a generalist generative framework for synthesizing high-fidelity videos directly from multimodal visual language (MVL) inputs, unifying video generation, editing, and reasoning tasks in a single end-to-end system. Unlike staged pipeline approaches, Kling-Omni processes a broad spectrum of user prompts—including text instructions, reference images, and video contexts—by integrating them into a unified multimodal representation. The system is supported by a large-scale, rigorously-curated data system and is optimized through advanced pre-training methodologies and infrastructure techniques, leading to state-of-the-art performance in both quantitative and qualitative video content creation tasks (Team et al., 18 Dec 2025).

1. System Composition and Data Flow

Kling-Omni's architecture comprises distinct but tightly integrated modules:

• Prompt Enhancer (PE): Receives raw text, image, and video inputs, converting them to optimized MVL prompts. It uses a specialized Multimodal LLM (MLLM), outputting token sequences $\{p_i\}$ and an auxiliary "reasoning trace" to encode semantic priors.
• Multimodal Encoder: A shared vision-language transformer encodes text ($x^t$), image ($x^i$), and video context ($x^v$) tokens to unified $d$-dimensional embeddings: $e^t = W_t \cdot Enc_T(x^t)$, $e^i = W_i \cdot Enc_I(x^i)$, $e^v = W_v \cdot Enc_V(x^v)$.
• Unified Representation Layer: Concatenates modality embeddings $[e^t; e^i; e^v]$ and applies multi-head cross-modal attention conditioning on instructions:

$$Q = W^Q h,\ K = W^K h,\ V = W^V h;\ h' = \mathrm{Softmax}(Q K^\top /\sqrt{d}) V$$

• Video Decoder: Implements a diffusion-transformer U-Net backbone. The denoising process is defined as $x^t$0 with loss $x^t$1.
• Reasoning/Editing Module: Adapts cross-attention by introducing learned offsets $x^t$2 to highlight edited regions in response to editing instructions.
• Multimodal Super-Resolution: Enhances generated video frames to cinematic quality.

The system executes as follows: user inputs → Prompt Enhancer → Multimodal Encoder → Unified Representation Layer → Video Decoder → Super-Resolution → output video.

2. Dataset Construction and Curation

The Kling-Omni data system is tailored for robust and flexible multimodal learning:

• Corpus Composition: 80M text–video pairs (web-mined), 20M image–video pairs (image-to-video tasks), and 15M synthetic editing/video references (in-house generated). Resolution ranges from $x^t$3 to $x^t$4, at $x^t$5–$x^t$6 fps, with video durations averaging $x^t$7 seconds.
• Annotations: Each sample contains a caption, modality tags, optional editing instructions, and alignment data (bounding-box/mask for reference-to-video).
• Preprocessing and Quality Control: Filtering enforces resolution $x^t$8 and duration $x^t$9–$x^i$0 s, removes duplicates via frame fingerprinting, and excludes NSFW. Additional checks include temporal blur/jitter detection and CLIP-based semantic alignment for cross-modal consistency.
• Data Augmentation: Includes spatial (random crops, flips, color jitter), temporal (frame-rate jitter, random offsets), and modality mixing (replacing video frames with static images).

This regimen ensures high alignment and diversity across modalities and tasks.

3. Training and Optimization Paradigm

Kling-Omni's development is organized in four progressive stages:

• Stage 1: Large-Scale Pre-training. Joint training on text-to-video (T2V) and image-to-video (I2V) tasks with a $x^i$1 mix ratio, optimizing

$x^i$2

• Stage 2: Supervised Fine-tuning. Continues training for reference-to-video, multi-image referencing, and editing tasks with a curriculum from unedited to complex edited samples. Quality-tuning employs balanced task sampling and precise instruction alignment, optimizing both cross-entropy on tokens and diffusion pixel loss.
• Stage 3: Reinforcement Learning (Direct Preference Optimization, DPO). Human raters construct preference pairs $x^i$3 for sampled MVL conditions. The DPO loss used is

$x^i$4

where $x^i$5 is the model-predicted log-probability.

• Stage 4: Model Distillation for Acceleration. A two-stage protocol: (1) trajectory matching (Phased Consistency Models) with $x^i$6 sampling steps, (2) ODE-based diffusion student with trajectory regularization, yielding %%%%27$x^t$028%%%% acceleration (reducing NFE from $x^i$9 to $x^v$0 with $x^v$13% quality loss).

Additional details: batch size $x^v$2 tokens (64 videos) per iteration, cosine-decay learning rate $x^v$3, AdamW optimizer ($x^v$4, weight decay $x^v$5).

4. Inference and Systemic Infrastructure

Kling-Omni deploys multiple infrastructure advancements for efficient model execution:

• Parallelism: Ulysses parallelism combines pipeline and data parallelism; tensor parallelism (Megatron-style) distributes attention/MLP across GPUs. Overlapping computation and communication hides $x^v$6\% of NCCL overhead.
• Quantization: FP8 quantization for GEMMs and self-attention weights, supporting fused quant/dequant operations and inter-GPU FP8 communication.
• Caching: KV-cache retains reference image/video tokens (Q, K, V) per MVL input and reuses them across diffusion steps, achieving %%%%37$x^i$38%%%% speedup. Cache-offload to host memory moderates GPU utilization.
• Latency and Throughput: On $x^v$9A100 (40 GB), $d$0-step sampling for $d$1 frames realizes $d$2 s latency and $d$3 videos/min throughput; with FP8, overlap, and cache enabled, latency drops to $d$4 s per video.

5. Experimental Results and Comparative Analysis

Quantitative Evaluation

Kling-Omni demonstrates strong results on multiple fronts:

Category	Kling-Omni	Google Veo 3.1	Runway Aleph
Text→Video	FID 18.3	24.7	26.1
Ref.→Video	GSB_G 57%	42%	–
Editing	GSB_G 62%	–	49%

Other metrics include CLIPScore (text$d$5video): $d$6 versus $d$7 for prior state-of-the-art. Human evaluations (GSB: Good–Same–Bad) on OmniVideo-1.0 show $d$8 “Good” for image-reference and $d$9 for editing tasks.

Ablation Findings

• Removing Prompt Enhancer reduces CLIPScore by $e^t = W_t \cdot Enc_T(x^t)$0\%.
• Omitting super-resolution degrades FID by $e^t = W_t \cdot Enc_T(x^t)$1.
• Excluding DPO leads to $e^t = W_t \cdot Enc_T(x^t)$2\% lower preference win rate.

Qualitative Insights

Strengths include stable identity preservation in multi-image referencing, seamless editing free from ghosting, and narrative coherence from sparse keyframes. The primary failure cases are temporal flicker with dynamic textures and minor shape drift under complex occlusions.

6. Context and Implications

Kling-Omni’s unified architecture bridges prompt understanding, cross-modal fusion, diffusion-based synthesis, and super-resolution in a single, scalable framework. It brings together high-capacity data curation, sequential curriculum-based training, preference-driven reinforcement learning, accelerated inference, and robust system infrastructure. This operational integration enables both high-fidelity video generation and advanced multimodal instruction following. The framework substantiates a movement toward multimodal world simulators that can perceive, reason, generate, and interact with dynamically complex environments (Team et al., 18 Dec 2025).