CogOmniDiT: LoRA Diffusion for Video Synthesis

• CogOmniDiT is a LoRA-augmented latent-diffusion transformer that unifies multi-modal control signals for precise and professional video synthesis.
• It integrates pixel-level and semantic inputs via a transformer architecture, using reinforcement learning for alignment with creative intent.
• The system’s dual-stage processing merges reasoning and diffusion steps to effectively handle sparse inputs such as sketches, clay renders, and textual prompts.

CogOmniDiT is a LoRA-augmented latent-diffusion transformer that serves as the core generative component within the CogOmniControl framework for controllable video synthesis under sparse, abstract, or heterogeneous control conditions. Developed to advance professional video generation workflows—especially those incorporating modalities such as storyboard sketches, clay renders, and textual descriptions—CogOmniDiT unifies control signals at both pixel and semantic levels and is further aligned to creative intent through reinforcement learning mechanisms based on reasoning-driven vision-language modeling (Yang et al., 19 May 2026).

1. Role within CogOmniControl and System Interfaces

Within the factorized architecture of CogOmniControl, generative synthesis is structurally divided into a preceding “Reasoning” phase (implemented by CogVLM) and a subsequent “Generation” phase handled by CogOmniDiT (see Fig. 2 in (Yang et al., 19 May 2026)). Given a set of sparse or abstract user conditions $$C = \{V_\mathrm{ctrl}, I_\mathrm{ref}, T_\mathrm{desc}\}$$, the CogVLM subsystem produces a dense multi-modal reasoning output $R$ and, optionally, an evaluator harness $H$ (Eq. 7). CogOmniDiT then consumes a collection of control-video latents ($Z_\mathrm{ctrl}$), reference-image latents ($Z_\mathrm{ref}$), text embeddings from the user’s description ($T_\mathrm{desc}$), and high-level VLM features ($\mathrm{Emb}_\mathrm{VLM}$), generating the next DDPM/DDIM diffusion step for the evolving video. This bridges professional intent—extracted by a domain-trained VLM—from input modalities through to pixel-space synthesis, allowing unified, fine-grained control.

2. CogOmniDiT Architecture and Data Flow

The Core of CogOmniDiT’s design is a transformer-based diffusion generator ("DiT") that ingests per-modality latent encodings, VLM-derived semantic signals, and produces noise predictions to drive reverse diffusion. The following summarizes the model’s data flow:

1. Preprocessing:
   • VAE encoder: current noisy video latent $Z_t$.
   • VAE encoder: reference image latent $Z_\mathrm{ref}$.
   • Control-video encoder: $Z_\mathrm{ctrl}$.
   • CogVLM connector: $R$0 of dimension $R$1.
2. Concatenated Input Sequence (Eq. 5):

$R$2

1. Diffusion Transformer: Each layer applies LayerNorm, multi-head SelfAttention, and FeedForward operations in residual streams.
2. Output Projection: The updated $R$3 segment is passed through an output projection to predict the noise residual $R$4.

Pseudocode for a single diffusion step: $Z_\mathrm{ctrl}$7

3. Mathematical Formulation of Control and Modality Unification

CogOmniDiT’s modality-agnostic formulation proceeds as follows:

• Modality-Specific Projections: (see 3.1 in (Yang et al., 19 May 2026))
  • $R$5
  • $R$6
  • $R$7
  • $R$8
• Sequence & Positional Encoding:

$R$9

where $H$0 are learned or sinusoidal position embeddings.

• Diffusion Update:

$H$1 is predicted over the concatenated embedding; denoising proceeds per standard DDPM or DDIM schedules.

This explicit unification enables self-attention to jointly resolve dependencies between coarse controls and semantic intent signals.

4. Reinforcement-Learning Alignment and Training Methodology

Fine-tuning of CogOmniDiT is performed using reinforcement learning alignment guided by vision-LLM (VLM) reasoning:

• Objective: Maximize expected video-quality reward $H$2 as in Eq. (6):

$H$3

• Policy Gradient via GRPO (Flow-Factory):
  • Let $H$4 be the (implicit) distribution over DiT-sampled videos.
  • Draw $H$5 samples $H$6, compute rewards $H$7, and use advantage weighting $H$8.
  • The RL gradient estimate is:

  $H$9 - Practically, $Z_\mathrm{ctrl}$0 is implicitly defined by noise prediction, with $Z_\mathrm{ctrl}$1 weighted by $Z_\mathrm{ctrl}$2. - The loss function is augmented as $Z_\mathrm{ctrl}$3.

• Hyperparameters: LoRA rank of 256, joint LoRA+connector updates, RL at 256p resolution, cosine LR schedule (see Table 5 in (Yang et al., 19 May 2026)).

5. Processing Sparse and Abstract Controls

CogOmniDiT is specifically evaluated in workflows involving abstract, professional controls:

• Clay Render Controls: $Z_\mathrm{ctrl}$4 encodes coarse 3D volumes; $Z_\mathrm{ctrl}$5 supplies semantic cues including “identity,” “lighting dynamics,” and “cloth flutter.” DiT’s self-attention fuses silhouette and intent, yielding coherent motion characteristics.
• Storyboard Sketches: Line-art frames and text prompts are unified; $Z_\mathrm{ctrl}$6 encodes resolved narrative aspects such as “camera pan,” “facial close-up,” and “lighting mood,” facilitating coordinated motion and style across frames rather than naively frame-by-frame generation.

This capability addresses key deficits of prior methods when conditioned on sparse or cross-modal professional inputs.

6. Ablation Studies and Empirical Performance

Ablation analyses and benchmarks on CogControlBench and CogReasonBench substantiate the effectiveness of CogOmniDiT within the CogOmniControl pipeline:

System Variant	Multimodal Intent (MI)	Video Quality	Judge Score
Qwen3-Thinking + CogOmniDiT(SFT)	3.142	—	—
CogVLM-SFT + CogOmniDiT	3.397	—	—
CogVLM-RFT + CogOmniDiT	3.586	—	—
CogVLM(RFT) + CogOmniDiT(RFT) (full RL)	3.588	4.239	0.727 (judge)
CogOmniControl (Best-of-4, harness eval.)	—	—	0.742
VINO (open-source)	—	—	0.686
VACE-Wan2.1 (open-source)	—	—	0.665
Seedance2.0 (proprietary)	—	—	0.750

Supervised fine-tuning (SFT) followed by RL-based reward fine-tuning (RFT) on both CogVLM and CogOmniDiT components yields measurable improvements: RFT of CogOmniDiT alone results in a +0.09 MI and +0.08 AQ gain. The joint system closes the gap with proprietary best-in-class frameworks and increases performance further, with selection from multiple candidates (“Best-of-N”) facilitated by harness evaluators, as enabled by the pipeline's “closed-loop” design.

7. Summary and Significance

CogOmniDiT operationalizes a unified, contextually conditioned diffusion process that synthesizes video under variable, often abstract, control conditions. Its empirical superiority arises from: (a) joint ingestion and alignment of pixel and semantic modalities within a single self-attention stream, (b) advanced LoRA-based training followed by preference- and reasoning-driven RL alignment, and (c) demonstrated gains in both multimodal intent following and visual fidelity. This architecture is particularly suited for high-level creative production workflows where alignment with sparse, director-style plans and complex professional intentions is critical (Yang et al., 19 May 2026).