CAT-LVDM: Noise-Aware Video Diffusion Training

• The paper introduces CAT-LVDM, applying BCNI and SACN to inject structured noise, thereby improving temporal coherence in video diffusion.
• It demonstrates that aligning noise with low-dimensional semantic and spectral subspaces reduces error amplification and boosts data efficiency.
• Empirical results reveal significant reductions in Fréchet Video Distance, improved motion smoothness, and superior transferability across video tasks.

Corruption-Aware Training for Latent Video Diffusion Models (CAT-LVDM) is a training paradigm for video diffusion models designed to address the brittleness of current text-to-video generation systems under noisy multimodal conditioning. It introduces structured, data-aligned noise injection methods—Batch-Centered Noise Injection (BCNI) and Spectrum-Aware Contextual Noise (SACN)—that target low-dimensional semantic or spectral subspaces of conditioning embeddings, significantly improving robustness, temporal fidelity, and data efficiency in video synthesis tasks (Maduabuchi et al., 24 May 2025).

1. Motivation and Failure Modes of Isotropic Noise

Conventional image diffusion models apply isotropic Gaussian or Uniform noise independently to all embedding dimensions during training ("CEP" corruption), as seen in prior works (Chen et al. 2024; Daras et al. 2023). However, directly transferring such full-rank, unstructured noise to video diffusion corrupts temporal coherence: small, stepwise embedding perturbations ($\delta z$) are amplified through frame-by-frame denoising, causing unnatural flicker, semantic drift, and error accumulation across $T$ timesteps. The error amplification is governed by the denoiser's Lipschitz constant $L_\theta$, leading to drift magnitude $O(L_\theta^T\, \|\delta z\|)$ over the diffusion process. The 2-Wasserstein radius under uniform or Gaussian corruption scales as $O(p\sqrt{D})$, and the local score-drift as $O(p^2 D)$, with $p$ the noise scale and $D$ the embedding dimension. This rapid expansion quickly overwhelms the model's motion consistency, limiting the effectiveness of these noise strategies in video settings (Maduabuchi et al., 24 May 2025).

2. CAT-LVDM Framework: Structured Corruption for Video Diffusion

CAT-LVDM addresses these problems by training Latent Video Diffusion Models (LVDMs) to denoise under structured, low-rank perturbations that are semantically or spectrally aligned with the data. CAT-LVDM leverages two operators:

• Batch-Centered Noise Injection (BCNI): Perturbs conditioning vectors along the semantic directions characterized by batch deviations.
• Spectrum-Aware Contextual Noise (SACN): Injects noise along the principal spectral modes determined by singular value decomposition (SVD) of the embedding matrix.

By introducing corruptions in directions that preserve meaningful variations—such as temporal stride differences or low-frequency motion—these operators tightly constrain noise propagation, mitigating the drift and temporal instability observed with isotropic strategies. CAT-LVDM is thus fundamentally rooted in aligning noise injection with the natural manifold of video semantics.

3. Mechanisms: Batch-Centered and Spectrum-Aware Noise Operators

3.1 Batch-Centered Noise Injection (BCNI)

BCNI operates on the text or multimodal embedding $z \in \mathbb{R}^D$, leveraging the semantic structure within each batch. The operator is defined as:

$$\text{CBCNI}(z; p) = z + p\|z - \bar{z}\|_2 \cdot (2U(0,1) - 1)$$

where $T$0 is the batch mean, $T$1 controls noise scale ($T$2), and $T$3 is uniform on $T$4. Each sample is perturbed along its deviation from the batch mean, confining noise to the $T$5-dimensional span of batch semantics. This preserves intra-batch variation (such as pose or action differences) while minimizing artificial high-frequency drift, maintaining coherence across video frames.

3.2 Spectrum-Aware Contextual Noise (SACN)

SACN targets the spectral subspace of embeddings $T$6 by reshaping $T$7 to a $T$8 matrix and employing SVD:

$T$9

Spectral coefficients $L_\theta$0 are sampled for $L_\theta$1, leading to the perturbed embedding:

$L_\theta$2

This selectively injects higher-magnitude noise in low-frequency modes, leaving high-frequency details (e.g., fine appearance features) largely unperturbed. The low-rank property (effective noise rank $L_\theta$3) reduces global perturbation, yielding O($L_\theta$4) 2-Wasserstein scaling and constraining drift to temporally coherent motions.

4. Theoretical Foundations and Robustness Guarantees

CAT-LVDM's structured operators provide substantial theoretical improvements across multiple analytic perspectives by substituting the ambient embedding dimension $L_\theta$5 with the much smaller effective rank $L_\theta$6 in every bound:

• Entropy Gain: With BCNI/SACN, entropy increases as $L_\theta$7 (vs. $L_\theta$8 for isotropic noise).
• 2-Wasserstein Radius: Scales as $L_\theta$9, rather than $O(L_\theta^T\, \|\delta z\|)$0, tightening optimal transport bounds.
• Score-Drift: Expected squared difference in denoising outputs is $O(L_\theta^T\, \|\delta z\|)$1 (vs. $O(L_\theta^T\, \|\delta z\|)$2).
• Mixing Time/Spectral Gap: Accelerated convergence as the spectral gap is $O(L_\theta^T\, \|\delta z\|)$3 (vs. $O(L_\theta^T\, \|\delta z\|)$4).
• Log-Sobolev and Transport Inequalities: LSI constant $O(L_\theta^T\, \|\delta z\|)$5, ensuring $O(L_\theta^T\, \|\delta z\|)$6, improving over $O(L_\theta^T\, \|\delta z\|)$7 scaling.
• Rademacher Complexity: $O(L_\theta^T\, \|\delta z\|)$8.
• Minimax Gap: No-free-lunch bound grows only as $O(L_\theta^T\, \|\delta z\|)$9 if one used full-rank noise.

These improvements yield a compression factor of $O(p\sqrt{D})$0 in robustness and generalization guarantees over isotropic baselines (Maduabuchi et al., 24 May 2025).

5. Empirical Evaluation and Quantitative Gains

CAT-LVDM was evaluated across major text-to-video datasets:

Method	MSR-VTT FVD	UCF-101 FVD	Params	#Videos
DEMO (clean)	422.0	547.3	2.3B	10M
LaVie (3B)	526.3	526.3	3.0B	35M
Gaussian CEP	445.3	615.3	2.3B	2M
Uniform CEP	526.8	599.5	2.3B	2M
CAT-LVDM (BCNI)	396.3	505.5	2.3B	2M
CAT-LVDM (SACN)	440.3	440.3	2.3B	2M

Key outcomes:

• BCNI yielded a 31.9% reduction in Fréchet Video Distance (FVD) versus Gaussian/Uniform baselines across WebVid-2M, MSR-VTT, and MSVD.
• SACN improved UCF-101 FVD by 12.3% compared to Uniform and outperformed DEMO/LaVie despite 5× less data.
• Additional metrics reported VBench motion smoothness increase (↑10%), flicker decrease (↓15%), EvalCrafter aesthetic score (↑9%), and action accuracy (↑6%).

The backbone was DEMO—a 2.3B parameter latent U-Net model with decomposed content/motion conditioning—trained on WebVid-2M using Adam with OneCycle learning rate and classifier-free guidance.

6. Ablation Studies and Operator Analysis

Ablations demonstrated that embedding-level corruptions (Gaussian, Uniform, TANI, HSCAN) were consistently outperformed by BCNI/SACN in preserving motion consistency and reducing FVD. Token-level corruptions (e.g., swap/replace/add/remove/perturb) exhibited limited benefit. When isolating operators:

• BCNI excelled on appearance-diverse, caption-rich datasets (WebVid-2M, MSR-VTT, MSVD).
• SACN achieved superior results on class-label, motion-focused datasets (UCF-101).

BCNI also demonstrated greater robustness to hyperparameter variation, yielding flat FVD curves across a range of guidance scales and DDIM steps.

7. Transferability and Broader Applicability

CAT-LVDM's approach showed transferability across:

• Autoregressive Video Generation: Application of BCNI/SACN to MAGViT (473M params) and CogVideo (9.4B params) produced MSR-VTT FVD $O(p\sqrt{D})$1 358–393 with only 0.6B parameters and 2M videos, closely approaching larger AR models trained with 10–35M videos.
• Multimodal Video Understanding: CAT-trained LLaVA (0.5B) outperformed task-specific finetuning with a 24% increase in CIDEr score on Audio-Visual Scene-aware Dialog tasks.

Data efficiency was notable: state-of-the-art quality was achieved with 5× less training data, offering practical advantages for domains lacking massive video datasets.

In sum, CAT-LVDM redefines noise injection in LVDMs through alignment with semantic or spectral subspaces. By transforming noise from a disruptive factor into an effective regularizer, it enables backbone-agnostic, robust video diffusion and enhances transferability to other video generation and understanding architectures (Maduabuchi et al., 24 May 2025).