Decoupled Video Diffusion Model

• Decoupled Video Diffusion Models separate static content and dynamic motion within the generative process, addressing inefficiencies in traditional video diffusion.
• This decoupling improves temporal consistency, simplifies motion modeling, allows leveraging pretrained image models, and enhances training and inference efficiency.
• The approach enables scalable, controllable video generation, including text conditioning and long-sequence synthesis, demonstrating superior performance over conventional methods.

A decoupled video diffusion model is a class of generative models for video synthesis in which different factors inherent to video data—most commonly, static content and dynamic motion—are explicitly separated ("decoupled") in the probabilistic modeling framework. This approach addresses inefficiencies and limitations found in traditional, monolithic diffusion models for high-dimensional video data by factorizing the generative process, improving temporal consistency, sample quality, controllability, and leveraging pretrained image diffusion components.

1. Conceptual Foundations and Motivations

Standard diffusion models for video generation typically treat each frame as an independent sample during the noising process, with independent noise added per frame. This i.i.d. noising procedure effectively destroys temporal correlations present in natural videos, increasing the modeling complexity required of the denoising network and leading to artifacts or poor temporal coherence. Furthermore, these methods overlook redundancies in video data, as adjacent frames often share substantial content, with changes primarily manifesting as low-dimensional motion.

Decoupled video diffusion models, as exemplified by VideoFusion (2303.08320), seek to address these limitations by decomposing framewise noise into shared and unique components. Specifically:

• A base noise (common to all frames) captures static, content-related information
• A residual noise (unique to each frame) models dynamic, temporal variations

This hierarchical factorization leverages the redundancy and structure of video data, partitioning generative modeling into subproblems aligned with the data's causal structure: appearance and motion. This design improves sample quality, training and inference efficiency, and supports modular integration with powerful pretrained image diffusion models.

2. Mathematical Formulation and Model Structure

In VideoFusion, the forward diffusion process for a video sequence $\mathbf{x} = \{x^i\}_{i=1}^N$ is modified as follows:

For each frame $x^i$,

$$z^i_t = \sqrt{\hat{\alpha}_t} x^i + \sqrt{1 - \hat{\alpha}_t} \left( \sqrt{\lambda^i} b_t + \sqrt{1 - \lambda^i} r^i_t \right)$$

where $b_t \sim \mathcal{N}(0, 1)$ is base noise shared across frames, $r^i_t \sim \mathcal{N}(0,1)$ is residual noise for frame $i$, and $\lambda^i$ determines the relative proportion of shared versus residual content.

The underlying frame $x^i$ is itself expressible as: $$x^i = \sqrt{\lambda^i} x^0 + \sqrt{1 - \lambda^i} \Delta x^i$$ with $x^0$ as the base frame and $\Delta x^i$ as the per-frame variation.

During denoising, two networks operate in tandem:

• Base Generator $\mathbf{z}^b_{\phi}$: estimates $b_t$ from the "central" frame
• Residual Generator $\mathbf{z}^r_{\psi}$: predicts $r^i_t$ for each frame individually

The composite noise prediction and denoising pipeline enable controlled, explicit reconstruction of both static content and dynamic motion across frames.

3. Comparative Advantages Over Conventional Approaches

The decoupled framework offers several concrete benefits:

• Temporal Consistency: Sharing the base noise $b_t$ across frames naturally preserves inter-frame correlations, reducing flicker and enforcing global content coherence.
• Simplified Motion Modeling: The residual generator specializes in learning only motion-specific variations, reducing the dimensionality and complexity of sequence modeling.
• Efficient Parameterization: By separating content and motion, heavy-lifting pretrained image models can be employed for the base generator ($\mathbf{z}^b_{\phi}$), while the residual generator ($\mathbf{z}^r_{\psi}$) remains lightweight.
• Scalability: Partitioning the problem into coarse (content) and fine (motion) generation stages allows scalability to longer, higher-resolution, and more complex video sequences.

These attributes are confirmed empirically: VideoFusion outperforms strong GAN-based and prior diffusion-based models (e.g., VDM) in objective metrics (IS, FVD) and subjective assessments across a variety of datasets.

4. Empirical Evaluation and Practical Outcomes

Experiments with decoupled video diffusion models have been conducted on datasets such as UCF101, TaiChi-HD, Sky Time-lapse, WebVid-10M, and Weizmann Action.

Key results include:

• On UCF101 (unconditional): VideoFusion achieves IS 72.22 and FVD 220, besting TATS (IS 57.63, FVD 420) and VDM (IS 57.00, FVD 295).
• On class-conditioned UCF101: VideoFusion scores IS 80.03, FVD 173, outperforming baselines.
• The parameter-efficient decoupled approach reduces memory consumption by 21.8% and inference latency by 57.5% relative to VDM.
• VideoFusion supports the generation of sequences up to 512 frames with sustained temporal coherence by reusing the base noise component.
• Leveraging pretrained DALL-E 2 as the base generator leads to further improvements, highlighting the modular compatibility of the framework.

5. Design Extensions, Controllability, and Integration with Pretrained Models

The decoupling paradigm extends naturally toward controllable and scalable video synthesis:

• Adaptation of Pretrained Models: Large image diffusion models can be used "out of the box" as base generators, since only one inference per video per step is needed to estimate base noise. The residual network, being lightweight, incurs minimal extra overhead.
• Text-Conditioned Generation: By training with text-captioned videos (e.g., WebVid-10M), the model readily supports text-to-video generation via conditioning on natural language prompts.
• Content/Motion Control: Fixing base noise while varying residual noise modulates motion independently of content; conversely, fixing residual noise and varying base noise allows content variation with static dynamics.
• Long Sequence Generation: The base noise sharing mechanism enables extension of short clips to high-dimension, long-duration sequences with maintained coherence.

This modularity and compositionality lay the foundation for domain adaptation, transfer learning, and customizable synthesis tasks spanning a wide variety of settings.

6. Limitations and Future Directions

While the decoupled model yields clear advantages, several challenges and opportunities remain:

• Adaptive Decomposition: Fixed $\lambda^i$ parameters may not be optimal across diverse video types. Developing adaptive strategies for adjusting the base-to-residual ratio, either per video or per frame, could improve generalization and diversity.
• Advanced Conditioning Mechanisms: Enhancing the capacity of the residual generator for direct handling of lengthy textual descriptions may bridge the gap between complex language prompts and detailed motion.
• Explicit Content-Motion Disentanglement: Further architectural or loss-driven constraints may yield even greater control and interpretability over generated video structure.
• Domain Adaptation: Closing the gap between image-domain pretraining and target video domains remains a practical concern for maximizing cross-domain transfer and sample quality.

---

Decoupled video diffusion modeling, through partitioning static and dynamic elements of video data in the generative process, offers significant advances in sample fidelity, temporal stability, efficiency, and practical adaptability. The approach, as instantiated in VideoFusion, is increasingly relevant for the construction of scalable, controllable, and high-quality video generative systems, and serves as a foundation for a broad array of future research in factorized and modular video synthesis.