Latent Video Diffusion Backbone

• Latent Video Diffusion Backbone is the core framework that decouples appearance content and motion through denoising diffusion in a low-dimensional latent space.
• It employs dual U-Net denoisers with attention and positional group normalization to enhance spatial-temporal coherence in video generation.
• Autoregressive synthesis with robust loss functions and latent motion conditioning leads to state-of-the-art performance on large-scale, high-resolution video datasets.

A latent video diffusion backbone is the core architectural and mathematical infrastructure that enables generative modeling of videos via denoising diffusion processes in a learned, typically compressed, latent space instead of pixels. By decoupling appearance content and motion, reducing dimensionality, and leveraging tailored conditioning mechanisms, such backbones make high-fidelity and temporally coherent video synthesis tractable on large-scale datasets and at high resolutions. The backbone as realized in VIDM ("Video Implicit Diffusion Models") comprises framewise convolutional encoding, dual U-Net–style denoisers with attention and specialized normalization, robustness-enhanced objectives, and explicit latent motion conditioning, organized into an autoregressive video generation pipeline that yields state-of-the-art quality and efficiency (Mei et al., 2022).

1. Mathematical Formulation in Latent Space

The backbone relies on a forward noising process in a low-dimensional latent space. Given a frame-wise or latent representation $x_0$, each video frame undergoes a $T$-step Gaussian diffusion process: $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t}\,x_{t-1}, (1-\alpha_t) I)$$ or expressed in closed form,

$$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0, I), \quad \bar\alpha_t = \prod_{s=1}^t \alpha_s$$

The reverse process is learned via deep neural networks $\epsilon_\theta(x_t, t, h)$, with two key variants:

• Content denoiser $\epsilon_\theta$ for initial frame generation,
• Motion denoiser $\rho_\phi$ that incorporates a motion latent $z$ and residual $r$.

The reverse kernel is: $$p_\theta(x_{t-1} \mid x_t, h) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t, h), \sigma_t^2 I)$$ where

$T$0

The loss objective replaces the standard DDPM mean-squared error with a robust Charbonnier penalty: $T$1

2. Content and Motion-Focused U-Net Architecture

• Per-frame Encoder: Four strided convolutional blocks (GroupNorm + ReLU) downsample $T$2 to a latent $T$3 with $T$4 or $T$5, $T$6. This representation conditions both content and motion branches.
• U-Net Denoisers: Both content and motion denoisers share a U-Net backbone:
  • Down path: four spatial resolutions; each level employs two $T$7 convolutions, GroupNorm, SiLU activation, and multi-head self-attention at the coarsest level ($T$8). Downsampling is via $T$9 strided convolution.
  • Up path: nearest-neighbor upsampling and mirror convolution/normalization.
  • Timestep $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t}\,x_{t-1}, (1-\alpha_t) I)$$0 is injected through sinusoidal embeddings at each block. The motion network additionally injects the implicit motion code $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t}\,x_{t-1}, (1-\alpha_t) I)$$1 via MLP $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t}\,x_{t-1}, (1-\alpha_t) I)$$2 FiLM gating.
  • A learnable truncation constant $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t}\,x_{t-1}, (1-\alpha_t) I)$$3 is channel-wise concatenated with $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t}\,x_{t-1}, (1-\alpha_t) I)$$4 at every step.
• Positional Group Normalization (PosGN): Each GroupNorm layer is replaced by

$$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t}\,x_{t-1}, (1-\alpha_t) I)$$5

facilitating spatial and temporal modulation, especially important in the motion branch.

3. Latent Motion Conditioning and Implicit Dynamics

• Motion Latent $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t}\,x_{t-1}, (1-\alpha_t) I)$$6: Computed via a pretrained SpyNet network to estimate an optical-flow-like representation from $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t}\,x_{t-1}, (1-\alpha_t) I)$$7, sharing spatial resolution with the U-Net's bottleneck. $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t}\,x_{t-1}, (1-\alpha_t) I)$$8 is injected into all denoising blocks via FiLM layers.
• Residual Term $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t}\,x_{t-1}, (1-\alpha_t) I)$$9: An adaptive residual $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0, I), \quad \bar\alpha_t = \prod_{s=1}^t \alpha_s$$0 is obtained through a separate encoder applied to the first frame and timestep, enhancing the reverse kernel for the motion denoiser:

$$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0, I), \quad \bar\alpha_t = \prod_{s=1}^t \alpha_s$$1

4. Regularization and Sampling Improvements

• Sampling-Space Truncation: A learnable constant tensor $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0, I), \quad \bar\alpha_t = \prod_{s=1}^t \alpha_s$$2 is introduced in the input of the U-Net, constraining the generative noise space as in StyleGAN truncation. $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0, I), \quad \bar\alpha_t = \prod_{s=1}^t \alpha_s$$3 is fixed during inference.
• Robustness Penalty: The Charbonnier loss function prevents overfitting and eliminates the need for dropout.
• Positional GroupNorm: As in Section 2, provides coordinate-aware normalization across space and time.

5. Autoregressive Video Generation and Training Protocols

• Autoregressive Synthesis:
  • The first frame is generated via the content denoiser from pure noise.
  • Subsequent frames are sampled by the motion denoiser conditioned on computed $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0, I), \quad \bar\alpha_t = \prod_{s=1}^t \alpha_s$$4, residual $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0, I), \quad \bar\alpha_t = \prod_{s=1}^t \alpha_s$$5, and the already generated frames.
• Training Hyperparameters:
  • 1,000 diffusion steps, $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0, I), \quad \bar\alpha_t = \prod_{s=1}^t \alpha_s$$6 linearly scheduled from $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0, I), \quad \bar\alpha_t = \prod_{s=1}^t \alpha_s$$7 to $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0, I), \quad \bar\alpha_t = \prod_{s=1}^t \alpha_s$$8.
  • Batch size 32 per GPU.
  • Adam optimizer with learning rate $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0, I), \quad \bar\alpha_t = \prod_{s=1}^t \alpha_s$$9, no weight decay.
  • The content and motion networks are trained for approximately 1 million steps each, sequentially.
• Efficiency:
  • For $\epsilon_\theta(x_t, t, h)$0 frames and $\epsilon_\theta(x_t, t, h)$1 diffusion steps, the total cost is $\epsilon_\theta(x_t, t, h)$2 (cost of one U-Net forward).
  • On $\epsilon_\theta(x_t, t, h)$3 resolution, generating 16 frames with 1,000 steps per frame takes about 800 seconds on an A100.
  • Inference is commonly reduced to 50–100 steps using distillation or accelerated samplers.

6. Summary Data Flow and Implementation Blueprint

Content Training: $\epsilon_\theta(x_t, t, h)$4

Motion Training: $\epsilon_\theta(x_t, t, h)$5

Generation (Autoregressive): $\epsilon_\theta(x_t, t, h)$6

$\epsilon_\theta(x_t, t, h)$7

This modular backbone allows for experimentation with U-Net capacity, attention depth, noise schedule, and conditioning schemes.

7. Empirical Outcomes and Usability

Experiments demonstrate that VIDM significantly outperforms GAN-based methods on Fréchet Video Distance (FVD) and visual coherence, with improvements attributed to the four key strategies: latent-space diffusion, explicit motion/appearance separation, positional normalization, and the truncation/robustness enhancements. This enables tractable and scalable state-of-the-art video synthesis, providing a robust backbone for further research in latent video diffusion (Mei et al., 2022).