Latent Video Diffusion Backbone

• Latent Video Diffusion Backbone is the core framework that decouples appearance content and motion through denoising diffusion in a low-dimensional latent space.
• It employs dual U-Net denoisers with attention and positional group normalization to enhance spatial-temporal coherence in video generation.
• Autoregressive synthesis with robust loss functions and latent motion conditioning leads to state-of-the-art performance on large-scale, high-resolution video datasets.

A latent video diffusion backbone is the core architectural and mathematical infrastructure that enables generative modeling of videos via denoising diffusion processes in a learned, typically compressed, latent space instead of pixels. By decoupling appearance content and motion, reducing dimensionality, and leveraging tailored conditioning mechanisms, such backbones make high-fidelity and temporally coherent video synthesis tractable on large-scale datasets and at high resolutions. The backbone as realized in VIDM ("Video Implicit Diffusion Models") comprises framewise convolutional encoding, dual U-Net–style denoisers with attention and specialized normalization, robustness-enhanced objectives, and explicit latent motion conditioning, organized into an autoregressive video generation pipeline that yields state-of-the-art quality and efficiency (Mei et al., 2022).

1. Mathematical Formulation in Latent Space

The backbone relies on a forward noising process in a low-dimensional latent space. Given a frame-wise or latent representation $x_0$, each video frame undergoes a $T$-step Gaussian diffusion process: $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t}\,x_{t-1}, (1-\alpha_t) I)$$ or expressed in closed form,

$$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0, I), \quad \bar\alpha_t = \prod_{s=1}^t \alpha_s$$

The reverse process is learned via deep neural networks $\epsilon_\theta(x_t, t, h)$, with two key variants:

• Content denoiser $\epsilon_\theta$ for initial frame generation,
• Motion denoiser $\rho_\phi$ that incorporates a motion latent $z$ and residual $r$.

The reverse kernel is: $$p_\theta(x_{t-1} \mid x_t, h) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t, h), \sigma_t^2 I)$$ where

$$\mu_\theta(x_t, t, h) = \frac{1}{\sqrt{\alpha_t}} \left(x_t - \frac{1-\alpha_t}{\sqrt{1-\bar\alpha_t}} \epsilon_\theta(x_t, t, h)\right)$$

The loss objective replaces the standard DDPM mean-squared error with a robust Charbonnier penalty: $$\mathcal{L}(\theta) = \mathbb{E}[ \sqrt{ \|\epsilon - \epsilon_\theta(x_t, t, \cdot)\|^2 + \eta^2 } ], \quad \eta \approx 1 \times 10^{-8}$$

2. Content and Motion-Focused U-Net Architecture

• Per-frame Encoder: Four strided convolutional blocks (GroupNorm + ReLU) downsample $x_0 \in \mathbb{R}^{3 \times H \times W}$ to a latent $h = E(x_0) \in \mathbb{R}^{C \times h \times w}$ with $C=64$ or $128$, $h=w=16$. This representation conditions both content and motion branches.
• U-Net Denoisers: Both content and motion denoisers share a U-Net backbone:
  • Down path: four spatial resolutions; each level employs two $3 \times 3$ convolutions, GroupNorm, SiLU activation, and multi-head self-attention at the coarsest level ($C \times 16 \times 16$). Downsampling is via $2 \times 2$ strided convolution.
  • Up path: nearest-neighbor upsampling and mirror convolution/normalization.
  • Timestep $t$ is injected through sinusoidal embeddings at each block. The motion network additionally injects the implicit motion code $z$ via MLP $\rightarrow$ FiLM gating.
  • A learnable truncation constant $c$ is channel-wise concatenated with $x_t$ at every step.
• Positional Group Normalization (PosGN): Each GroupNorm layer is replaced by

$$\alpha, \beta = \text{MLP}(h, w, n, t) \qquad \text{PosGN}(x) = \alpha \cdot \text{GroupNorm}(x) + \beta,$$

facilitating spatial and temporal modulation, especially important in the motion branch.

3. Latent Motion Conditioning and Implicit Dynamics

• Motion Latent $z$: Computed via a pretrained SpyNet network to estimate an optical-flow-like representation from $(x_0^{(1)}, x_0^{(n-1)})$, sharing spatial resolution with the U-Net's bottleneck. $z$ is injected into all denoising blocks via FiLM layers.
• Residual Term $r$: An adaptive residual $r = \hat \rho_\phi(x_0^{(1)}, t)$ is obtained through a separate encoder applied to the first frame and timestep, enhancing the reverse kernel for the motion denoiser:

$$p_\phi(x_{t-1} \mid x_t, z) = \mathcal{N}(x_{t-1}; \mu_\phi(x_t, t, z) + r, \sigma_t^2 I)$$

4. Regularization and Sampling Improvements

• Sampling-Space Truncation: A learnable constant tensor $c$ is introduced in the input of the U-Net, constraining the generative noise space as in StyleGAN truncation. $c$ is fixed during inference.
• Robustness Penalty: The Charbonnier loss function prevents overfitting and eliminates the need for dropout.
• Positional GroupNorm: As in Section 2, provides coordinate-aware normalization across space and time.

5. Autoregressive Video Generation and Training Protocols

• Autoregressive Synthesis:
  • The first frame is generated via the content denoiser from pure noise.
  • Subsequent frames are sampled by the motion denoiser conditioned on computed $z$, residual $r$, and the already generated frames.
• Training Hyperparameters:
  • 1,000 diffusion steps, $\beta_t$ linearly scheduled from $10^{-4}$ to $0.02$.
  • Batch size 32 per GPU.
  • Adam optimizer with learning rate $1 \times 10^{-4}$, no weight decay.
  • The content and motion networks are trained for approximately 1 million steps each, sequentially.
• Efficiency:
  • For $N$ frames and $T$ diffusion steps, the total cost is $N \times T \times$ (cost of one U-Net forward).
  • On $256 \times 256$ resolution, generating 16 frames with 1,000 steps per frame takes about 800 seconds on an A100.
  • Inference is commonly reduced to 50–100 steps using distillation or accelerated samplers.

6. Summary Data Flow and Implementation Blueprint

Content Training: $$x_0 \xrightarrow{\text{add noise}} x_t \xrightarrow{[x_t ; c]} \text{Content-U-Net}(t) \rightarrow \hat{\epsilon} \rightarrow \text{robust loss}$$

Motion Training: $$\begin{align*} \text{Select } n; &\ x_0^{(n)} \xrightarrow{\text{add noise}} x_t^{(n)} \ &\ z = \text{SpyNet}(x_0^{(1)}, x_0^{(n-1)}) \ &\ r = \text{ResidualEncoder}(x_0^{(1)}, t) \ &\ [x_t^{(n)}; c] \rightarrow \text{Motion-U-Net}(z, t) + r \rightarrow \hat{\epsilon} \ &\text{loss} = \text{robust}(\hat{\epsilon}, \epsilon) \end{align*}$$

Generation (Autoregressive): $$\text{for } n=1 \ldots N: \quad x_T^{(n)} \sim \mathcal{N}(0, I)$$

$$\text{for } t=T \ldots 1: \begin{cases} n==1: x_{t-1} = \text{sample from Content-U-Net} \ \text{else: } z, r; x_{t-1} = \text{sample from Motion-U-Net} + r \end{cases}$$

This modular backbone allows for experimentation with U-Net capacity, attention depth, noise schedule, and conditioning schemes.

7. Empirical Outcomes and Usability

Experiments demonstrate that VIDM significantly outperforms GAN-based methods on Fréchet Video Distance (FVD) and visual coherence, with improvements attributed to the four key strategies: latent-space diffusion, explicit motion/appearance separation, positional normalization, and the truncation/robustness enhancements. This enables tractable and scalable state-of-the-art video synthesis, providing a robust backbone for further research in latent video diffusion (Mei et al., 2022).