---
title: 'DiViD: Unsupervised Static–Dynamic Video Diffusion'
url: https://www.emergentmind.com/topics/divid
type: topic
---

# DiViD: Unsupervised Static–Dynamic Video Diffusion

DiViD, short for **“Disentangled Video Diffusion,”** is a method for **unsupervised static–dynamic factorization in video**. It is presented as the **first end-to-end video diffusion framework for explicit static-dynamic factorization**, with the goal of decomposing a video into a **global static representation** \(s\), capturing identity or appearance, and **frame-specific dynamic representations** \(\{d_i\}_{i=1}^N\), capturing motion, expression, or other time-varying content. A conditional **DDPM** decoder then reconstructs frames or performs factor swapping while aiming for sharpness, temporal consistency, and reduced static–dynamic leakage [2507.13934].

## 1. Problem setting and conceptual motivation

Given a video
\[
v = \{x_1,\dots,x_N\},
\]
the task is to learn, **without supervision**, a factorization such that \(s\) captures what stays the same across the sequence, \(d_i\) captures what changes across time, and the decoder can reconstruct frames or recombine static and dynamic factors from different videos.

The method is motivated by several failure modes identified in prior sequential disentanglement models. In many **VAE-style** systems, both static and dynamic latents are conditioned on the whole sequence, so **dynamic codes can encode static appearance** and **static codes can encode action or motion**. This information leakage degrades swap performance. The paper also argues that **VAE-based video models often produce oversmoothed outputs**, while **GAN-based methods** can generate sharper samples but remain difficult to disentangle reliably. Video additionally imposes a constraint absent in static-image disentanglement: the model must separate factors while still preserving **temporal coherence** [2507.13934].

DiViD’s core claim is that effective static–dynamic factorization requires explicit **architectural and training inductive biases**, not only generic regularization. The method therefore combines a residual encoder for static removal, a conditional diffusion decoder, a shared-noise schedule, a time-varying KL-based bottleneck, structured cross-attention routing, and an orthogonality regularizer. This combination is intended to preserve static fidelity, improve dynamic transfer, and reduce average cross-leakage.

## 2. Sequence encoder and latent factorization

DiViD is organized around a **sequence encoder** \(\tau_\phi\) and a **conditional diffusion decoder**. The encoder maps a video to one static token and per-frame dynamic tokens:
\[
\tau_\phi(v) \mapsto \bigl(s,\{d_1,\dots,d_N\}\bigr).
\]

Each frame is independently processed by a convolutional image encoder:
\[
f_i = E_{\text{img}}(x_i).
\]
The implementation described for this image encoder uses **input frames of \(128\times128\) RGB**, **3 resolution levels**, **channel multipliers \((1,2,4)\)**, **2 residual blocks per level**, **base channels \(128\)**, and an output projected by a **\(1\times1\) convolution**.

The **static token** is extracted from the **first frame**:
\[
s = \mathrm{MLP}(f_1).
\]
The MLP is specified as **2 layers**, with **hidden dimension 1024**, **ReLU**, and **final output dimension \(256\)**. The model follows a “first-frame static” intuition: the first frame often contains sufficient identity or appearance information for the sequence.

The **dynamic pathway** explicitly removes static content before temporal modeling. Rather than constructing motion codes from \(f_i\) directly, the encoder forms residuals
\[
r_i = f_i - f_1, \qquad i>1.
\]
These residuals are then processed by a **bidirectional LSTM**, followed by **multi-head self-attention** with **8 heads** and **embedding dimension 256**, plus **residual connection**, **layer normalization**, and a final projection:
\[
d_i = \mathrm{Proj}\!\left(\mathrm{Attn}\!\left(\mathrm{BiLSTM}(r_i)\right)\right).
\]

This subtraction mechanism is one of the method’s central architectural biases. The paper’s interpretation is that static appearance is shared across frames and is therefore largely canceled by \(f_i-f_1\), while pose, expression, and movement remain in the residual. This is meant to discourage the dynamic code from storing identity or appearance information.

## 3. Conditional DDPM decoder and video-specific inductive biases

The decoder is a **conditional DDPM** that reconstructs each frame from noisy inputs conditioned on the static token \(s\) and the frame-specific dynamic token \(d_i\):
\[
p_\theta(x_{i,0:T}\mid s, d_i) = p(x_{i,T}) \prod_{t=1}^{T} p_\theta(x_{i,t-1}\mid x_{i,t}, s, d_i).
\]

For the forward process, the method uses the standard DDPM corruption form:
\[
x_{t,i} = \sqrt{\bar\alpha_t}\,x_i + \sqrt{1-\bar\alpha_t}\,\epsilon.
\]
A defining DiViD choice is that **the same noise realization \(\epsilon\) is shared across all frames**. The paper argues that if each frame receives unrelated noise, denoising trajectories may become frame-inconsistent, whereas shared noise provides a common stochastic scaffold across the video and should improve temporal consistency [2507.13934].

The denoiser is a **UNet** with **4 resolution levels**, **channel multipliers \((1,2,4,4)\)**, **base channel width \(128\)**, **2 residual blocks per level**, **self-attention at spatial resolutions 32, 16, 8**, **FiLM-like scale-shift normalization**, **learned up/downsampling**, and **spatial transformer blocks for cross-attention conditioning**. Time embeddings are encoded by a **2-layer MLP** and injected into residual blocks.

Conditioning is injected by **cross-attention**. The operational routing is:
- the **global static token \(s\)** conditions **every frame**,
- the **dynamic token \(d_i\)** conditions **only frame \(i\)**.

A concise formulation is
\[
c_i = [\,s;\,d_i\,],
\]
with the denoiser predicting
\[
\epsilon_\theta(x_{t,i}, t, c_i).
\]
This routing is intended to broadcast sequence-wide appearance while keeping motion local and frame-specific.

The paper also characterizes DiViD as using a **time-varying KL-based bottleneck** inspired by EncDiff. Conceptually, the bottleneck is described as **tight at early timesteps / high noise** and **relaxed at later timesteps / lower noise**. The stated interpretation is that early denoising encourages the static token to carry only robust, global, time-invariant appearance information, while later denoising allows richer use of dynamic tokens. However, the method section does **not** provide a standalone DiViD-specific KL formula beyond this conceptual description.

To further suppress leakage, DiViD adds an **orthogonality regularizer**:
\[
\mathcal{L}_{\text{orth}} = \sum_{i=1}^{N}(s^\top d_i)^2.
\]
This penalizes overlap between the static token and each dynamic token.

## 4. Optimization, training objective, and evaluation protocol

DiViD is trained **end-to-end in one stage**. Its explicit denoising objective is the simplified DDPM loss
\[
\mathcal{L}_{\text{simple}} = \mathbb{E}_{x_0\sim q(x_0),\,\epsilon_t\sim\mathcal{N}(0,I),\,t}
\left[ \|\epsilon_\theta(x_t,t,s,d_i)-\epsilon_t\|_1 \right].
\]
This is a noise-prediction objective, but notably the paper uses **\(\ell_1\)** rather than the more common \(\ell_2\) form. The final training objective is
\[
\mathcal{L} = \mathcal{L}_{\text{simple}} + \lambda \mathcal{L}_{\text{orth}}.
\]

The practical pipeline is: encode the video into \(s\) and \(\{d_i\}\), sample a diffusion timestep and Gaussian noise, corrupt each frame using the shared-noise DDPM forward process, predict the noise with the conditional UNet, and jointly update encoder and decoder.

The method is evaluated on **MHAD / UTD-MHAD** and **MEAD**. MHAD contains **861 video sequences**, has **resolution \(640\times480\)**, **8 subjects**, **27 actions**, and each action is **repeated 4 times**; sequences are standardized by random sampling of **10-frame clips**. MEAD contains **30 subjects** and **8 facial expressions**—**anger, fear, disgust, happiness, sadness, surprise, contempt, neutral**—with variable-length videos standardized by random sampling of **15 frames**; faces are detected by **Haar Cascades** and cropped to **\(128\times128\)**.

The paper compares DiViD against **SPYL** and **DBSE**, both sequential disentanglement methods built on **VAE-style formulations**. For fairness, the authors report replacing the original encoders and decoders of the baselines with the same image encoder and decoder backbone used in DiViD, and they also add a **VGG perceptual loss** to baseline reconstruction losses. The reported baseline loss weights are **reconstruction loss: 10**, **static KL: 5**, and **dynamic KL: 1**.

Evaluation uses two families of metrics. The first is **swap-based evaluation**. Given two clips, the model swaps static and dynamic factors to generate outputs such as
\[
\tilde x^{d_1s_2}_{1:N} = \mathrm{Decode}(\{d^1_i\}, s_2), \qquad
\tilde x^{d_2s_1}_{1:N} = \mathrm{Decode}(\{d^2_i\}, s_1).
\]
A pretrained classifier then measures **Static-only accuracy**, **Dynamic-only accuracy**, and **Joint accuracy**. The second is **cross-leakage classification**, with **Static \(\rightarrow\) Dynamic** and **Dynamic \(\rightarrow\) Static** probes; the reported average leakage is
\[
\mathrm{Average\ leakage} = \tfrac12\bigl(\mathrm{Acc}_{S\to D} + \mathrm{Acc}_{D\to S}\bigr).
\]
Lower leakage is better.

## 5. Quantitative results, qualitative behavior, and limitations

On **MHAD**, the key reported results are as follows [2507.13934].

| Model | Static Only | Dynamic Only | Joint Acc. | Information Leakage |
|---|---:|---:|---:|---:|
| DBSE | 99.0 | 13.7 | 13.7 | 83.5 |
| SPYL | 46.1 | 44.1 | 17.6 | 98.5 |
| DiViD | 94.1 | 29.4 | 28.4 | 70.9 |

These numbers establish the paper’s central empirical claim. **DBSE** preserves identity almost perfectly but transfers motion poorly. **SPYL** achieves a relatively higher dynamic-only score but low static-only accuracy, indicating that it often transfers motion while losing identity. **DiViD** achieves the **highest joint accuracy**, at **28.4**, and the **lowest average leakage**, at **70.9**. The reported interpretation is that DiViD provides a better balance of **static fidelity** and **dynamic transfer** than the VAE-based baselines.

Qualitative findings on **MHAD** and **MEAD** are aligned with the swap metrics. On MHAD, DiViD is described as preserving source identity while transferring target action; on MEAD, it preserves face identity and transfers expression well. A notable qualitative detail is that DiViD often omits irrelevant static scene elements such as a **chair** from the source sequence, which the paper interprets as evidence that it focuses more on actual dynamics rather than entangled background content. By contrast, **DBSE** is described as under-transferring motion, while **SPYL** may mix source and target characteristics or even produce a “swapped” output that is essentially a copy of the target sequence.

At the same time, the paper leaves several issues unresolved. It explicitly states that **systematic ablations are future work**, so there is **no dedicated ablation table** isolating the impact of the shared-noise schedule, the time-varying KL bottleneck, the cross-attention routing, or the orthogonality regularizer. The evaluation is limited to **MHAD** and **MEAD**. Some reported metrics remain modest in absolute terms—most notably **29.4** for dynamic-only accuracy and **28.4** for joint accuracy on MHAD—which indicates that static–dynamic disentanglement in real video remains challenging. The KL bottleneck, while prominent in the conceptual framing, is not accompanied by an explicit standalone optimization term in the method section.

## 6. Distinction from similarly named methods

DiViD should not be conflated with several near-homographic or acronymically similar methods in adjacent areas.

**DIVD** refers to **“Deblurring with Improved Video Diffusion Model,”** a conditional diffusion model for **video deblurring** that introduces **Window-based Temporal Self-Attention (WTSA)** and **Multi-frame Relative Positional Encoding (MRPE)** [2412.00773]. Despite the visual similarity of the names, its task is restoration rather than disentanglement.

**VIDI** is **“Video Dataset of Incidents,”** a multilingual benchmark for **incident and disaster video classification** with **4,534 video clips** and **43 incident categories** [2205.13277]. It is a dataset rather than a generative model.

**VEViD** denotes **“Vision Enhancement via Virtual diffraction and coherent Detection,”** an optics-inspired algorithm for **low-light enhancement** and **color enhancement** based on virtual diffraction and phase readout [2208.12366]. Its domain is image and video enhancement, not latent factorization.

**DiVE** is **“Difference Vector Equalization,”** a method for **robust fine-tuning of vision-language models** that regularizes embedding shifts during CLIP-like adaptation [2511.09973]. It is not a video diffusion model.

**ViDiDi** stands for **“Video Time-Differentiation for Instance Discrimination,”** a self-supervised video representation learning framework based on temporal derivatives and instance discrimination [2409.02371]. A plausible source of confusion is acronym similarity, but the method is explicitly named **ViDiDi**, not DiViD.

Within this naming landscape, **DiViD** specifically denotes the 2025 method for **unsupervised static–dynamic factorization via a conditional video DDPM**, with a residual encoder, structured conditioning, shared temporal noise, and orthogonality-based leakage reduction [2507.13934].

Source: https://www.emergentmind.com/topics/divid