DiViD: Unsupervised Static–Dynamic Video Diffusion

• The paper introduces DiViD, an end-to-end video diffusion method that factorizes videos into a global static token for appearance and frame-specific dynamic tokens for motion.
• It employs a residual encoder with shared-noise diffusion and an orthogonality regularizer to minimize static–dynamic leakage and maintain temporal consistency.
• Experimental evaluations on MHAD and MEAD show that DiViD achieves improved joint accuracy and reduced information leakage compared to VAE-style baselines.

DiViD, short for “Disentangled Video Diffusion,” is a method for unsupervised static–dynamic factorization in video. It is presented as the first end-to-end video diffusion framework for explicit static-dynamic factorization, with the goal of decomposing a video into a global static representation $s$, capturing identity or appearance, and frame-specific dynamic representations $\{d_i\}_{i=1}^N$, capturing motion, expression, or other time-varying content. A conditional DDPM decoder then reconstructs frames or performs factor swapping while aiming for sharpness, temporal consistency, and reduced static–dynamic leakage (Gheisari et al., 18 Jul 2025).

1. Problem setting and conceptual motivation

Given a video

$v = \{x_1,\dots,x_N\},$

the task is to learn, without supervision, a factorization such that $s$ captures what stays the same across the sequence, $d_i$ captures what changes across time, and the decoder can reconstruct frames or recombine static and dynamic factors from different videos.

The method is motivated by several failure modes identified in prior sequential disentanglement models. In many VAE-style systems, both static and dynamic latents are conditioned on the whole sequence, so dynamic codes can encode static appearance and static codes can encode action or motion. This information leakage degrades swap performance. The paper also argues that VAE-based video models often produce oversmoothed outputs, while GAN-based methods can generate sharper samples but remain difficult to disentangle reliably. Video additionally imposes a constraint absent in static-image disentanglement: the model must separate factors while still preserving temporal coherence (Gheisari et al., 18 Jul 2025).

DiViD’s core claim is that effective static–dynamic factorization requires explicit architectural and training inductive biases, not only generic regularization. The method therefore combines a residual encoder for static removal, a conditional diffusion decoder, a shared-noise schedule, a time-varying KL-based bottleneck, structured cross-attention routing, and an orthogonality regularizer. This combination is intended to preserve static fidelity, improve dynamic transfer, and reduce average cross-leakage.

2. Sequence encoder and latent factorization

DiViD is organized around a sequence encoder $\tau_\phi$ and a conditional diffusion decoder. The encoder maps a video to one static token and per-frame dynamic tokens: $$\tau_\phi(v) \mapsto \bigl(s,\{d_1,\dots,d_N\}\bigr).$$

Each frame is independently processed by a convolutional image encoder: $f_i = E_{\text{img}}(x_i).$ The implementation described for this image encoder uses input frames of $128\times128$ RGB, 3 resolution levels, channel multipliers $(1,2,4)$, 2 residual blocks per level, base channels $\{d_i\}_{i=1}^N$0, and an output projected by a $\{d_i\}_{i=1}^N$1 convolution.

The static token is extracted from the first frame: $\{d_i\}_{i=1}^N$2 The MLP is specified as 2 layers, with hidden dimension 1024, ReLU, and final output dimension $\{d_i\}_{i=1}^N$3. The model follows a “first-frame static” intuition: the first frame often contains sufficient identity or appearance information for the sequence.

The dynamic pathway explicitly removes static content before temporal modeling. Rather than constructing motion codes from $\{d_i\}_{i=1}^N$4 directly, the encoder forms residuals

$\{d_i\}_{i=1}^N$5

These residuals are then processed by a bidirectional LSTM, followed by multi-head self-attention with 8 heads and embedding dimension 256, plus residual connection, layer normalization, and a final projection: $\{d_i\}_{i=1}^N$6

This subtraction mechanism is one of the method’s central architectural biases. The paper’s interpretation is that static appearance is shared across frames and is therefore largely canceled by $\{d_i\}_{i=1}^N$7, while pose, expression, and movement remain in the residual. This is meant to discourage the dynamic code from storing identity or appearance information.

3. Conditional DDPM decoder and video-specific inductive biases

The decoder is a conditional DDPM that reconstructs each frame from noisy inputs conditioned on the static token $\{d_i\}_{i=1}^N$8 and the frame-specific dynamic token $\{d_i\}_{i=1}^N$9: $v = \{x_1,\dots,x_N\},$0

For the forward process, the method uses the standard DDPM corruption form: $v = \{x_1,\dots,x_N\},$1 A defining DiViD choice is that the same noise realization $v = \{x_1,\dots,x_N\},$2 is shared across all frames. The paper argues that if each frame receives unrelated noise, denoising trajectories may become frame-inconsistent, whereas shared noise provides a common stochastic scaffold across the video and should improve temporal consistency (Gheisari et al., 18 Jul 2025).

The denoiser is a UNet with 4 resolution levels, channel multipliers $v = \{x_1,\dots,x_N\},$3, base channel width $v = \{x_1,\dots,x_N\},$4, 2 residual blocks per level, self-attention at spatial resolutions 32, 16, 8, FiLM-like scale-shift normalization, learned up/downsampling, and spatial transformer blocks for cross-attention conditioning. Time embeddings are encoded by a 2-layer MLP and injected into residual blocks.

Conditioning is injected by cross-attention. The operational routing is:

• the global static token $v = \{x_1,\dots,x_N\},$5 conditions every frame,
• the dynamic token $v = \{x_1,\dots,x_N\},$6 conditions only frame $v = \{x_1,\dots,x_N\},$7.

A concise formulation is

$v = \{x_1,\dots,x_N\},$8

with the denoiser predicting

$v = \{x_1,\dots,x_N\},$9

This routing is intended to broadcast sequence-wide appearance while keeping motion local and frame-specific.

The paper also characterizes DiViD as using a time-varying KL-based bottleneck inspired by EncDiff. Conceptually, the bottleneck is described as tight at early timesteps / high noise and relaxed at later timesteps / lower noise. The stated interpretation is that early denoising encourages the static token to carry only robust, global, time-invariant appearance information, while later denoising allows richer use of dynamic tokens. However, the method section does not provide a standalone DiViD-specific KL formula beyond this conceptual description.

To further suppress leakage, DiViD adds an orthogonality regularizer: $s$0 This penalizes overlap between the static token and each dynamic token.

4. Optimization, training objective, and evaluation protocol

DiViD is trained end-to-end in one stage. Its explicit denoising objective is the simplified DDPM loss

$s$1

This is a noise-prediction objective, but notably the paper uses $s$2 rather than the more common $s$3 form. The final training objective is

$s$4

The practical pipeline is: encode the video into $s$5 and $s$6, sample a diffusion timestep and Gaussian noise, corrupt each frame using the shared-noise DDPM forward process, predict the noise with the conditional UNet, and jointly update encoder and decoder.

The method is evaluated on MHAD / UTD-MHAD and MEAD. MHAD contains 861 video sequences, has resolution $s$7, 8 subjects, 27 actions, and each action is repeated 4 times; sequences are standardized by random sampling of 10-frame clips. MEAD contains 30 subjects and 8 facial expressions—anger, fear, disgust, happiness, sadness, surprise, contempt, neutral—with variable-length videos standardized by random sampling of 15 frames; faces are detected by Haar Cascades and cropped to $s$8.

The paper compares DiViD against SPYL and DBSE, both sequential disentanglement methods built on VAE-style formulations. For fairness, the authors report replacing the original encoders and decoders of the baselines with the same image encoder and decoder backbone used in DiViD, and they also add a VGG perceptual loss to baseline reconstruction losses. The reported baseline loss weights are reconstruction loss: 10, static KL: 5, and dynamic KL: 1.

Evaluation uses two families of metrics. The first is swap-based evaluation. Given two clips, the model swaps static and dynamic factors to generate outputs such as

$s$9

A pretrained classifier then measures Static-only accuracy, Dynamic-only accuracy, and Joint accuracy. The second is cross-leakage classification, with Static $d_i$0 Dynamic and Dynamic $d_i$1 Static probes; the reported average leakage is

$d_i$2

Lower leakage is better.

5. Quantitative results, qualitative behavior, and limitations

On MHAD, the key reported results are as follows (Gheisari et al., 18 Jul 2025).

Model	Static Only	Dynamic Only	Joint Acc.	Information Leakage
DBSE	99.0	13.7	13.7	83.5
SPYL	46.1	44.1	17.6	98.5
DiViD	94.1	29.4	28.4	70.9

These numbers establish the paper’s central empirical claim. DBSE preserves identity almost perfectly but transfers motion poorly. SPYL achieves a relatively higher dynamic-only score but low static-only accuracy, indicating that it often transfers motion while losing identity. DiViD achieves the highest joint accuracy, at 28.4, and the lowest average leakage, at 70.9. The reported interpretation is that DiViD provides a better balance of static fidelity and dynamic transfer than the VAE-based baselines.

Qualitative findings on MHAD and MEAD are aligned with the swap metrics. On MHAD, DiViD is described as preserving source identity while transferring target action; on MEAD, it preserves face identity and transfers expression well. A notable qualitative detail is that DiViD often omits irrelevant static scene elements such as a chair from the source sequence, which the paper interprets as evidence that it focuses more on actual dynamics rather than entangled background content. By contrast, DBSE is described as under-transferring motion, while SPYL may mix source and target characteristics or even produce a “swapped” output that is essentially a copy of the target sequence.

At the same time, the paper leaves several issues unresolved. It explicitly states that systematic ablations are future work, so there is no dedicated ablation table isolating the impact of the shared-noise schedule, the time-varying KL bottleneck, the cross-attention routing, or the orthogonality regularizer. The evaluation is limited to MHAD and MEAD. Some reported metrics remain modest in absolute terms—most notably 29.4 for dynamic-only accuracy and 28.4 for joint accuracy on MHAD—which indicates that static–dynamic disentanglement in real video remains challenging. The KL bottleneck, while prominent in the conceptual framing, is not accompanied by an explicit standalone optimization term in the method section.

6. Distinction from similarly named methods

DiViD should not be conflated with several near-homographic or acronymically similar methods in adjacent areas.

DIVD refers to “Deblurring with Improved Video Diffusion Model,” a conditional diffusion model for video deblurring that introduces Window-based Temporal Self-Attention (WTSA) and Multi-frame Relative Positional Encoding (MRPE) (Long et al., 2024). Despite the visual similarity of the names, its task is restoration rather than disentanglement.

VIDI is “Video Dataset of Incidents,” a multilingual benchmark for incident and disaster video classification with 4,534 video clips and 43 incident categories (Sesver et al., 2022). It is a dataset rather than a generative model.

VEViD denotes “Vision Enhancement via Virtual diffraction and coherent Detection,” an optics-inspired algorithm for low-light enhancement and color enhancement based on virtual diffraction and phase readout (MacPhee et al., 2022). Its domain is image and video enhancement, not latent factorization.

DiVE is “Difference Vector Equalization,” a method for robust fine-tuning of vision-LLMs that regularizes embedding shifts during CLIP-like adaptation (Suzuki et al., 13 Nov 2025). It is not a video diffusion model.

ViDiDi stands for “Video Time-Differentiation for Instance Discrimination,” a self-supervised video representation learning framework based on temporal derivatives and instance discrimination (Chen et al., 2024). A plausible source of confusion is acronym similarity, but the method is explicitly named ViDiDi, not DiViD.

Within this naming landscape, DiViD specifically denotes the 2025 method for unsupervised static–dynamic factorization via a conditional video DDPM, with a residual encoder, structured conditioning, shared temporal noise, and orthogonality-based leakage reduction (Gheisari et al., 18 Jul 2025).