Diffusion Sequential Disentanglement Autoencoder

• The paper DiffSDA proposes a diffusion-based framework that factorizes sequences into time-invariant static and time-varying dynamic latents using a single L2 regression loss.
• Its architecture employs modality-specific backbones and LSTMs to separately encode global identity and dynamic content, enabling non-autoregressive parallel processing.
• Evaluation across video, audio, and time series shows significant improvements in metrics like AED, AKD, MSE, and FVD compared to VAE/GAN-based approaches.

Diffusion Sequential Disentanglement Autoencoder (DiffSDA) is an unsupervised, modal-agnostic framework for sequential disentanglement that factorizes a sequence into a time-invariant static latent and time-varying dynamic latents, and uses a conditional diffusion decoder to reconstruct or generate observations across video, audio, and time-series modalities. It was introduced to address two coupled gaps: the reliance of earlier sequential disentanglement methods on VAE- or GAN-based objectives with many loss terms, and the absence of a probabilistic diffusion-based formalization for separating static and dynamic factors in real-world sequential data (Zisling et al., 7 Oct 2025).

1. Formal definition and probabilistic structure

DiffSDA studies unsupervised sequential disentanglement on sequences

$$x^{1:V}=\{x^1,\dots,x^V\},\qquad x^\tau\in\mathbb{R}^d,$$

with the goal of learning a representation

$$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$

where $s$ is a static factor shared across the sequence and $d^\tau$ are dynamic factors that vary with time. In the formulation used by DiffSDA, $s$ is intended to encode time-invariant information such as identity or global characteristics, whereas $d^\tau$ encodes time-varying content such as pose, facial expression, phonetic content, or short-term signal variation (Zisling et al., 7 Oct 2025).

The generative model introduces both disentangled latents and diffusion trajectories. For a sequence $x_0^{1:V}$, its joint prior is written as

$$p(x_0^{1:V},x_T^{1:V},s_0,s_T,d_0^{1:V},d_T^{1:V}) = p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V}) \prod_{\tau=1}^{V} p_{T0}(x_0^\tau\mid x_T^\tau,s_0,d_0^\tau).$$

Here, $p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V})$ is a latent diffusion prior over the concatenated latent $z_0^{1:V}=(s_0,d_0^{1:V})$, and each $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$0 is a conditional reverse diffusion process from noisy observation $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$1 to clean observation $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$2 conditioned on the disentangled factors (Zisling et al., 7 Oct 2025).

Its posterior factorization is

$$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$3

which makes the asymmetry between static and dynamic inference explicit: the static code depends on the whole sequence, while each dynamic code depends on previous dynamics and past or current observations. A central design choice is that the prior over static and dynamic latents is dependent rather than factorized; the reported effect is an approximately $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$4 FVD improvement on VoxCeleb relative to independent priors, and the paper attributes this to higher expressivity, non-autoregressive parallel sampling of all dynamics, and the ability to capture causal relationships between static and dynamic latent trajectories (Zisling et al., 7 Oct 2025).

2. Architecture and latent parameterization

DiffSDA is built from four components: a sequential semantic encoder, a stochastic encoder implementing forward diffusion, a stochastic decoder implementing conditional reverse diffusion, and a separate latent DDIM prior over disentangled latents (Zisling et al., 7 Oct 2025).

The sequential semantic encoder maps the clean sequence to $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$5 and $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$6. Each observation $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$7 is first processed by a modality-specific backbone: for video, a U-Net operating on VQ-VAE latents; for audio and time series, an MLP-based network. The resulting per-step outputs are passed through linear layers and an LSTM to obtain hidden states $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$8. The last hidden state is projected to the static code,

$$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$9

while the full hidden-state sequence is processed by another LSTM and linear layer to obtain the dynamic trajectory,

$s$0

This architecture directly matches the posterior factorization: static information is pooled globally, dynamic information is produced sequentially (Zisling et al., 7 Oct 2025).

The stochastic encoder follows EDM-style perturbation. For each step $s$1, noise is sampled as

$s$2

with $s$3. For high-dimensional video, DiffSDA uses latent diffusion: raw frames $s$4 are first mapped to latent variables $s$5 by a pre-trained VQ-VAE encoder, and reconstructions are obtained with the corresponding decoder $s$6 (Zisling et al., 7 Oct 2025).

The stochastic decoder denoises conditionally on the disentangled latents. For each $s$7,

$s$8

with EDM parameterization

$s$9

The conditioning is injected through AdaGN, so both $d^\tau$0 and $d^\tau$1 modulate intermediate feature maps inside the denoiser. The same factorization is used across modalities; only the backbone family differs (Zisling et al., 7 Oct 2025).

Finally, DiffSDA trains a separate latent DDIM prior over $d^\tau$2 with the standard noise-prediction objective

$d^\tau$3

At generation time, this prior produces new static and dynamic latent trajectories, which are then fed to the conditional decoder (Zisling et al., 7 Oct 2025).

3. Training objective, samplers, and the disentanglement mechanism

The theoretical objective is conditional score matching: $d^\tau$4 which in EDM form becomes the implemented regression loss

$d^\tau$5

The paper emphasizes that this is a single $d^\tau$6 regression loss per noisy sample. DiffSDA does not add explicit disentanglement regularizers such as mutual-information penalties or adversarial content-motion objectives; disentanglement is induced architecturally by sharing $d^\tau$7 across time and constraining each $d^\tau$8 to be low-dimensional (Zisling et al., 7 Oct 2025).

The reported mechanism is capacity asymmetry. Since the static code is shared across all timesteps, it is the natural carrier for what is common across the sequence. Since the dynamic code is low-dimensional, using it to encode static appearance is inefficient and empirically harmful. The decoder must reconstruct each $d^\tau$9 from the pair $s$0, so the structurally cheapest solution is to place time-invariant information in $s$1 and residual time-varying information in $s$2. The appendix ablations reported in the paper support this reading: when $s$3 is not shared, dynamics collapse and disentanglement degrades; when the dynamic dimension is too large, static information leaks into dynamics and verification accuracy together with AED and AKD metrics worsens (Zisling et al., 7 Oct 2025).

DiffSDA uses two EDM-style samplers. The ConditionedStochasticSampler performs reverse sampling conditioned on $s$4, with optional stochastic churn $s$5, Euler updates, and an optional second-order correction. The StochasticEncoder runs the reverse of that procedure to map clean observations to a noisy state consistent with the decoder. The paper states that high-quality samples are typically obtained with only 63 NFEs, or 71 in another setting, rather than hundreds of naive DDPM steps. The samplers are non-autoregressive across the sequence index $s$6, so all frame or timestep latents can be updated in parallel (Zisling et al., 7 Oct 2025).

4. Evaluation protocol across modalities

A major contribution of DiffSDA is its evaluation protocol for real-world sequential disentanglement, motivated by the claim that no standard protocol previously existed for high-resolution video and related modalities (Zisling et al., 7 Oct 2025).

Modality	Datasets	Principal metrics
Video	MUG, TaiChi-HD, VoxCeleb, CelebV-HQ	AED, AKD, MSE, FVD, Acc, IS, inter-entropy, intra-entropy
Audio	TIMIT, LibriSpeech	Static EER, Dynamic EER, Disentanglement Gap, spectrogram MSE, DNSMOS
Time series	PhysioNet ICU, Air Quality, ETTh1	AUPRC, AUROC, MAE, accuracy

For video, the protocol departs from judge-based classification metrics alone and introduces two unsupervised swap-based criteria. Average Euclidean Distance (AED) measures identity or object preservation by comparing embeddings of original and swapped sequences; lower AED under static preservation indicates better retention of identity. Average Keypoint Distance (AKD) measures motion preservation by comparing facial landmarks or body keypoints; lower AKD under dynamic preservation indicates better retention of motion. Reconstruction is evaluated with AED, AKD, and MSE, while generative quality is measured with FVD. On MUG, the paper also reports legacy judge-based metrics—Accuracy, Inception Score, inter-entropy, and intra-entropy—for comparability with earlier work, but explicitly argues that AED and AKD are more reliable for real-world, unseen variations (Zisling et al., 7 Oct 2025).

For audio, the main disentanglement protocol is speaker verification. A well-disentangled representation should yield low EER from static latents and high EER from dynamic latents, since dynamics are supposed not to encode speaker identity. The paper defines the Disentanglement Gap as

$s$7

Speech quality and reconstruction are further assessed by spectrogram MSE and DNSMOS (Zisling et al., 7 Oct 2025).

For time series, DiffSDA is evaluated not only as a generator but as a representation learner. On PhysioNet, static and dynamic latents together are used for mortality prediction, reported by AUPRC and AUROC. Static latents alone are used for ICU type classification. On Air Quality, static latents are used for month classification. On ETTh1, latents support forecasting of oil temperature, evaluated by MAE (Zisling et al., 7 Oct 2025).

5. Empirical findings, downstream utility, and controllable generation

Across modalities, DiffSDA is reported to outperform recent state-of-the-art sequential disentanglement baselines, primarily SPYL and DBSE, on both disentanglement and reconstruction quality (Zisling et al., 7 Oct 2025).

On video, DiffSDA achieves the best or near-best swap metrics on all four datasets. For MUG, conditional swaps yield AED $s$8 versus $s$9 for SPYL and $d^\tau$0 for DBSE, and AKD $d^\tau$1 versus $d^\tau$2 and $d^\tau$3. For VoxCeleb, AED is $d^\tau$4 versus $d^\tau$5 and $d^\tau$6, and AKD is $d^\tau$7 versus $d^\tau$8 and $d^\tau$9. For CelebV-HQ, AED is $x_0^{1:V}$0 versus $x_0^{1:V}$1 and $x_0^{1:V}$2, while AKD is $x_0^{1:V}$3 versus $x_0^{1:V}$4 and $x_0^{1:V}$5. For TaiChi-HD, AED is $x_0^{1:V}$6, compared with $x_0^{1:V}$7 for SPYL and $x_0^{1:V}$8 for DBSE, while AKD is $x_0^{1:V}$9 versus $$p(x_0^{1:V},x_T^{1:V},s_0,s_T,d_0^{1:V},d_T^{1:V}) = p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V}) \prod_{\tau=1}^{V} p_{T0}(x_0^\tau\mid x_T^\tau,s_0,d_0^\tau).$$0 and $$p(x_0^{1:V},x_T^{1:V},s_0,s_T,d_0^{1:V},d_T^{1:V}) = p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V}) \prod_{\tau=1}^{V} p_{T0}(x_0^\tau\mid x_T^\tau,s_0,d_0^\tau).$$1. Reconstruction errors are likewise substantially lower; examples reported include VoxCeleb MSE $$p(x_0^{1:V},x_T^{1:V},s_0,s_T,d_0^{1:V},d_T^{1:V}) = p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V}) \prod_{\tau=1}^{V} p_{T0}(x_0^\tau\mid x_T^\tau,s_0,d_0^\tau).$$2 versus $$p(x_0^{1:V},x_T^{1:V},s_0,s_T,d_0^{1:V},d_T^{1:V}) = p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V}) \prod_{\tau=1}^{V} p_{T0}(x_0^\tau\mid x_T^\tau,s_0,d_0^\tau).$$3 and $$p(x_0^{1:V},x_T^{1:V},s_0,s_T,d_0^{1:V},d_T^{1:V}) = p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V}) \prod_{\tau=1}^{V} p_{T0}(x_0^\tau\mid x_T^\tau,s_0,d_0^\tau).$$4, CelebV-HQ MSE $$p(x_0^{1:V},x_T^{1:V},s_0,s_T,d_0^{1:V},d_T^{1:V}) = p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V}) \prod_{\tau=1}^{V} p_{T0}(x_0^\tau\mid x_T^\tau,s_0,d_0^\tau).$$5 versus $$p(x_0^{1:V},x_T^{1:V},s_0,s_T,d_0^{1:V},d_T^{1:V}) = p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V}) \prod_{\tau=1}^{V} p_{T0}(x_0^\tau\mid x_T^\tau,s_0,d_0^\tau).$$6 and $$p(x_0^{1:V},x_T^{1:V},s_0,s_T,d_0^{1:V},d_T^{1:V}) = p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V}) \prod_{\tau=1}^{V} p_{T0}(x_0^\tau\mid x_T^\tau,s_0,d_0^\tau).$$7, and TaiChi-HD MSE $$p(x_0^{1:V},x_T^{1:V},s_0,s_T,d_0^{1:V},d_T^{1:V}) = p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V}) \prod_{\tau=1}^{V} p_{T0}(x_0^\tau\mid x_T^\tau,s_0,d_0^\tau).$$8 versus $$p(x_0^{1:V},x_T^{1:V},s_0,s_T,d_0^{1:V},d_T^{1:V}) = p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V}) \prod_{\tau=1}^{V} p_{T0}(x_0^\tau\mid x_T^\tau,s_0,d_0^\tau).$$9 and $p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V})$0. On VoxCeleb generation, FVD improves from $p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V})$1 for SPYL and $p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V})$2 for DBSE to $p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V})$3 for DiffSDA (Zisling et al., 7 Oct 2025).

On audio, the main pattern is a larger static-dynamic separation. On TIMIT, DiffSDA reports static EER $p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V})$4, dynamic EER $p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V})$5, and a gap of $p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V})$6, versus $p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V})$7 for DBSE and $p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V})$8 for SPYL. On LibriSpeech, it reports static EER $p_{T0}(s_0,d_0^{1:V}\mid s_T,d_T^{1:V})$9, dynamic EER $z_0^{1:V}=(s_0,d_0^{1:V})$0, and a gap of $z_0^{1:V}=(s_0,d_0^{1:V})$1, compared with $z_0^{1:V}=(s_0,d_0^{1:V})$2 for DBSE and $z_0^{1:V}=(s_0,d_0^{1:V})$3 for SPYL. Reconstruction quality on LibriSpeech improves to MSE $z_0^{1:V}=(s_0,d_0^{1:V})$4, compared with $z_0^{1:V}=(s_0,d_0^{1:V})$5 for DBSE and $z_0^{1:V}=(s_0,d_0^{1:V})$6 for SPYL, while DNSMOS rises to $z_0^{1:V}=(s_0,d_0^{1:V})$7, compared with $z_0^{1:V}=(s_0,d_0^{1:V})$8 and $z_0^{1:V}=(s_0,d_0^{1:V})$9, with the reference value noted as approximately $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$00 (Zisling et al., 7 Oct 2025).

On time series, DiffSDA also functions as a strong representation learner. On PhysioNet mortality prediction, it reports AUPRC $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$01 and AUROC $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$02, exceeding DBSE, GLR, and SPYL. On ETTh1 forecasting, it reports MAE $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$03, compared with $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$04 for DBSE and $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$05 for SPYL, and the paper notes that this even surpasses a supervised baseline of approximately $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$06. On static-only classification, PhysioNet ICU type accuracy reaches $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$07, compared with $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$08 for DBSE, $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$09 for SPYL, $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$10 for GLR, and $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$11 for the supervised baseline; Air Quality month classification reaches $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$12, compared with $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$13, $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$14, and a supervised baseline of $$z^{1:V}=(s,d^{1:V}),\qquad d^{1:V}=\{d^1,\dots,d^V\},$$15 (Zisling et al., 7 Oct 2025).

The paper also reports two broader capabilities. First, zero-shot disentanglement: a model trained on VoxCeleb can be qualitatively transferred to MUG or CelebV-HQ, with static and dynamic swaps continuing to preserve identity while transferring expressions or poses. Second, multifactor structure beyond the static-dynamic split: PCA on static latents reveals directions associated with male↔female, age, skin tone, and blur level, while PCA on dynamic latents reveals head pose, mouth or eye openness, and expression intensity. The paper treats these as evidence that the learned representation contains richer factorization than the nominal two-way split alone (Zisling et al., 7 Oct 2025).

6. Relation to earlier diffusion disentanglement research and open problems

DiffSDA belongs to a broader transition from VAE- and GAN-centered disentanglement to diffusion-based representation learning. Earlier work on score-based representation learning reformulated denoising score matching as a representation-learning objective and argued that diffusion behaves like a family of multi-scale denoising autoencoders, with controllable detail levels through the noise schedule (Mittal et al., 2021). Subsequent diffusion autoencoder work split latent spaces architecturally rather than through heavy explicit regularization: DDAE separated a known-variance latent from an unknown-variance latent for controllable harmonization of neuroimaging data (Ijishakin et al., 2024), and EncDiff showed that cross-attention over concept tokens can itself act as an inductive bias for disentanglement without additional regularization (Yang et al., 2024). Other methods focused on stronger inductive biases or training dynamics, such as Dynamic Gaussian Anchoring and Skip Dropout for attribute-separated latent units (Jun et al., 2024), and gated residual U-Nets with an SNR curriculum to steer optimization toward stronger representations (Rasal et al., 6 Jul 2026). A separate but relevant line is GDM, which assigns different groups of variables to different diffusion intervals and thereby makes latent groups structurally interpretable and hierarchically organized in the frequency domain (Lee et al., 2023). In a different application area, SIDDA factorized sequential histories with factor prototypes and Gumbel-Softmax assignments, illustrating another route to sequential latent-factor specialization (Wang et al., 2020).

Within that landscape, DiffSDA’s distinctive claim is not merely that diffusion can support disentanglement, but that sequential disentanglement can be given a diffusion-based probabilistic formulation that is explicitly static-dynamic, trained with a single score-matching loss, and evaluated across video, audio, and time series under one protocol (Zisling et al., 7 Oct 2025). A plausible implication is that DiffSDA occupies the intersection of diffusion autoencoders, latent dynamical models, and unsupervised sequential factorization, rather than being a direct extension of any one of those lines alone.

The paper also states several limitations. Diffusion remains computationally heavy, even with EDM-style 63-step samplers. Multifactor disentanglement beyond the static-dynamic split remains open despite promising PCA analyses. Although the probabilistic formulation is modal-agnostic, unusual domains may still require architecture-specific changes. Finally, evaluation remains only partially stabilized: the paper argues that AED and AKD improve on judge-based metrics, but also states that more comprehensive and standardized benchmarks are still needed, especially for sequential representations with more than two factors (Zisling et al., 7 Oct 2025).