SIEDD: Sparse Diffusion for Video Compression

• SIEDD is a sparsity-enhanced, diffusion-based video compression framework that selectively encodes backbone frames and motion cues for ultra-low bitrate regimes.
• It employs a two-stage pipeline with the Sparse Temporal Encoding Module (STEM) for selective frame encoding and a one-step diffusion transformer for global refinement, ensuring perceptual fidelity and temporal coherence.
• Empirical evaluations show that SIEDD outperforms traditional codecs and prior generative models with improved LPIPS, temporal metrics, and BD-rate savings at sub-0.04 bpp.

SIEDD for Video Compression

Sparse Information–Enhanced Diffusion for Dynamic video (SIEDD) refers to a high-efficiency, generative video compression methodology for ultra-low bitrate regimes. It operates by aggressively reducing transmitted information to the minimal set needed to reconstruct video sequences with high perceptual fidelity and temporal coherence, utilizing a two-stage pipeline: an initial sparse temporal encoding module (STEM) for information selection and transmission, followed by a one-step, type-conditioned diffusion-transformer network for global sequence refinement. The term SIEDD has also appeared in earlier contexts as "Soft-guided Intra-Edge Decomposition and Delivery," signifying a different pipeline based on soft edge detection and adversarial inpainting. This article focuses on the diffusion-driven SIEDD scheme that advances the state-of-the-art in ultra-low bitrate video compression (Zhou et al., 19 Mar 2026).

1. Theoretical Foundations and Motivations

SIEDD emerges from the observation that at sub-0.04 bpp rates, traditional codecs and standard end-to-end learned compression architectures exhibit two dominant failure modes: loss of perceptual detail and incoherent temporal artifacts. Classic codecs allocate the majority of available bits to motion compensation and residual coding. Generative methods, especially those using per-frame autoencoding or adversarial approaches, often fail to maintain temporal synchronization and introduce hallucinated artifacts due to limited temporal modeling. SIEDD sidesteps these limitations by (a) encoding only information-rich "backbone" frames and sparse motion cues, and (b) using a global, sequence-level diffusion model conditioned on sparse frame types, ensuring both cross-frame detail coherence and visual plausibility at very low rates (Zhou et al., 19 Mar 2026).

2. Sparse Temporal Encoding Module (STEM)

STEM forms the first stage and is responsible for identifying and encoding only the backbone information required to initialize sequence-wide reconstruction. Given an input sequence $x = \{x_1, \dots, x_T\}$, STEM selects backbone frames on an evenly spaced grid, such as $X_B = \{x_2, x_5, \ldots\}$. These are divided into two categories:

• I-frame: The initial backbone frame ($x_2$), compressed using a pretrained HiFiC encoder.
• P-frames: Subsequent backbone frames, each encoded conditionally using a latent-space entropy codec with learned context model $f_{con}$. The resulting posterior emits a bitstream at rate $R_B$.

The non-backbone frames, termed MV-frames, are reconstructed at the decoder via motion propagation. For every decoded backbone $\tilde{x}_{3t+2}$, optical flows $F_{3t+2 \to 3t\pm1}$ are estimated using SpyNet and quantized, incurring a minimal $R_{\mathrm{MV}}$. Warping these flows yields the intermediate reconstructions $\tilde{x}_{3t\pm1}$. Finally, the union of backbone and motion-propagated intermediates is reassembled as the intermediate decoded sequence $\tilde{x} = \{\tilde{x}_1, ..., \tilde{x}_T\}$ (Zhou et al., 19 Mar 2026).

3. One-Step Diffusion with Frame Type Embedder (ODFTE)

The ODFTE stage employs a single-pass diffusion-transformer for sequence-wide denoising and visual quality calibration. Its technical components are as follows:

• Frame Type Embedder (FTE): For every frame, its compression type (I, P, MV) is converted to a one-hot channel. These are stacked temporally; a 3D convolution ($X_B = \{x_2, x_5, \ldots\}$0) aggregates this information into a global embedding vector $X_B = \{x_2, x_5, \ldots\}$1.
• One-Step Diffusion Process: The intermediate sequence $X_B = \{x_2, x_5, \ldots\}$2 is encoded into latent tokens $X_B = \{x_2, x_5, \ldots\}$3 at a fixed diffusion timestep $X_B = \{x_2, x_5, \ldots\}$4 via a DiT/Wan encoder. The type embedding $X_B = \{x_2, x_5, \ldots\}$5 is incorporated as an additive bias, yielding $X_B = \{x_2, x_5, \ldots\}$6. A single analytic reverse step computes the cleaned token:

$X_B = \{x_2, x_5, \ldots\}$7

with $X_B = \{x_2, x_5, \ldots\}$8 denoting the cumulative noise schedule. The denoised latent is then mapped back to pixel space using the Wan decoder.

• Temporal Modeling: The DiT/Wan backbone, being a 3D transformer, applies self-attention jointly over spatial and temporal axes, allowing the ODFTE to propagate high-frequency details from high-bit backbone frames to underdetermined MV frames (Zhou et al., 19 Mar 2026).

4. Loss Functions and Training Paradigm

SIEDD employs distinct objectives across its two pipeline stages:

• Stage 1 (STEM) Loss:

$X_B = \{x_2, x_5, \ldots\}$9

Here $x_2$0 are quantized backbone latents, $x_2$1 the quantized motion bits.

• Stage 2 (ODFTE) Loss:

$x_2$2

where $x_2$3 measures mean squared frame-difference error for temporal consistency:

$x_2$4

The diffusion denoiser is pretrained with a standard noise-prediction loss over timesteps.

• Frame Type Conditioning: The token bias from the FTE embedding is broadcast to all transformer tokens, allowing the denoiser to adaptively reconstruct each frame based on its coarse type assignment (Zhou et al., 19 Mar 2026).

5. Quantitative Performance and Empirical Evaluation

Benchmarked on HEVC Class B, MCL-JCV, and UVG sequences, SIEDD attains the following at ultra-low rates (approximately 0.03 bpp):

• LPIPS = 0.1689, outperforming DCVC-DC’s 0.2319 by ~27%
• Ewarp, a temporal consistency metric, improves by ~15% relative to the strongest GAN-based baseline
• BD-rate savings in LPIPS of approximately 23.6% compared to direct one-step diffusion without sparse encoding

In comparisons with HM/VTM (traditional codecs) and learned systems such as DCVC-DC, DCVC-FM, DVC-P, and HNeRV, SIEDD consistently demonstrates state-of-the-art perceptual and temporal metrics, though expectedly yields lower PSNR typical of generative models in extreme low-bitrate contexts (Zhou et al., 19 Mar 2026).

6. Applications, Flexibility, and Open Problems

SIEDD supports dynamic adaptation to various video delivery requirements:

• GOP Adaptation: The backbone-to-MV frame ratio can be tuned, enabling seamless reconfiguration for different GOP sizes, on-demand streaming rates, or random-access capabilities.
• Integration of Generative Priors: The clean pipeline division (STEM + ODFTE) provides for modular integration of future generative priors, such as more efficient video diffusion backbones or GANs.
• Limitations: Decoding remains bottlenecked by the DiT/Wan, accounting for roughly 52% of inference time per frame. The system is reliant on image-diffusion pretraining, with end-to-end video diffusion still underexplored. Degradation under extreme scene shifts or motion patterns indicates prospective value in learned motion priors or event-based approaches.

A plausible implication is that SIEDD’s design suggests both immediate applicability for ultra-low-bitrate streaming and research avenues in computational efficiency and robustness to complex temporal dynamics (Zhou et al., 19 Mar 2026).

7. Historical Context and Relationship to Alternative SIEDD Approaches

The SIEDD acronym has denoted different conceptions in video compression literature. In one earlier context, "Soft-guided Intra-Edge Decomposition and Delivery" describes a two-stage codec based on conditional GANs guided by low-rate soft edge maps extracted after spatial downsampling and quantization (Kim et al., 2018). Here, a small set of H.264-coded key frames serves as the generative training signal, and the majority of frames are synthesized from compressed edge maps, allowing plausible reconstructions at bitrates well below H.264 failure thresholds.

While both approaches embody the principle of sparse guidance (key frames and map transmission in (Kim et al., 2018), backbone+motion in (Zhou et al., 19 Mar 2026)), diffusional SIEDD introduces a principled, data-driven diffusion prior to ensure perceptual and temporal fidelity. These foundational differences are summarized in the table below.

Approach	Key Transmission	Main Generative Prior
SIEDD (Diffusion)	Backbone frames, motion fields	One-step diffusion DiT/Wan
SIEDD (Edge-Guided)	Keyframes, soft edge maps	U-Net conditional GAN

The convergence of these ideas reflects the broader trend toward selective, information-efficient guiding signals in learned, generative compression architectures. The contemporary SIEDD framework sets a new benchmark for sub-0.04 bpp compression, substantiating the role of sparse information enhancement in next-generation video codecs (Zhou et al., 19 Mar 2026).