DiffVC-OSD: One-Step Neural Video Compression

• The paper introduces a one-step diffusion model that refines video latents while preserving structural information, drastically reducing inference time compared to multi-step methods.
• It integrates a Temporal Context Adapter that fuses multi-scale temporal features into a pre-trained UNet, significantly enhancing perceptual quality and compression efficiency.
• End-to-end optimization with LoRA adaptation and a combined rate–distortion–perception loss sets a new standard for efficient and high-quality neural video compression.

DiffVC-OSD is a One-Step Diffusion-based Perceptual Neural Video Compression framework that advances the state of neural video compression (NVC) by performing perceptual enhancement of video latents via a single-step diffusion process, rather than the conventional multi-step approach. By feeding the reconstructed latent representation directly into a diffusion model conditioned on both the latent and multi-scale temporal context, DiffVC-OSD achieves substantial gains in perceptual quality, compression efficiency, and computational speed compared to prior diffusion-based methods (Ma et al., 11 Aug 2025).

1. Conceptual Framework and Innovations

Traditional perceptual NVC relies on generative models such as GANs or diffusion models to enhance the realism of compressed videos, balancing the standard rate–distortion trade-off. Multi-step diffusion-based approaches first corrupt the reconstructed latent with significant Gaussian noise, followed by numerous denoising steps (20–50 per frame), a process that—while effective for perceptual fidelity—discards structural latent information, is computationally expensive at inference, and renders end-to-end finetuning impractical due to backpropagation constraints.

DiffVC-OSD replaces this multi-step denoising with a one-step diffusion pass, taking as input the noise-free reconstructed latent, and conditions the process with both that latent and a temporal context tensor from the previous frame. This one-step architecture retains maximal structural information, leverages a high-capacity pre-trained UNet in a refinement (rather than denoising) role, permits full end-to-end optimization of the NVC system, and accelerates inference by ≈20× over multi-step baselines.

The pipeline consists of:

• Motion estimation and coding, yielding reconstructed motion $\hat v_t$.
• Latent encoding and reconstruction, outputting $\bar y_t$.
• One-step diffusion: $\bar y_t \xrightarrow{\text{UNet+TCA}} \hat y_t$.
• Latent decoding and synthesis of the final frame $\hat x_t$.

2. One-Step Diffusion and Mathematical Formalism

DiffVC-OSD’s one-step diffusion process builds on DDPM machinery, but diverges by applying a single denoising UNet pass directly to the noise-free latent. The system forms a conditioning vector by concatenating the reconstructed latent $\bar y_t$ and a large-scale temporal context tensor $C_t^0$ (from the prior frame): $c_t = \mathrm{Concat}(\bar y_t, C_t^0).$ The UNet model processes $(\bar y_t, c_t, n)$, where $n$ is a selected timestep in a fixed noise schedule (hyperparameter), yielding a predicted noise $\epsilon_\theta$. The enhanced latent is then obtained as: $\bar y_t$0 with $\bar y_t$1, $\bar y_t$2 denoting forward-diffusion coefficients. No additional noise is injected; the denoising is limited to one UNet evaluation. During initial adaptation, a standard $\bar y_t$3 denoising loss is employed.

3. Temporal Context Adapter (TCA)

To enhance the temporal modeling in the one-step paradigm, DiffVC-OSD introduces the Temporal Context Adapter (TCA), a sub-module integrating temporal context at multiple feature scales. TCA takes as input the concatenated tensor $\bar y_t$4, where $\bar y_t$5 and $\bar y_t$6 are channel counts for the latent and context tensors.

TCA architecture:

• Input convolution (3×3): Maps $\bar y_t$7 inputs to $\bar y_t$8 outputs. Filters are initialized following a “ControlNet” scheme: the first $\bar y_t$9 initialized with UNet’s initial conv weights, the rest zeroed.
• Stack of 4 ResBlocks (with SiLU activation and GroupNorm): Each block preserves dimension.
• Zero-initialized 3×3 conv heads after each ResBlock: Produce multi-scale features $\bar y_t \xrightarrow{\text{UNet+TCA}} \hat y_t$0, each mapped to a UNet stage. At inference, these features are point-wise added to their respective UNet feature maps, enabling the denoising model to exploit temporal continuity.

This design allows gradual, stable incorporation of temporal conditioning, mitigating risk of destabilizing the pre-trained denoising model.

4. Denoising UNet Design and Fusion Mechanism

DiffVC-OSD employs the Stable Diffusion V2.1-base UNet as its backbone. The UNet accepts the input latent $\bar y_t \xrightarrow{\text{UNet+TCA}} \hat y_t$1 of shape $\bar y_t \xrightarrow{\text{UNet+TCA}} \hat y_t$2, with a learnable sinusoidal embedding for the timestep $\bar y_t \xrightarrow{\text{UNet+TCA}} \hat y_t$3. The model follows a standard U-shaped encoder–decoder configuration, containing four down/up blocks and a central block, with channel widths set as $\bar y_t \xrightarrow{\text{UNet+TCA}} \hat y_t$4.

Fusion with TCA: At each of the four main scales $\bar y_t \xrightarrow{\text{UNet+TCA}} \hat y_t$5, the corresponding TCA output $\bar y_t \xrightarrow{\text{UNet+TCA}} \hat y_t$6 is point-wise added to the UNet feature map $\bar y_t \xrightarrow{\text{UNet+TCA}} \hat y_t$7. Thus,

$\bar y_t \xrightarrow{\text{UNet+TCA}} \hat y_t$8

LoRA Adaptation: During finetuning, only LoRA (Low-Rank Adaptation) matrices ($\bar y_t \xrightarrow{\text{UNet+TCA}} \hat y_t$9) within cross-attention or MLP projections are trainable; all other UNet weights are frozen. This low-rank strategy provides efficient, memory-light adaptation paths without jeopardizing overall model stability.

5. Training Procedure and Optimization

A nine-stage training schedule is implemented:

• Stages 1–7: Follow the DiffVC pre-training regime (motion codecs, base codec, context modules).
• Stage 8 (TCA+LoRA Finetuning): All components except TCA and LoRA adapters are frozen. The training loss combines MSE, LPIPS, and DISTS, with weights $\hat x_t$0. AdamW is used (batch size 30), training 7 epochs with staged learning rate decay from $\hat x_t$1 to $\hat x_t$2.
• Stage 9 (Full End-to-End Finetuning): All parameters are unfrozen for optimization under a joint rate–distortion–perception objective: $\hat x_t$3 where $\hat x_t$4 denotes bitrates; $\hat x_t$5 tunes perceptual trade-offs; $\hat x_t$6; $\hat x_t$7 cycles every 4 P-frames $\hat x_t$8 (batch 10, 1 epoch, LR $\hat x_t$9).

The one-step design facilitates full-gradient backpropagation through the denoising UNet, enabling global optimization, which was computationally prohibitive in multi-step setups.

6. Comparative Results and Component Analysis

Extensive evaluation demonstrates the efficacy of DiffVC-OSD on standard datasets and perception metrics, as summarized below.

Comparison	DiffVC-OSD Improvement	Metric
Decoding Speed (P-frame)	$\bar y_t$0 s vs. $\bar y_t$1 s	$\bar y_t$220× faster
Perceptual BD-rate (HEVC, LPIPS)	$\bar y_t$3	Best among anchor-based methods
FID Reduction (HEVC/MCL-JCV/UVG)	$\bar y_t$4, $\bar y_t$5, $\bar y_t$6	Lower is better
Avg. Perceptual BD-rate vs. Multi-step	$\bar y_t$7 bitrate reduction	Table 2 ablation (D vs. DiffVC-OSD)
TCA Ablation (BD-rate gain)	$\bar y_t$8 from $\bar y_t$9	$C_t^0$010.6% improvement at negligible cost

Component ablations indicate that both TCA insertion and the end-to-end strategy yield measurable BD-rate improvements. LoRA adaptation was most effective when applied exclusively to UNet, with $C_t^0$1 selected for optimal trade-off (see Figure 1 in the cited work).

7. Implementation and Experimental Protocols

Key implementation details:

• Base model: Stable Diffusion V2.1-base UNet, input $C_t^0$2 for $C_t^0$3 input.
• Motion and context modules: Transfer directly from DiffVC.
• TCA configuration: Four ResBlocks, $C_t^0$4 hidden channels.
• LoRA rank: $C_t^0$5.
• Training data: Vimeo-90k, random $C_t^0$6 crops ($C_t^0$7 latents).
• Evaluation: First 96 frames of HEVC, MCL-JCV, and UVG datasets; intra period $C_t^0$8, low-delay P.
• Metrics: LPIPS, DISTS, KID, FID; PSNR and MS-SSIM for distortion; BD-rate/BD-metric for comparison.
• Decoding time breakdown: UNet ($C_t^0$9 s), dequantization/motion compensation/decoder ($c_t = \mathrm{Concat}(\bar y_t, C_t^0).$0 s).

8. Summary and Significance

DiffVC-OSD demonstrates that single-step diffusion, jointly conditioned on raw latent and multi-scale temporal context via TCA, suffices to achieve state-of-the-art perceptual neural video compression. It achieves a 20× speed increase in inference and an $c_t = \mathrm{Concat}(\bar y_t, C_t^0).$1 bitrate reduction relative to multi-step diffusion NVC, while supporting end-to-end optimization using a combined rate–distortion–perception loss. This approach establishes a new paradigm for integrating fast, perceptually-guided generative refinement within conditional NVC systems (Ma et al., 11 Aug 2025).