DiV-INR: Extreme Low-Bitrate Diffusion Video Compression with INR Conditioning

Abstract: We present a perceptually-driven video compression framework integrating implicit neural representations (INRs) and pre-trained video diffusion models to address the extremely low bitrate regime (<0.05 bpp). Our approach exploits the complementary strengths of INRs, which provide a compact video representation, and diffusion models, which offer rich generative priors learned from large-scale datasets. The INR-based conditioning replaces traditional intra-coded keyframes with bit-efficient neural representations trained to estimate latent features and guide the diffusion process. Our joint optimization of INR weights and parameter-efficient adapters for diffusion models allows the model to learn reliable conditioning signals while encoding video-specific information with minimal parameter overhead. Our experiments on UVG, MCL-JCV, and JVET Class-B benchmarks demonstrate substantial improvements in perceptual metrics (LPIPS, DISTS, and FID) at extremely low bitrates, including improvements on BD-LPIPS up to 0.214 and BD-FID up to 91.14 relative to HEVC, while also outperforming VVC and previous strong state-of-the-art neural and INR-only video codecs. Moreover, our analysis shows that INR-conditioned diffusion-based video compression first composes the scene layout and object identities before refining textural accuracy, exposing the semantic-to-visual hierarchy that enables perceptually faithful compression at extremely low bitrates.

• The paper introduces DiV-INR, which combines a quantized implicit neural representation with parameter-efficient NOLA adapters to provide dense temporal conditioning for a frozen video diffusion model without transmitting keyframes.
• DiV-INR consistently outperforms HEVC, VVC, DCVC-FM, Relic, and HiNeRV on perceptual metrics such as LPIPS, DISTS, and FID from 0.005–0.05 bpp, including BD-FID gains of up to 91.14 over HEVC.
• The method improves texture and temporal consistency at extreme compression rates but requires about 15 hours to encode each video, decodes at roughly 1 FPS, and sacrifices at least 1 dB of BD-PSNR compared with distortion-focused codecs.

Motivation and problem setting

At bitrates below approximately 0.05 bits per pixel, both conventional codecs (HEVC, VVC) and learned video codecs (DCVC-FM, HiNeRV) fail to preserve high-frequency detail, producing blur, blocking, and banding. Existing diffusion-based generative codecs mitigate this by conditioning a pre-trained video diffusion model on heavily compressed keyframes or explicit optical flow, but such sparse, localized signals provide weak temporal guidance far from keyframes and fail under occlusion or newly appearing content. DiV-INR addresses this regime by replacing keyframe conditioning entirely with an implicit neural representation (INR) that encodes the full sequence compactly and is optimized jointly with parameter-efficient adapters of the diffusion backbone.

Method

The framework consists of two transmitted components per group of pictures (GoP): the quantized weights of an INR conditioning network ($\theta_{\text{INR}}$) and the coefficients of NOLA adapters ($\theta_{\text{PEFT}}$) injected into a frozen Wan2.1 1.3B-parameter diffusion transformer operating in the latent space of a 3D causal VAE with $4\times8\times8$ downsampling.

INR-based adaptive conditioning. The INR follows the HiNeRV architecture (learned feature grid plus convolutional decoder) but maps normalized temporal coordinates to latent-space conditioning rather than RGB frames. For each latent frame it outputs a predicted conditioning signal $y_f \in \mathbb{R}^{C\times H'\times W'}$ together with an adaptive 4-channel mask $M_f \in [0,1]^{C_m\times H'\times W'}$ expressing confidence in the prediction. The concatenation of $y$ and $M$ conditions the denoising process channel-wise, providing dense temporal guidance across the entire GoP without any intra-coded I-frames.

Parameter-efficient adaptation. Rather than LoRA, the authors use NOLA adapters, which reparameterize weight updates as linear combinations of frozen pseudo-random basis matrices; only scalar mixing coefficients are trained. With rank 64 across 30 DiT blocks (self-attention output projections, feed-forward layers, final output head) and 500 bases, this amounts to only 91K trainable parameters — roughly 25× fewer than rank-16 LoRA per layer — and decouples bitrate from the backbone's hidden dimensions. Adapters are merged at inference so decoding cost is unaffected.

Compression pipeline. The INR decoder undergoes adaptive magnitude pruning (15% of parameters, scored as $|\theta_i|/\sqrt{P}$ to protect narrow layers), followed by Quant-Noise-based quantization-aware training with $\rho=0.9$, and 6-bit uniform quantization applied to both INR weights and NOLA coefficients at inference.

Training objective. A dual loss combines a flow-matching loss $\mathcal{L}_{\text{flow}}$ on the diffusion model with a latent reconstruction loss $\theta_{\text{PEFT}}$0 supervising the INR, weighted by a cosine-annealed $\theta_{\text{PEFT}}$1 so the INR converges first before flow matching dominates. A three-stage curriculum (dense training for 300 epochs, pruning-aware fine-tuning for 120, quantization-aware fine-tuning for 60) runs per GoP on a single RTX 4090 in about 15 hours, with the Wan2.1 backbone frozen throughout.

Results

Evaluations on UVG, JVET Class-B, and MCL-JCV (all resized to $\theta_{\text{PEFT}}$2) compare against HEVC (x265 veryslow), VVC (VTM-23.11 low-delay P), DCVC-FM, Relic et al.'s flow-conditioned diffusion codec, and HiNeRV. BD-metric deltas relative to DiV-INR show consistent perceptual advantages:

Codec	UVG BD-LPIPS	JVET-B BD-LPIPS	MCL-JCV BD-LPIPS
HEVC	+0.195	+0.214	+0.069
VVC	+0.131	+0.160	+0.052
DCVC-FM	+0.156	+0.175	+0.049
Relic et al.	+0.064	+0.055	+0.013
HiNeRV	+0.108	+0.165	+0.082

Positive values indicate the baseline's perceptual disadvantage. Corresponding BD-FID gains reach 91.14 over HEVC on JVET-B and 47.95 over VVC on UVG. Notably, DiV-INR outperforms Relic et al. even though the latter uses the same class of generative prior; swapping Wan2.1 into the Relic et al. pipeline actually degrades its BD-PSNR by 0.89 dB relative to their SVD-based result, which the authors cite as evidence that the INR-plus-PEFT integration, not the backbone choice, drives the gains.

As expected from the rate-distortion-perception trade-off, DiV-INR trails distortion-oriented codecs by at least 1 dB BD-PSNR while dominating all baselines on LPIPS, DISTS, and FID across the 0.005–0.05 bpp range. Qualitatively, the method preserves texture sharpness and temporal consistency on sequences with high-frequency motion (ShakeNDry, YachtRide, RitualDance) where baselines exhibit blur or blocking.

Ablations

Three ablations isolate the contributions of the design. INR vs. keyframe conditioning: replacing the INR with JPEG-compressed or generatively coded keyframes yields strictly worse rate-distortion behavior, confirming that continuous temporal conditioning is superior to sparse keyframes at these rates. PEFT size: removing adapters entirely costs up to 0.239 LPIPS, establishing that instance-specific adaptation is necessary; returns diminish beyond 500 bases (only ~0.02 LPIPS improvement from 500 to 1000), motivating the chosen budget. Adaptive masks: replacing learned masks with uniform confidence degrades BD-PSNR by 0.056 dB and BD-LPIPS by 0.003 on UVG, indicating the learned uncertainty weighting matters most in complex-motion regions.

Hierarchical convergence

A notable empirical observation is a semantic-first convergence pattern during joint optimization: early checkpoints produce semantically coherent but visually divergent content (correct scene layout and object categories, plausible-but-different appearances) before textural fidelity emerges. The authors argue this explains why the diffusion prior tolerates imperfect conditioning and suggest semantic-consistency-based early stopping as a route to cheaper encoding, though they do not implement or evaluate such a criterion.

Limitations and open questions

The paper concedes several constraints. Encoding requires roughly 15 hours per video on consumer hardware, restricting applicability to offline archival use; decoding runs at ~1 FPS with 20 UniPC steps, far from real time, and the claimed path to practical frame rates via distillation or efficient attention is asserted rather than demonstrated. All evaluations are at $\theta_{\text{PEFT}}$3 to match the backbone's pre-training distribution and 24 GB VRAM limits, leaving scaling behavior at higher resolutions untested. The PSNR deficit inherent to the perception-distortion trade-off means the method is unsuitable when pixel-exact fidelity is required. Finally, the proposed semantic-aware stopping strategy remains an open question, as does generalization beyond the three benchmarks and the single evaluated backbone family.

Conclusion

DiV-INR demonstrates that a compact INR can serve simultaneously as the video bitstream and as an optimized conditioning signal for a pre-trained video diffusion model, with instance-specific NOLA adaptation adding negligible bitrate overhead. The approach achieves state-of-the-art perceptual quality below 0.05 bpp, outperforming traditional, learned, and prior diffusion-based codecs on LPIPS, DISTS, and FID, while the observed semantic-to-textural convergence pattern offers a mechanistic account of why generative decoding remains robust under extreme compression.