Neural Video Compression with Domain Transfer

Abstract: Content-adaptive compression has always been a key direction in neural video coding (NVC), aiming to mitigate the domain gap between training and testing data. Such gaps often arise from distributional discrepancies between training and inference data, which may cause noticeable performance degradation when the testing content differs from the training distribution. To tackle this challenge, we propose DCVC-DT, a domain transfer enhanced neural video compression framework. Specifically, we design a lightweight online domain transfer (DT) mechanism that dynamically adapts the encoded latent representation during inference, effectively bridging the domain gap without modifying the encoder or decoder parameters. In addition, we develop a frame-level dynamic RD (Rate and Distortion) adjustment scheme that actively regulates the ratio of R and D in the loss function based on quality fluctuation, thereby improving rate-distortion performance. Extensive experiments demonstrate that DCVC-DT achieves up to 6.21% bitrate savings over the baseline DCVC-DC, while significantly enhancing generalization to unseen testing data and alleviating error propagation. Our code is available at https://github.com/SunnyMass/DCVC-DT.

• The paper presents DCVC-DT, which integrates online latent refinement with dynamic RD adjustment to bridge domain gaps in neural video compression.
• The method refines latent representations using gradient descent and stochastic Gumbel annealing, achieving up to 6.21% BD-rate reduction on HEVC datasets.
• Encoder-side optimization mitigates error propagation across frames while maintaining unchanged decoder complexity, offering a practical solution for diverse video domains.

Neural Video Compression with Domain Transfer: Authoritative Summary

Introduction and Motivation

Neural Video Compression (NVC) models have demonstrated superior performance to classical codecs by leveraging feature-based motion compensation, contextual coding, and advanced entropy modeling. However, these models have been limited by domain gaps between training and inference data, particularly in scenarios involving non-natural video sources (screen recordings, medical, animation). Existing approaches to domain adaptation, such as parameter fine-tuning and modular adaptation, tend to increase encoder complexity and are primarily focused on motion coding, often neglecting context coding components that contribute significantly to overall bitrate. The paper "Neural Video Compression with Domain Transfer" (2605.13476) introduces DCVC-DT, a domain transfer-enhanced NVC framework that leverages lightweight latent refinement at inference and frame-level dynamic rate-distortion (RD) adjustment to effectively bridge this gap and improve generalization.

Figure 1: The DCVC-DT architecture integrates Online Domain Transfer and Dynamic RD Adjustment mechanisms, optimizing latent features exclusively at the encoder side.

Methodology

Online Latent Refinement for Domain Transfer

DCVC-DT extends the context-based DCVC-DC baseline with a novel online latent refinement mechanism. Instead of updating network parameters, the model fine-tunes the latent representations ($y_t$) during inference by minimizing the RD loss:

$$\mathcal{L}_t = R + w_t \cdot \lambda \cdot D(x_t, \hat{x}_t)$$

where $D(\cdot)$ is the PSNR-based distortion measure, $R$ denotes estimated bitrate, and $w_t$ is a hierarchical frame weighting coefficient. Latent vectors are updated via gradient descent, utilizing stochastic Gumbel annealing quantization to ensure compatibility with the entropy coding pipeline. This strategy enables adaptive encoding—conforming to the statistics of domain-specific content, while maintaining fixed encoder and decoder parameters (thus not increasing decoding complexity).

Figure 2: Overview of Online Latent Refinement and Frame-level Dynamic RD Adjustment, where latent representation and RD weighting are iteratively updated during encoding.

Frame-level Dynamic RD Adjustment

To further optimize RD performance, DCVC-DT implements a frame-level dynamic RD adjustment module. This component dynamically modulates the bitrate weighting factor ($\beta_t$) within the loss function according to inter-frame distortion fluctuations ($\Delta Q_t$). The iterative update

$$\beta_t = \beta_0 - 0.2 \cdot \text{sign}(\Delta Q_t)$$

allows responsive rate allocation and quality smoothing, enhancing the model's ability to mitigate error propagation and quality instability across frames, especially when reference frames degrade due to domain variation.

Experimental Evaluation

Extensive experiments on HEVC Class C and D datasets, evaluated with PSNR under BT.709 color space, demonstrate the effectiveness of DCVC-DT. The proposed method achieves up to 6.21% average BD-rate reduction versus the DCVC-DC anchor, outperforming other DCVC variants across all test sequences. The ablation study confirms that online latent refinement is independently effective, yielding a 2.35% BD-rate reduction, with the dynamic RD adjustment contributing an additional 3.61% improvement.

Figure 3: RD performance comparison shows DCVC-DT's superior PSNR-bitrate curves on HEVC Class C and D datasets.

Error propagation is significantly mitigated, as evidenced by the BasketballPass sequence results: DCVC-DT displays smoother PSNR and BPP trajectories across the first 32 frames, demonstrating that improved reference frame quality leads to consistently better reconstruction in subsequent frames.

Figure 4: Frame-level analysis reveals reduced error propagation and stabilized reconstruction quality.

Complexity analysis indicates that the online refinement process increases encoder-side computation and memory requirements (e.g., ~25 minutes for 500 iterations per sequence on a single A100 GPU), while decoder complexity remains unchanged. This makes DCVC-DT especially suitable for scenarios requiring encoder-side optimization such as high-value archival, medical, or production-grade compression.

Implications and Future Directions

DCVC-DT represents a shift towards domain transfer techniques that operate entirely in the latent space, sidestepping the overhead associated with parameter updating or modular adaptation. The approach offers practical benefits in terms of generalization, RD efficiency, and complexity separation (encoder vs. decoder). Theoretically, optimizing latent space representations at inference is a promising direction for aligning the model's output distribution with arbitrary domains, thus improving compression quality and robustness.

Future work could explore:

• Autoregressive or diffusion-based latent refinement for improved convergence.
• Adaptive dynamic RD adjustment policies based on multi-dimensional quality metrics (e.g., perceptual quality, task-specific utility).
• Extension to hybrid codecs and real-time settings by reducing encoder-side optimization time or leveraging distributed computation.

Conclusion

DCVC-DT introduces an efficient, effective domain transfer framework for neural video compression, based on online latent refinement and frame-level dynamic RD adjustment. These mechanisms enable superior generalization and RD performance across heterogeneous video domains without parameter updates, making the approach scalable and practical for diverse deployment scenarios. This work underscores the importance of adaptive inference-time optimization in NVC and paves the way for further developments in content-adaptive compression strategies.