Differentiable Proxy Learning for Adaptive Quantization Control in H.264 Video Coding

Abstract: H.264 has been the most widely used video coding format for the past two decades due to its relative simplicity, efficiency, and wide availability of software and hardware implementations. However, optimizing codec parameters such as the quantization parameter (QP) for specific objectives (e.g., perceptual quality or machine vision tasks) is challenging due to the non-differentiable nature of standard video codecs. While differentiable proxies have recently been used to enable gradient-based optimization around standard codecs, their fidelity to the target codec is rarely explicitly characterized. In this paper, we propose a differentiable proxy learning method for H.264 intra codec to enable adaptive quantization control. Built upon a variable-rate learned compression model, the proposed proxy is made differentiable with respect to codec QP through a soft-indexing mechanism. It is then trained to approximate the rate-distortion behavior of H.264 under two quantization settings: global-QP, which uses one QP per image, and spatial-QP, which assigns QPs at the macroblock level. Using the frozen trained proxy, we develop a proxy-based adaptive quantization (AQ) framework for both perceptual optimization and machine vision tasks. Experimental results demonstrate that the proposed proxies closely approximate the rate-distortion behavior of H.264 intra codec. The resulting proxy-based AQ framework consistently improves rate-task trade-offs over fixed-QP H.264 baselines, achieving BD-rate reduction of up to 17.12% for semantic segmentation and 15.30% for MS-SSIM.

• The paper proposes a novel differentiable proxy framework that enables gradient-based adaptive quantization for H.264 intra coding.
• It employs a DCVC-FM based soft-indexing mechanism to learn both global and spatial quantization parameter mappings accurately.
• Experimental results demonstrate significant BD-rate reductions and enhanced performance for downstream tasks like object detection and segmentation.

Differentiable Proxy Learning for Adaptive Quantization Control in H.264 Video Coding

Introduction and Motivation

Optimizing quantization parameters (QPs) in standard block-based video codecs such as H.264 is critical for achieving optimal trade-offs between bitrate, visual quality, and downstream machine vision task performance. However, the discrete, non-differentiable nature of codec operations prevents direct gradient-based optimization, constraining the ability to adapt bit allocation dynamically at either the image or block level. Recent advances utilizing differentiable proxies—learned neural networks that simulate the behavior of lossy codecs—have enabled backpropagation-based joint optimization and parameter search in video compression pipelines. However, prior works generally treat proxies as auxiliary modules for optimizing pre-/post-processing or downstream adaptation and rarely address the fidelity between the proxy and the original codec, especially under fine-grained adaptive quantization.

This paper addresses these limitations via a framework for differentiable proxy learning explicitly tuned to the intra mode of H.264 codec. The approach supports both global-QP (image-level) and spatial-QP (macroblock-level) control by constructing a proxy based on a variable-rate learned compression model (DCVC-FM), augmented with a differentiable soft-indexing mechanism for continuous QP assignment. The fidelity of the proxy to the reference codec is systematically quantified, and the trained proxy is utilized in a closed-loop, gradient-based adaptive quantization (AQ) control—facilitating both perceptual and downstream task (e.g., object detection, segmentation) optimizations.

Figure 1: Training pipeline of the proposed differentiable proxy learning for H.264 Intra Codec.

Methodology

Differentiable Proxy for H.264: Architecture and Training

The proxy model is built atop DCVC-FM, a neural compression model supporting variable-rate coding via discrete quantization index selection. To reconcile the codec's QP space with the proxy's internal quality index, a mapping is learned by aligning distributions in the rate-distortion (e.g., BPP–PSNR) manifold on the Kodak benchmark set. The proxy operates on both scalar (global-QP) and spatially varying (spatial-QP) input QPs.

A soft-indexing mechanism replaces hard quantization index selection, exposing the quantization scaling factors as weighted interpolations over candidate scale values, which makes the proxy differentiable with respect to QP inputs, thus enabling efficient gradient flow for downstream AQ network training.

The proxy’s loss combines absolute bitrate prediction error and mean squared error distortion, with adaptive Lagrange multipliers linked to QP in the DCVC-FM design.

Figure 2: Proxy-based adaptive quantization control framework.

Proxy-Based Adaptive Quantization Framework

With the proxy frozen, separate AQ networks are trained for global-QP and spatial-QP quantization control. For global-QP control, a lightweight CNN predicts a scalar QP per image; for spatial-QP, a fully convolutional network infers a macroblock-level QP map, modulated by a user-tunable rate-task parameter $\beta$. Both settings involve joint minimization of bitrate and downstream perceptual or task losses (e.g., MS-SSIM for perceptual quality, cross-entropy for segmentation).

Figure 3: Structure of the AQ network for adaptive spatial-QP control.

Experimental Results

Fidelity of the Learned Proxy

On the Kodak dataset, the global-QP proxy achieves high fidelity to the H.264 intra codec, with BD-rate differences of $-4.75\%$ (PSNR) and $-2.56\%$ (MS-SSIM). The spatial-QP proxy displays a larger but acceptable gap ($-15.01\%$ for PSNR, $-12.91\%$ for MS-SSIM), reflecting increased complexity in matching macroblock-adaptive quantization.

Figure 4: Rate-distortion curves of the global-QP and spatial-QP proxies in terms of PSNR and MS-SSIM tested on Kodak.

Qualitative analysis indicates that, although less capable of emulating severe blocking artifacts seen in H.264, the proxy reconstructions capture overall rate-distortion trends and artifact spatial localization—especially under spatial-QP settings.

Figure 5: Visual comparison between H.264 and the proposed proxy under global-QP and spatial-QP settings.

Adaptive Quantization for Downstream Objectives

Global-QP Control

On COCO object detection benchmarks with YOLOv5 backbones, the AQ framework substantially improves rate–accuracy trade-offs over fixed-QP H.264. BD-rate reductions are $-8.96\%$, $-8.19\%$, and $-9.50\%$ respectively for S/M/L models.

Figure 6: Rate-accuracy comparison on COCO using YOLOv5 backbones with three model sizes.

For perceptual optimization (rate–MS-SSIM), the gains are less pronounced ($-3.75\%$ BD-rate), indicating that bit allocation at the image level is less effective for purely perceptual objectives.

Figure 7: Rate--MS-SSIM comparison on Kodak.

Spatial-QP Control

For semantic segmentation (Pascal VOC, DeepLabv3-ResNet50), spatial-QP control yields a $-17.12\%$ BD-rate reduction relative to fixed-QP baselines, outperforming global-QP control by a considerable margin. Visualizations confirm that the network allocates lower QP (thus more bits) to task-critical regions.

Figure 8: Visual comparison of segmentation results. (a) Original image. (b) Ground truth. (c) Reconstructed image using adaptive spatial-QP, with the QP map shown in the top-right corner. (d) Corresponding segmentation result. (e) Reconstructed image using fixed QP. (f) Corresponding segmentation result.

Figure 9: Rate-Accuracy comparison on Pascal VOC using Deeplabv3 with ResNet-50 backbone.

For MS-SSIM optimization (Vimeo-90k), spatial-QP adaptive training achieves a $-4.75\%$0 BD-rate improvement. In all cases, the adaptive AQ infrastructure reconfirms that fine-grained, differentiable codec control through accurate proxies directly benefits rate-task performance.

Figure 10: Visual examples of adaptive spatial-QP control optimized for MS-SSIM.

Implications and Future Work

This work establishes that differentiable codec proxies with explicit QP control facilitate effective, scalable AQ strategies for standard codecs, even when the proxy is not perfectly matching all artifacts of the original H.264 codec. The successful extension to spatial-QP maps demonstrates the utility of learned proxies for fine-grained, practical encoder parameter tuning—a longstanding challenge in classical video coding.

Potential ramifications include the adoption of proxy-driven neural AQ in hardware-friendly, standard-compatible streaming environments, and the integration of proxy-based AQ with deep preprocessing and neural wrappers for broader rate-distortion-perception-task trade-off optimization. Extension to inter-frame coding with temporal dynamics and cross-video rate allocation represents a logical subsequent direction.

Conclusion

Differentiable proxy learning, leveraging soft-indexing in a variable-rate neural compression architecture, enables practical, gradient-based adaptive quantization for H.264 intra codecs. The framework achieves up to $-4.75\%$1 BD-rate reduction for semantic segmentation and $-4.75\%$2 for perceptual (MS-SSIM) objectives in empirical evaluation. This demonstrates that highly accurate, codec-fidelity-aligned proxies are sufficient for effective AQ, even under block-level adaptation. The methods and insights here offer a viable blueprint for neural control of legacy codecs in both vision- and perception-centric compressed video delivery.

---

Cited paper: "Differentiable Proxy Learning for Adaptive Quantization Control in H.264 Video Coding" (2607.10478)