Bitrate Proxy in Video and Image Compression

• Bitrate proxy is a computational model that estimates encoding bitrates using learned features and neural networks, reducing the need for exhaustive encoding passes.
• It leverages rate–distortion curves and perceptual quality metrics such as VMAF to balance quality, bitrate, and resource fairness in adaptive encoding setups.
• By employing efficient feature extraction and neural approximations, bitrate proxies achieve high prediction accuracy with sub-10 ms latency across diverse codec systems.

A bitrate proxy is a computational construct or learned model that serves as a stand-in for direct measurement, prediction, or estimation of the bitrate associated with a given video or image compression configuration, typically under constraints of rate–distortion, perceptual quality, or resource fairness. Rather than requiring exhaustive encoding passes at every candidate parameter, a bitrate proxy predicts, interpolates, or otherwise infers the output bitrate (or related mapping, such as rate–quality or rate–distortion curves), often leveraging features, auxiliary neural networks, or cross-session statistics to minimize operational cost while maintaining accurate steering of encoding or streaming workflows. This concept spans learned video and image codec systems, adaptive streaming, and network control, serving both streaming and storage scenarios.

1. Mathematical and Algorithmic Foundations

Most bitrate proxies formalize some mapping between control parameters of the encoder (rate factor, quantization parameter, Lagrangian multiplier, or derived neural coding bottleneck) and the expected output bitrate, often conditioned on input content features or side information.

In content-adaptive rate–quality proxy models, let $RF$ denote the encoder’s rate factor (e.g., CRF in HEVC). The central objects are two discrete curves:

• $Q(RF) = \{\hat{Q}(RF_1), ..., \hat{Q}(RF_n)\}$, where $\hat{Q}$ denotes predicted quality (e.g., VMAF) at each RF.
• $B(RF) = \{\hat{B}(RF_1), ..., \hat{B}(RF_n)\}$, the predicted bitrate.

These curves are learned jointly by fitting neural networks $f_Q$ and $f_B$, with features $F$ and parameters $\theta_Q$, $\theta_B$. The loss combines per-point squared errors:

$$L = \| \hat{Q} - Q_{GT} \|_2^2 + \lambda \| \hat{B} - B_{GT} \|_2^2,$$

where $\lambda$ controls trade-off between quality and bitrate fit. The predicted $B(Q)$ bitrate–quality mapping is obtained via monotonic inversion or lookup on these curves (Yin et al., 2024).

For learned image compression, differentiable bitrate proxies (e.g., ProxIQA) enable direct back-propagation through otherwise non-differentiable or computationally expensive quality metrics. The total training loss takes the form

$$L_t(\theta; \phi) = \lambda [ \alpha L_p(\theta; \phi) + (1-\alpha) L_d(\theta) ] + L_r(\theta),$$

where $L_r$ is a (proxy) bitrate loss, $L_d$ is a pixel-level distortion loss, and $L_p$ is a perceptual (proxy) loss derived from a surrogate network $f_p$ trained to mimic a target quality metric (e.g., VMAF) (Chen et al., 2019).

For networked streaming, proxies such as DDA (Data-Driven Aggregation) predict available throughput $T_p$ using cross-session feature similarity. The aggregate prediction

$$\tilde{T}_p = k_p \cdot \mathrm{median}\{ T_j : j \in S(p, M_p) \}$$

directly guides bitrate ladder selection, ensuring reliability while maintaining elevated goodput (Jiang et al., 2015).

2. Feature Engineering and Architectural Variants

Input features for bitrate proxies are domain-adapted to maximize predictive power at minimal computational cost. For content-adaptive encoding:

• Codec features (e.g., per-frame PSNR, I/P/B-frame ratios, partition sizes) are extracted by low-bitrate, low-resolution pre-encodes.
• Content features (e.g., motion vector statistics, gray-level co-occurrence, no-reference quality estimators) capture spatial/temporal complexity.
• Anchor features (e.g., VMAF, bitrate at anchor RF) "lock" predicted curves for drift robustness (Yin et al., 2024).

In low-resolution proxy systems, features from 144p encodes or fast presets are regressed to predict optimal Lagrange multiplier scaling factors for rate control, with Random Forests (or other regressors) trained to match ground-truth BD-Rate minimizers (Ringis et al., 2022).

Neural bitrate proxies in context of learned codecs feature shallow or deep convolutional architectures, e.g., three-stage convolutional blocks with channel expansions and pooling, or four-layer CNNs for latent space projection (as in BAM of CBANet) (Guo et al., 2021, Chen et al., 2019).

3. Proxy Integration into Video and Image Encoding Systems

Bitrate proxies are deployed in diverse settings:

• Encoding and transcoding: The predicted $B(Q)$ mapping enables constant-quality or constant-bitrate encoding, custom rate–quality ladders, and minimizes model retraining overhead via feature injection and curve inversion (Yin et al., 2024).
• Single-pass rate control: At the GOP level, shallow neural proxies adapt CRF values per-group, meeting bitrate constraints in one pass without budget overshoot, using lookahead features computed in standard encoders (Cheng et al., 2019).
• Rate–distortion optimization: Bitrate proxies facilitate Lagrangian or entropy-constrained RD estimation without multiple full-resolution encodes, both for classical codecs and end-to-end deep image encoders (Ringis et al., 2022, Chen et al., 2019).
• Streaming and networking: Bitrate proxies predict sustainable delivery rates before session start via cross-session correlation, increasing initial bitrate for video streaming with minimal rebuffering (Jiang et al., 2015).

In online systems, runtime overhead is kept low via pre-extraction of features at reduced complexity and inference using compact neural networks, yielding sub-10 ms CPU/GPU latency for curve evaluation (Yin et al., 2024, Guo et al., 2021).

4. Theoretical Guarantees and Performance Evaluation

Empirical studies report high predictive accuracy and operational gains for bitrate proxies:

• Content-adaptive bitrate proxy models achieve a VMAF mean absolute error (MAE) of 0.230, MAE_Rate of 32.25 kbps, and a VMAF-Accuracy-Within-Choice (VACC) of 99.14%. Anchor suspension further guarantees zero error at anchor points and reduces drift under codec changes (Yin et al., 2024).
• Single-pass CARF encoding: 84.5% of test segments are within 20% of the target bitrate, and BD-rate reductions over standard ABR exceed 5% (Cheng et al., 2019).
• Low-resolution proxy regression: 60–80% of the possible BD-rate gain is recovered using only one proxy encode per clip and one regression, offering 22–1,000× speedups versus brute force (Ringis et al., 2022).
• Proxy-based streaming selection: DDA reduces 80-percentile throughput prediction errors by 50–80% compared to baselines, supporting a 4× higher sustainable starting bitrate with nearly 100% “good” sessions (Jiang et al., 2015).
• Proxy fairness frameworks: In multi-client scenarios, in-network proxy enforcement guarantees nearly equal per-client bitrates, bounded bitrate degradation, and zero rebuffering, with measured Jain unfairness index $F = 0.03$–$0.24$ in controlled topologies (Tran et al., 2020).

5. Proxy Strategies in End-to-End Learned and Differentiable Systems

Bitrate proxies in deep learning architectures often address the challenge of incorporating non-differentiable objective functions:

• Perceptual proxies: Networks such as ProxIQA learn to approximate VMAF or other perceptual quality metrics via a trainable proxy, allowing quality awareness in backpropagation and delivering up to 31% bitrate reduction at fixed perceptual quality (Chen et al., 2019).
• Differentiable bitrate estimation for codecs: Closed-form differentiable proxies for standard codecs (e.g., HEVC/H.265) model block coefficient distributions with a spatially varying (hyper-prior) Laplacian, yielding rate estimates and gradients compatible with automatic differentiation (Said et al., 2023).
• Bitrate adaptive latent space projection: The Bitrate Adaptive Module (BAM) maps full-rate latent spaces to reduced-rate spaces and recovers back, enabling a single codec to support multiple rates and compute budgets with massive storage and deployment efficiencies (Guo et al., 2021).

6. Proxy-Based Control in Streaming and Network Fairness

Bitrate proxies are instrumental in network-fairness and adaptive streaming contexts:

• In the FAURAS proxy, bandwidth allocation and request overwrite modules enforce fair per-client allocations; re-writing requests when a client selects a bitrate above its share and risks underflow, while maintaining HTTP/2 push semantics via custom headers (Tran et al., 2020).
• This guarantees fairness (as measured by Jain’s index), maintains buffer health, and eliminates bandwidth waste—all without sacrificing push efficiency or introducing complex per-client optimization.

7. Limitations, Future Directions, and Extensions

Bitrate proxies depend on accurate feature selection, representativeness of training data, and robustness to changes in encoder implementations. While anchor suspension methods and online fine-tuning can stabilize proxies under mild drifts, major codec changes necessitate full retraining. In sparse or nonstationary environments (e.g., highly variable cross-session network traces), performance may degrade due to insufficiently close matching sessions or overfitting to irrelevant features (Jiang et al., 2015).

Extensions under exploration include learned similarity metrics for session matching, direct integration of active feedback, hybrid proxy strategies combining low-resolution prescreening with sparse full-resolution search, and multi-objective optimization to balance rate–quality, fairness, and computational complexity (Ringis et al., 2022, Guo et al., 2021, Tran et al., 2020).

In summary, bitrate proxies unify a family of techniques for efficiently steering bitrate selection, encoding configuration, rate–distortion control, and network fairness, spanning both traditional codec pipelines and modern learned image/video compression systems. Their adoption enables flexible, content- and context-aware trade-offs without the prohibitive computational burden of exhaustive search or repeated encoding passes, and with precision close to direct measurement or ground truth.