Spatially Adaptive Bit Rates (SABR)

• Spatially Adaptive Bit Rates (SABR) are techniques that allocate bits non-uniformly across visual content based on local signal complexity, semantics, or perceptual weighting.
• SABR employs analytical STAR rate models combined with neural architectures, such as tiled recurrent autoencoders and reinforcement learning, to optimize compression and streaming quality.
• Practical SABR implementations tackle challenges like artifact avoidance and effective spatial segmentation by integrating spatial context predictors and heuristic bitrate constraints.

Spatially Adaptive Bit Rates (SABR) denote a class of methodologies for allocating bits non-uniformly across the spatial or temporal extent of visual signals—such as images and video—based on local signal complexity, semantics, or perceptual weighting. SABR frameworks adapt quantization, spatial/temporal resolution, or codeword allocation in a region-dependent fashion to maximize rate-distortion or perceptual efficiency under a constrained total bitrate budget. Foundational approaches span signal-analytic, deep learning, and reinforcement learning paradigms and have been validated in both image/video compression and adaptive streaming applications (Ma et al., 2012, Johnston et al., 2017, Minnen et al., 2018, Gao et al., 2018, Li et al., 2023).

1. Analytical Foundations and Rate Modeling

The analytical basis for SABR is the explicit and separable modeling of the dependencies of bit rate on amplitude, spatial, and temporal resolution (collectively, STAR parameters). The rate of a compressed stream can be written as a product of power functions:

$$R(q, s, t) = R_{\max} \left( \frac{q}{q_{\min}} \right)^{-a} \left( \frac{t}{t_{\max}} \right)^b \left( \frac{s}{s_{\max}} \right)^c$$

where

• $q$: quantization stepsize (amplitude resolution),
• $s$: frame size (spatial resolution),
• $t$: frame rate (temporal resolution),
• $R_{\max}$: maximum rate at maximum spatial/temporal and minimum quantization,
• $a, b, c$: content-dependent scaling exponents (Ma et al., 2012).

This separability permits direct estimation of regional bitrate as spatial parameters vary. The model parameters can be efficiently estimated either empirically—via least squares fitting to observed rates across coding points—or predicted from content features, including the mean displaced frame difference, standard deviation of motion vector magnitude, and standard deviation of motion direction activity. This analytical tractability enables systematic optimization of STAR for any given bitrate constraint, facilitating SABR strategies in coding and adaptation workflows.

2. Neural Architectures for Spatially Adaptive Bitrate Allocation

Neural compression approaches exploiting SABR deploy spatially segmented encoding workflows. A common architectural theme is the use of non-overlapping tiles—e.g., $16 \times 16$ (Johnston et al., 2017), $32 \times 32$ pixels (Minnen et al., 2018)—which are processed independently or with limited context.

In tiled recurrent auto-encoder systems:

• Each tile is progressively encoded: the encoder recurrently reduces local residuals; the number of iterations (hence bits) per tile is adjusted to achieve target reconstruction quality.
• For each tile $i$, bit depth $b_i$ is set as

$$b_{i} = \min\left(b_{\text{max}},\ \text{smallest } b : Q_{\text{tile}} \leq Q_{\text{target}}\right)$$

where $Q_{\text{tile}}$ is a tile-wise distortion criterion, typically maximum mean $L_1$ error over spatial sub-tiles, compared against a threshold.

Spatial context prediction networks, operating on surrounding decoded tiles, are integrated in more advanced tiled models to smooth out block artifacts and further localize bitrate allocation, as described in (Minnen et al., 2018).

3. SABR Algorithms and Control Policies

The practical SABR bit allocation process is governed by a quality or complexity criterion:

• Tiles or spatial regions exceeding a local distortion threshold receive more bits (longer encoding iterations or finer quantization); low-complexity regions are encoded with fewer bits.
• Additional heuristic constraints are often employed for artifact avoidance (e.g., bounding allocated bits per tile to 50–120% of target rate, as in (Johnston et al., 2017)).

For network-based streaming and adaptive rate control, SABR can leverage content-aware feature prediction and reinforcement learning. Deep Q-Network (DQN) agents choose chunk or region-level bitrates based on predicted semantic “interestingness” (via 3D ConvNets), buffer state, and expected bandwidth, aligning bitrate allocation with content value (Gao et al., 2018). Multi-agent RL architectures separate prefetching and bitrate adaptation (BM-agent, BA-agent) in short-video streaming scenarios, combining expert-guided imitation and MARL fine-tuning for high efficiency and generalization (Li et al., 2023).

4. SABR in Streaming and Adaptive Video

SABR principles extend beyond still image and classic video compression into adaptive streaming. In content-aware personalized streaming, region or chunk-level “interestingness” is inferred using learned features; bitrates are then allocated to maximize cumulative perceptual quality (QoE), with explicit penalty or reward for user engagement and bandwidth wastage (Gao et al., 2018, Li et al., 2023).

Notably, in short-video platforms where rapid user interaction and prefetch unpredictability dominate, SABR methodologies can jointly optimize which videos to download and at which bitrate, using hierarchical multi-agent reinforcement learning and compound utility functions combining QoE and bandwidth usage (Li et al., 2023).

5. Quantitative Impact and Comparative Performance

SABR provides significant, empirically verified improvements over fixed-rate baselines and classical codecs:

• Recurrent network-based image compression with SABR achieves up to 43–45% rate savings over JPEG at equivalent MS-SSIM on benchmark datasets (Johnston et al., 2017).
• Tiled adaptive neural architectures yield up to 2 dB PSNR improvements versus constant bit rate versions and outperform JPEG at aggressive compression ratios (Minnen et al., 2018).
• In adaptive streaming, content-aware SABR methods align subjective quality with user engagement without degradation in objective metrics and outperform content-agnostic ABR across diverse datasets (Gao et al., 2018).
• Hierarchical multi-agent SABR for short video streaming achieves a 53.2% utility score improvement over state-of-the-art rule-based methods, with reduced rebuffering and bandwidth wastage, and real-time decision latency (Li et al., 2023).

6. Limitations and Practical Considerations

SABR systems require mechanisms to avoid visible artifacts due to aggressive local bitrate variation; empirical and formal constraints on per-region bitrate budget are typically enforced. For neural architectures, tile dependency modeling (spatial context predictors) is critical for artifact suppression, and parameter signaling (e.g., per-tile bit allocation maps) must be embedded in the bitstream for correct decoding. While neural SABR can be implemented as a post-processing step, retraining may further improve quality.

The complexity of region- or chunk-level bitrate optimization rises with the granularity of segmentation and depth of perceptual weighting. Analytical models based on STAR separability and linear feature prediction mitigate the need for per-content empirical measurement and provide tractable optimization in high-dimensional adaptation spaces (Ma et al., 2012).

7. Summary Table of SABR Applications and Methods

Application Domain	SABR Methodology	Key Mechanism / Metric
Image Compression (DNN)	Tiled recurrent encoding	Tile PSNR, progressive bits
Image Compression (DNN)	Post-process bit allocation	Per-tile $L_1$/MS-SSIM
Video Compression	Analytical STAR rate models	STAR parameter optimization
Video Streaming (chunk-level)	DQN/Content-of-Interest (CoI)	Semantic score-weighted QoE
Short Video Streaming	Hierarchical MARL+Imitation	Compound utility, modular RL

SABR methodologies provide a comprehensive and flexible framework for adaptive bit allocation across spatial or temporal domains in image and video coding. The approaches are theoretically grounded in separable rate models and have demonstrated experimentally validated gains in compression efficiency and perceptual quality in both classic and learning-based frameworks (Ma et al., 2012, Johnston et al., 2017, Minnen et al., 2018, Gao et al., 2018, Li et al., 2023).