JND-Aware Per-Scene Bitrate Laddering

• The paper introduces JASLA, a framework that generates content-adaptive encoding ladders by integrating just-noticeable-difference (JND) modeling per scene to eliminate redundant bitrate-resolution pairs.
• It leverages DCT-based scene features and support vector regression to accurately predict JND thresholds, achieving significant bitrate savings and storage reduction.
• Empirical results show improved PSNR, VMAF gains, and faster processing, demonstrating JASLA’s practical impact for real-time integration in video-on-demand workflows.

JND-aware Per-Scene Bitrate Laddering (JASLA) is a framework for adaptive video streaming designed to produce content-adaptive encoding ladders that minimize bitrate and storage without introducing perceptible quality degradation. By integrating just-noticeable-difference (JND) modeling per scene, JASLA explicitly identifies and eliminates bitrate-resolution pairs that provide redundant or imperceptibly minor improvements in perceptual quality, thus optimizing both quality of experience and resource utilization (Menon et al., 2023).

1. Motivation and Conceptual Overview

Traditional HTTP adaptive streaming solutions employ a fixed collection of bitrate-resolution pairs (the bitrate ladder) for all video content, ignoring marked variability in spatial and temporal complexity across scenes. This strategy is suboptimal, as complex scenes require higher bitrates for the same perceptual quality than simpler ones. JASLA addresses this by generating for each scene a distinct, optimized ladder composed of:

• An optimal resolution $r^*(b)$ for each target bitrate $b \in B$ that maximizes predicted perceptual quality (VMAF).
• A corresponding constant rate factor $c^*(b)$ for rate control.
• A JND threshold CRF $C_T$ to identify and eliminate perceptually indistinguishable representations.

This per-scene, JND-aware ladder ensures that the encoding process meets stringent perceptual quality constraints while yielding significant bitrate and storage reductions (Menon et al., 2023).

2. Feature Extraction and JND Prediction

Bitrate ladder optimization in JASLA relies crucially on low-complexity, content-derived features capable of capturing scene complexity and perceptual redundancy.

DCT-Based Scene Features

Seven features are extracted for each scene using the VCA analyzer:

• $E_y$: average luma-texture energy
• $h$: average gradient of luma-texture energy
• $L_y$: average pixel-luminance
• $E_u$, $E_v$: average chroma-texture energies (U and V planes)
• $L_u$, $L_v$: average chrominance means

Conventional spatial (SI) and temporal (TI) information, as defined by ITU-P.910, are also considered but found to be insufficient for JND threshold prediction.

JND Threshold Feature Construction

For robust CRF threshold (JND) prediction, JASLA aggregates three categories of features:

• Scene complexity features $X_s$: the seven DCT-based measures above.
• Bitstream features $X_b$: framerate, stream bitrate, frame size, average motion (computed from a near-lossless CRF=5 encode).
• GLCM (gray-level co-occurrence matrix) features $X_g$: contrast, dissimilarity, homogeneity, ASM, energy, and correlation.

A spatial-temporal pooling strategy yields fixed-length vectors. Forward Sequential Feature Selection (F-SFS) reduces this set to the 15 most predictive features. These vectors are input to a support vector regression (SVR, RBF kernel) that predicts the JND threshold $C_T$ for the given scene (Menon et al., 2023).

3. Algorithmic Structure and Elimination Criteria

JASLA's encoding ladder generation consists of a joint prediction and pruning process:

1. Feature Extraction: Compute the seven DCT features for the scene.
2. JND Threshold Prediction: Use the selected 15-dimensional feature vector to predict $C_T$ via SVR.
3. Ladder Optimization: For each target bitrate $b_t \in B$:
   • Predict $U_{i,t}$ (VMAF) for each $r_i \in R$ using random forest regressors.
   • Select $r_t = \arg\max_{i} U_{i,t}$.
   • Predict CRF $c_t$ for $(r_t, b_t)$.
4. Representation Elimination: For all $t$ where $r_t = r_{max}$ and $c_t < C_T$, only retain the lowest-bitrate such $t$, discarding higher-bitrate representations as perceptually redundant.

This approach ensures that only a single representation operates in the "perceptually lossless" regime at the highest resolution, pruning all others below the JND threshold. No retraining is required if the set of supported resolutions expands, as each resolution maintains independent regression models for both VMAF and CRF (Menon et al., 2023).

4. Optimization Formulation and Constraints

The optimization process can be formalized as follows. For a given set of bitrates $\{b_t\}_{t=1}^N$ and resolutions $R$, JASLA solves, for each scene:

$r_t^* = \arg\max_{r \in R} VMAF_{pred}(r, b_t)$

$c_t^* = CRF_{pred}(r_t^*, b_t)$

with constraints:

• $r_t \in R$
• $C_{min} \leq c_t \leq C_{max}$ (HEVC: $0 \leq c_t \leq 51$)
• Enforce at most one $r_{max}$ representation with $c_t < C_T$.

Bitrate-respecting encoding is achieved by invoking x265's constrained VBR mode with $CRF = c_t$ and maximum bitrate set to $b_t$ (Menon et al., 2023).

5. Empirical Evaluation and Results

JASLA was evaluated on the Video Complexity Dataset (500 scenes, 80% training/20% test split) with x265 HEVC v3.5 at 30 fps. The fixed target ladder employed the Apple HLS specification:

• Resolutions: $\{360p,432p,540p,720p,1080p\}$
• Bitrates: $\{145,300,600,900,1600,2400,3400,4500,5800,8100\}$ kb/s

Objective and perceptual quality metrics included PSNR, VMAF (measured after upscaling all outputs to 1080p), and Bjøntegaard-delta bitrate (BD-rate). Storage reduction was indexed as $AS = (\sum b_t^{opt} / \sum b_t^{ref}) - 1$.

Key Results:

Metric	Value
VMAF-prediction $R^2$	0.93
MAE (VMAF)	3.25
CRF-prediction $R^2$	0.97
MAE (CRF)	1.86
JND $C_T$ prediction MAE	0.96 (CRF units)
Runtime	~0.4 s/segment
BD-rate$_{PSNR}$	–34.42%
BD-rate$_{VMAF}$	–42.67%
Storage reduction $AS$	–54.34%
PSNR gain at fixed bitrate	+2.90 dB
VMAF gain at fixed bitrate	+9.51 units

JASLA’s JND thresholding stage is 97.2% faster than prior SVR–masking–feature approaches. Per-scene RD curves confirm that JASLA-generated ladders outperform fixed ladders in all measured scenarios (Menon et al., 2023).

6. Comparative Developments and Extensions

Several recent frameworks implement JND-aware per-scene or per-segment bitrate laddering. The Quality-Aware Dynamic Resolution Adaptation (QADRA) framework (Premkumar et al., 2024) shares core algorithmic elements with JASLA:

• Use of the same seven VCA-generated DCT-energy features to model spatiotemporal complexity.
• Learning-based quality prediction, though QADRA regresses XPSNR via XGBoost, whereas JASLA employs per-resolution Random Forests for VMAF and CRF.
• JND-based pruning: QADRA uses a user-settable $\Delta_{JND}$ (default 0.0 dB) to discard representations on the ladder whose quality is within one JND of lower-bitrate points.
• Upper convex hull determination for final ladder selection.
• Explicit latency-constrained resolution selection.

Prior work, such as Menon et al. (2023) and Zhu et al. (2022) (referenced in QADRA), has empirically demonstrated that JND elimination can reduce ladder size by 20–30% and achieve 10–15% bitrate savings for equivalent perceptual quality (Premkumar et al., 2024).

7. Practical Implications, Deployment, and Limitations

JASLA achieves content-aware bitrate and storage efficiency by leveraging computationally inexpensive features and lightweight regression models, enabling real-time integration into VoD workflows as a preprocessing step that outputs detailed per-scene encoding configurations. The approach—trained on 1080p—requires retraining or subjective testing for robust extension to UHD (2160p) content. Furthermore, JASLA presumes static sets of bitrate and resolution targets and relies on pre-segmented scene boundaries.

A plausible implication is that such per-scene, JND-aware laddering frameworks, by removing perceptually redundant high-bitrate representations, can substantially decrease delivery and storage costs for content distributors while maintaining (or improving) user-experienced quality. The same principles guide QADRA and similar frameworks, suggesting a general trend toward granularity and perceptual modeling in adaptive streaming workflows (Menon et al., 2023, Premkumar et al., 2024).