ABRBench-4G+ Evaluation Benchmark

• ABRBench-4G+ is a benchmark that aggregates diverse 4G and above network traces to train and evaluate adaptive bitrate controllers on both in-distribution and out-of-distribution scenarios.
• It employs a controlled train/test/OOD split by withholding entire trace sets during training to rigorously assess robustness across varied network environments.
• The benchmark provides standardized QoE metrics and reproducible evaluation protocols, enabling fair comparisons among state-of-the-art ABR methods.

ABRBench-4G+ is a benchmark within the ABRBench suite introduced in “SABR: A Stable Adaptive Bitrate Framework Using Behavior Cloning Pretraining and Reinforcement Learning Fine-Tuning” (Luo et al., 30 Aug 2025). It is designed to provide a large-scale, wide-coverage collection of 4G and above network traces together with a held-out out-of-distribution (OOD) test set, so that learning-based Adaptive Bitrate (ABR) controllers can be trained on realistic, diverse throughput profiles and then evaluated on both in-distribution and unseen conditions. Within the SABR framework, the benchmark serves two roles simultaneously: it enlarges the training distribution beyond limited trace sets, and it operationalizes robustness evaluation through source-level withholding of entire trace sets (Luo et al., 30 Aug 2025).

1. Scope and design objectives

ABRBench-4G+ was constructed around four explicit design goals. The first is wide coverage: real mobile networks exhibit bandwidths from tens of kilobits to multiple gigabits per second, and the benchmark brings together multiple public trace repositories, including Lumos 4G/5G and Solis Wi-Fi, to cover this dynamic range (Luo et al., 30 Aug 2025). The second is robust OOD evaluation: entire trace sets, specifically Ghent and Lab, are carved out so that they never appear during training or in-distribution testing. The third is consistency with previous work: by matching the methodology of ABRBench-3G and early simulators such as Pensieve, the benchmark can be substituted into established experimental pipelines. The fourth is reproducibility: train/test/OOD splits, evaluation code, and a reference simulator are publicly released.

These goals position ABRBench-4G+ as more than a trace archive. It is a controlled evaluation protocol for learning-based ABR under distributional breadth and explicit generalization stress. This suggests that the benchmark is intended to separate two questions that are often conflated in ABR evaluation: whether a policy optimizes QoE on seen-like conditions, and whether it remains effective when the source distribution changes at the trace-set level.

2. Trace sources and data construction

The benchmark aggregates traces from several public repositories. Lumos 4G and Lumos 5G, identified as SIGCOMM ’20 and OACM IMC ’21 sources, captured commercial cellular throughput in hundreds of users’ smartphones. Solis Wi-Fi, identified as an INFOCOM ’22 source, measured public Wi-Fi hotspots. Ghent and Lab traces, identified as PiTree ’19 sources, were recorded on-campus 4G cells and dedicated lab testbeds (Luo et al., 30 Aug 2025).

The underlying collection environments are heterogeneous. Lumos data was collected on Android phones across multiple U.S. and European carriers. Ghent was gathered in a Belgian university campus, and Lab traces were collected in a campus office environment. All throughput samples are second-level aggregates obtained via standard ping-based or HTTP-download probes, as described in the original trace papers. The benchmark post-processes each raw trace into a time series of instantaneous available bandwidth in Mbps (Luo et al., 30 Aug 2025).

This composition matters methodologically. The benchmark does not restrict itself to a single radio access technology or a single deployment setting. Instead, it combines commercial cellular, 5G, public Wi-Fi, campus 4G, and lab environments. A plausible implication is that ABR policies evaluated on ABRBench-4G+ are exposed to broader temporal dynamics and measurement regimes than in narrower trace collections, although the benchmark itself remains defined by the included source sets rather than by any universal notion of “real-world” coverage.

3. Dataset composition and statistical profile

ABRBench-4G+ specifies the size and bandwidth profile of each split. The training set contains 262 distinct traces, each approximately 200–1,200 seconds long, with average length approximately 600 seconds. The in-distribution test set contains 114 traces drawn from three source sets: Lumos 4G (53), Lumos 5G (37), and Solis Wi-Fi (24). The OOD set contains 101 traces: Ghent (40) and Lab (61) (Luo et al., 30 Aug 2025).

Group	Traces	Bandwidth summary
Training	262	Min 0.00 Mbps, Max 1890.00 Mbps, Mean ≈ 250, $\sigma \approx 300$
Lumos 4G	53	Min 0.00 Mbps, Max 270.00 Mbps, Mean ≈ 35, $\sigma \approx 40$
Lumos 5G	37	Min 0.00 Mbps, Max 1920.00 Mbps, Mean ≈ 450, $\sigma \approx 500$
Solis Wi-Fi	24	Min 0.00 Mbps, Max 124.00 Mbps, Mean ≈ 20, $\sigma \approx 25$
Ghent (OOD)	40	Min 0.00 Mbps, Max 110.97 Mbps, Mean ≈ 15, $\sigma \approx 18$
Lab (OOD)	61	Min 0.16 Mbps, Max 175.91 Mbps, Mean ≈ 30, $\sigma \approx 35$

The mean and standard deviation values are described as approximate, reconstructed from the original per-trace histograms (Luo et al., 30 Aug 2025). Even with that caveat, the aggregate picture is clear: the training distribution spans from zero to 1890.00 Mbps, while the constituent evaluation subsets differ substantially in scale and variance.

The benchmark also defines per-trace variability metrics. The coefficient of variation is

$CV = \sigma / \mu.$

The burstiness index, following Narayanan et al. 2021, is

$B = \sqrt{E[(b_t - b_{t-1})^2]} / \mu,$

where $b_t$ is the bandwidth sample at time $t$ (Luo et al., 30 Aug 2025). Averaged across traces, the overall burstiness is reported as 1.2 for training, 1.1 for in-distribution test, and 1.5 for OOD. This suggests that OOD traces are not merely distributionally distinct in origin; they also exhibit stronger temporal fluctuation under the benchmark’s chosen burstiness statistic.

4. In-distribution and OOD split semantics

ABRBench-4G+ defines OOD at the level of withheld source sets. Any trace belonging to a source set that was withheld entirely during training and in-distribution testing is designated OOD. The data states this formally as

$\sigma \approx 40$0

with the quantitative breakdown

$\sigma \approx 40$1

and the in-distribution decomposition $\sigma \approx 40$2 and OOD decomposition $\sigma \approx 40$3 (Luo et al., 30 Aug 2025).

This split construction is stricter than random per-trace partitioning. Because whole source sets are withheld, a policy cannot benefit from partial overlap in collection environment, measurement idiosyncrasies, or site-specific dynamics between training and OOD evaluation. A common misconception is to treat OOD as simply “harder test traces.” In ABRBench-4G+, OOD is defined structurally by provenance exclusion, not only by low throughput or high variance.

The benchmark also gives an OOD detection definition using the Mahalanobis distance over a two-dimensional feature vector formed by the trace mean and standard deviation:

$\sigma \approx 40$4

where

$\sigma \approx 40$5

A threshold $\sigma \approx 40$6 classifies $\sigma \approx 40$7 as OOD if $\sigma \approx 40$8 (Luo et al., 30 Aug 2025). This definition is not the split criterion itself; rather, it provides a feature-based mechanism for flagging OOD traces relative to the empirical training distribution.

5. QoE model and supported evaluation metrics

ABRBench-4G+ uses the canonical QoE formulation

$\sigma \approx 40$9

For the 4G+ setting,

$\sigma \approx 500$0

with $\sigma \approx 500$1 measured in kbps, $\sigma \approx 500$2, $\sigma \approx 500$3, and $\sigma \approx 500$4 as the rebuffering-penalty coefficient used for 4G+ (Luo et al., 30 Aug 2025). Here $\sigma \approx 500$5 is the rebuffering time in seconds during the $\sigma \approx 500$6-th chunk.

The benchmark further supports three additional metrics. Average bitrate is

$\sigma \approx 500$7

Rebuffering ratio is

$\sigma \approx 500$8

where $\sigma \approx 500$9 is the nominal chunk length of 4 seconds. Smoothness, expressed as total variation, is

$\sigma \approx 25$0

These definitions make the benchmark directly comparable across controllers under a fixed bitrate ladder, fixed chunk duration, and fixed penalty weights. They also clarify the intended trade-off surface: higher bitrate contributes positively, while both bitrate switching and rebuffering reduce QoE, with rebuffering assigned a coefficient of 40 in the 4G+ regime (Luo et al., 30 Aug 2025).

6. Experimental protocol and reproducibility requirements

The benchmark prescribes a concrete training and reporting protocol. During training, the 262 training traces are shuffled, and the ABR algorithm is run end-to-end over each trace for 49 chunks of 4 seconds each (Luo et al., 30 Aug 2025). Evaluation on in-distribution data is performed separately on Lumos 4G, Lumos 5G, and Solis Wi-Fi, with per-set QoE reported alongside the average rank

$\sigma \approx 25$1

OOD evaluation is performed analogously on Ghent and Lab, with average rank computed over those two sets.

For learning-based approaches, the benchmark requires at least 10 independent models with different random seeds, and reporting of the mean and standard error of QoE (Luo et al., 30 Aug 2025). It also requires granular reporting: test and OOD results should not be aggregated, so that high-bandwidth sets do not mask performance in lower-bandwidth environments. Public code and data are released through the ABRBench simulator and trace splits at github.com/luopeng69131/ABRBench to ensure bit-exact reproducibility of splits and QoE calculations.

The benchmark was introduced together with SABR, and the accompanying study reports that SABR achieves the best average rank compared with Pensieve, Comyco, and NetLLM across the proposed benchmarks (Luo et al., 30 Aug 2025). For ABRBench-4G+, the more general significance is infrastructural rather than algorithm-specific: any new ABR controller can be compared against Pensieve, Comyco, NetLLM, and SABR on the same 4G+ distributions and on truly unseen OOD traffic patterns. This suggests that ABRBench-4G+ is intended as a common evaluation substrate for stability, generalization, and source-aware robustness in learning-based ABR.