---
title: ABRBench-4G+ Evaluation Benchmark
url: https://www.emergentmind.com/topics/abrbench-4g
type: topic
---

# ABRBench-4G+ Evaluation Benchmark

ABRBench-4G+ is a benchmark within the ABRBench suite introduced in “SABR: A Stable Adaptive Bitrate Framework Using Behavior Cloning Pretraining and Reinforcement Learning Fine-Tuning” [2509.10486]. It is designed to provide a large-scale, wide-coverage collection of 4G and above network traces together with a held-out out-of-distribution (OOD) test set, so that learning-based Adaptive Bitrate (ABR) controllers can be trained on realistic, diverse throughput profiles and then evaluated on both in-distribution and unseen conditions. Within the SABR framework, the benchmark serves two roles simultaneously: it enlarges the training distribution beyond limited trace sets, and it operationalizes robustness evaluation through source-level withholding of entire trace sets [2509.10486].

## 1. Scope and design objectives

ABRBench-4G+ was constructed around four explicit design goals. The first is **wide coverage**: real mobile networks exhibit bandwidths from tens of kilobits to multiple gigabits per second, and the benchmark brings together multiple public trace repositories, including Lumos 4G/5G and Solis Wi-Fi, to cover this dynamic range [2509.10486]. The second is **robust OOD evaluation**: entire trace sets, specifically Ghent and Lab, are carved out so that they never appear during training or in-distribution testing. The third is **consistency with previous work**: by matching the methodology of ABRBench-3G and early simulators such as Pensieve, the benchmark can be substituted into established experimental pipelines. The fourth is **reproducibility**: train/test/OOD splits, evaluation code, and a reference simulator are publicly released.

These goals position ABRBench-4G+ as more than a trace archive. It is a controlled evaluation protocol for learning-based ABR under distributional breadth and explicit generalization stress. This suggests that the benchmark is intended to separate two questions that are often conflated in ABR evaluation: whether a policy optimizes QoE on seen-like conditions, and whether it remains effective when the source distribution changes at the trace-set level.

## 2. Trace sources and data construction

The benchmark aggregates traces from several public repositories. Lumos 4G and Lumos 5G, identified as SIGCOMM ’20 and OACM IMC ’21 sources, captured commercial cellular throughput in hundreds of users’ smartphones. Solis Wi-Fi, identified as an INFOCOM ’22 source, measured public Wi-Fi hotspots. Ghent and Lab traces, identified as PiTree ’19 sources, were recorded on-campus 4G cells and dedicated lab testbeds [2509.10486].

The underlying collection environments are heterogeneous. Lumos data was collected on Android phones across multiple U.S. and European carriers. Ghent was gathered in a Belgian university campus, and Lab traces were collected in a campus office environment. All throughput samples are second-level aggregates obtained via standard ping-based or HTTP-download probes, as described in the original trace papers. The benchmark post-processes each raw trace into a time series of instantaneous available bandwidth in Mbps [2509.10486].

This composition matters methodologically. The benchmark does not restrict itself to a single radio access technology or a single deployment setting. Instead, it combines commercial cellular, 5G, public Wi-Fi, campus 4G, and lab environments. A plausible implication is that ABR policies evaluated on ABRBench-4G+ are exposed to broader temporal dynamics and measurement regimes than in narrower trace collections, although the benchmark itself remains defined by the included source sets rather than by any universal notion of “real-world” coverage.

## 3. Dataset composition and statistical profile

ABRBench-4G+ specifies the size and bandwidth profile of each split. The training set contains 262 distinct traces, each approximately 200–1,200 seconds long, with average length approximately 600 seconds. The in-distribution test set contains 114 traces drawn from three source sets: Lumos 4G (53), Lumos 5G (37), and Solis Wi-Fi (24). The OOD set contains 101 traces: Ghent (40) and Lab (61) [2509.10486].

| Group | Traces | Bandwidth summary |
|---|---:|---|
| Training | 262 | Min 0.00 Mbps, Max 1890.00 Mbps, Mean ≈ 250, $\sigma \approx 300$ |
| Lumos 4G | 53 | Min 0.00 Mbps, Max 270.00 Mbps, Mean ≈ 35, $\sigma \approx 40$ |
| Lumos 5G | 37 | Min 0.00 Mbps, Max 1920.00 Mbps, Mean ≈ 450, $\sigma \approx 500$ |
| Solis Wi-Fi | 24 | Min 0.00 Mbps, Max 124.00 Mbps, Mean ≈ 20, $\sigma \approx 25$ |
| Ghent (OOD) | 40 | Min 0.00 Mbps, Max 110.97 Mbps, Mean ≈ 15, $\sigma \approx 18$ |
| Lab (OOD) | 61 | Min 0.16 Mbps, Max 175.91 Mbps, Mean ≈ 30, $\sigma \approx 35$ |

The mean and standard deviation values are described as approximate, reconstructed from the original per-trace histograms [2509.10486]. Even with that caveat, the aggregate picture is clear: the training distribution spans from zero to 1890.00 Mbps, while the constituent evaluation subsets differ substantially in scale and variance.

The benchmark also defines per-trace variability metrics. The coefficient of variation is
$$
CV = \sigma / \mu.
$$
The burstiness index, following Narayanan et al. 2021, is
$$
B = \sqrt{E[(b_t - b_{t-1})^2]} / \mu,
$$
where $b_t$ is the bandwidth sample at time $t$ [2509.10486]. Averaged across traces, the overall burstiness is reported as 1.2 for training, 1.1 for in-distribution test, and 1.5 for OOD. This suggests that OOD traces are not merely distributionally distinct in origin; they also exhibit stronger temporal fluctuation under the benchmark’s chosen burstiness statistic.

## 4. In-distribution and OOD split semantics

ABRBench-4G+ defines OOD at the level of withheld source sets. Any trace belonging to a source set that was withheld entirely during training and in-distribution testing is designated OOD. The data states this formally as
$$
D_{\text{train}} \cup D_{\text{test}} \cap D_{\text{OOD}} = \varnothing,
$$
with the quantitative breakdown
$$
|D_{\text{train}}| = 262,\quad |D_{\text{test}}| = 114,\quad |D_{\text{OOD}}| = 101
$$
and the in-distribution decomposition $114 = 53 + 37 + 24$ and OOD decomposition $101 = 40 + 61$ [2509.10486].

This split construction is stricter than random per-trace partitioning. Because whole source sets are withheld, a policy cannot benefit from partial overlap in collection environment, measurement idiosyncrasies, or site-specific dynamics between training and OOD evaluation. A common misconception is to treat OOD as simply “harder test traces.” In ABRBench-4G+, OOD is defined structurally by provenance exclusion, not only by low throughput or high variance.

The benchmark also gives an OOD detection definition using the Mahalanobis distance over a two-dimensional feature vector formed by the trace mean and standard deviation:
$$
M(\tau) = \sqrt{(f_\tau - \mu_{\text{train}})^\top \Sigma_{\text{train}}^{-1} (f_\tau - \mu_{\text{train}})},
$$
where
$$
f_\tau = [\mu_\tau; \sigma_\tau].
$$
A threshold $\eta$ classifies $\tau$ as OOD if $M(\tau) > \eta$ [2509.10486]. This definition is not the split criterion itself; rather, it provides a feature-based mechanism for flagging OOD traces relative to the empirical training distribution.

## 5. QoE model and supported evaluation metrics

ABRBench-4G+ uses the canonical QoE formulation
$$
QoE = \sum_{n=1}^N q(R_n)
- \delta \sum_{n=1}^{N-1} |q(R_{n+1}) - q(R_n)|
- \mu \sum_{n=1}^N T_n.
$$
For the 4G+ setting,
$$
R_n \in R^{4G+} = \{1000,2500,5000,8000,16000,40000\},
$$
with $R_n$ measured in kbps, $q(R_n)=R_n$, $\delta=1$, and $\mu=40$ as the rebuffering-penalty coefficient used for 4G+ [2509.10486]. Here $T_n$ is the rebuffering time in seconds during the $n$-th chunk.

The benchmark further supports three additional metrics. Average bitrate is
$$
\bar R = \frac{1}{N}\sum_{n=1}^N R_n.
$$
Rebuffering ratio is
$$
RR = \frac{\sum_{n=1}^N T_n}{\sum_{n=1}^N L_n},
$$
where $L_n$ is the nominal chunk length of 4 seconds. Smoothness, expressed as total variation, is
$$
S = \sum_{n=1}^{N-1} \bigl| q(R_{n+1}) - q(R_n)\bigr|.
$$

These definitions make the benchmark directly comparable across controllers under a fixed bitrate ladder, fixed chunk duration, and fixed penalty weights. They also clarify the intended trade-off surface: higher bitrate contributes positively, while both bitrate switching and rebuffering reduce QoE, with rebuffering assigned a coefficient of 40 in the 4G+ regime [2509.10486].

## 6. Experimental protocol and reproducibility requirements

The benchmark prescribes a concrete training and reporting protocol. During training, the 262 training traces are shuffled, and the ABR algorithm is run end-to-end over each trace for 49 chunks of 4 seconds each [2509.10486]. Evaluation on in-distribution data is performed separately on Lumos 4G, Lumos 5G, and Solis Wi-Fi, with per-set QoE reported alongside the average rank
$$
AveRank = (r_{L4G} + r_{L5G} + r_{WiFi}) / 3.
$$
OOD evaluation is performed analogously on Ghent and Lab, with average rank computed over those two sets.

For learning-based approaches, the benchmark requires at least 10 independent models with different random seeds, and reporting of the mean and standard error of QoE [2509.10486]. It also requires granular reporting: test and OOD results should not be aggregated, so that high-bandwidth sets do not mask performance in lower-bandwidth environments. Public code and data are released through the ABRBench simulator and trace splits at `github.com/luopeng69131/ABRBench` to ensure bit-exact reproducibility of splits and QoE calculations.

The benchmark was introduced together with SABR, and the accompanying study reports that SABR achieves the best average rank compared with Pensieve, Comyco, and NetLLM across the proposed benchmarks [2509.10486]. For ABRBench-4G+, the more general significance is infrastructural rather than algorithm-specific: any new ABR controller can be compared against Pensieve, Comyco, NetLLM, and SABR on the same 4G+ distributions and on truly unseen OOD traffic patterns. This suggests that ABRBench-4G+ is intended as a common evaluation substrate for stability, generalization, and source-aware robustness in learning-based ABR.

Source: https://www.emergentmind.com/topics/abrbench-4g