ABRBench-3G Benchmark

• ABRBench-3G is a benchmark for adaptive bitrate research that standardizes network traces, video content, and evaluation protocols.
• It organizes data into training, test, and dedicated out-of-distribution sets to measure algorithm robustness.
• The benchmark integrates a fixed bitrate ladder and rank-based QoE metrics to enable fair, repeatable comparisons.

Searching arXiv for the benchmark and associated ABR paper to ground the article in current literature. ABRBench-3G is a benchmark for Adaptive Bitrate (ABR) research introduced alongside the SABR framework to support training, testing, and rigorous evaluation under wide and realistic network conditions (Luo et al., 30 Aug 2025). It is designed to address a specific limitation in prior learning-based ABR work: reliance on limited network trace sets during training and the resulting poor generalization in out-of-distribution (OOD) scenarios. Within the benchmark, traces are organized into training, test, and OOD sets; each benchmark also contains video content, enabling standardized end-to-end evaluation of ABR control policies on both seen and unseen network conditions (Luo et al., 30 Aug 2025).

1. Scope, purpose, and research setting

ABRBench-3G is intended for the development and fair comparison of ABR algorithms that must perform robustly not only on distributions represented during training, but also on network conditions excluded from training and standard testing (Luo et al., 30 Aug 2025). Its stated purpose is to improve and measure robustness and generalization under wide and realistic network conditions.

The benchmark is introduced in a setting where learning-based ABR methods are assessed for Quality of Experience (QoE) under heterogeneous bandwidth regimes. In that context, ABRBench-3G functions as an evaluation substrate rather than as a single algorithmic contribution. It provides network traces, video content, prescribed data splits, and an evaluation protocol that explicitly separates in-distribution assessment from OOD generalization assessment (Luo et al., 30 Aug 2025).

A central feature is the dedicated OOD split. The benchmark’s design makes the OOD set unavailable to training and standard test procedures, so performance on that split serves as a direct measure of generalization to unseen network conditions. This distinguishes ABRBench-3G from evaluations that use only train/test partitions drawn from the same effective distribution (Luo et al., 30 Aug 2025).

2. Benchmark organization and data splits

ABRBench-3G contains both network traces and video content. Its traces are divided into three sets: a training set for model fitting, a test set for standard performance evaluation on distributions similar to the training set, and an OOD set for evaluation on distributions not included in training or test (Luo et al., 30 Aug 2025).

The trace statistics reported for ABRBench-3G are as follows (Luo et al., 30 Aug 2025):

Split / trace set	Count	Range (Mbps)
Training / same as test	1828	0.00–45.38
Test / FCC-16	69	0.00–8.95
Test / FCC-18	100	0.00–41.76
Test / Oboe	100	0.16–9.01
Test / Puffer-21	100	0.00–25.14
Test / Puffer-22	100	0.00–9.29
OOD / HSR	34	0.00–44.68

The benchmark description states that the training set consists of 1828 traces with bandwidths ranging from 0.00 to 45.38 Mbps, and that “the training set traces are the same as the test traces, except for OOD” (Luo et al., 30 Aug 2025). The same source also specifies that traces are reorganized and curated from the original sets and that each trace set is assigned proportionally for training and testing, with FCC-18 split as 75% for training and 25% for testing as an example. Read together, these statements indicate a curated benchmark-level reorganization rather than simple reuse of an untouched source corpus.

The OOD split is the HSR set, with 34 traces spanning 0.00–44.68 Mbps. These traces are never included in training or standard test evaluation and are reserved purely for generalization assessment (Luo et al., 30 Aug 2025).

3. Trace provenance, coverage, and curation

ABRBench-3G is assembled from multiple public trace sources in order to provide broad bandwidth coverage and realistic diversity (Luo et al., 30 Aug 2025). The trace sets named in the benchmark are FCC-16, FCC-18, Oboe, Puffer-21, Puffer-22, and HSR. The benchmark description identifies these as widely recognized academic or public datasets and states that they provide realistic, geographically diverse, and temporally varied trace samples.

The benchmark’s curation procedure reorganizes traces from the original datasets to fit benchmark requirements. Each trace set is assigned proportionally for training and testing, while the OOD set is composed of trace sets that are not split or reused elsewhere. This strict isolation is the mechanism by which the benchmark enforces OOD integrity (Luo et al., 30 Aug 2025).

This structure matters methodologically because the benchmark evaluates performance per trace set rather than by pooling all traces into a single aggregate score. The stated rationale is to prevent high-bandwidth trace sets from skewing overall results and to ensure fair assessment across bandwidth regimes (Luo et al., 30 Aug 2025). In practical terms, ABRBench-3G therefore treats heterogeneity across trace families as a first-class experimental variable rather than as incidental noise.

4. Video asset, bitrate ladder, and execution protocol

ABRBench-3G standardizes not only the network environment but also the streamed content. The video used is Envivio-Dash3, and the available bitrate set is

$$R^{3G} = \{300, 750, 1200, 1850, 2850, 4300\} \text{ kbps}.$$

Each experiment uses a fixed video with 49 chunks, and each chunk has duration 4 seconds (Luo et al., 30 Aug 2025).

The training protocol specifies that models are trained using traces from the training set and that training data is randomly shuffled to prevent overfitting to specifics of temporal ordering (Luo et al., 30 Aug 2025). The testing protocol is trace-driven: for each trace, the full video is streamed under the network conditions from that trace. Evaluation is performed per trace set in both the test and OOD partitions. The OOD set is used only for out-of-distribution generalization tests (Luo et al., 30 Aug 2025).

A common misunderstanding would be to treat ABRBench-3G as only a trace repository. The benchmark definition explicitly includes video content, a fixed bitrate ladder, chunk structure, and execution rules for training and testing. In that sense, it is a benchmarked evaluation environment rather than merely a collection of bandwidth traces (Luo et al., 30 Aug 2025).

5. QoE objective and ranking methodology

ABRBench-3G uses a QoE metric defined as

$$\text{QoE} = \sum_{n=1}^N q(R_n) - \delta \sum_{n=1}^{N-1} \left| q(R_{n+1}) - q(R_n) \right| - \mu \sum_{n=1}^N T_n.$$

For ABRBench-3G, the benchmark specifies the following quantities: $N = 49$, $R_n \in R^{3G}$, $q(R_n) = R_n$, $\delta = 1$, and $\mu = 4.3$ (Luo et al., 30 Aug 2025). Here, $T_n$ denotes the rebuffering time at step $n$.

Beyond raw QoE, the benchmark adopts a rank-based comparison procedure. Each algorithm is ranked per trace set, and the reported aggregate is the average rank

$$\text{Ave Rank}(i) = \frac{1}{M} \sum_{j=1}^M r_{i, j},$$

where $$\text{QoE} = \sum_{n=1}^N q(R_n) - \delta \sum_{n=1}^{N-1} \left| q(R_{n+1}) - q(R_n) \right| - \mu \sum_{n=1}^N T_n.$$0 is the rank of algorithm $$\text{QoE} = \sum_{n=1}^N q(R_n) - \delta \sum_{n=1}^{N-1} \left| q(R_{n+1}) - q(R_n) \right| - \mu \sum_{n=1}^N T_n.$$1 on trace set $$\text{QoE} = \sum_{n=1}^N q(R_n) - \delta \sum_{n=1}^{N-1} \left| q(R_{n+1}) - q(R_n) \right| - \mu \sum_{n=1}^N T_n.$$2, and $$\text{QoE} = \sum_{n=1}^N q(R_n) - \delta \sum_{n=1}^{N-1} \left| q(R_{n+1}) - q(R_n) \right| - \mu \sum_{n=1}^N T_n.$$3 is the number of trace sets in the benchmark (Luo et al., 30 Aug 2025). Lower average rank is better.

The benchmark also prescribes repeated execution to reduce variance. Each method is run 10 times on each trace and the results are averaged. For learning-based approaches, including SABR, 10 separate models are trained and averaged for reporting (Luo et al., 30 Aug 2025). This protocol makes ABRBench-3G a repeated-trials benchmark rather than a single-seed leaderboard.

6. OOD evaluation and role in ABR generalization research

The most distinctive research role of ABRBench-3G is its support for strict OOD evaluation (Luo et al., 30 Aug 2025). Its HSR set is entirely excluded from training and from the standard test split, so performance on HSR measures the ability of an ABR method to handle unseen network conditions without leakage from model development.

In the SABR study, algorithms including SABR and several baselines are evaluated on HSR, and average rank is used to determine generalization performance on ABRBench-3G OOD (Luo et al., 30 Aug 2025). The benchmark is therefore not only a source of in-distribution comparisons but also an instrument for distinguishing methods that are robust to distribution shift from those that are primarily adapted to the observed training/test regime.

This emphasis reflects the motivating critique of earlier learning-based ABR systems: most of them rely on limited network trace sets during training and consequently generalize poorly in OOD scenarios (Luo et al., 30 Aug 2025). ABRBench-3G operationalizes that critique into a reproducible benchmark design by reserving a dedicated unseen trace family for evaluation.

7. Significance, release status, and relation to SABR

ABRBench-3G is publicly released for the research community at https://github.com/luopeng69131/ABRBench (Luo et al., 30 Aug 2025). Its stated significance lies in providing a standardized foundation for consistent and fair comparison of ABR solutions, especially with respect to the trade-off between in-distribution performance and out-of-distribution robustness.

The benchmark is paired in the same work with the SABR training framework, which combines behavior cloning pretraining with reinforcement learning fine-tuning. Within the reported experiments, SABR achieves the best average rank compared with Pensieve, Comyco, and NetLLM across the proposed benchmarks, including the OOD evaluations built around ABRBench-3G (Luo et al., 30 Aug 2025). That empirical result pertains to SABR rather than to the benchmark itself, but it also illustrates the benchmark’s intended use: evaluating whether training strategies produce stable learning across wide distributions and improved generalization to unseen network conditions.

A plausible implication is that ABRBench-3G is best understood as an infrastructure contribution to ABR methodology. Its value lies in codifying wide-coverage training traces, per-trace-set evaluation, fixed content and bitrate conditions, and a strict OOD protocol into a single reproducible benchmark. Within ABR research, that combination makes it a reference setting for studying robustness and generalization rather than only average QoE on familiar traces (Luo et al., 30 Aug 2025).