BO4Mob: High-Dim Bayesian Optimization Benchmark

• BO4Mob is a benchmark framework for high-dimensional Bayesian optimization that addresses urban OD travel demand estimation using a realistic traffic simulation setup.
• It integrates open-source SUMO simulations with real PeMS sensor data, offering five scenarios that span from debugging to full urban-scale calibration.
• The framework challenges optimization methods with expensive, stochastic, and non-differentiable evaluations while ensuring most variables are influential.

BO4Mob is a benchmark framework for high-dimensional Bayesian optimization (BO) centered on origin–destination (OD) travel demand estimation in large urban road networks. Introduced in 2025, it couples open-source, high-fidelity SUMO traffic simulation with real PeMS sensor data from the San Jose, California freeway system, and exposes five benchmark scenarios whose input dimensionality ranges from $3$ to $10{,}100$ continuous decision variables. The framework targets a class of inverse optimization problems in which each objective evaluation is computationally expensive, stochastic, and non-differentiable, making BO4Mob simultaneously a transportation benchmark, a systems benchmark, and a stress test for scalable black-box optimization methods (Ryu et al., 21 Oct 2025).

1. Benchmark identity and motivation

BO4Mob, expanded in the source material as “Bayesian Optimization for Mobility,” is both a benchmark suite and a software framework. As a benchmark suite, it contains five OD-estimation problems defined on subnetworks of the San Jose freeway system, with dimensionality ranging from $3$ to $10{,}100$. As a framework, it connects BO libraries implemented with BoTorch to a SUMO-based traffic simulation and real PeMS sensor data. Its intended use is the evaluation of high-dimensional black-box optimization algorithms under expensive evaluations, stochastic and non-differentiable dynamics, noisy and under-determined objectives, and realistic network and data constraints (Ryu et al., 21 Oct 2025).

The benchmark addresses a specific gap between two research traditions. In BO, high-dimensional benchmarks have often been either cheap analytic functions such as Ackley, Rastrigin, or BBOB, usually noiseless and unconstrained, or hyperparameter optimization problems that are often below $10$ dimensions or only nominally high-dimensional because very few variables are influential. In transportation and digital-twin research, by contrast, OD calibration is already recognized as high-dimensional, stochastic, computationally heavy, and under-determined, yet there had been no standardized, open benchmark linking such realistic instances to the BO community. BO4Mob fills that gap by releasing SUMO networks, PeMS data, code, and baseline methods drawn from both transportation and BO (Ryu et al., 21 Oct 2025).

A central point of the benchmark is that it is not merely large in ambient dimension. The authors report that most variables matter, verified via sensitivity analysis, and therefore the benchmark is intended to avoid the “fake” high-dimensional regime in which only a small inactive subspace drives performance. This gives BO4Mob particular relevance for work on scalable surrogates, trust-region BO, sparsity assumptions, simulation-based calibration, and data-driven urban mobility digital twins (Ryu et al., 21 Oct 2025).

2. Mathematical formulation as OD demand estimation

BO4Mob formulates OD demand estimation as a continuous black-box optimization problem over a box-constrained feasible set. A metropolitan freeway network is partitioned into traffic analysis zones (TAZes), and each feasible origin–destination zone pair defines a continuous decision variable representing expected trips over a time window. Let $\tau$ denote the number of TAZes, let $$\mathcal{P}\subset\{(o,d):o,d\in\{1,\dots,\tau\},\,o\neq d\}$$ be the set of feasible OD pairs after connectivity filtering, and let $d=|\mathcal{P}|$ be the number of OD variables. The decision vector is $\mathbf{x}\in\mathbb{R}^d$, where $x_j$ is the demand for OD pair $10{,}100$0 (Ryu et al., 21 Oct 2025).

The simulator maps $10{,}100$1 to link-level traffic statistics. With $10{,}100$2 the index set of links equipped with sensors, $10{,}100$3 the ground-truth traffic statistic on link $10{,}100$4 from PeMS data, and $10{,}100$5 the random simulator output on link $10{,}100$6, the canonical objective is a least-squares fit to ground-truth link statistics: $10{,}100$7 with box constraints

$10{,}100$8

OD upper bounds are typically between $10{,}100$9 and $3$0, or $3$1 for the smallest instance. The problem is explicitly characterized as an inverse optimization problem: OD demands are adjusted so that the simulated network reproduces observed traffic (Ryu et al., 21 Oct 2025).

For cross-network evaluation, BO4Mob also defines the normalized root mean squared error (NRMSE) between simulated and ground-truth traffic counts: $3$2 Here $3$3 is the simulated average count on link $3$4 for OD vector $3$5, used as an estimator of $3$6 (Ryu et al., 21 Oct 2025).

The difficulty of the search problem comes from several simultaneous properties. Dimensionality ranges from $3$7 to $3$8; each evaluation requires one or more SUMO simulations; runtime ranges from below one second to approximately $3$9 seconds; SUMO’s mesoscopic model introduces stochasticity and non-differentiability; and the objective is under-determined because many distinct OD matrices can induce similar aggregated sensor counts. This under-determination implies flat or multimodal structure along multiple directions in the search space (Ryu et al., 21 Oct 2025).

3. Scenario construction and simulation regime

BO4Mob provides five benchmark scenarios of increasing scale and complexity, all derived from the San Jose freeway system. These are designed to span debugging-scale, corridor-scale, junction-scale, regional, and full urban-scale calibration tasks (Ryu et al., 21 Oct 2025).

Network	Scale summary	Avg. sim time
Simple Ramp	10 nodes, 10 links, 3 TAZes, 3 OD pairs, 3 sensors	0.8 s
One-Way Corridor	66 nodes, 68 links, 7 TAZes, 21 OD pairs, 5 sensors	5.9 s
Junction	137 nodes, 152 links, 9 TAZes, 44 OD pairs, 18 sensors	11.9 s
Small Region	251 nodes, 270 links, 16 TAZes, 151 OD pairs, 27 sensors	82.7 s
Full Region	1,977 nodes, 2,173 links, 101 TAZes, 10,100 OD pairs, 219 sensors	40,099 s

The Simple Ramp instance is the only deterministic case. It is deliberately tiny, but all three OD variables are reported to be influential, making it useful for debugging BO pipelines and hyperparameter settings. One-Way Corridor introduces overlapping OD flows and the onset of congestion propagation while keeping evaluation cost modest. Junction adds freeway intersections, ramps, and direct connectors, so merging and diverging flows induce localized nonlinearities and sharp congestion transitions. Small Region extends these interactions over multiple corridors and junctions and remains computationally manageable at approximately $10{,}100$0 seconds per evaluation, while still being genuinely high-dimensional. Full Region covers the full San Jose freeway system and becomes an urban-scale, ultra-high-dimensional problem with evaluation time above $10{,}100$1 hours per call (Ryu et al., 21 Oct 2025).

All scenarios use SUMO in mesoscopic mode. In this regime, individual trips are represented, but car-following and lane-changing behavior are aggregated. The resulting dynamics remain nonlinear and stochastic through congestion, queue spillback, merging, diverging, departures, and route-choice realizations. For most networks, simulations are configured over a one-hour OD generation window, while sensor observations are taken over overlapping periods such as sensor time $10{,}100$2–$10{,}100$3 s and OD time $10{,}100$4–$10{,}100$5 s. Mesoscopic mode is chosen to keep runtimes tractable while preserving realistic traffic counts (Ryu et al., 21 Oct 2025).

4. Data pipeline, observability, and evaluation protocol

A BO4Mob evaluation begins with trip generation from a candidate OD vector $10{,}100$6. For each OD pair and time window, trips are generated from the corresponding origin and destination TAZes using SUMO’s od2trips tool and routing. Only OD pairs with at least one feasible path, validated by repeated od2trips sampling, are included in the benchmark. OD bounds such as $10{,}100$7–$10{,}100$8 are enforced at this stage (Ryu et al., 21 Oct 2025).

SUMO then runs in mesoscopic mode over the specified simulation interval, for example $10{,}100$9–$10$0 seconds. In the published benchmark experiments, each run uses a fixed random seed in order to isolate algorithmic variability. This is an important technical nuance: the underlying simulator remains stochastic in principle, but the main experimental configuration makes the objective effectively deterministic for a fixed candidate $10$1. A common misconception is therefore to treat BO4Mob as either fully deterministic or fully noisy; the benchmark is better understood as a stochastic simulator used under a fixed-seed evaluation protocol, with explicit room for future replicated or multi-seed studies (Ryu et al., 21 Oct 2025).

Ground-truth data are obtained from PeMS mainline loop detectors at five-minute resolution. Each detector is matched to the nearest freeway link with consistent direction. Sensors are then filtered for physical consistency, including upstream/downstream conservation around on- and off-ramps, and for ambiguity due to TAZ granularity, with sensors between multiple ramps from the same TAZ excluded. For each retained sensor-link pair, the ground-truth statistic is computed for a chosen day and time window, such as a representative day in October 2022 (Ryu et al., 21 Oct 2025).

From the simulation output, link counts and optionally speeds are aggregated over the sensor time window and inserted into the least-squares objective or the NRMSE metric. In practice, the benchmark methods optimize NRMSE and report percentage improvement relative to a shared initial pool. Improvement is defined as

$10$2

The computational setup uses a dual-CPU, $10$3-core AMD server with $10$4 TB RAM; SUMO is CPU-only; and up to six simulations are run in parallel with Python multiprocessing. Even under this configuration, Full Region remains sufficiently expensive that experiments there use only $10$5 epochs with batch size $10$6 (Ryu et al., 21 Oct 2025).

Observability is a recurrent theme. Some OD pairs may be unobservable because their flows never pass through sensor-equipped links, which complicates identifiability. An auxiliary experiment on One-Way Corridor removed four such unobservable OD pairs, reducing dimensionality from $10$7 to $10$8, and BO methods then obtained consistently better final performance, with TuRBO reaching best NRMSE down to approximately $10$9 and roughly $\tau$0 improvement. This suggests that latent or weakly observed demand components materially affect high-dimensional BO behavior in transportation calibration (Ryu et al., 21 Oct 2025).

5. Optimization methods and empirical findings

BO4Mob evaluates five optimization methods: Random Search, Simultaneous Perturbation Stochastic Approximation (SPSA), Vanilla Bayesian Optimization, SAASBO, and TuRBO. All BO variants are implemented in BoTorch with minor adaptations (Ryu et al., 21 Oct 2025).

Vanilla BO uses a Gaussian Process surrogate in the full $\tau$1-dimensional space with a Matérn $\tau$2 kernel and ARD lengthscales constrained to $\tau$3, Gaussian noise variance inferred in $\tau$4, inputs normalized to $\tau$5, outputs standardized, and multi-point Log Expected Improvement (qLogEI) as the acquisition function. SAASBO uses BoTorch’s SaasFullyBayesianSingleTaskGP, again with a Matérn $\tau$6 kernel, but imposes a hierarchical horseshoe prior on inverse lengthscales to encourage sparsity; posterior inference uses NUTS with $\tau$7 warm-up steps, $\tau$8 posterior samples, and thinning $\tau$9, while acquisition is standard qEI. TuRBO uses a local adaptive trust region in normalized space, initialized with side lengths $$\mathcal{P}\subset\{(o,d):o,d\in\{1,\dots,\tau\},\,o\neq d\}$$0, shaped anisotropically by GP lengthscales, expanded after three consecutive improvements, contracted when a failure counter reaches $$\mathcal{P}\subset\{(o,d):o,d\in\{1,\dots,\tau\},\,o\neq d\}$$1, restarted when the region length falls below $$\mathcal{P}\subset\{(o,d):o,d\in\{1,\dots,\tau\},\,o\neq d\}$$2, and populated through Sobol candidates with masking, where each dimension is perturbed with probability $$\mathcal{P}\subset\{(o,d):o,d\in\{1,\dots,\tau\},\,o\neq d\}$$3; final candidate choice uses Thompson sampling (Ryu et al., 21 Oct 2025).

SPSA serves as a transportation baseline. At iteration $$\mathcal{P}\subset\{(o,d):o,d\in\{1,\dots,\tau\},\,o\neq d\}$$4, it samples $$\mathcal{P}\subset\{(o,d):o,d\in\{1,\dots,\tau\},\,o\neq d\}$$5, evaluates the function at $$\mathcal{P}\subset\{(o,d):o,d\in\{1,\dots,\tau\},\,o\neq d\}$$6, estimates a gradient from the symmetric finite-difference expression, and updates with decaying step sizes using the classical exponents $$\mathcal{P}\subset\{(o,d):o,d\in\{1,\dots,\tau\},\,o\neq d\}$$7 and $$\mathcal{P}\subset\{(o,d):o,d\in\{1,\dots,\tau\},\,o\neq d\}$$8. Its appeal is that it requires only two evaluations per iteration regardless of $$\mathcal{P}\subset\{(o,d):o,d\in\{1,\dots,\tau\},\,o\neq d\}$$9, but BO4Mob reports that in these landscapes the resulting gradient estimates are often too noisy and unstable (Ryu et al., 21 Oct 2025).

Across Simple Ramp, One-Way Corridor, Junction, and Small Region, all BO methods significantly outperform Random Search and SPSA in both final NRMSE and convergence speed. TuRBO is often best: on Simple Ramp it achieves NRMSE approximately $d=|\mathcal{P}|$0 with about $d=|\mathcal{P}|$1 improvement, and on Small Region it reaches NRMSE approximately $d=|\mathcal{P}|$2 with about $d=|\mathcal{P}|$3 improvement. On Junction, TuRBO and Vanilla BO are nearly tied at approximately $d=|\mathcal{P}|$4–$d=|\mathcal{P}|$5 NRMSE and about $d=|\mathcal{P}|$6 improvement. SAASBO performs strongly at moderate dimensionality, for example on One-Way Corridor with best NRMSE about $d=|\mathcal{P}|$7 and around $d=|\mathcal{P}|$8 improvement, but it underperforms TuRBO and sometimes Vanilla BO as network complexity grows. A notable empirical result is that Vanilla BO remains surprisingly competitive even at $d=|\mathcal{P}|$9 and $\mathbf{x}\in\mathbb{R}^d$0 dimensions, whereas the sparsity assumptions underlying SAASBO can misalign with BO4Mob’s many-influential-variables structure (Ryu et al., 21 Oct 2025).

As network size increases, the problem becomes intrinsically harder. The best NRMSE found in the initial random pools rises from $\mathbf{x}\in\mathbb{R}^d$1 on Simple Ramp to $\mathbf{x}\in\mathbb{R}^d$2 on Small Region, and residual error remains higher after optimization. For the $\mathbf{x}\in\mathbb{R}^d$3-dimensional Full Region, no method improves over the initial solution under the heavily truncated experimental budget; all methods remain around NRMSE $\mathbf{x}\in\mathbb{R}^d$4–$\mathbf{x}\in\mathbb{R}^d$5. The dominant bottleneck is computational, not merely statistical, because each evaluation costs approximately $\mathbf{x}\in\mathbb{R}^d$6 hours (Ryu et al., 21 Oct 2025).

Additional analyses reinforce the benchmark’s intended difficulty profile. Convergence curves show roughly exponential-like early decline in NRMSE followed by plateauing. Fit plots of simulated versus ground-truth counts show TuRBO producing points tightly clustered around the diagonal, while Vanilla BO and SAASBO exhibit systematic underestimation at higher counts and SPSA deviates more strongly. A kernel comparison among Matérn $\mathbf{x}\in\mathbb{R}^d$7, Matérn $\mathbf{x}\in\mathbb{R}^d$8, and RBF yields only modest differences, suggesting that trust-region logic and broader structural assumptions matter more than fine kernel choice (Ryu et al., 21 Oct 2025).

6. High-dimensional significance, applications, limitations, and related mobility BO

BO4Mob is designed to be genuinely high-dimensional in three senses: it reaches very large ambient dimension, most variables are influential, and the variables are not abstract coordinates but OD flows linked by conservation, capacity, and route-connectivity structure. In Small Region, for example, the authors report an average of $\mathbf{x}\in\mathbb{R}^d$9 dominant plus $x_j$0 secondary variables out of $x_j$1; in Junction, about $x_j$2 dominant plus $x_j$3 secondary out of $x_j$4. This distinguishes BO4Mob from high-dimensional benchmarks whose behavior is largely controlled by a small active subset (Ryu et al., 21 Oct 2025).

The framework is practically motivated by calibration of digital twins for urban mobility. It can be used to calibrate detailed freeway simulators from limited sensor data, study under-determination and sensor placement, experiment with counts-versus-speeds metrics, and test new scalable BO algorithms, including random embeddings, deep BO, or physics-informed priors. The implementation is modular and configurable through command-line arguments, with network choice, model, kernel, simulation date and time, and metric all adjustable without code changes (Ryu et al., 21 Oct 2025).

The benchmark also has clear limitations. Its geographic scope is restricted to five fixed subnetworks from a single region; users cannot arbitrarily re-extract or reshape networks within the distributed benchmark. The Full Region case is so expensive that experiments there are illustrative rather than exhaustive. The simulation stack is fixed to SUMO in mesoscopic mode with specific behavior settings, and the published experiments rely on single-seed evaluations, so the empirical objective is nearly deterministic per configuration even though the underlying simulator is stochastic. These choices make the benchmark practical and reproducible, but they also delimit the class of noise models and simulator behaviors currently represented (Ryu et al., 21 Oct 2025).

Future work proposed by the authors includes uncertainty quantification for under-determined OD estimation, physics-informed BO with traffic-theoretic kernels or hybrid analytical–simulation meta-models, improved initialization strategies, sensor placement optimization, multi-objective and multi-class OD calibration, and methods capable of handling the Full Region regime under extremely tight evaluation budgets through multi-fidelity approximations, topology-informed dimensionality reduction, or more advanced trust-region and batch-selection strategies (Ryu et al., 21 Oct 2025).

In the broader mobility-optimization literature, BO4Mob is related to earlier work on Bayesian optimization for wireless mobility management, but the problem class is markedly different. “Towards Mobility Management with Multi-Objective Bayesian Optimization” studies a one-dimensional handover-threshold tuning problem in 3GPP indoor factory scenarios, using MOBO with qEHVI to trade off counts of handovers that are “too early” and “too late” over a discrete threshold domain $x_j$5 dB (Rodrigues et al., 2023). A plausible implication is that the term “BO4Mob” now points to at least two distinct strands of mobility-oriented BO research: low-dimensional multi-objective control of radio handover parameters, and ultra-high-dimensional inverse optimization for urban traffic-demand calibration. The 2025 benchmark formalizes the latter strand as an open, reproducible testbed for high-dimensional BO (Ryu et al., 21 Oct 2025).