Closed-Loop Benchmarking Suite

• Closed-loop benchmarking suites are evaluation platforms that integrate real-time feedback between algorithmic actions and simulated or physical environments for causally relevant performance assessment.
• They employ high-fidelity simulators, detailed sensor streams, and modular APIs to ensure reproducible and fair comparisons across robotics, autonomous driving, and optimization tasks.
• They quantify dynamic metrics such as trajectory error, latency, and safety parameters to identify bottlenecks and inform future research enhancements.

A closed-loop benchmarking suite is a structured evaluation platform that measures the real-time performance of algorithms, typically in robotics, optimization, or autonomous systems, under conditions where the system’s actions directly influence subsequent observations and outcomes. Unlike open-loop analysis, which assesses components in isolation without feedback, closed-loop benchmarking explicitly quantifies performance—including feedback effects—across full perception-to-action pipelines or black-box optimization loops, with the aim of robust, repeatable, and causally relevant assessment.

1. Architectural Principles of Closed-Loop Benchmarking

The defining aspect of a closed-loop benchmarking suite is the inclusion of real-time feedback between algorithmic modules and the simulated or physical environment. For robotic and control systems, the architecture generally comprises:

• Simulator Environment: High-fidelity simulator (e.g., Gazebo/ROS for SLAM (Zhao et al., 2020), CARLA for autonomous driving (Gerstenecker et al., 9 Apr 2026), PyBullet/MuJoCo/PhysX for manipulation (Jiang et al., 23 Oct 2025)) providing continuous physics, sensor emulation, and scenario orchestration.
• Sensing Stream: Sensor outputs such as stereo/RGB images, LiDAR point clouds, inertial measurements, and proprioceptive data, often with explicit noise and timestamping to mirror real-world uncertainty.
• Estimator or Policy Module: The algorithm under test (e.g., VI-SLAM, E2E-AD, RL/IL policy, black-box optimizer) receives streaming sensory input and outputs actions in real time.
• Controller/Actuator Layer: Translates high-level actions or planned trajectories into motor commands or function evaluations, which influence environment state or task progress.
• Performance Logger: Persistent storage of time-stamped system state, estimator output, reference trajectory, and all performance metrics for offline analysis and reproducibility.

In suites like Bencher, the closed-loop is instantiated as an optimizer loop where each candidate $x_t$ triggers a full benchmark execution and receives $y_t = f(x_t)$ via gRPC, isolating the evaluation from the optimizer logic (Papenmeier et al., 27 May 2025).

2. Metrics and Performance Evaluation

Closed-loop benchmarking suites quantify multiple aspects of system performance, capturing both steady-state and dynamic behavior.

• Trajectory Error Metrics: For SLAM and AD tasks, typical closed-loop metrics include instantaneous error $e(t)$, root-mean-square error (RMSE), and maximum deviation $E_{\text{max}}$, computed between executed and desired trajectories $g(t)$ vs. $g^*(t)$ (Zhao et al., 2020, Jia et al., 2024).
• Success Rate and Driving Score: In autonomous driving, success rate (SR) is the fraction of routes completed without failure (collision or timeout). The driving score (DS) exponentially penalizes infractions:

$$DS = \frac{1}{n_{total}}\sum_{i=1}^{n_{total}} \left(\text{RouteCompletion}_i \times \prod_{j=1}^{n_{i,inf}} p_{i,j}\right)$$

where $p_{i,j}$ are infraction penalty factors (Jia et al., 2024, Gerstenecker et al., 9 Apr 2026).

• Latency and Drift: Time delay (latency) between sensor acquisition and estimator output, and estimator drift (average error per meter) are explicitly disentangled. Latency impacts closed-loop tracking directly, with performance degrading as $\propto\tau^2$ (Zhao et al., 2020).
• Collision Rate and Safety Metrics: Closed-loop collision rates, minimum time-to-collision (TTC), and impact speed for AD and manipulation tasks are directly measured per episode (Ljungbergh et al., 2024, Caesar et al., 2021).
• Comfort and Feasibility: Aggregate jerk, longitudinal/lateral deviation, and action feasibility (e.g., violation of actuation or dynamics limits) provide secondary axes of performance (Caesar et al., 2021).
• Generalization Gap: Delta success rate ($\Delta SR$) across in-distribution and OOD scenario pairs formally measures distributional robustness (Gerstenecker et al., 9 Apr 2026).

3. Representative Benchmarking Suites and Their Protocols

Several state-of-the-art closed-loop benchmarking frameworks exist across robotics, optimization, and autonomous driving domains:

Suite	Domain	Core Feature
Bencher (Papenmeier et al., 27 May 2025)	Black-box optimization	Containerized, RPC-based, multi-benchmark isolation
Fail2Drive (Gerstenecker et al., 9 Apr 2026)	E2E autonomous driving	Paired-route OOD generalization, scenario taxonomy
Bench2Drive (Jia et al., 2024)	E2E autonomous driving	44-scenario, multi-ability, fair training split
NeuroNCAP (Ljungbergh et al., 2024)	Sensor-realistic AD safety	NeRF-based, photorealistic, safety-critical scenarios
GSWorld (Jiang et al., 23 Oct 2025)	Robotic manipulation	3DGS rendering, sim2real, reproducible MDP
nuPlan (Caesar et al., 2021)	ML-based planning	Large-scale real driving data, closed/open-loop, multi-metric

Distinctives include strict benchmark/environment isolation (Bencher), sensor-realistic rendering and actor reconfigurability (NeuroNCAP, GSWorld), comprehensive scenario taxonomies and OOD paired comparisons (Fail2Drive), MDP-structured closed-loop policy evaluation (GSWorld), and open-source APIs supporting integration and extension.

4. Experimental Design and Scenario Taxonomies

Scenario diversity and OOD robustness assessment are central to contemporary closed-loop suites:

• Paired In-Distribution/OOD Matching: Fail2Drive constructs 100 paired routes in CARLA’s Town13, matching each OOD scenario (single distributional shift) with an in-distribution twin, isolating the causal impact of the shift (Gerstenecker et al., 9 Apr 2026).
• Scenario Decomposition: Bench2Drive decomposes assessments by 44 atomic interactive scenarios (cut-in, overtaking, detour, emergency brake, etc.) with fine-grained per-skill metrics (Jia et al., 2024).
• Safety-Critical Scenarios: NeuroNCAP defines stationary, frontal drifting, and side-crossing vehicle scenarios, parameterized over TTC, velocity, and actor positioning, to probe real-world AV safety gaps (Ljungbergh et al., 2024).
• Manipulation Task Set: GSWorld benchmarks 7 diverse tasks spanning single-arm and bimanual manipulation with reproducible object initializations via GSDF assets (Jiang et al., 23 Oct 2025).
• Simulation and Logging: Suites specify deterministic seeds, short scenario-focused episodes, standardized sensor/actuator models, and unified training/test splits to enforce comparability and reproducibility (Jia et al., 2024, Gerstenecker et al., 9 Apr 2026).

5. Implementation, Isolation, and Reproducibility

Closed-loop suites emphasize reproducibility and modular isolation:

• Benchmark Isolation: Bencher encapsulates each benchmark in a Poetry-managed Python environment and exposes the API over gRPC; there is no dependency cross-talk, enabling integration of benchmarks with incompatible requirements (Papenmeier et al., 27 May 2025).
• Containerization: Standardization via Docker/Singularity for seamless cloud and HPC deployment; rebuilding the container images allows roll-forward of benchmarks, reproducibility, and CI-driven integration (Papenmeier et al., 27 May 2025, Jiang et al., 23 Oct 2025).
• Scenario Packaging: GSDF files (GSWorld), scenario scripts (NeuroNCAP), and benchmark registries (Bencher) allow new tasks to be added with no cross-interference or manual API wiring (Jiang et al., 23 Oct 2025, Ljungbergh et al., 2024).
• API Access: Lightweight Python or CLI interfaces enable users to conduct closed-loop runs, extract metrics, or inject custom policies for evaluation with minimal effort (Papenmeier et al., 27 May 2025, Jiang et al., 23 Oct 2025, Ljungbergh et al., 2024, Caesar et al., 2021).
• Alignment and Logging: Offline log alignment by timestamp, fixed evaluation scripts, and public release of ground truth annotations are standard for rigorous post-hoc analysis (Zhao et al., 2020, Jia et al., 2024).

6. Limitations, Failure Modes, and Future Directions

Current closed-loop benchmarking approaches are characterized by several open limitations:

• Latency and Feedback Loops: Estimator/tracker latency imposes a strict performance bottleneck in real-time control; for VI-SLAM, latency over 30 ms leads to significant closed-loop degradation even with low drift (Zhao et al., 2020).
• Domain Coverage: nuPlan, for instance, is geographically limited to four cities and captures limited long-tail events (emergency vehicles, rare weathers) (Caesar et al., 2021). CARLA-based suites have finite sensor and traffic model realism.
• Metric Selection: Composite metrics (e.g., DS, HM) may induce degenerate policies (e.g., over-braking in Bench2Drive) or mask specific skill deficits. Suggested refinements include decomposing safety, comfort, and efficiency (Jia et al., 2024).
• Scenario Generalization: Even strong planners exhibit >50% performance drops in behavioral OOD scenarios; OOD robustness, especially to compositional appearance or semantic shifts, remains unsolved (Gerstenecker et al., 9 Apr 2026).
• Evaluation Scope: Bencher handles only unconstrained, single-objective problems; multi-objective, multi-fidelity, and graph-structured inputs are cited as explicit future directions (Papenmeier et al., 27 May 2025).
• Simulator Fidelity: Some frameworks lack novel-view sensor synthesis after agent drift (nuPlan), potentially leading to open-/closed-loop discrepancy for strongly interactive methods (Caesar et al., 2021).

Planned enhancements include hardware-in-the-loop validation, orientation/error metric extensions (e.g., orientation ATE, maximum yaw error), expanded scenario libraries, improved sim-to-real realism, and high-efficiency in-process communication APIs (Zhao et al., 2020, Papenmeier et al., 27 May 2025, Jia et al., 2024).

7. Significance and Impact on the Research Landscape

Closed-loop benchmarking suites systematically advance the field by:

• Enabling robust, fair comparison of algorithms under feedback—surfacing performance bottlenecks (e.g., estimator latency, failure to generalize OOD, sim2real transfer gaps) not revealed by open-loop metrics (Zhao et al., 2020, Jia et al., 2024, Jiang et al., 23 Oct 2025).
• Driving explicit scenario taxonomy and reproducibility standards, which support reproducible, meaningful progress in safety-critical domains.
• Creating modular, extensible APIs, which foster integration by third-party methods and facilitate rapid testbed evolution.
• Highlighting failure modes—sensor/estimator latency, policy overfitting, poor generalization—and thus sharpening future research focus by specifying quantifiable targets for improvement.

By formalizing the impact of feedback, scenario diversity, and real-world uncertainty, closed-loop benchmarking suites are now essential infrastructure in comparative algorithm evaluation across autonomous driving, robotics, and algorithmic optimization (Zhao et al., 2020, Papenmeier et al., 27 May 2025, Jia et al., 2024, Gerstenecker et al., 9 Apr 2026, Jiang et al., 23 Oct 2025, Ljungbergh et al., 2024, Caesar et al., 2021).