Bench2ADVLM: Closed-Loop ADVLM Benchmark
- Bench2ADVLM is a unified, hierarchical benchmark designed for evaluating vision-language autonomous driving systems through closed-loop interaction.
- Its dual-system adaptation decouples high-level semantic decision-making from low-level control execution, ensuring rigorous assessment of interactive behavior and safety.
- Operating in both simulation and hardware, Bench2ADVLM employs self-reflective scenario generation to expose failure modes that open-loop protocols miss.
Searching arXiv for the target paper and closely related autonomous-driving VLM benchmarks to ground the article. arXiv search query: "Bench2ADVLM autonomous driving closed-loop benchmark vision-LLMs" Bench2ADVLM is a unified, hierarchical closed-loop benchmark for evaluating vision-LLM-based autonomous driving systems (ADVLMs) in both simulation and the physical world. It is designed to assess the end-to-end chain from high-level, language-grounded reasoning to mid-level driving commands and low-level vehicle actuation under closed-loop interaction, thereby exposing interactive behavior, feedback resilience, and safety properties that open-loop protocols miss (Zhang et al., 4 Aug 2025). The framework introduces a dual-system adaptation architecture, a physical control abstraction layer, and a self-reflective scenario generation module, and its reported experiments indicate that existing ADVLMs still exhibit limited performance under closed-loop conditions (Zhang et al., 4 Aug 2025).
1. Definition, motivation, and scope
Bench2ADVLM was introduced to address a specific limitation in the evaluation of ADVLMs: most prior protocols were open-loop, meaning that model outputs were produced from static inputs and were not fed back into the environment. In autonomous driving, this omits the interactive nature of decisions, resilience to feedback, and safety under cascading errors. Bench2ADVLM instead closes the loop from language-based decision-making to actuation, enabling dynamic interaction with the environment and revealing failure modes that open-loop evaluation neglects (Zhang et al., 4 Aug 2025).
The benchmark evaluates tasks spanning multimodal reasoning and planning, including action prediction, trajectory or behavioral intent forecasting, and semantic reasoning tied to driving goals. Its scope is explicitly hierarchical. At the high level, target ADVLMs produce heterogeneous semantic driving commands in text form. At the mid level, a general-purpose vision-LLM interprets these outputs into simulator-executable control actions. At the low level, a physical control abstraction layer translates those actions into platform-specific steering, throttle, and brake signals for real vehicles (Zhang et al., 4 Aug 2025).
This hierarchical framing distinguishes Bench2ADVLM from two neighboring benchmark families. Open-loop ADVLM benchmarks such as DriveLM, Dolphins, and related static reasoning sets emphasize question answering or planning on fixed data, but cannot assess feedback effects, interaction, or safety under execution (Zhang et al., 4 Aug 2025). Closed-loop driving benchmarks such as CARLA Leaderboard, Bench2Drive, and later Bench2Drive-VL focus on interactive evaluation, but either target end-to-end control models or define alternative VLM-specific interfaces and annotation pipelines rather than the dual-system adaptation used here (Jia et al., 1 Apr 2026). Bench2ADVLM is described as, to the authors’ knowledge, the first closed-loop benchmark tailored for ADVLMs (Zhang et al., 4 Aug 2025).
2. Hierarchical architecture and dual-system adaptation
Bench2ADVLM is inspired by dual-process theories of cognition and decouples decision generation from semantic execution. The target ADVLM functions as the “fast system,” receiving image sequences and task prompts and outputting heterogeneous textual commands across tasks in a task set . This is formalized as
$\mathbb{T}_t = \left\{ \mathbf{t}_t^{\tau} = \mathcal{F}_{\text{fast}(\{\mathbf{I}_{t-k}, \dots, \mathbf{I}_t\}, \mathbf{Q}_t^\tau) \;\middle|\; \tau \in \mathcal{T} \right\}.$
The abstraction is intended to accommodate output heterogeneity, including structured commands, trajectories-as-text, and free-form chain-of-thought, without requiring model-specific parsers (Zhang et al., 4 Aug 2025).
The “slow system” is a general-purpose VLM acting as semantic executor. It takes a task-specific textual output , current observations, a control prompt template , and an optional candidate control set , and produces a standardized CARLA-compatible control vector:
$\mathbf{u}_t = \mathcal{F}_{\text{slow}(\mathbf{t}_t^\tau, \{\mathbf{I}_{t-k}, \dots, \mathbf{I}_t\}, \mathbf{P}, \mathbb{U}),$
with . The components are mapped to CARLA’s vehicle control API, with for steering, for throttle, and for brake (Zhang et al., 4 Aug 2025).
Two operational modes are defined for this semantic execution layer. In Continuous Numerical Generation (CNG), the candidate set $\mathbb{T}_t = \left\{ \mathbf{t}_t^{\tau} = \mathcal{F}_{\text{fast}(\{\mathbf{I}_{t-k}, \dots, \mathbf{I}_t\}, \mathbf{Q}_t^\tau) \;\middle|\; \tau \in \mathcal{T} \right\}.$0 is empty and the slow system directly regresses continuous control values, which are then clamped to CARLA’s valid ranges. In Discrete Classification Selection (DCS), $\mathbb{T}_t = \left\{ \mathbf{t}_t^{\tau} = \mathcal{F}_{\text{fast}(\{\mathbf{I}_{t-k}, \dots, \mathbf{I}_t\}, \mathbf{Q}_t^\tau) \;\middle|\; \tau \in \mathcal{T} \right\}.$1 consists of a domain-knowledge-derived discrete set of candidate control triplets, and the slow system selects the semantically best-matching candidate. The benchmark states that DCS is intended to provide robust, smoother controls, while CNG allows finer-grained control outputs (Zhang et al., 4 Aug 2025).
For VLM-based slow systems, visual inputs are provided directly. For LLM-based slow systems, images are first encoded into text and appended to the prompt. This design standardizes the actuation interface while preserving heterogeneity across both fast ADVLMs and slow executors. A plausible implication is that the benchmark treats semantic control translation as a separate locus of evaluation rather than assuming that language-native driving outputs are directly executable.
3. Closed-loop execution in simulation and hardware
Bench2ADVLM defines a closed-loop pipeline in both simulation and hardware. In the simulation loop, the fast system consumes recent visual observations and task prompts, produces textual commands, and the slow system converts them into control vectors $\mathbb{T}_t = \left\{ \mathbf{t}_t^{\tau} = \mathcal{F}_{\text{fast}(\{\mathbf{I}_{t-k}, \dots, \mathbf{I}_t\}, \mathbf{Q}_t^\tau) \;\middle|\; \tau \in \mathcal{T} \right\}.$2. These controls are executed through CARLA’s vehicle interface, after which the simulator advances and new observations are fed back to the model stack (Zhang et al., 4 Aug 2025).
The hardware-in-the-loop extension is a central contribution. To bridge simulation and reality, Bench2ADVLM introduces a physical control abstraction layer that maps mid-level actions to platform-specific low-level signals. Reported physical platforms are Jetbot and LIMO on the AGILE·X sandbox, equipped with cameras, LiDAR, and IMU, with ROS used for onboard communication and actuation (Zhang et al., 4 Aug 2025).
The hardware deployment follows a client–server design. The server hosts the fast and slow systems along with semantic parsing, while the client on the vehicle handles real-time sensing and motor commands. Communication is implemented over TCP sockets with SSH tunnels for low-latency, reliable feedback. The hardware loop runs at a fixed $\mathbb{T}_t = \left\{ \mathbf{t}_t^{\tau} = \mathcal{F}_{\text{fast}(\{\mathbf{I}_{t-k}, \dots, \mathbf{I}_t\}, \mathbf{Q}_t^\tau) \;\middle|\; \tau \in \mathcal{T} \right\}.$3 control cycle: each cycle, the vehicle executes $\mathbb{T}_t = \left\{ \mathbf{t}_t^{\tau} = \mathcal{F}_{\text{fast}(\{\mathbf{I}_{t-k}, \dots, \mathbf{I}_t\}, \mathbf{Q}_t^\tau) \;\middle|\; \tau \in \mathcal{T} \right\}.$4 for $\mathbb{T}_t = \left\{ \mathbf{t}_t^{\tau} = \mathcal{F}_{\text{fast}(\{\mathbf{I}_{t-k}, \dots, \mathbf{I}_t\}, \mathbf{Q}_t^\tau) \;\middle|\; \tau \in \mathcal{T} \right\}.$5, collects new observations, and returns them for the next inference step (Zhang et al., 4 Aug 2025).
Notably, the benchmark states that it directly maps normalized $\mathbb{T}_t = \left\{ \mathbf{t}_t^{\tau} = \mathcal{F}_{\text{fast}(\{\mathbf{I}_{t-k}, \dots, \mathbf{I}_t\}, \mathbf{Q}_t^\tau) \;\middle|\; \tau \in \mathcal{T} \right\}.$6 to ROS control topics and CARLA APIs, and does not introduce explicit PID or model-based low-level control laws. This places the methodological focus on evaluation rather than on controller design. The benchmark further states that no domain randomization is reported, and that evaluation, not training, is the focus (Zhang et al., 4 Aug 2025).
Compared with Bench2Drive-VL, which also supports closed-loop VLM evaluation in CARLA but relies on behavior-grounded question answering, graph-based chain-of-thought execution, and action keys extracted from language (Jia et al., 1 Apr 2026), Bench2ADVLM instead standardizes evaluation through a mid-level control interface mediated by a slow semantic executor. This suggests a different decomposition of the VLM-for-driving problem: Bench2Drive-VL centers dynamic VQA and behavior prompting, whereas Bench2ADVLM centers semantic-to-control translation.
4. Self-reflective scenario generation and threat-oriented testing
To probe model-specific weaknesses, Bench2ADVLM introduces a self-reflective scenario generation module. This module actively involves the target ADVLM in constructing its own threat scenarios through a structured P3 reasoning pipeline consisting of Perception, Prediction, and Planning prompts. First, the model describes the current scene; second, it anticipates near-future evolution and salient risk factors; third, it proposes the next action or policy (Zhang et al., 4 Aug 2025).
A GVLM then fuses these stage-wise responses into a coherent, controllable threat description that captures the model’s own perceived hazards and priorities. This fused description conditions a text-to-scenario instantiation pipeline that generates safety-critical scenes aligned with the Bench2Drive route set. Starting from 220 standard Bench2Drive routes, the module generates a corresponding 220 threat-critical scenarios, thereby enlarging adversarial and long-tail coverage beyond static templates (Zhang et al., 4 Aug 2025).
The paper emphasizes that this is an active testing procedure, but it does not use reinforcement learning optimization, explicit objective functions, or pseudocode. Instead, it operationalizes structured prompting and GVLM fusion for targeted scenario synthesis (Zhang et al., 4 Aug 2025). This design differs from benchmarks that rely on fixed scenario banks or purely manually curated adverse conditions.
A useful contrast is AD$\mathbb{T}_t = \left\{ \mathbf{t}_t^{\tau} = \mathcal{F}_{\text{fast}(\{\mathbf{I}_{t-k}, \dots, \mathbf{I}_t\}, \mathbf{Q}_t^\tau) \;\middle|\; \tau \in \mathcal{T} \right\}.$7-Bench, which focuses on autonomous driving under adverse weather and complex scenes using hierarchical chain-of-thought annotations over real images, with fine-grained step-level evaluation but without closed-loop control execution (Wei et al., 11 Jun 2025). Bench2ADVLM addresses a different axis of difficulty: it is not primarily a benchmark for adverse-weather reasoning traces, but for interactive execution under feedback and self-induced state shifts. The self-reflective generator is intended to surface long-tail and safety-critical failures that appear only once a model’s own reasoning is placed in the loop.
5. Evaluation protocol, metrics, and benchmark dimensions
Bench2ADVLM integrates CARLA Leaderboard and Bench2Drive metrics to evaluate three progressively different aspects of closed-loop performance: basic task success, behavioral quality, and specialized skill capability. Each experiment is repeated 10 times, with average and standard deviation reported (Zhang et al., 4 Aug 2025).
Basic performance is measured with Success Rate and Driving Score. Success Rate is defined as the percentage of routes completed without violations:
$\mathbb{T}_t = \left\{ \mathbf{t}_t^{\tau} = \mathcal{F}_{\text{fast}(\{\mathbf{I}_{t-k}, \dots, \mathbf{I}_t\}, \mathbf{Q}_t^\tau) \;\middle|\; \tau \in \mathcal{T} \right\}.$8
Driving Score is a benchmark-defined aggregate route completion score weighted by penalties, following CARLA and Bench2Drive definitions (Zhang et al., 4 Aug 2025).
Behavioral quality is measured by Efficiency and Comfortness. Efficiency is relative speed sampled at every 5% of route progress and then averaged. Comfortness is the proportion of smooth 20-frame segments with stable control signals; higher values indicate smoother, less jerky behavior (Zhang et al., 4 Aug 2025).
Specialized capability is summarized by Skill Score, defined as the average over five interactive skills: lane merging, overtaking, emergency braking, yielding, and traffic sign recognition. Standard deviations across runs are also reported, with lower variance interpreted as better closed-loop stability (Zhang et al., 4 Aug 2025).
The benchmark’s evaluation space spans both standard and threat-oriented scenarios, as well as physical and simulated execution. The principal evaluation settings can be organized as follows:
| Dimension | Bench2ADVLM specification | Reported scope |
|---|---|---|
| Simulation | CARLA-based evaluation aligned with CARLA Leaderboard and Bench2Drive | 220 standard routes + 220 self-reflective threat scenarios |
| Skills | Interactive skill evaluation | lane merging, overtaking, emergency braking, yielding, traffic sign recognition |
| Hardware | Physical HIL evaluation on AGILE·X sandbox | Jetbot and LIMO, Ackermann mode in reported results |
This protocol differs from process-centric multimodal benchmarks such as AdaptMMBench, which evaluates adaptive mode selection using Matthews Correlation Coefficient and process metrics such as key step coverage and tool effectiveness rather than closed-loop actuation (Zhang et al., 2 Feb 2026). Bench2ADVLM is therefore a benchmark of executional robustness and control integration, not of mode-selection meta-cognition.
6. Experimental results, ablations, and diagnostic findings
Bench2ADVLM reports results for four fast systems—Dolphins, DriveLM, EM-VLM4AD, and OmniDrive—and two slow systems, LLaMA-3-8B and LLaVA-1.5-13B, under both CNG and DCS parsing modes (Zhang et al., 4 Aug 2025). Across common closed-loop simulation scenarios, the overall performance remains modest. OmniDrive is reported as the strongest model, reaching Success Rate up to 12.99%, Driving Score up to 43.75, and Skill Score up to 19.38 depending on slow system and parsing mode. DriveLM and Dolphins obtain Driving Scores around 34–39 and Skill Scores below 18, while EM-VLM4AD shows competitive Comfortness, up to 71.16, despite being lightweight (Zhang et al., 4 Aug 2025).
The comparison between slow executors reveals a trade-off. LLaMA often yields higher Success Rates, while LLaVA can produce higher Driving Scores in some settings. Parsing mode also has a clear effect: CNG tends to increase Driving Score through fine-grained control, while DCS substantially improves Comfortness. One reported example is DriveLM combined with LLaMA, where Comfortness improves from 31.95 to 65.20 under DCS, suggesting that DCS dampens control jitter typical of direct regression (Zhang et al., 4 Aug 2025).
The threat scenarios generated by the self-reflective module produce a broad degradation in robustness. Average Driving Score decreases by approximately 26.1%, and Efficiency and Comfortness degrade more sharply than Success Rate. Example degradations include Dolphins with CNG and LLaMA dropping in Success Rate from 9.09% to 7.91%, and OmniDrive’s Comfortness falling by about 12 on average. Variance also increases for stronger models such as OmniDrive, exposing stability issues under adversarial conditions. EM-VLM4AD maintains relatively lower variance, indicating steadier behavior at lower absolute performance (Zhang et al., 4 Aug 2025).
Real-world hardware-in-the-loop results are reported on simplified lane-following routes, using route completion rate without boundary crossings or collisions as the metric. Jetbot averages 55.5%, and LIMO averages 54.5%. On Jetbot and LIMO respectively, DriveLM achieves 46% and 42%, Dolphins 34% and 30%, EM-VLM4AD 60% and 60%, and OmniDrive 82% and 86% (Zhang et al., 4 Aug 2025). Even in these simplified routes, the benchmark reports a substantial failure rate, which it uses to motivate closed-loop hardware evaluation.
Several ablations and sensitivity studies are also reported. Fine-grained control prompting, termed “Potential Pathways,” augments high-level commands with simple control suffixes such as “keep straight with speed = 0.5” and improves DriveLM plus LLaMA Success Rate from 9.09 to 12.52 on the 220 standard routes (Zhang et al., 4 Aug 2025). A hybrid mode switching strategy between LLaMA and LLaVA, selected by a simple threat query—“Is there a security threat ahead?”—improves DriveLM’s Success and Efficiency simultaneously, from 9.09 and 127.55 under LLaMA, and 6.82 and 129.83 under LLaVA, to 9.23 and 131.29 in the hybrid setting (Zhang et al., 4 Aug 2025).
These findings support a central diagnostic claim of the benchmark: open-loop promise does not translate directly into closed-loop reliability. Bench2Drive-VL reports a related observation, namely that open-loop VQA correctness does not directly translate into closed-loop success and that chain-of-thought may even degrade performance under long-context and latency constraints (Jia et al., 1 Apr 2026). Bench2ADVLM reaches an analogous conclusion through a different architecture and metric suite.
7. Significance, limitations, and relation to adjacent benchmarks
Bench2ADVLM’s main significance lies in establishing a hierarchical evaluation pipeline that seamlessly integrates high-level abstract reasoning, mid-level simulation actions, and low-level real-world execution (Zhang et al., 4 Aug 2025). Its novelty is stated in four parts: dual-system adaptation for heterogeneous ADVLM outputs, a physical control abstraction layer for hardware-in-the-loop evaluation, a self-reflective P3-structured scenario generator, and a unified metric suite spanning task success, behavior quality, and specialized skills (Zhang et al., 4 Aug 2025).
Its limitations are equally explicit. Real-world experiments are conducted in a sandbox with small-scale platforms and simplified routes, and therefore do not reflect the full complexity of commercial autonomous vehicle stacks or dense multi-agent urban traffic. Current experiments are single-agent and lack coordinated multi-agent interactions, which limits assessment of negotiation and cooperation. All hardware tests are conducted in controlled environments, and the benchmark is described as research-only; physical deployment beyond the sandbox would require additional engineering safeguards and independent safety controllers (Zhang et al., 4 Aug 2025).
The benchmark also inherits broader open questions in sim-to-real generalization. Although the abstraction layer and dual-system design mitigate part of the semantic gap between simulation and hardware actuation, robustness under broader conditions such as weather, sensors, and vehicle dynamics remains open. The reported degradation on self-reflective threat scenarios is presented as evidence of this unresolved challenge (Zhang et al., 4 Aug 2025).
Within the broader benchmark landscape, Bench2ADVLM occupies a specific niche. AD$\mathbb{T}_t = \left\{ \mathbf{t}_t^{\tau} = \mathcal{F}_{\text{fast}(\{\mathbf{I}_{t-k}, \dots, \mathbf{I}_t\}, \mathbf{Q}_t^\tau) \;\middle|\; \tau \in \mathcal{T} \right\}.$9-Bench concentrates on hierarchical chain-of-thought evaluation for adverse-weather autonomous driving on real images, emphasizing step-wise correctness and interpretability rather than interactive control (Wei et al., 11 Jun 2025). Bench2Drive-VL brings VLMs into closed-loop CARLA evaluation with dynamic VQA generation and graph-based chain-of-thought execution (Jia et al., 1 Apr 2026). AdaptMMBench evaluates adaptive multimodal reasoning using dynamic difficulty labels and process-level metrics, but not driving execution (Zhang et al., 2 Feb 2026). Bench2ADVLM differs from all three by centering the closed-loop transformation from language-based driving reasoning to standardized control in both simulation and physical vehicles (Zhang et al., 4 Aug 2025).
This suggests that Bench2ADVLM should be understood less as a dataset in the conventional static sense and more as an evaluation infrastructure for probing whether language-grounded autonomous driving systems remain coherent, recoverable, and safe once their decisions have consequences.
