AgentDrive: A Unified Benchmark for Autonomous Driving

• AgentDrive is an open benchmark suite for autonomous driving that combines simulation-ready JSON scenarios with MCQ-based reasoning tests.
• It generates 300,000 structured driving scenarios and 100,000 multiple-choice questions to assess safety, policy adherence, and contextual understanding.
• The framework unifies prompt-to-JSON conversion, rigorous schema validation, simulation rollouts, and surrogate safety metrics for comprehensive LLM evaluation.

AgentDrive is an open benchmark suite for evaluating and training agentic AI and LLM-based autonomous driving systems. It combines two components: AgentDrive-Gen / AgentDrive-Sim, a dataset of 300,000 LLM-generated driving scenarios represented as structured, simulation-ready JSON and paired with simulation rollouts and safety labels, and AgentDrive-MCQ, a 100,000-question multiple-choice benchmark for driving-related reasoning. Its defining contribution is the unification of generative scenario creation, simulation-grounded evaluation, and reasoning-oriented benchmarking within a single framework for autonomous agents (Ferrag et al., 23 Jan 2026).

1. Benchmark scope and motivation

AgentDrive was introduced in response to several limitations attributed to existing autonomous driving benchmarks: many are static and do not test interactive behavior; many focus only on perception or theory questions; many lack large-scale diversity and rare safety-critical cases; many do not support training or fine-tuning for agentic reasoning; and few connect natural-language reasoning to closed-loop simulation outcomes. The benchmark is therefore designed to support training, fine-tuning, evaluation, stress testing, and safety and alignment analysis for LLM-driven autonomous agents (Ferrag et al., 23 Jan 2026).

The benchmark is organized around two complementary artifacts.

Component	Scale	Function
AgentDrive-Gen / AgentDrive-Sim	300,000 scenarios	Structured generation, simulation rollout, safety labeling
AgentDrive-MCQ	100,000 MCQs	Reasoning evaluation across five dimensions

This division is central to the benchmark’s intended use. Simulation assesses whether an agent behaves safely in rollout, whereas MCQ evaluation assesses whether a model can reason about safety, policy, physics, and maneuver choice before acting. The paper presents this combination as broader than benchmarks limited to static knowledge, text QA, perception, or open-loop prediction (Ferrag et al., 23 Jan 2026).

2. Factorized scenario model and dataset composition

AgentDrive formalizes each scenario as a tuple

$s = (t, b, e, r, o, d, q),$

drawn from the factorized scenario space

$$\mathcal{S} = \mathcal{T} \times \mathcal{B} \times \mathcal{E} \times \mathcal{R} \times \mathcal{O} \times \mathcal{D} \times \mathcal{Q}.$$

The seven orthogonal axes are scenario type, driver behavior, environment, road layout, objective, difficulty, and traffic density (Ferrag et al., 23 Jan 2026).

Axis	Symbol	Representative values
Scenario type	$t$	lane change, intersection crossing, merging, pedestrian jaywalking, mechanical failures
Driver behavior	$b$	compliant, cooperative, aggressive, distracted, impaired
Environment	$e$	rain, fog, low visibility, clear daytime, foggy night
Road layout	$r$	highway, urban intersection, roundabout, rural roads, tunnels
Objective	$o$	safe navigation, overtaking, emergency stop, obey red light
Difficulty	$d$	easy, medium, hard
Traffic density	$q$	low / sparse, medium, high / congested

Difficulty is not only categorical. It is mapped to numeric safety constraints $H(d)$: for easy, $$\mathcal{S} = \mathcal{T} \times \mathcal{B} \times \mathcal{E} \times \mathcal{R} \times \mathcal{O} \times \mathcal{D} \times \mathcal{Q}.$$0 and $$\mathcal{S} = \mathcal{T} \times \mathcal{B} \times \mathcal{E} \times \mathcal{R} \times \mathcal{O} \times \mathcal{D} \times \mathcal{Q}.$$1; for medium, $$\mathcal{S} = \mathcal{T} \times \mathcal{B} \times \mathcal{E} \times \mathcal{R} \times \mathcal{O} \times \mathcal{D} \times \mathcal{Q}.$$2 and $$\mathcal{S} = \mathcal{T} \times \mathcal{B} \times \mathcal{E} \times \mathcal{R} \times \mathcal{O} \times \mathcal{D} \times \mathcal{Q}.$$3; and for hard, $$\mathcal{S} = \mathcal{T} \times \mathcal{B} \times \mathcal{E} \times \mathcal{R} \times \mathcal{O} \times \mathcal{D} \times \mathcal{Q}.$$4 and $$\mathcal{S} = \mathcal{T} \times \mathcal{B} \times \mathcal{E} \times \mathcal{R} \times \mathcal{O} \times \mathcal{D} \times \mathcal{Q}.$$5. These numeric hints are used during generation to make scenarios quantitatively consistent with the intended difficulty level (Ferrag et al., 23 Jan 2026).

Each dataset entry is stored as

$$\mathcal{S} = \mathcal{T} \times \mathcal{B} \times \mathcal{E} \times \mathcal{R} \times \mathcal{O} \times \mathcal{D} \times \mathcal{Q}.$$6

where $$\mathcal{S} = \mathcal{T} \times \mathcal{B} \times \mathcal{E} \times \mathcal{R} \times \mathcal{O} \times \mathcal{D} \times \mathcal{Q}.$$7 is a sampled scenario tuple, $$\mathcal{S} = \mathcal{T} \times \mathcal{B} \times \mathcal{E} \times \mathcal{R} \times \mathcal{O} \times \mathcal{D} \times \mathcal{Q}.$$8 is the generated JSON scenario specification, $$\mathcal{S} = \mathcal{T} \times \mathcal{B} \times \mathcal{E} \times \mathcal{R} \times \mathcal{O} \times \mathcal{D} \times \mathcal{Q}.$$9 is the simulation rollout, and $t$0 is the categorical label. The dataset supports both scenario-level and segment-level analysis. The paper reports that episode-level labels are roughly balanced across unsafe, safe_goal, safe_stop, and inefficient, while segment-level labels are dominated by safe-goal, with millions of segments (Ferrag et al., 23 Jan 2026).

3. Prompt-to-JSON generation, validation, and simulation grounding

The generation pipeline begins by sampling a scenario tuple $t$1 from the factorized space. A prompt is then formed as

$t$2

where $t$3 is the natural-language encoding of the tuple and $t$4 injects the difficulty-specific numeric constraints. An LLM generates a JSON specification,

$t$5

which is then checked against schema constraints $t$6. If validation fails, the output is repaired or retried:

$t$7

with $t$8 denoting the maximum number of repair or retry attempts (Ferrag et al., 23 Jan 2026).

The paper provides explicit prompt constraints. The model must “Return a SINGLE valid JSON object,” include no comments or explanations, respect physics constraints such as $t$9 and $b$0, use 0-indexed lanes, and satisfy traffic-light-specific conditions such as ego spawn at least 30 m before the stopline and duration_steps >= red_steps + green_steps when a traffic light exists. Required fields include seed, policy_frequency, and all schema-mandated elements (Ferrag et al., 23 Jan 2026).

The schema includes fields such as name, seed, duration_steps, road, traffic_light, environment, layout, objective, difficulty, traffic_density, ego.spawn, traffic[], events[], and metrics[]. Validation imposes constraints including duration_steps >= 60, road.lanes >= 2, speed_limit_kph in [60, 140], and ego.spawn.v_mps >= 0 (Ferrag et al., 23 Jan 2026).

After validation, each scenario is executed in highway-env as a rollout

$b$1

where $b$2 is determined by duration_steps and each state $b$3 includes ego position, velocity, acceleration, surrounding traffic, traffic light state, and environment parameters. The timestep is

$b$4

The benchmark also explicitly seeks diversity by maximizing entropy across the seven sampled axes, subject to schema compliance and valid rollouts. This design is intended to avoid repetitive or collapsed scenario distributions and to broaden coverage across routine driving, rare edge cases, and safety-critical combinations (Ferrag et al., 23 Jan 2026).

4. Surrogate safety metrics and categorical labeling

The central surrogate safety metric in AgentDrive is Time-to-Collision (TTC):

$b$5

At the episode level,

$b$6

The paper specifies thresholds $b$7 and $b$8. Headway distance and collisions are also mentioned, but TTC is treated as the primary surrogate safety indicator (Ferrag et al., 23 Jan 2026).

Outcome labeling is rule-based and event-driven. The event definitions are collision for ego crash, red_violation for crossing the stopline on red, stopped_on_red for a full stop before the stopline on red, and crossed_green for crossing the stopline on green. Final labels are assigned as follows: unsafe if a collision occurs, a red violation occurs, or $b$9 falls below $e$0; safe_goal if the vehicle stopped on red and later crossed on green; safe_stop if it stopped on red and did not cross on green; and inefficient otherwise (Ferrag et al., 23 Jan 2026).

A common misunderstanding would be to treat LLM-generated scenarios as purely textual artifacts. AgentDrive’s construction explicitly counters that interpretation through schema validation, repair, simulation rollout, surrogate safety computation, and rule-based labeling. The benchmark is therefore not limited to prompt generation; it binds generated descriptions to executable dynamics and interpretable supervision. A plausible implication is that it is designed to support downstream learning of both success conditions and failure modes, rather than only natural-language plausibility (Ferrag et al., 23 Jan 2026).

5. AgentDrive-MCQ and the evaluation of driving reasoning

AgentDrive-MCQ is the reasoning companion to the simulation dataset. It contains 100,000 multiple-choice questions and is designed to test whether LLM-based agents can reason correctly about driving situations without relying only on simulation. The benchmark spans five reasoning dimensions: physics, policy, hybrid, scenario, and comparative reasoning (Ferrag et al., 23 Jan 2026).

The MCQ pipeline proceeds in three stages. First, a validated scenario JSON is converted into a 10–12 sentence plain-text description. Second, for each reasoning style, the model generates a question $e$1, four options $e$2, the correct choice $e$3, and a rationale $e$4. Third, the MCQ is stored with a hash-based identifier,

$e$5

to improve traceability and reproducibility (Ferrag et al., 23 Jan 2026).

The validation constraints for MCQ generation are highly specific: $e$6; labels must be exactly A/B/C/D; options must be distinct; question length must be at most 25 words; the rationale must be non-empty; physics and hybrid questions must include numeric tokens; scenario questions should be qualitative rather than all-numeric; and comparative questions must include at least two action-oriented options (Ferrag et al., 23 Jan 2026).

The five reasoning styles partition the evaluation space in a deliberate way. Physics covers numerical kinematics, stopping distance, time-to-collision, deceleration, and safe headway. Policy covers traffic rules, right-of-way, defensive driving, and safety regulations. Hybrid combines physics and policy. Scenario focuses on qualitative interpretation of risk and contextual priority. Comparative requires selecting the safest maneuver among alternatives such as braking, accelerating, lane changing, or maintaining speed (Ferrag et al., 23 Jan 2026).

To summarize performance, the paper defines per-style accuracy, overall accuracy as the mean across the five styles, Safety Compliance Rate (SCR) as a weighted combination of policy and scenario performance, and Situational Awareness Score (SAS) as a weighted combination of comparative, hybrid, and physics performance, with default weights set to 1. This separation is intended to distinguish safety alignment and rule adherence from physical and contextual situational understanding (Ferrag et al., 23 Jan 2026).

6. Large-scale LLM evaluation and empirical findings

The paper evaluates 50 LLMs on AgentDrive-MCQ using 2,000 benchmark samples with deterministic decoding parameters: temperature = 0.0, max_tokens = 16, top_p = 1.0, and max_retries = 5. The stated goal is reproducible comparison across models (Ferrag et al., 23 Jan 2026).

Model	Overall accuracy
ChatGPT 4o	82.5
GPT-5 Chat	81.0
Qwen3 235B A22B (2507)	81.0
Mistral Medium 3.1	80.0
GPT-4.1 Mini	77.5

The broad empirical pattern is that proprietary frontier models generally lead overall, while advanced open models are rapidly closing the gap in physics-grounded and structured reasoning. Proprietary models are reported as especially strong in policy, scenario, and comparative reasoning. Examples given include ChatGPT 4o with policy 100 and scenario 95, and GPT-5 Chat with policy 100 and scenario 92.5. Among open models, Qwen3 235B A22B reaches overall 81.0 and physics 67.5, ERNIE 4.5 300B A47B reaches scenario 97.5, Phi 4 Reasoning Plus reaches comparative 95 and scenario 97.5, and Qwen3 Max reaches comparative 95, policy 95, and scenario 95 (Ferrag et al., 23 Jan 2026).

Hybrid reasoning is identified as the hardest category overall. Even strong models show materially lower performance there: ChatGPT 4o scores 72.5, GPT-5 Chat 70.0, and Qwen3 235B A22B 60.0. The paper interprets this as evidence that combining numerical physics, policy constraints, and contextual reasoning remains difficult. Smaller models perform much worse overall, particularly on policy and hybrid reasoning; examples include GPT-3.5 Turbo Instruct at 34.5, Llama 3.1 8B Instruct at 42.0, Qwen 2.5 7B Instruct at 37.5, GPT-OSS-20B at 9.5, and GPT-OSS-120B at 11.5 (Ferrag et al., 23 Jan 2026).

7. Uses, limitations, and relation to autonomous-systems research

AgentDrive is positioned for training, fine-tuning, evaluation, stress testing, and safety analysis. The paper explicitly lists supervised fine-tuning on scenario descriptions, JSON generation, reasoning Q&A, and safety-aligned decision-making, as well as closed-loop evaluation in simulation and zero-shot or few-shot reasoning evaluation via MCQs. The dataset, labeled scenarios, AgentDrive-MCQ, evaluation scripts, and related materials are released at https://github.com/maferrag/AgentDrive (Ferrag et al., 23 Jan 2026).

The benchmark also has clearly stated limitations. It is currently text-based; visual contexts are represented as textual descriptions; and multimodal vision-language integration is future work. Further, scenarios are simulation-derived rather than fully real-world observed, the reasoning benchmark is based on generated text rather than direct sensor grounding, and autonomous driving complexity is broader than the current scenario factorization (Ferrag et al., 23 Jan 2026).

AgentDrive should be distinguished from systems such as DriveAgent, which is a modular, LLM-guided multi-agent framework that ingests synchronized camera, LiDAR, GPS, and IMU data to support structured, interpretable driving reasoning and urgency-aware response generation (Hou et al., 4 May 2025). AgentDrive is not that kind of operational sensor-fusion system; it is a benchmark and training substrate. This suggests a complementary relationship: AgentDrive provides large-scale, structured, safety-critical scenarios and reasoning tests, while frameworks such as DriveAgent target multimodal autonomous driving inference and decision support. In that sense, AgentDrive functions as infrastructure for evaluating and stress-testing agentic autonomous-system reasoning rather than as a deployed driving stack (Ferrag et al., 23 Jan 2026).