AgentDrive Benchmark Suite for Autonomous AI

• AgentDrive Benchmark Suite is an open framework featuring 300,000 LLM-generated, factorized driving scenarios paired with rigorous simulations.
• It employs a seven-axis factorized scenario space and a prompt-to-JSON pipeline to ensure diverse and reproducible simulation scenarios.
• The suite integrates surrogate safety metrics and an MCQ evaluation system to assess LLM performance in physics, policy, and risk assessment.

AgentDrive is an open benchmark suite designed to advance the evaluation and training of agentic artificial intelligence in autonomous driving contexts, particularly targeting LLM-based reasoning. It provides a structured, large-scale dataset of 300,000 LLM-generated driving scenarios, each paired with physical simulations, surrogate safety metrics, rule-based outcome labeling, and an associated multiple-choice benchmark focused on diverse reasoning skills. The suite aims to fill a gap in rigorous, safety-critical benchmarking essential for the research and development of AI reasoning in autonomous systems (Ferrag et al., 23 Jan 2026).

1. Factorized Scenario Space

A core architectural feature of AgentDrive is the formalization of driving scenarios as factorized tuples along seven orthogonal axes. Each scenario $s$ is a 7-tuple: $s = (t, b, e, r, o, d, q)$ with axes and examples:

Axis	Symbol	Interpretation/Examples
Scenario type	$t\in\mathcal{T}$	lane change, braking failure, intersection crossing
Driver behavior	$b\in\mathcal{B}$	aggressive, compliant, impaired, convoy-following
Environment	$e\in\mathcal{E}$	clear, fog, rain, night, sandstorm
Road layout/topology	$r\in\mathcal{R}$	roundabout, cloverleaf, mountain pass, urban intersection
Ego-vehicle objective	$o\in\mathcal{O}$	safe navigation, overtake, U-turn, emergency stop
Difficulty level	$d\in\mathcal{D}$	easy, medium, hard (with associated constraints $H(d)$)
Traffic density	$q\in\mathcal{Q}$	low, medium, high

The full scenario space is defined as the Cartesian product: $s = (t, b, e, r, o, d, q)$0 The total number of distinct scenarios is: $s = (t, b, e, r, o, d, q)$1 with $s = (t, b, e, r, o, d, q)$2 the cardinalities of each set. Axis cardinalities are tuned such that $s = (t, b, e, r, o, d, q)$3. For example, $s = (t, b, e, r, o, d, q)$4 yield $s = (t, b, e, r, o, d, q)$5, with adjustments achieving $s = (t, b, e, r, o, d, q)$6 exact. Entropy-maximized sampling ensures diverse, balanced coverage. Each axis is treated independently, enabling stress-testing along isolated or compound scenario dimensions (e.g., impaired driver in fog on a cloverleaf at high density) (Ferrag et al., 23 Jan 2026).

2. LLM-Driven Prompt-to-JSON Scenario Generation

AgentDrive adopts a prompt-to-JSON pipeline leveraging LLMs for semantically rich, simulation-ready scenario specification. The process comprises:

1. Prompt Construction: For each scenario $s = (t, b, e, r, o, d, q)$7 and difficulty mapping $s = (t, b, e, r, o, d, q)$8, a natural-language encoding $s = (t, b, e, r, o, d, q)$9 is constructed, augmented with numeric constraints from $t\in\mathcal{T}$0. This yields the composite prompt $t\in\mathcal{T}$1.
2. LLM Generation: An LLM generator $t\in\mathcal{T}$2 produces a candidate JSON $t\in\mathcal{T}$3.
3. Schema Validation and Repair: Each JSON is validated against schema $t\in\mathcal{T}$4. If invalid, a repair module $t\in\mathcal{T}$5 is applied up to $t\in\mathcal{T}$6 retries, yielding a validated sample $t\in\mathcal{T}$7.

Technical and domain constraints ensure simulation correctness, including velocity and acceleration bounds, minimum spawn distances before stop lines, step count consistency with traffic light phases, and schema compliance for road topology, agent definitions, and event fields. Outputs comprise a complete, schema-conforming JSON description for each scenario, suitable for direct ingestion by the downstream simulation pipeline (Ferrag et al., 23 Jan 2026).

3. Simulation Rollouts and Surrogate Safety Metrics

Each validated scenario is executed in the highway-env simulator, producing a trajectory rollout $t\in\mathcal{T}$8 with $t\in\mathcal{T}$9 representing ego vehicle and environment kinematics. The principal surrogate safety metric is scenario-minimum time-to-collision ($b\in\mathcal{B}$0) with the lead vehicle: $b\in\mathcal{B}$1

$b\in\mathcal{B}$2

Thresholds $b\in\mathcal{B}$3 and $b\in\mathcal{B}$4 are enforced. Rule-based checks detect collision, red-light violations, and correct stopping/crossing behaviors. The final outcome label $b\in\mathcal{B}$5 is assigned via:

$b\in\mathcal{B}$6

The full simulation corpus is

$b\in\mathcal{B}$7

where each entry contains the scenario axes, JSON file, simulated trajectory, and label. This enables systematic analysis of policy robustness and safety-critical failure modes (Ferrag et al., 23 Jan 2026).

4. AgentDrive-MCQ: LLM Reasoning Benchmark

AgentDrive-MCQ augments the dataset with a structured, multiple-choice evaluation suite targeting model reasoning spanning five dimensions:

• Physics: Quantitative reasoning over vehicle kinematics (e.g., explicit TTC calculation).
• Policy: Normative reasoning over traffic laws and safety protocols.
• Hybrid: Compositional reasoning blending physics-based calculations and policy constraints.
• Scenario-interpretive: Qualitative risk and hazard analysis.
• Comparative: Multi-alternative selection of safest actions.

For each of $b\in\mathcal{B}$8 selected scenarios, 5 MCQs are generated, yielding $b\in\mathcal{B}$9 questions in total. The MCQ generation pipeline involves: LLM-generated 10–12 sentence scenario description, followed by constrained prompt-to-MCQ generation per style (physics, policy, hybrid, scenario, comparative), with enforced answer/rationale completeness and type checks. Every MCQ specifies: succinct question (≤25 words), four options, correct index, and rationale—in strict JSON schema, with meta fields for scenario and traceability.

This design permits fine-grained, style-specific assessment of LLM and agentic reasoning in policy, ethical, and physical driving contexts (Ferrag et al., 23 Jan 2026).

5. Evaluation Protocols and Model Comparison

AgentDrive supports benchmarking across proprietary and open LLMs. Fifty leading models were evaluated on a $e\in\mathcal{E}$0-sample subset of AgentDrive-MCQ. For model $e\in\mathcal{E}$1 and question style $e\in\mathcal{E}$2: $e\in\mathcal{E}$3 Overall accuracy is the unweighted mean across five styles.

Two composite metrics are defined:

• Safety Compliance Rate (SCR):

$e\in\mathcal{E}$4

(measuring rule- and qualitative safety alignment)

• Situational Awareness Score (SAS):

$e\in\mathcal{E}$5

(scoring numerical and decision-selection competence)

Key empirical findings:

Model	Comparative	Hybrid	Physics	Policy	Scenario	Overall
ChatGPT 4o	90.0	72.5	55.0	100	95.0	82.5
GPT-5 Chat	92.5	70.0	50.0	100	92.5	81.0
Qwen3 235B A22B	92.5	60.0	67.5	87.5	97.5	81.0
ERNIE 4.5 300B	85.0	45.0	52.5	95.0	97.5	75.0
Mistral Medium 3.1	95.0	60.0	52.5	97.5	95.0	80.0

• Proprietary models excel in policy and scenario reasoning ($e\in\mathcal{E}$6).
• Open models approach parity in physics and hybrid, with Qwen3 235B scoring $e\in\mathcal{E}$7 in physics.
• Hybrid questions remain the most challenging ($e\in\mathcal{E}$8 for all), reflecting the difficulty of joint numeric-symbolic reasoning.
• Comparative reasoning is high ($e\in\mathcal{E}$9 for several models), signifying the effectiveness of instruction tuning on textual choice tasks.

SCR and SAS exhibit strong positive correlation, suggesting situational awareness advances are closely linked with safety compliance capabilities in current LLMs (Ferrag et al., 23 Jan 2026).

6. Dataset, Code Availability, and Reproducibility

The entire AgentDrive benchmark suite is openly released at https://github.com/maferrag/AgentDrive under an open-source license. The release includes:

• AgentDrive-Gen & AgentDrive-Sim: 300,000 JSON scenario files (conforming to schema $r\in\mathcal{R}$0), corresponding simulation logs and surrogate safety metrics, and full outcome labels.
• AgentDrive-MCQ: 100,000 MCQs with scenario metadata, question, options, gold answer, rationale.
• Evaluation scripts: Colab-ready Python notebooks for parsing, simulation, surrogate metric computation (including TTC, headway, rule-based labeling), LLM question answering, and benchmark aggregation (Accuracy, SCR, SAS).
• Benchmarks and logs: CSV-format summaries of all model runs, and plotting scripts for detailed performance visualization.

Schema definitions, sample prompts, and workflow documentation are provided to ensure experimental reproducibility and encourage downstream research in generative scenario generation, cognitive and ethical evaluation of agentic models, and simulation-based safety assessment for autonomous systems (Ferrag et al., 23 Jan 2026).