DriveQA: Driving Knowledge Benchmark

• DriveQA is a multimodal benchmark that assesses both text and vision models on comprehensive U.S. driving regulations and safety-critical scenarios.
• It comprises two components—DriveQA-T with 26K text QA pairs and DriveQA-V with 448K multimodal QA pairs—spanning traffic rules, signage, and spatial reasoning.
• Empirical results reveal that fine-tuning on DriveQA enhances models' regulatory reasoning, sign recognition, and real-world simulation performance.

DriveQA is a large-scale, multimodal benchmark specifically designed to assess whether LLMs and multimodal LLMs (MLLMs) possess the regulatory, semantic, and spatial knowledge required to pass a U.S. driving knowledge test. Unlike conventional perception- or planning-focused autonomous driving QA datasets, DriveQA explicitly targets the full range of traffic regulations, signage, right-of-way, edge-case scenarios, and environmental robustness encountered on licensing exams. Its central aim is to provide a diagnostic testbed for both text-based and vision-LLMs, probing their ability not just to perceive, but also to reason about, codified traffic law and safety principles (Wei et al., 29 Aug 2025, Xiu et al., 14 Sep 2025).

1. Benchmark Scope and Structure

DriveQA comprises two primary components, each targeting distinct but complementary aspects of driver knowledge:

• DriveQA-T: A text-only multiple-choice question answering (QA) benchmark, constructed from official driver’s handbooks from all 50 U.S. states and Washington, D.C. This subset targets regulatory knowledge, traffic rules, right-of-way, limits, penalties, and edge-case scenarios, totaling approximately 26,000 QA pairs. Questions cover categories such as speed/distance limits, parking, intersection behavior, alcohol law, passing and turning, and safety rules. Each sample includes the question, distractors, a correct answer, and an annotated explanation.
• DriveQA-V: A vision-language QA benchmark centered on traffic sign recognition, sign semantics, and intersection/right-of-way reasoning. This component includes about 68,000 diverse images and 448,000 multimodal QA pairs. It spans over 220 U.S. traffic sign types and leverages both procedurally generated CARLA simulation scenes (where sign placement, weather, lighting, camera perspective, and traffic layout are systematically varied for controlled robustness analysis) and a Mapillary-sourced real-world image evaluation set with manual sign annotations for sim-to-real generalization testing.

Combined, DriveQA offers around 474,000 QAs across 19 text-based categories and four sign classes (Regulatory, Warning, Guide, Temporary Control), with explicit metadata for categories, sign types, scenario variables, and explanations.

2. Motivation, Design Principles, and Coverage

The introduction of DriveQA addresses documented shortcomings in prior driving QA and VQA benchmarks, which predominantly emphasize perception and scene grounding in naturalistic driving data, but lack comprehensive evaluation of traffic-rule competence, edge-case regulatory reasoning, and sign interpretation. Key differentiators in the DriveQA benchmark include:

• Exhaustive Rule Coverage: Using 51 official handbooks, DriveQA-T represents broad U.S.-centric regulatory coverage, capturing rare but safety-critical scenarios absent from real driving logs (e.g., specific right-of-way edge cases, rare signage, legal exceptions).
• Procedural and Manual Annotation: DriveQA-V supplements the CARLA simulation pipeline with 220 custom 3D traffic sign models, scenario scripting for complex intersection logic, and manual Mapillary label curation, allowing for high-density, balanced, and scenario-targeted QA generation.
• Robustness to Environmental Variation: DriveQA explicitly varies lighting, weather, camera viewpoint (forward, oblique, top-down), distance, and geometric scene layout to probe the sensitivity and failure modes of MLLMs under real and synthetic perturbations.
• Answer Explanation: Both components include textual rationales to support not simply answer correctness, but also model reasoning transparency and explanation quality measurement (BLEU-4, ROUGE-L).

DriveQA’s design paradigm is that passing a licensing test demands not only accurate perception but also precise application of codified rules, numerical reasoning, and interpretation of rare or corner-case traffic scenarios.

3. Task Format, Question Taxonomy, and Distinctive Evaluation

DriveQA’s QA format is grounded in real-world driver testing: all questions are presented in multiple-choice form, carefully balanced with semantically plausible distractors, especially for rare categories (e.g., warning/regulatory sign confusions, numeric distractors for speed/distance/alcohol limits).

DriveQA-T employs a cluster-derived taxonomy—hierarchical BERT-based clustering and KeyBERT labeling—grouping 19 subcategories into five broad question families: traffic signals, limits, parking, intersection/right-of-way, and special subjects (alcohol, passing, turning, etc.).

DriveQA-V intersects sign category (Regulatory, Warning, Guide, Temporary Control), intersection type (T/Cross), and viewing perspective (front, top-down), allowing for fine-grained performance analysis. Each question is paired with a visual input and, where appropriate, an explanation.

Evaluation metrics are:

• Accuracy (primary, for both T and V)
• BLEU-4 and ROUGE-L (for explanation rationales)
• Downstream transfer metrics: waypoint planning L2 errors, action/explanation mF1/F1ₐₗₗ on external driving datasets

Multiple prompted evaluation strategies are compared:

• Basic (direct answer)
• CoT (Chain-of-Thought, stepwise reasoning)
• RAG (retrieval-augmented generation)
• CoT + RAG (combined structured prompting and retrieval grounding)

4. Model Performance and Core Findings

DriveQA’s diagnostic focus reveals consistent, multi-faceted weaknesses in state-of-the-art models:

	Numerical/Limit	Parking	Intersection	Signs (Reg./Warn.)	Explanation BLEU-4	Real-World Sim2Real
Off-the-shelf LLMs/MLLMs	Poor	Poor	Very poor	Poor to moderate	Low	Poor to moderate
GPT-4o	Good/Best	Very good	Very good	Excellent	Best-in-class	Best-in-class
Fine-tuned open models	Improved	Good	Strong	Strong	Moderate to good	Improved

• Numerical Reasoning: All open-source models struggle with limits (speed, distance, alcohol), even post-finetuning. GPT-4o is strongest but not perfect.
• Right-of-Way/Intersections: Intersection and right-of-way QA remains the most challenging, especially for egocentric/front-view images. Off-the-shelf visual models perform at or near random; only fine-tuned or GPT-4o-level models achieve high accuracy.
• Sign Recognition: Regulatory and warning signs are persistently harder than guide or temporary control. Off-the-shelf open models perform poorly here; fine-tuning yields large gains, especially on rare signs.
• Explanation Quality: Even with RAG and fine-tuning, BLEU-4 and ROUGE-L do not approach those of human references or GPT-4o, indicating incomplete reasoning chains.
• Robustness: Environmental and layout variation systematically degrades performance, especially for models not fine-tuned on the exact variations present.
• Difficulty-Adjusted Performance: Controlled negative sampling, hard distractors, and challenge sets produce pronounced accuracy drops—15–29% even for top models—showing broad reliance on pattern-matching rather than robust discrimination.

Empirical results also demonstrate that fine-tuning on DriveQA confers substantial gains:

• Visual model accuracy on intersections jumps from ~20–40% to 70–86% after adaptation.
• Sign recognition, especially for rare or difficult categories, rises from near-zero to >80%+.
• Transfer learning from DriveQA pretraining improves real-world sign QA (Mapillary), waypoint planning (nuScenes), and action/explanation outputs (BDD-OIA).

5. Transfer, Generalization, and Downstream Impact

A central claim, substantiated experimentally, is that LLMs/MLLMs trained or adapted on DriveQA data can internalize driving knowledge that is both generalizable and transferrable:

• Sim-to-Real Transfer: Models fine-tuned on CARLA-based DriveQA-V transfer knowledge to real-world Mapillary annotations (gain of 10–25% accuracy).
• Downstream Task Enhancement: Pretraining on DriveQA improves autonomous waypoint planning L2 accuracy on nuScenes trajectory datasets (notably for smaller models).
• Action and Explanation Generation: On BDD-OIA, joint pretraining yields improvements in action and, notably, explanation mF1/F1ₐₗₗ metrics when compared to training only on the downstream task.

This direct linkage from driving knowledge test supervision to practical autonomous driving modules is a key foundational finding.

6. Limitations, Open Problems, and Research Directions

While DriveQA significantly broadens the assessment of driving-domain LLM/MLLM capabilities, several limitations and open research needs are explicitly noted:

• Static Scene Focus: The benchmark privileges static images and textual questions, lacking comprehensive temporal/video reasoning essential for complex driving scenarios and multi-agent causality (Xiu et al., 14 Sep 2025).
• US-Centric Regulations: Coverage is explicitly U.S.-centric; cross-jurisdictional generalization is untested.
• Synthetic-Real Gap: While sim2real improvements are demonstrated, further evidence and adaptation methods are needed, particularly for edge-case real-world events.
• Incomplete Edge-Case Visuals: Emergency vehicle scenarios, pedestrian intent, and globally diverse regulatory signs are not exhaustively included.
• Lack of Explicit Causal Formulation: Although some versions (e.g., Traffic-MLLM (Xiu et al., 14 Sep 2025)) add causal inference and retrieval-augmented reasoning layers, DriveQA’s own visual evaluation does not implement graphical, counterfactual, or interventionist modules.
• Quantitative Robustness Reporting: While controlled perturbations are designed, per-factor breakdowns of performance are not fully tabulated.

The benchmark thus serves as both an evaluative frontier and as a generative tool for research on explainable, regulation-grounded, and safety-critical driving AI.

7. Relation to Broader Driving QA and Model Innovation

DriveQA has become the canonical reference for regulation and rule-grounded driving QA evaluation. Model innovations in the domain—such as Traffic-MLLM's (Xiu et al., 14 Sep 2025) LoRA-based, knowledge-injection, and Chain-of-Thought plus Retrieval-augmented prompting architecture—demonstrate sharply improved performance on DriveQA’s most discriminating tasks compared to generic MLLMs. Empirical benchmarks show that traffic-domain adaptation, regulatory retrieval, and stepwise reasoning yield aggregate accuracy gains between 7-25% across regulatory, warning, guide, and temporary control sign categories. These findings anchor DriveQA as the primary benchmark for future multimodal, knowledge-intensive, and explainably robust autonomous driving AI.

---

References

• "Traffic-MLLM: A Spatio-Temporal MLLM with Retrieval-Augmented Generation for Causal Inference in Traffic" (Xiu et al., 14 Sep 2025)
• "DriveQA: Passing the Driving Knowledge Test" (Wei et al., 29 Aug 2025)