DriveQA-V: Visual Benchmark for Driving Knowledge

• DriveQA-V is a visual evaluation suite that assesses vision-and-language models’ understanding of 220 U.S. traffic sign models and spatial reasoning in intersections.
• It uses procedural 3D simulation via CARLA and real-world Mapillary imagery to generate 448,000 QA pairs with controlled variations in camera, distance, weather, and lighting.
• It provides quantitative metrics like per-factor accuracy drop and robustness scores to guide fine-tuning strategies and improve model generalization for complex driving tasks.

DriveQA-V is the visual evaluation suite of the DriveQA benchmark, designed to systematically probe the capabilities and weaknesses of modern vision-and-LLMs (MLLMs) in context-grounded driving knowledge. It focuses on comprehensive coverage of U.S. traffic signage and intersection reasoning under controlled and challenging visual conditions, providing a basis for robust model development and quantitative comparison across a wide range of perceptual and cognitive driving tasks (Wei et al., 29 Aug 2025).

1. Scope and Sub-Tasks

DriveQA-V targets two fundamental tasks in machine driving knowledge:

1. Traffic-sign comprehension: Models are tested on their ability to identify or interpret 220 distinct U.S. sign models representing regulatory, warning, guide, and temporary-control categories. Each example prompts the model with, “What does this sign indicate?” and provides one correct answer plus three distractors drawn from the same category.
2. Right-of-way judgment: Involves answering “Which vehicle has the right-of-way?” in realistic intersection layouts containing up to four vehicles, with diverse and ambiguous spatial arrangements. Vehicles are colored for explicit reference, and questions rely on precise spatial reasoning.

Unlike prior driving-VQA datasets, DriveQA-V exhaustively covers long-tail sign types and complex multi-vehicle intersections while systematically varying camera perspective, distance, weather, lighting, and scene geometry to probe model robustness and generalization (Wei et al., 29 Aug 2025).

2. Dataset Construction and Statistics

The pipeline for constructing DriveQA-V involves procedural 3D synthetic generation with CARLA alongside manual annotation over Mapillary street-level imagery. This supports closed- and open-domain testing, adaptation, and sim-to-real transfer analysis.

• Asset Integration: 220 3D sign models imported into CARLA, enabling simulation of all major U.S. sign classes in diverse layouts. Two intersection templates (T-junction, four-way) are instantiated across five distinct CARLA towns.
• Scene Sampling:
  • For sign recognition: Each sign is placed at a roadside location. Camera angle ($\theta_{\text{yaw}}$, $\theta_{\text{pitch}}$), distance ($d$), weather, and lighting are systematically sampled and images rendered.
  • For intersection reasoning: Vehicles are placed at predefined stop/yield points with randomized colors and camera/environmental parameters.
• Question Generation: For every image, a multiple-choice QA pair is auto-generated. Distractors are sampled from in-category options for sign tasks or, for intersections, based on vehicle color/position to require spatial analysis.
• Real-World Extension: 1,303 Mapillary images with manual annotation cover 166 unique sign types, using identical QA protocols to evaluate sim-to-real model transfer.

Summary statistics:

Metric	Value
Total CARLA images	68,000
Total QA pairs (CARLA+Mapillary)	448,000
Sign-recognition QA pairs	240,000
Intersection QA pairs	208,000
Average questions per image	≈ 6.6
Synthetic (CARLA) / Real (Mapillary) split	90% / 10%

3. Controlled Variation and Metadata

DriveQA-V is explicitly designed for analyzing sensitivity to specific environmental factors. For every image and QA-pair, four principal factors are controlled and logged:

• Camera Perspective: Front view ($$\theta_{\text{yaw}}=0^\circ, \theta_{\text{pitch}}=0^\circ$$), Oblique ($$\theta_{\text{yaw}}\sim\mathcal{U}(30^\circ,60^\circ)$$, $$\theta_{\text{pitch}}\sim\mathcal{U}(15^\circ,30^\circ)$$), and Top-down ($\theta_{\text{pitch}}=75^\circ$) perspectives are selected through fixed intrinsics and sampled extrinsics via the pinhole model.
• Distance:
  • Sign scenes: $d \in \{10, 20, 30, 50\}$ meters
  • Intersections: nominally 25 m, optionally varied in $\pm5$ m
• Weather Conditions: Five CARLA presets (ClearNoon, CloudyNoon, WetNoon, RainNoon, WetCloudyNoon) manipulate rain ($r_{\text{rain}}$), fog ($\theta_{\text{pitch}}$0), and puddles ($\theta_{\text{pitch}}$1) independently.
• Lighting/Time of Day: Four diurnal bins (Dawn, Noon, Dusk, Night) realized via $\theta_{\text{pitch}}$2 h, together with global pixel brightness scaling and shift ($\theta_{\text{pitch}}$3, $\theta_{\text{pitch}}$4).

All variable values are recorded in image metadata, supporting fine-grained ablation and sensitivity studies.

4. Evaluation Metrics and Experimental Analysis

DriveQA-V introduces factor-wise accuracy drop and a robustness score:

• Per-factor accuracy drop: For factor $\theta_{\text{pitch}}$5, accuracy drop is $\theta_{\text{pitch}}$6, where $\theta_{\text{pitch}}$7 is under reference conditions (ClearNoon, front-view, 25 m).
• Robustness Score:

$\theta_{\text{pitch}}$8

with $\theta_{\text{pitch}}$9 denoting insensitivity to the factor.

Empirical findings:

Model	F-Front→F-Front	T-Top→T-Top	Reg. Signs	Warning Signs	Guide Signs	Temp. Control	Avg
Mini-InternVL (2B)	27.8 → 86.7	24.8 → 82.1	64.1 → 93.8	55.3 → 92.2	65.8 → 91.1	45.0 → 96.5	41.8 → 86.6
LLaVA-1.5 (7B)	23.3 → 64.2	23.1 → 70.6	23.5 → 72.2	26.6 → 73.0	22.3 → 76.8	21.1 → 89.3	23.5 → 69.7
GPT-4o	55.1	60.4	93.8	94.0	95.1	94.4	75.3

Notable patterns:

• Intersection reasoning exhibits pronounced accuracy drops for camera perspective changes (Mini-InternVL: 27.8% to 24.8%; LLaVA: 23.3% to 23.1% zero-shot), while sign classification is less affected.
• Regulatory and Warning signs suffer the largest zero-shot degradation compared to Guide and Temporary-Control signs (up to 20 percentage points lower).
• Fine-tuning on DriveQA-V enables open-source models to exceed 80–90% across categories.
• The hardest sign types (Playground, Trauma Center, Low Ground Clearance, etc.) have 0–20% accuracy off-the-shelf but reach 80–100% post-fine-tuning.

5. Degradation Sources and Factor Interactions

Models are most sensitive to:

• Rare sign categories: Zero-shot classification performance for uncommon Regulatory/Warning signs drops by up to 30 percentage points.
• Intersection layout changes: Shifts from front to top-down angles reduce raw accuracy, especially for intersection reasoning, with models defaulting to near-random guesses absent fine-tuning.
• Adverse visual conditions: Night, heavy rain, or fog induce an additional 5–10 percentage point decrease in zero-shot accuracy.

Observed factor interactions include compounding effects:

• Heavy fog at longer distances ($d$0, $d$1 m) leads to near-random sign guesses.
• Night-time oblique views accentuate occlusion and glare, impairing right-of-way judgments.

6. Recommendations for Model Robustness

To mitigate failures revealed by DriveQA-V:

1. Domain-randomized pre-training: Systematic exposure to broad ranges in camera parameterization, environmental variation, and lighting in synthetic data.
2. Long-tail oversampling: Increasing sampling weights for rare Regulatory/Warning sign types during fine-tuning or initial pre-training.
3. Multi-angle inference: Using ensemble predictions across multiple camera views (e.g., across front, oblique, and top-down perspectives).
4. Feature-level adaptation: Applying contrastive objectives that encourage consistent features between clear-day and adverse-weather images.

A plausible implication is that model robustness to real driving-domain QA tasks critically depends on both data diversity (including synthetic augmentation) and task-specific fine-tuning on challenging, long-tail scenarios.

7. Significance and Open-Source Resources

DriveQA-V constitutes the first large-scale, systematically varied visual benchmark for multimodal driving knowledge acquisition. Its exhaustive coverage, controlled variability, and integration of real and synthetic data provide a rigorous platform for evaluating MLLM spatial grounding, adaptation, and real-world transfer. All code, assets, parameterizations, and reproducibility scripts are openly available, enabling the research community to extend, re-assess, or ablate every aspect of the benchmark, fostering ongoing improvements in model generalization and robustness (Wei et al., 29 Aug 2025).