---
title: 'DriveQA-V: Visual Benchmark for Driving Knowledge'
url: https://www.emergentmind.com/topics/driveqa-v
type: topic
---

# DriveQA-V: Visual Benchmark for Driving Knowledge

DriveQA-V is the visual evaluation suite of the DriveQA benchmark, designed to systematically probe the capabilities and weaknesses of modern vision-and-language models (MLLMs) in context-grounded driving knowledge. It focuses on comprehensive coverage of U.S. traffic signage and intersection reasoning under controlled and challenging visual conditions, providing a basis for robust model development and quantitative comparison across a wide range of perceptual and cognitive driving tasks [2508.21824].

## 1. Scope and Sub-Tasks

DriveQA-V targets two fundamental tasks in machine driving knowledge:

1. **Traffic-sign comprehension**: Models are tested on their ability to identify or interpret 220 distinct U.S. sign models representing regulatory, warning, guide, and temporary-control categories. Each example prompts the model with, “What does this sign indicate?” and provides one correct answer plus three distractors drawn from the same category.

2. **Right-of-way judgment**: Involves answering “Which vehicle has the right-of-way?” in realistic intersection layouts containing up to four vehicles, with diverse and ambiguous spatial arrangements. Vehicles are colored for explicit reference, and questions rely on precise spatial reasoning.

Unlike prior driving-VQA datasets, DriveQA-V exhaustively covers long-tail sign types and complex multi-vehicle intersections while systematically varying camera perspective, distance, weather, lighting, and scene geometry to probe model robustness and generalization [2508.21824].

## 2. Dataset Construction and Statistics

The pipeline for constructing DriveQA-V involves procedural 3D synthetic generation with CARLA alongside manual annotation over Mapillary street-level imagery. This supports closed- and open-domain testing, adaptation, and sim-to-real transfer analysis.

- **Asset Integration**: 220 3D sign models imported into CARLA, enabling simulation of all major U.S. sign classes in diverse layouts. Two intersection templates (T-junction, four-way) are instantiated across five distinct CARLA towns.
- **Scene Sampling**:
  - For sign recognition: Each sign is placed at a roadside location. Camera angle ($\theta_{\text{yaw}}$, $\theta_{\text{pitch}}$), distance ($d$), weather, and lighting are systematically sampled and images rendered.
  - For intersection reasoning: Vehicles are placed at predefined stop/yield points with randomized colors and camera/environmental parameters.
- **Question Generation**: For every image, a multiple-choice QA pair is auto-generated. Distractors are sampled from in-category options for sign tasks or, for intersections, based on vehicle color/position to require spatial analysis.
- **Real-World Extension**: 1,303 Mapillary images with manual annotation cover 166 unique sign types, using identical QA protocols to evaluate sim-to-real model transfer.

Summary statistics:

| Metric                        | Value        |
|-------------------------------|-------------|
| Total CARLA images            | 68,000      |
| Total QA pairs (CARLA+Mapillary) | 448,000 |
| Sign-recognition QA pairs     | 240,000     |
| Intersection QA pairs         | 208,000     |
| Average questions per image   | ≈ 6.6       |
| Synthetic (CARLA) / Real (Mapillary) split | 90% / 10% |

## 3. Controlled Variation and Metadata

DriveQA-V is explicitly designed for analyzing sensitivity to specific environmental factors. For every image and QA-pair, four principal factors are controlled and logged:

- **Camera Perspective**: Front view ($\theta_{\text{yaw}}=0^\circ, \theta_{\text{pitch}}=0^\circ$), Oblique ($\theta_{\text{yaw}}\sim\mathcal{U}(30^\circ,60^\circ)$, $\theta_{\text{pitch}}\sim\mathcal{U}(15^\circ,30^\circ)$), and Top-down ($\theta_{\text{pitch}}=75^\circ$) perspectives are selected through fixed intrinsics and sampled extrinsics via the pinhole model.
- **Distance**:
  - Sign scenes: $d \in \{10, 20, 30, 50\}$ meters
  - Intersections: nominally 25 m, optionally varied in $\pm5$ m
- **Weather Conditions**: Five CARLA presets (ClearNoon, CloudyNoon, WetNoon, RainNoon, WetCloudyNoon) manipulate rain ($r_{\text{rain}}$), fog ($f_{\text{fog}}$), and puddles ($p_{\text{puddles}}$) independently.
- **Lighting/Time of Day**: Four diurnal bins (Dawn, Noon, Dusk, Night) realized via $t \in [0,24]$ h, together with global pixel brightness scaling and shift ($c\sim \mathcal{U}(0.7,1.3)$, $\Delta B\sim\mathcal{U}(-30,+30)$).

All variable values are recorded in image metadata, supporting fine-grained ablation and sensitivity studies.

## 4. Evaluation Metrics and Experimental Analysis

DriveQA-V introduces factor-wise accuracy drop and a robustness score:

- **Per-factor accuracy drop**: For factor $i$, accuracy drop is $\Delta_{\text{acc}}^i = \mathrm{Acc}_{\text{clean}} - \mathrm{Acc}_i$, where $\mathrm{Acc}_{\text{clean}}$ is under reference conditions (ClearNoon, front-view, 25 m).
- **Robustness Score**:
  $$
  R = 1 - \frac{1}{N}\sum_{i=1}^N |\mathrm{Acc}_i - \mathrm{Acc}_{\text{clean}}|
  $$
  with $R\approx1$ denoting insensitivity to the factor.

Empirical findings:

| Model                | F-Front→F-Front | T-Top→T-Top | Reg. Signs | Warning Signs | Guide Signs | Temp. Control | Avg              |
|----------------------|-----------------|-------------|------------|--------------|-------------|---------------|------------------|
| Mini-InternVL (2B)   | 27.8 → 86.7     | 24.8 → 82.1 | 64.1 → 93.8| 55.3 → 92.2  | 65.8 → 91.1 | 45.0 → 96.5   | 41.8 → 86.6      |
| LLaVA-1.5 (7B)       | 23.3 → 64.2     | 23.1 → 70.6 | 23.5 → 72.2| 26.6 → 73.0  | 22.3 → 76.8 | 21.1 → 89.3   | 23.5 → 69.7      |
| GPT-4o               | 55.1            | 60.4        | 93.8       | 94.0         | 95.1        | 94.4          | 75.3             |

Notable patterns:
- Intersection reasoning exhibits pronounced accuracy drops for camera perspective changes (Mini-InternVL: 27.8% to 24.8%; LLaVA: 23.3% to 23.1% zero-shot), while sign classification is less affected.
- Regulatory and Warning signs suffer the largest zero-shot degradation compared to Guide and Temporary-Control signs (up to 20 percentage points lower).
- Fine-tuning on DriveQA-V enables open-source models to exceed 80–90% across categories.
- The hardest sign types (Playground, Trauma Center, Low Ground Clearance, etc.) have 0–20% accuracy off-the-shelf but reach 80–100% post-fine-tuning.

## 5. Degradation Sources and Factor Interactions

Models are most sensitive to:

- **Rare sign categories**: Zero-shot classification performance for uncommon Regulatory/Warning signs drops by up to 30 percentage points.
- **Intersection layout changes**: Shifts from front to top-down angles reduce raw accuracy, especially for intersection reasoning, with models defaulting to near-random guesses absent fine-tuning.
- **Adverse visual conditions**: Night, heavy rain, or fog induce an additional 5–10 percentage point decrease in zero-shot accuracy.

Observed factor interactions include compounding effects:
- Heavy fog at longer distances ($f>0.7$, $d>30$ m) leads to near-random sign guesses.
- Night-time oblique views accentuate occlusion and glare, impairing right-of-way judgments.

## 6. Recommendations for Model Robustness

To mitigate failures revealed by DriveQA-V:

1. **Domain-randomized pre-training**: Systematic exposure to broad ranges in camera parameterization, environmental variation, and lighting in synthetic data.
2. **Long-tail oversampling**: Increasing sampling weights for rare Regulatory/Warning sign types during fine-tuning or initial pre-training.
3. **Multi-angle inference**: Using ensemble predictions across multiple camera views (e.g., across front, oblique, and top-down perspectives).
4. **Feature-level adaptation**: Applying contrastive objectives that encourage consistent features between clear-day and adverse-weather images.

*A plausible implication is that model robustness to real driving-domain QA tasks critically depends on both data diversity (including synthetic augmentation) and task-specific fine-tuning on challenging, long-tail scenarios*.

## 7. Significance and Open-Source Resources

DriveQA-V constitutes the first large-scale, systematically varied visual benchmark for multimodal driving knowledge acquisition. Its exhaustive coverage, controlled variability, and integration of real and synthetic data provide a rigorous platform for evaluating MLLM spatial grounding, adaptation, and real-world transfer. All code, assets, parameterizations, and reproducibility scripts are openly available, enabling the research community to extend, re-assess, or ablate every aspect of the benchmark, fostering ongoing improvements in model generalization and robustness [2508.21824].

Source: https://www.emergentmind.com/topics/driveqa-v