---
title: 'NuScenes-SpatialQA: Spatial Reasoning in Driving'
url: https://www.emergentmind.com/topics/nuscenes-spatialqa-spatialqa-e
type: topic
---

# NuScenes-SpatialQA: Spatial Reasoning in Driving

NuScenes-SpatialQA (SpatialQA-E) is a large-scale, ground-truth-based question answering (QA) benchmark designed specifically to evaluate spatial understanding and reasoning capabilities of Vision-Language Models (VLMs) in real-world autonomous driving scenarios. Built upon the nuScenes dataset, NuScenes-SpatialQA leverages a fully automated 3D scene graph pipeline and an extensive set of task-driven QA templates to systematically probe both qualitative and quantitative facets of spatial understanding, as well as multi-hop and situational reasoning critical to autonomous driving decision-making. Unlike prior benchmarks that focus on toy or synthetic indoor scenes, or that rely on monocular depth estimation, NuScenes-SpatialQA exploits precise LiDAR-based ground truth to provide unambiguous, metric-accurate supervision for spatial relations, distances, and object sizes, filling a long-standing gap in the evaluation of spatially-aware VLMs [2504.03164].

## 1. Objectives and Scope

NuScenes-SpatialQA targets two core objectives: (1) to provide the first large-scale, ground-truth-driven QA benchmark for systematically evaluating VLMs on spatial understanding and reasoning in urban driving environments, and (2) to focus on scene-centric spatial tasks directly relevant to critical driving decisions, such as navigation, obstacle negotiation, and occlusion handling.

The benchmark addresses deficiencies in conventional VQA datasets, which either lack driving-specific spatial question types, utilize noisy or depth-biased estimates, or are limited to non-driving domains. By leveraging the nuScenes dataset's LiDAR annotations, SpatialQA eliminates depth estimation biases and supports both qualitative (categorical/relational) and quantitative (metric, numeric) QA tasks spanning multi-hop reasoning.

## 2. Dataset Construction and Scene Graph Pipeline

SpatialQA-E is built on the validation split of nuScenes trainval-v1.0, yielding 150 scenes × 40 keyframes (at 2 Hz), each captured using six cameras with precise 3D bounding boxes.

The automated 3D scene graph construction pipeline proceeds as follows:
- Each ground-truth 3D object is projected onto all available camera views (using nuscenes-devkit). Objects are filtered by minimum visibility and 2D bounding box size.
- All appearances of a distinct instance across keyframes are grouped, and the best single crop is selected via a metric combining 2D bounding box area and brightness.
- Concise attribute-centric captions (e.g., "red sedan with black roof") are synthesized using LLaMA-3.2 with structured prompting.
- Scene graph nodes are identified by unique instance_token, and include attributes for category_name, translation (in ego-vehicle coordinates), dimensions, and caption.
- Scene graph edges capture spatial relations: Euclidean distance, longitudinal and lateral offsets relative to the ego vehicle's heading, and relative bearing angle computed as θ = atan2$(y_2 - y_1, x_2 - x_1)$ in degrees.

## 3. QA Generation, Coverage, and Categories

NuScenes-SpatialQA features an automated QA generation pipeline, utilizing more than 35 reusable templates spanning spatial understanding and reasoning. The dataset contains approximately 3.5 million QA pairs:
- Qualitative spatial understanding (≈2.5M): Binary yes/no comparisons of position (above/below, left/right, front/behind) and size (larger/smaller, taller/shorter, wider/thinner, longer/shorter).
- Quantitative spatial understanding (≈0.6M): Numeric requests for absolute distances, offsets, object dimensions, and relative angles.
- Direct and situational spatial reasoning (≈0.2M): Multi-choice or binary reasoning about proximity, presence in defined regions, and scenario-based inference (e.g., collision prediction, occlusion, or parking fit).

QA generation operates at the level of 6,000 keyframes × 6 cameras per scene, ensuring broad coverage and redundancy for both frequent and rare object configurations.

## 4. Task Structure, Question Taxonomy, and Evaluation

SpatialQA-E evaluates VLMs along multiple axes:
- **Relative positioning:** Queries about categorical spatial relations, e.g., "Is the pedestrian to the left of the red car?"
- **Size comparison:** Pairwise/comparative questions about volume or specific dimensions.
- **Distance and offset estimation:** Numeric questions requiring precise metric inference, such as determining the Euclidean distance or relative offset in the ego-vehicle frame.
- **Dimension and angle measurement:** Extraction of object size or orientation, e.g., “What is the bearing angle of the truck relative to the car?”
- **Counting and selection:** Identification of nearest, farthest, or specific objects from a candidate set.
- **Scenario-based inference:** Relational or temporal/logical queries, such as "Will these objects collide in five seconds given specific velocities?"

Evaluation is performed using:
- **Closed-ended accuracy** for binary and categorical queries.
- **Tolerance-based accuracy (TAcc):** Proportion of numeric predictions within [75%, 125%] of ground truth.
- **Mean absolute error (MAE):** Average absolute error for each numeric prediction dimension.

## 5. Vision-Language Model Baselines and Analysis

The benchmark includes extensive evaluation of both general-purpose and spatially-enhanced VLMs:
- LLaVA-v1.6 (Mistral-7B, Vicuna-7B/13B/34B), Llama-3.2-11B-Vision-Instruct, BLIP-2-Flan-T5-XL, Qwen2.5-VL-7B-Instruct, Deepseek-vl2-tiny, and SpatialRGPT (spatially fine-tuned with synthetic spatial data + CLIP-L encoder).

Notable empirical findings:
- Qualitative spatial understanding (accuracy ≈50–60%) remains modest across all baselines; spatially-enhanced VLMs, particularly SpatialRGPT, excel in size-based binary questions.
- Quantitative tasks are significantly more difficult (TAcc <20% for most models on distance/offset, with higher scores in dimension estimation for certain models).
- Spatial enhancement confers improvements for qualitative tasks (especially size/shape), but not for numeric estimation.
- Direct geometric reasoning is consistently more challenging than situational/semantic reasoning—models achieve >73% on situational reasoning but only ~41–58% on direct spatial queries.

An ablation study reveals that backbone choice (Mistral-7B > Vicuna-7B), larger parameter count, and prompt engineering (CoT generally degrades performance by 6–13 percentage points in spatial reasoning) all influence performance.

| Model               | Qualitative Accuracy (%) | Quantitative TAcc (%) | Direct Reasoning TAcc (%) | Situational Reasoning TAcc (%) |
|---------------------|-------------------------|-----------------------|---------------------------|-------------------------------|
| LLaVA-v1.6 (M7B)    | 53.3                    | 35.5                  | 48.5                      | 73.5                          |
| Llama-3.2-11B       | 54.3                    | 16.1                  | 41.8                      | 37.3                          |
| Qwen2.5-VL-7B       | 58.0                    | 19.4                  | 58.2                      | 84.1                          |
| SpatialRGPT         | 59.8                    | 14.6                  | 45.5                      | 80.8                          |

## 6. Error Patterns, Challenges, and Underlying Model Limitations

Analysis of error modes reveals:
- Confusion in distinguishing axes (front/behind versus above/below) in cluttered, occluded scenes.
- High MAE and instability in numeric outputs, particularly for dimensions with high variance (width, height).
- Substantial challenges with long-range metric inference (e.g., distances >20 m), multi-object comparatives, and angle estimations (errors exceeding 100° MAE in some instances).

Underlying causes include the absence of dense, metric spatial supervision during pretraining, inductive bias from prompts or language model reasoning, and insufficient extrapolation capability given existing spatially-enhanced fine-tuning regimes.

## 7. Implications, Comparative Context, and Future Directions

NuScenes-SpatialQA provides, for the first time, a comprehensive ground-truth, LiDAR-based QA testbed for spatially-oriented VLMs in urban driving. General VLMs suffer pronounced deficiencies in quantitative spatial inference, while current spatially-enhanced models only marginally improve qualitative or categorical tasks. The relative tractability of situational reasoning likely reflects reliance on pretrained commonsense or world-knowledge, rather than direct metric geometry.

Future work suggested by these findings includes:
- Incorporation of metric spatial objectives (e.g., 3D contrastive learning) into pretraining,
- Development of spatially-guided or geometric step-wise prompting,
- Extension of evaluation to broader traffic domains (highway, rural, adverse weather),
- Integration of explicit 3D scene representation modules (e.g., BEV encoders) within multi-modal foundation models.

NuScenes-SpatialQA establishes a new paradigm for rigorous, fine-grained assessment of spatial reasoning in vision-language systems, and directly addresses limitations highlighted in related graph-structured spatial representations (e.g., nuScenes Knowledge Graph [2312.09676]), laying a robust groundwork for the next generation of spatially competent, physically grounded VLMs in autonomous systems.

Source: https://www.emergentmind.com/topics/nuscenes-spatialqa-spatialqa-e