Ego3D-Bench: 3D Spatial Reasoning Benchmark

• Ego3D-Bench is a benchmark for evaluating 3D spatial reasoning in multi-view outdoor scenes through over 8,600 human-annotated spatial QA pairs.
• The dataset leverages multi-view data from NuScenes, Waymo, and Argoverse 1 to capture diverse urban, highway, and rural scenes in ego-centric settings.
• It introduces Ego3D-VLM, a modular framework that enhances VLM spatial understanding by integrating cognitive maps derived from 3D object coordinates.

Ego3D-Bench is a benchmark for evaluating the spatial reasoning abilities of Vision-LLMs in ego-centric, multi-view outdoor scenes. It was introduced to address the mismatch between prior spatial question-answering datasets, which largely emphasize single images or indoor videos, and the perceptual regime of embodied AI agents such as robots and self-driving cars, which operate from ego-centric, multi-view observations. The benchmark comprises over 8,600 spatial QA pairs, includes substantial human annotation, and was released together with Ego3D-VLM, a post-training framework that injects a cognitive map derived from estimated global 3D coordinates into a VLM pipeline. Reported results show a notable gap between human performance and contemporary VLMs, indicating that current systems remain below human level spatial understanding in this setting (Gholami et al., 8 Sep 2025).

1. Conceptual scope and benchmark identity

Ego3D-Bench targets 3D spatial reasoning rather than generic visual question answering. Its design centers on real-world, ego-centric, multi-view outdoor data and on questions that require spatial integration across views. The stated motivation is that existing spatial reasoning benchmarks focus primarily on single-view images or static indoor video, whereas embodied agents rely on ego-centric, multi-view observations in dynamic scenes. Accordingly, Ego3D-Bench is framed as an evaluation substrate for spatial understanding under outdoor, multi-agent, multi-object conditions (Gholami et al., 8 Sep 2025).

The benchmark should be distinguished from the similarly named Ego3DT Benchmark, which was introduced for zero-shot 3D multi-object tracking in egocentric videos and consists of Ego3DT-daily and Ego3DT-indoor for tracking and reconstruction evaluation. Ego3DT Benchmark emphasizes localization and identity maintenance of objects in video, whereas Ego3D-Bench emphasizes spatial QA for VLMs in multi-view scenes (Hao et al., 2024).

A central design principle is exclusion: Ego3D-Bench omits questions that can be answered from a single view, by object counting, by generic visual recognition, or by external world knowledge such as typical object size. This makes the benchmark a test of explicit spatial integration rather than a proxy for recognition or commonsense priors (Gholami et al., 8 Sep 2025).

2. Dataset construction and annotation protocol

Ego3D-Bench is built from the validation sets of three outdoor driving datasets: NuScenes with 6 views, Waymo Open Dataset with 7 views, and Argoverse 1 with 5 views. These sources provide urban, highway, and rural scenes with overlapping camera viewpoints, which the benchmark uses to instantiate ego-centric, multi-view outdoor scenarios (Gholami et al., 8 Sep 2025).

The QA construction process incorporates extensive human involvement. Human annotators first identify unique objects in each scene, a step described as challenging in outdoor settings because many objects may share the same category, such as cars or pedestrians. Annotators then write discriminative, short object captions for unique identification. Ground-truth 3D bounding box annotations are collected and associated with specific camera views. Question templates are then instantiated by inserting object descriptions and views, and answers are generated by rule-based functions leveraging the ground-truth 3D annotations. For challenging questions, especially motion-related ones, a final human review is performed to ensure correctness (Gholami et al., 8 Sep 2025).

This construction protocol is intended to ensure both diversity and answer fidelity. The use of predefined templates constrains question semantics, while the dependence on ground-truth 3D annotations ties answers to geometric scene structure rather than annotation heuristics. A plausible implication is that the benchmark is designed to reduce both linguistic ambiguity and shortcut exploitation.

3. Task taxonomy and evaluative principles

Ego3D-Bench organizes its QA pairs into five task categories, with ego-centric and object-centric variants when applicable. These categories are absolute distance measurement, relative distance measurement, localization, motion reasoning, and travel time (Gholami et al., 8 Sep 2025).

Absolute distance measurement asks for metric distance either between ego and another object or between two objects, and appears in both multi-choice and open numerical forms. Relative distance measurement asks which of two objects is closer, either to ego or to another object, and is always two-choice. Localization is object-centric only and requires inferring the location of one object from a second object’s perspective in multi-choice form. Motion reasoning asks whether ego or an object moving in a cardinal direction, $N/E/S/W$, gets closer to or farther from a second object, typically in yes/no form. Travel time asks for approximate or multi-choice time to move from ego or object-1 to object-2 at a given speed (Gholami et al., 8 Sep 2025).

Only questions requiring true multi-view, 3D spatial integration are retained. Questions answerable through visual recognition, counting, or external knowledge are excluded. Human performance is estimated by sampling 10% of questions per category and having annotators answer them. This evaluation design positions the benchmark as a measurement of compositional spatial reasoning rather than broad multimodal competence (Gholami et al., 8 Sep 2025).

4. Evaluation protocol and empirical findings

The benchmark evaluates 16 state-of-the-art VLMs. The reported set includes closed-source models such as GPT-4o and Gemini1.5-Pro, open-source models such as InternVL3, Qwen2.5-VL, Ovis2, and LLaVA-One-Vision, and 3D-specific VLMs such as SpaceThinker-Qwen2.5-3B, SpatialRGPT (SRGPT-VILA1.5-8B), and SpaceQwen2.5-3B (Gholami et al., 8 Sep 2025).

Two metrics are used. Multi-choice QA is evaluated with accuracy in percent, and absolute distance estimation is evaluated with root mean squared error in meters. Chance level is computed per category and is typically about 25–50%. Human performance is reported as well above all VLMs, at approximately 85% average accuracy (Gholami et al., 8 Sep 2025).

The reported performance spread is substantial but still far from human level. GPT-4o reaches about 56.7% average accuracy with about 19.2 m average RMSE; Gemini1.5-Pro reaches about 57.5% average accuracy with about 19.6 m average RMSE; InternVL3-78B reaches about 59.9% accuracy with about 13.8 m RMSE; and Qwen2.5-72B reaches about 58.0% accuracy with about 16.2 m RMSE. Smaller models in the 3B–8B range operate near chance level, around 25–40%. Large models show only moderate gains. Travel time, localization, and object-centric absolute distance estimation are identified as particularly difficult, with even the best models averaging only about 40–45% accuracy in those categories. Blind text-only VLMs perform about 16% above chance but about 5% below vision-enabled models, which the benchmark authors interpret as confirmation that visual context remains important (Gholami et al., 8 Sep 2025).

These results support a narrow conclusion: scaling alone does not close the spatial reasoning gap in ego-centric, multi-view environments. A plausible implication is that current VLM deficiencies are not reducible to language modeling capacity alone.

5. Ego3D-VLM and cognitive-map augmentation

Ego3D-Bench was introduced together with Ego3D-VLM, a post-training framework whose goal is to enhance 3D spatial reasoning of any VLM by adding a cognitive map derived from multi-view images and estimated object locations in 3D space, without modifying the base model’s weights. The framework is modular and can be integrated with existing VLM pipelines (Gholami et al., 8 Sep 2025).

The pipeline begins with multi-view images and a language prompt. A referring expression comprehension model, exemplified by Grounding-DINO-Base, locates prompt-mentioned objects in each view:

$$\mathcal{O}^{(v)} = \{ (b_i^{(v)}, c_i^{(v)}) \}_{i=1}^{N^{(v)}}$$

where $b_i^{(v)}$ denotes a 2D bounding box for object $i$ in camera view $v$, and $c_i^{(v)}$ denotes the matched referring expression. A metric depth estimator, exemplified by Depth-Anything-V2-Metric-Large, predicts dense depth for each view. Depth at the center pixel of each bounding box is then used to project 2D detections into 3D camera coordinates:

$$\mathbf{p}_{\text{cam},i}^{(v)} = d_i^{(v)} \cdot \left(K^{(v)}\right)^{-1} \begin{bmatrix} x_i \ y_i \ 1 \end{bmatrix}$$

These coordinates are transformed into a global front-camera ego-centric frame using view-specific rotation $R^{(v)}$ and translation $T^{(v)}$:

$$\mathbf{p}_{\text{global},i}^{(v)} = \begin{bmatrix} R^{(v)} & T^{(v)} \ 0 & 1 \end{bmatrix} \begin{bmatrix} \mathbf{p}_{\text{cam},i}^{(v)} \ 1 \end{bmatrix}$$

The framework then applies relational scaling to mitigate outdoor metric-depth error by leveraging familiar object heights. A cognitive map generation function, $$\mathcal{O}^{(v)} = \{ (b_i^{(v)}, c_i^{(v)}) \}_{i=1}^{N^{(v)}}$$0, converts the referenced objects and their 3D positions into a compact structured representation, typically textual but optionally JSON or visual:

$$\mathcal{O}^{(v)} = \{ (b_i^{(v)}, c_i^{(v)}) \}_{i=1}^{N^{(v)}}$$1

The final answer is produced by conditioning the VLM on the cognitive map $$\mathcal{O}^{(v)} = \{ (b_i^{(v)}, c_i^{(v)}) \}_{i=1}^{N^{(v)}}$$2, the image set $$\mathcal{O}^{(v)} = \{ (b_i^{(v)}, c_i^{(v)}) \}_{i=1}^{N^{(v)}}$$3, and the query $$\mathcal{O}^{(v)} = \{ (b_i^{(v)}, c_i^{(v)}) \}_{i=1}^{N^{(v)}}$$4:

$$\mathcal{O}^{(v)} = \{ (b_i^{(v)}, c_i^{(v)}) \}_{i=1}^{N^{(v)}}$$5

The benchmark reports that textual and JSON cognitive maps work best, both at about 60% accuracy, whereas visual maps reach about 51%. The stated rationale is that the map compresses spatial information to referenced objects only, reducing token overhead relative to point-cloud or bird’s-eye-view representations and making the approach plug-and-play and approximately 10 times faster than those denser alternatives (Gholami et al., 8 Sep 2025).

6. Measured gains, remaining gap, and benchmark significance

Ego3D-VLM yields an average improvement of 12% on multi-choice QA and a 56% average improvement on absolute distance estimation, reported as a reduction from about 19 m RMSE to about 8.5 m RMSE for large VLMs. For GPT-4o specifically, the benchmark reports a shift from 56.7% accuracy and 19.2 m RMSE to 73.2% accuracy and 7.4 m RMSE when augmented with Ego3D-VLM. Gains are described as consistent across closed-source, open-source, and 3D-oriented VLM families, independent of parameter count (Gholami et al., 8 Sep 2025).

Even with these improvements, the human-model gap is not closed. Across categories, humans are at about 85% accuracy; the best unaugmented VLMs are at about 55–60%; and the best models with Ego3D-VLM reach about 68–73%. An upper bound using ground-truth cognitive maps reaches about 79%, leaving a residual gap attributed to VLM deficiencies in reasoning, perception errors, and out-of-distribution scenarios. The benchmark also notes an asymmetry across task types: models augmented with Ego3D-VLM can exceed human performance on numeric object-centric absolute distance estimation because the models are given explicit 3D location data, yet they still trail humans on spatial localization (Gholami et al., 8 Sep 2025).

The authors further report that providing ground-truth 2D bounding boxes helps only marginally, which they interpret as evidence that the major bottleneck is 3D spatial reasoning rather than detection. Ego3D-VLM also transfers to All-Angle Bench and VSI-Bench, with consistent gains in other multi-view settings. This suggests that the benchmark is not merely diagnosing one dataset-specific weakness but a broader limitation of contemporary VLMs in explicit spatial grounding (Gholami et al., 8 Sep 2025).

7. Position within egocentric 3D research

Ego3D-Bench occupies a specific niche within egocentric 3D research: evaluation of VLM spatial reasoning in ego-centric, multi-view outdoor environments. Adjacent benchmarks emphasize different tasks. EgoLoc addresses Visual Queries with 3D Localization in egocentric video and reports an overall success rate of up to 87.12% on VQ3D through improved pose estimation and multi-view aggregation (Mai et al., 2022). EgoHumans is an egocentric 3D multi-human benchmark for detection, tracking, 2D/3D pose estimation, and mesh recovery, with more than 125k egocentric images and a focus on challenging multi-human activities (Khirodkar et al., 2023). Ego3DT Benchmark evaluates zero-shot 3D multi-object tracking and reconstruction in egocentric video rather than QA-based spatial reasoning (Hao et al., 2024).

The benchmark also connects to synthetic-data efforts. EgoGen is an egocentric synthetic data generator that is explicitly described as enabling scalable and flexible generation of data for benchmarks like Ego3D-Bench, with support for mapping, localization, tracking, and mesh recovery tasks in first-person settings (Li et al., 2024).

Taken together, these works delineate complementary strands of egocentric 3D research: localization, tracking, pose and mesh recovery, synthetic data generation, and spatial reasoning evaluation. Ego3D-Bench is the component in this landscape that isolates and measures the capacity of VLMs to answer multi-view spatial questions under outdoor, dynamic, ego-centric conditions.