---
title: 'SpatialUncertain: A Benchmark for Spatial Reasoning'
url: https://www.emergentmind.com/topics/spatialuncertain
type: topic
---

# SpatialUncertain: A Benchmark for Spatial Reasoning

SpatialUncertain refers to a class of benchmarks, metrics, and tasks for evaluating spatial reasoning under observation uncertainty, with a focus on scenarios involving occlusion and perspective ambiguity. The SpatialUncertain framework was introduced to systematically assess whether vision-language models (VLMs) deployed in real-world 3D environments can recognize when a spatial question is unanswerable from a given observation and whether they can actively seek additional evidence rather than producing incorrect or overconfident answers [2605.30557]. This marks a shift from traditional spatial VQA and scene understanding benchmarks, which focus mainly on answer correctness for fully observable scenes and typically neglect uncertainty quantification, abstention, and evidence-seeking capabilities.

## 1. Motivation and Problem Definition

SpatialUncertain addresses the central limitation that visual observations (e.g., RGB images) are inherently partial representations of 3D worlds. Occlusion may hide relevant objects completely, and perspective effects can systematically distort geometric cues, leading to ambiguous or indeterminate evidence.

Formally, for a given spatial question $Q$ and a rendered observation $O$ from a specified camera pose in a fixed scene, the ground-truth answerability function $A(O, Q) \in \{0, 1\}$ encodes whether unambiguous information for $Q$ is present in $O$. The benchmark tasks VLMs with returning either one of the candidate answers or the abstention token "Cannot determine," enforcing explicit recognition of observational uncertainty.

This benchmark diverges from prior VQA/spatial analysis frameworks, which typically assume that every question is answerable from the provided observation and optimize models exclusively for answer accuracy.

## 2. Categories of Uncertainty: Occlusion and Perspective Ambiguity

SpatialUncertain introduces two primary controlled sources of unanswerability:

- **Occlusion:** A scene is augmented so that a target object is positioned behind an occluder along the line of sight, rendering all, or a significant portion, of the object invisible. The extent of occlusion (none, partial, or full) is systematically annotated.

- **Perspective Ambiguity:** Camera views are selected that create systematic ambiguities due to perspective projection. For example, by shifting the camera laterally toward one of two similarly sized objects, projected visual cues like apparent size or shape become unreliable for answering certain spatial questions (e.g., "Which chair is larger?"), even if the objects themselves are fully visible.

By explicitly labeling which (observation, question) pairs are answerable $(A=1)$ or unanswerable $(A=0)$, the benchmark directly quantifies model behavior in the presence of epistemic spatial uncertainty.

## 3. Task Design and Evaluation Protocols

SpatialUncertain defines several core tasks designed to probe both passive and active model uncertainty awareness:

- **Answer–Abstain:** Given $(O,Q)$, the model must select either the correct answer (if $A=1$) or "Cannot determine" (if $A=0$). This assesses selective abstention in the face of uncertain or misleading evidence.
  
- **Viewpoint Selection (ViewSel):** Given a set of candidate views $V = \{v_1, ..., v_5\}$ for a fixed $Q$, the model is asked to choose the view $v_j$ where $A(O_{v_j}, Q) = 1$, i.e., which view provides sufficient evidence. This probes active evidence-seeking.

- **Abstain–then–ViewSel (AbsViewSel):** A two-stage protocol where the model must (1) abstain if presented with an ambiguous view, and (2) subsequently identify an informative view that would resolve ambiguity.

Evaluation metrics include:
- Accuracy on answerable cases (Ans.)
- Abstention accuracy on unanswerable (Unans.)
- Overall accuracy (All)
- ViewSel and AbsViewSel scores as defined above.

The rigorous construction of these tasks enables a quantitative assessment of both error-avoidance and evidence-seeking under spatial uncertainty.

## 4. Model Failure Modes and Empirical Results

An extensive evaluation of eight state-of-the-art VLMs (including closed- and open-source models such as GPT-5.4, Gemini-3.0-Flash, Qwen2.5-VL) reveals two consistent failures [2605.30557]:

- **Overconfident Answering:** In unanswerable scenarios ($A=0$, due to occlusion or ambiguity), models overwhelmingly attempt to pick a concrete answer, with "Cannot determine" selected only $\sim30\%$ of the time under occlusion and below $10\%$ under perspective ambiguity.

- **Deficient Active Evidence-Seeking:** Even when provided with multiple viewpoints, model accuracy in selecting an informative view (ViewSel) peaks at $\sim70\%$, and in the joint abstain–then-view selection protocol, performance collapses toward random guessing ($<25\%$), indicating poor integration of uncertainty detection and evidence search.

Partial occlusion (objects still visible) still induces major degradation, suggesting that models are insensitive to finer shades of missing or unreliable evidence.

These results demonstrate that current VLMs do not have calibrated spatial epistemic uncertainty and systematically overcommit in ambiguous observation regimes.

## 5. Implications for Robust Spatial Reasoning

SpatialUncertain establishes new desiderata for spatial reasoning benchmarks and model behavior:

- **Epistemic Uncertainty Awareness:** Models must learn when to abstain (i.e., recognize regions in observation space where predictions are indeterminate or misleading).
  
- **Active Evidence Gathering:** Beyond passively recognizing uncertainty, robust models must identify which additional observations or actions (e.g., viewpoint changes) would provide conclusive evidence.
  
- **Evaluation Beyond Accuracy:** Measures must account for uncertainty calibration (selective abstention), not just answer correctness on fully observable instances.

Empirical findings indicate that uncertainty awareness and evidence-seeking behaviors do not emerge purely from large-scale multimodal pretraining; targeted fine-tuning on mixed observation conditions can improve performance, but only when explicitly trained to recognize and respond to different forms of spatial uncertainty.

## 6. Recommendations and Future Directions

SpatialUncertain provides concrete guidance for developing the next generation of spatial reasoning benchmarks and VLM architectures:

- Include explicit unanswerable cases (through controlled occlusion, ambiguous projections, or sensor noise) in training and evaluation.
- Supply explicit abstention options rather than requiring forced-choice guessing.
- Incorporate active information-seeking tasks—multi-view selection, action recommendations—to measure a model's ability to resolve uncertainty by gathering new evidence.
- Use structured reasoning prompts ("Is the target visible?") to scaffold uncertainty detection, though empirical results show that this introduces a trade-off with answerable accuracy.

Generalizing, robust embodied AI and spatial VLMs should move beyond "seeing to knowing": discriminating the limits of observable evidence, deciding when not to answer, and actively seeking the observations required for an unambiguous judgment [2605.30557].

Source: https://www.emergentmind.com/topics/spatialuncertain