Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?
Abstract: Spatial reasoning is a fundamental capability for vision-LLMs (VLMs) deployed in real-world environments. However, visual observations are inherently limited representations of a 3D world: occlusion can render objects invisible, and perspective can make geometric properties misleading. Despite this, existing spatial reasoning benchmarks typically assume that observations are sufficient and reliable, focusing on whether models produce correct answers rather than whether they recognize when a question cannot be answered and what additional observations would be needed. In this work, we challenge this assumption by constructing a controlled evaluation framework, SpatialUncertain, and introducing two types of observation challenges: (1) occlusion, which hides target information, and (2) perspective ambiguity, which produces misleading visual cues. For each configuration, we design spatial questions that are answerable under clean observations but require abstention under the introduced challenges. We further evaluate whether models can identify which additional viewpoints would resolve perspective ambiguity. Our results across a diverse set of frontier open- and closed-source VLMs reveal two consistent failure modes. First, models are prone to overconfident answering, attempting to solve spatial reasoning tasks even when visual evidence is incomplete or misleading, with average accuracy around 30\% under occlusion and below 10\% under perspective ambiguity. Second, even when additional views are available, some models perform near random chance in identifying which would provide reliable evidence. Together, our findings call for moving beyond answer correctness toward evaluating whether models know when to abstain and how to seek reliable evidence.
Summary
- The paper introduces SpatialUncertain, a controlled framework that evaluates VLMs on spatial questions under varying levels of occlusion and perspective ambiguity.
- It demonstrates that state-of-the-art VLMs often overestimate certainty, yielding only 10%-30% accuracy on unanswerable cases while providing confidently wrong answers.
- Mitigation attempts with structured prompting and fine-tuning reveal that targeted training on diverse spatial ambiguities can improve safe abstention and viewpoint selection despite inherent trade-offs.
Spatial Reasoning Under Observational Uncertainty in Vision-LLMs
Motivation and Problem Formulation
Spatial reasoning is foundational for Vision-LLMs (VLMs) deployed in embodied and interactive applications, yet prevailing benchmarks assume that visual observations are both sufficient and reliable. In real-world settings, visual input is only a 2D projection of a 3D scene, leading to information loss through occlusion and geometrical distortion through viewpoint-induced perspective. This work formulates two critical and underexplored aspects of VLMs’ capabilities: (1) recognizing when available evidence is insufficient or unreliable for spatial reasoning, and (2) identifying additional informative observations that would resolve ambiguity.
SpatialUncertain, a controlled evaluation framework, is introduced to probe these questions. The diagnostic framework systematically varies scene observation conditions via occlusion and perspective-induced ambiguity in photorealistic 3D simulations. For each controlled configuration, ground-truth answerability is established, allowing models to be evaluated on both the correctness of their answers and their ability to abstain or seek more data when appropriate. The framework additionally incorporates explicit viewpoint selection tasks.
Figure 1: Visual observations as 2D projections: sufficient for clean views, but inherently unreliable under occlusion and perspective ambiguity.
Framework and Experimental Setup
Scene and Observation Construction
SpatialUncertain generates 3D indoor environments using Holodeck, constructing a range of realistic room layouts rendered via AI2-THOR. Clean baseline observations are systematically corrupted either by inserting occluders between camera and target objects or by introducing lateral camera shifts that induce geometric ambiguity without altering underlying object geometry.
Figure 2: Overview of the SpatialUncertain evaluation framework, contrasting occlusion (top) and perspective (bottom) perturbations and introducing ViewSel and AbstainViewSel tasks.
Camera placement is tightly controlled: occlusion conditions vary from partial to full, with human annotation to guarantee label accuracy; perspective ambiguity leverages same-category object pairs and deliberately manipulates camera position to induce foreshortening, scale illusion, or loss of visual comparability.
Figure 3: Schematic of camera placement strategies for introducing controlled occlusion and perspective ambiguity.
Task Design
Four principal classes of spatial questions are posed:
- Visibility: Is the target object observable from the supplied viewpoint?
- Relative Position: What is the spatial relationship between referenced objects?
- Depth Ordering: Which object is closer/deeper?
- Size/Shape Reasoning: Which object is larger or differently shaped?
Answerability—binary classification of whether a question can be reliably resolved from the observation—is used alongside accuracy as an evaluation metric. For ambiguous observations, "Cannot determine" is the only correct response.
Additionally, two explicit viewpoint selection tasks are defined:
- ViewSel: Given ambiguous evidence and several alternative candidate views (one informative), select the most diagnostic view.
- AbstainViewSel: First abstain from answering (when appropriate), then identify a resolving viewpoint.
Figure 4: Examples of controlled evaluation scenes, including occlusion and perspective configurations.
Key Empirical Findings
Systematic Failure Modes
Evaluation of eight strong open- and closed-source VLMs—including GPT-4o, GPT-5 variants, Gemini-2.5/3.0-Flash, Qwen2.5-VL, InternVL3—reveals two dominant, robust limitations in spatial reasoning under reliability-challenged conditions.
- Overconfident Answering Under Unreliable Evidence: Across all models, accuracy on unanswerable (i.e., should abstain) cases is low—around 30% under occlusion, and below 10% under perspective ambiguity—despite models exceeding random performance (typically ~25–32%) on answerable cases. Models often supply confident but incorrect answers instead of correctly abstaining.
- Limited Ability to Identify Informative Viewpoints: Even when presented with candidate images that could resolve ambiguity, state-of-the-art VLMs perform near random chance or display a significant answer-abstention trade-off. GPT-5.4 achieves 70.9% on isolated ViewSel but only 22.6% on the joint AbstainViewSel.
Figure 5: Model accuracy across question types and conditions. Blue/orange backgrounds distinguish answerable and unanswerable cases; bold lines highlight best closed-source models.
Ablation comparing text-only vs. vision-enabled modalities identifies a striking asymmetry: visual context is helpful for missing information (occlusion) but misleading for unreliable information (perspective ambiguity), which actively suppresses appropriate abstention.
Mitigation Attempts—Prompting vs. Fine-tuning
Structured reasoning in prompting (explicitly instructing models to validate viewpoint reliability and target visibility before answering) yields partial gains in abstention accuracy but with an inherent trade-off: increased abstention comes at the cost of reduced correct answering (i.e., models become overly conservative). The answer–abstain trade-off persists across architectures.
Supervised fine-tuning with LoRA adapters on purpose-built spatial ambiguity data demonstrates that observational uncertainty is learnable: when exposed to diverse forms of data ambiguity, models substantially improve both answerable and unanswerable performance, and the above trade-off can be mitigated. However, fine-tuning on one ambiguity domain (e.g., only occlusion) fails to generalize—robust abstention under observational uncertainty necessitates heterogeneous ambiguity exposure during training.
Implications and Theoretical Consequences
The work underscores a foundational limitation of current large-scale VLMs: the inability to calibrate their inferences when facing incomplete or misleading visual evidence. This deficiency is particularly acute for embodied AI systems where spatial reasoning errors—via unrecognized uncertainty—can lead to unsafe or suboptimal behavior in control, navigation, or interaction tasks.
The data also shows that correctness-based benchmarks inadequately measure true spatial proficiency; models can succeed on answerable cases yet fail catastrophically on unanswerable/ambiguous ones. In safety-critical settings, this gap is consequential.
On a cognitive-theoretic level, these findings indicate that the current generation of VLMs operates as pattern matchers on observed evidence, with limited meta-reasoning over epistemic validity. Progress toward "self-aware" inference may require explicit modeling of projection geometry, multi-view integration, or active information-seeking routines.
Future Directions
Observational reliability should become a core criterion for VLM evaluation and advancement, not just spatial answer accuracy. Practically, benchmarks must present models with ambiguous or unanswerable cases and require abstention, uncertainty detection, and active inquiry.
Architectural innovations could involve integrating learned geometric priors, spatial consistency checks, or hierarchical inference that conditions decisions on explicit evidence validation. Further, the efficacy of curriculum training regimes with controlled exposure to diverse observational ambiguities could foster more robust meta-cognitive capabilities in VLMs.
Conclusion
SpatialUncertain defines a new standard for evaluating and developing vision-LLMs by emphasizing observational awareness, not just answer correctness. Current models—even state-of-the-art—systematically overestimate their certainty and are unable to reliably detect when spatial questions cannot be solved from available views. Effective abstention and viewpoint selection are learnable with proper supervision, but require frameworks that systematically expose models to the limits of their own perception.
Development of VLMs capable of explicit uncertainty assessment and evidence-seeking will be essential for the safe, reliable deployment of embodied AI in open-world settings.
Figure 6: Occlusion annotation examples, demonstrating validation for category assignments of clean, partial, and full occlusion.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
Easy-to-Understand Summary of “Seeing Isn’t Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?”
What is this paper about?
This paper studies whether AI systems that “look” at pictures and “read” text (called vision-LLMs, or VLMs) know when they can’t trust what they see. In the real world, cameras only capture 2D images of a 3D world. That can cause problems:
- Occlusion: something blocks the view, so important parts are hidden.
- Perspective: the camera angle makes things look bigger/smaller or differently shaped than they really are.
The authors build a new test, called SpatialUncertain, to check if VLMs can:
- realize when a spatial question can’t be answered from a given view,
- say “Cannot determine” instead of guessing, and
- figure out which new viewpoint would make the answer clear.
What questions were the researchers trying to answer?
In simple terms, they asked:
- Can AI tell the difference between “I can answer this from what I see” and “I can’t tell from this view”?
- Will AI avoid guessing and instead say “Cannot determine” when the view is blocked or misleading?
- If the view is misleading, can AI pick a better camera angle to see the truth?
How did they study it?
They created realistic 3D indoor rooms (bedrooms, kitchens, museums, etc.) in a simulator and rendered images from different camera positions. This let them precisely control what the camera sees.
They tested two kinds of challenges:
- Occlusion: adding an object in front of the target item so it’s partially or fully hidden.
- Perspective ambiguity: moving the camera so one of two same-type objects (like two chairs) looks bigger or oddly shaped just because of the angle, not because it actually is.
They asked four types of simple spatial questions (e.g., “Is the lamp visible?”, “Which object is closer?”, “Which is bigger?”). Each question was answerable from a clear view but sometimes unanswerable from a blocked or biased view. The correct behavior in those “bad views” is to say “Cannot determine.”
They also added viewpoint-choosing tasks:
- ViewSel: given five possible new views, pick the one that would best answer a size question.
- AbstainViewSel: first, decide if the current view is too misleading (and abstain); then, choose the right new view.
They tested several well-known VLMs (both open- and closed-source) using consistent prompts and tracked:
- Accuracy on answerable questions,
- Accuracy on unanswerable ones (i.e., correctly saying “Cannot determine”),
- View selection accuracy.
Think of it like testing whether someone looking through a window knows when the window view isn’t good enough—and whether they know where to move to get a better look.
What did they find, and why is it important?
The results showed two big problems:
- Overconfident guessing
- When objects were hidden (occlusion), models often tried to answer anyway and were right only about a third of the time.
- When the camera angle was misleading (perspective), models still tried to answer and did even worse—often under 10% on the questions that should have been unanswerable.
- In short: many models don’t know when they don’t know. They guess instead of abstaining.
- Trouble choosing better views
- Even when asked to pick a helpful new camera angle, several models performed near random.
- In the two-step test (first abstain, then pick a view), performance dropped a lot, showing it’s hard for models both to recognize bad evidence and to seek good evidence.
They also found an interesting twist about images:
- Visual input helps when information is missing (like occlusion). Models were better at realizing “I can’t see it.”
- But visual input can hurt when the view is misleading (perspective). The image tricks the model, making it less likely to say “Cannot determine.”
Can we fix this?
- Better prompts (structured instructions) helped some models abstain more but often reduced accuracy on questions that were actually answerable. So prompting alone isn’t enough.
- Fine-tuning (extra training) on a mixture of different uncertainty cases worked much better. Models got more reliable at both answering when it’s possible and abstaining when it’s not. Training only on one kind of uncertainty (just occlusion or just perspective) didn’t generalize well.
What does this mean for the future?
If we want safe, reliable AI assistants and robots that move around and make decisions, they must:
- Recognize when a camera view isn’t good enough to answer a question,
- Avoid guessing,
- Ask for—or seek out—a better view.
This paper argues that we should not only ask, “Did the model get the answer right?” but also, “Did the model know when it couldn’t know?” Their new testbed, SpatialUncertain, helps researchers build and evaluate AI that understands its own limits—leading to smarter, safer systems in real-world settings like home robots, AR assistants, and autonomous tools.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a concise list of what remains missing, uncertain, or unexplored in the paper, structured to guide actionable future research.
- Sim-to-real generalization: Validate whether findings from AI2-THOR/Holodeck-rendered scenes transfer to real images/videos captured in physical environments, including off-the-shelf cameras and robotic platforms.
- Static single-view limitation: Extend beyond single RGB frames to dynamic video and multi-step observations where partial occlusions vary over time; assess whether temporal cues reduce the need to abstain.
- Restricted uncertainty sources: Incorporate additional observation challenges beyond occlusion and perspective bias—e.g., lighting/shadows, specular/transparent materials, reflections, motion blur, sensor noise, depth-of-field, and textureless surfaces—and quantify their impact on abstention behavior.
- Limited task coverage: Expand spatial questions beyond visibility/relative position/depth/size-shape to include containment/support, contact, symmetry, angle/pose, metric distances, and topology, and test how answerability shifts across these.
- Camera metadata omission: Evaluate whether providing camera intrinsics/extrinsics (or estimated poses) enables projective-geometry-aware reasoning that resolves some “unanswerable” perspective cases.
- Missing 3D modalities: Test whether augmenting inputs with depth maps, stereo, point clouds, or predicted geometry (e.g., monocular depth, normal maps) improves uncertainty awareness and abstention under perspective ambiguity.
- Multi-view aggregation: Move from picking a single “best” view to integrating multiple views jointly; measure how performance scales with the number and diversity of views and whether aggregation reduces overconfidence.
- Active perception in continuous space: Replace five pre-specified candidate views with continuous, physically feasible camera actions; evaluate learned policies that decide when/where to move to collect reliable evidence.
- Coverage–risk and calibration: Go beyond accuracy metrics to report selective risk/coverage curves, area under the risk-coverage curve, and calibration (e.g., ECE) for both answerable and unanswerable cases; study how abstention thresholds trade off with risk.
- Abstention without an explicit option: Test open-ended answering without a “Cannot determine” choice and evaluate whether models spontaneously abstain, hedge, or ask for more views; analyze sensitivity to prompt wording.
- Rationale quality for abstention: Require models to justify why a question is unanswerable (e.g., occluded target, foreshortening); evaluate correctness of rationales and whether rationale generation correlates with correct abstentions.
- Fine-grained difficulty controls: Parameterize and report performance as a function of occlusion fraction, occluder category/size, camera shift magnitude, object size ratio, and object separation; identify thresholds where abstention becomes appropriate.
- Broader scene diversity: Assess generalization to outdoor, industrial, and cluttered spaces; include unseen room types, unseen object categories, and alternative simulators to stress-test domain shift.
- Language variability: Replace templated QA with diverse natural language paraphrases and compositional queries; evaluate robustness to linguistic ambiguity and instruction phrasing.
- View selection task scope: Extend view selection beyond size/shape questions to other ambiguity-prone tasks (e.g., depth ordering, pose, aspect ratio) and verify uniqueness of the “best” view when multiple candidates are informative.
- Physical feasibility and planning constraints: Couple view recommendations with kinematic/obstacle constraints; evaluate whether selected views are reachable and still informative under real navigation limits.
- Model-internal attribution: Diagnose whether failures stem from the vision encoder (e.g., scale sensitivity), cross-modal alignment, or the LLM head (e.g., prior-driven guessing); perform ablations or swap components to locate error sources.
- Training methods for abstention: Develop and test principled selective prediction objectives (e.g., risk-coverage, abstention-penalized losses, entropy regularization) rather than supervised fine-tuning alone; compare to post-hoc thresholding.
- Epistemic vs. aleatoric uncertainty: Disentangle uncertainty due to insufficient views (epistemic) versus inherent projection ambiguity (aleatoric); design experiments that manipulate each independently and measure model responses.
- Downstream impact on embodied tasks: Quantify how abstention and view selection affect success/safety in navigation/manipulation pipelines (e.g., fewer collisions, better grasp success), not just QA accuracy.
- Open-ended view guidance: Evaluate models’ ability to propose next-view directions/distances in natural language (e.g., “move 1 m to the left and 0.5 m closer”) and verify that such guidance increases answerability.
- Prompt and API sensitivity: Assess robustness of results to prompt wording, answer format (free-form vs. MCQ), and API/version drift for closed-source models; establish reproducibility baselines.
- Dataset artifacts and biases: Audit scene layouts generated by Holodeck for systematic biases (object co-locations, unrealistic placements) that might cue answers; evaluate whether models exploit such priors instead of visual evidence.
- Generalization of fine-tuning: Reproduce LoRA-mixed results across different backbones and scales; test out-of-distribution scenes and stricter held-out categories to ensure gains reflect true uncertainty awareness rather than overfitting.
- Camera/FOV robustness: Vary focal length, FOV, and lens distortion to test whether conclusions about perspective ambiguity hold across camera models and whether calibration data mitigates failures.
Practical Applications
Practical Applications of “Seeing Isn’t Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?”
The paper introduces SpatialUncertain—a controlled 3D evaluation framework that stresses VLMs with occlusion and perspective ambiguity—and demonstrates two key practical insights: (1) current VLMs are overconfident under unreliable observations and (2) abstention and next-view selection are learnable but require exposure to diverse ambiguity conditions. Below are actionable applications derived from the benchmark, findings, and methods.
Immediate Applications
The following applications can be deployed now with modest integration and/or data/compute resources.
- Uncertainty-aware VLM evaluation and CI gating
- Sectors: software/AI tooling, robotics, AR/VR, autonomous systems
- What: Integrate SpatialUncertain scenes and metrics (Ans, Unans, ViewSel, AbstainViewSel) into model validation, regression tests, and release gates to prevent shipping overconfident spatial reasoning.
- Tools/workflows: Benchmark harness, nightly CI jobs, dashboards tracking Unans and AbsViewSel; “perspective stress tests” in model cards.
- Assumptions/dependencies: Access to 3D simulation (AI2-THOR) or use the provided images; acceptance that “Cannot determine” is a valid production outcome; compute for batch evaluation.
- Risk-aware prompting and UI for camera/AR assistants
- Sectors: mobile apps, AR/VR, consumer software, customer support
- What: Deploy structured prompts that evaluate visibility/viewpoint reliability before answering; surface “Cannot determine” with clear rationales; add UI affordances (e.g., warning badges for unreliable views).
- Tools/workflows: Prompt libraries, abstention tokens, UI components showing uncertainty; telemetry on abstain rates vs. accuracy.
- Assumptions/dependencies: Minor accuracy–abstention trade-off (as observed); user acceptance of abstentions; prompt adherence in closed-source APIs.
- Viewpoint recommendation (“next-best-view”) in interactive systems
- Sectors: warehouse/logistics, retail picking, field service, consumer scanning apps
- What: Use ViewSel-style ranking to suggest where a user/operator should move a camera to resolve ambiguity (e.g., “Step 30 cm left to estimate size”).
- Tools/workflows: Candidate view generation (precomputed or on-device via SLAM), simple view-ranking prompts, UI arrows/voice cues.
- Assumptions/dependencies: Ability to propose/approximate candidate views; user or robot can reposition; latency acceptable for interactive use.
- Data augmentation and fine-tuning recipes for abstention
- Sectors: model providers, platform teams, research labs
- What: Adopt LoRA-Mixed-style training (diverse occlusion + perspective conditions) to improve both answerable accuracy and calibrated abstention.
- Tools/workflows: Holodeck+AI2-THOR pipelines to synthesize ambiguity, LoRA adapters, evaluation on Unans/AbsViewSel to tune trade-offs.
- Assumptions/dependencies: Training compute; rights to train; transfer from simulated indoor scenes to target domains.
- Safety acceptance criteria for embodied agents and vision pipelines
- Sectors: robotics, drones, industrial automation, autonomous retail
- What: Define product gates requiring minimum Unans and AbsViewSel thresholds on domain-specific suites before enabling autonomy levels or features.
- Tools/workflows: Domain-tailored SpatialUncertain subsets (e.g., shelf picking, narrow-aisle occlusions), release checklists, fail-safe fallbacks (slow/stop/request help).
- Assumptions/dependencies: Agreement with safety and compliance teams; scenario coverage approximates in-field conditions.
- Red-teaming for spatial illusions and occlusions
- Sectors: AI assurance, model auditing, QA
- What: Systematically attack models with occlusion/perspective “illusions” to map failure modes and improve calibration.
- Tools/workflows: Scenario generators, adversarial suites, failure taxonomy (overconfident under occlusion vs. perspective), issue tracking.
- Assumptions/dependencies: Simulator skillset; alignment with internal audit criteria.
- Bug reproduction and dataset curation via controllable 3D scenes
- Sectors: product engineering, QA, applied research
- What: Reproduce user-reported misperceptions (e.g., “app misjudged distance behind pillar”) by constructing matched 3D scenes with adjustable viewpoint and occluders.
- Tools/workflows: Holodeck layouts, AI2-THOR cameras, reproducible test artifacts; “before/after” for model versions.
- Assumptions/dependencies: Simulator assets approximate real scenes; staff trained in scene authoring.
- Analytics for “answer vs. abstain” calibration
- Sectors: AI product analytics, reliability engineering
- What: Monitor changes in Unans and answerable accuracy as prompts/decoders/models change; tune operating points by use-case (e.g., cautious mode in high risk).
- Tools/workflows: Experimentation platform, calibration dashboards, thresholded abstention policies.
- Assumptions/dependencies: Reliable logging/telemetry; consent to instrument production flows.
- Course modules and demos on perspective illusions and uncertainty
- Sectors: education, training, workforce upskilling
- What: Classroom labs showing why 2D projections mislead spatial reasoning and how abstention or new views help.
- Tools/workflows: Packaged scenes, worksheets, small leaderboards.
- Assumptions/dependencies: Education licenses for assets; basic GPU access optional.
- Procurement/testing language in contracts and RFPs
- Sectors: public sector, enterprise IT, safety-critical buyers
- What: Require bidders to report Unans/AbsViewSel performance on specified scenes and document abstention behavior and fail-safes.
- Tools/workflows: Template test suites, standardized scoring, acceptance criteria.
- Assumptions/dependencies: Agreement on test domain; tolerance for abstention in SLAs.
Long-Term Applications
These applications require further research, scaling, or multi-component development (e.g., sensors, autonomy stacks, standardization).
- Active perception planners that autonomously resolve ambiguity
- Sectors: robotics, AR/VR, industrial inspection
- What: Close the loop from abstention to action—agents detect unreliable evidence, plan and execute next-best-view, then answer.
- Tools/products: NBV planners, geometry-aware policies, embodied simulators for training; “uncertainty-to-motion” APIs.
- Dependencies: Robust SLAM, safe navigation, low-latency control; domain adaptation beyond synthetic rooms.
- Geometry- and uncertainty-aware VLM architectures
- Sectors: foundational model providers, research
- What: Models that natively encode 3D priors (differentiable rendering, multi-view fusion, depth cues) and calibrated abstention heads.
- Tools/products: 3D-aware pretraining objectives, multi-view encoders, uncertainty estimation modules.
- Dependencies: Large-scale multi-view/3D corpora; compute; benchmarks beyond SpatialUncertain.
- Cross-industry standards and certification for observational uncertainty
- Sectors: policy/regulation, safety certification bodies, consortia
- What: Standard test methods, reports, and thresholds for occlusion/perspective robustness and abstention; akin to “spatial truthfulness” certifications.
- Tools/products: Reference test suites, compliance protocols, third-party audits.
- Dependencies: Multi-stakeholder alignment; domain-specific annexes (e.g., warehouses vs. hospitals).
- Sensor and hardware co-design for ambiguity reduction
- Sectors: robotics, consumer devices, AR glasses, mobile OEMs
- What: Pair algorithms with sensor suites (stereo, ToF/LiDAR, multi-fisheye) that reduce perspective-induced misperception and blind spots.
- Tools/products: Sensor fusion pipelines, uncertainty-aware perception SoCs.
- Dependencies: Cost/weight/power constraints; calibration; manufacturability.
- Safer autonomy policies in transportation and mobile robots
- Sectors: autonomous driving, last-mile delivery, service robots
- What: Policies that detect occlusion-induced uncertainty and adjust behavior (slow/stop/reposition/request V2X info).
- Tools/products: Policy stacks with abstention gates, map-based NBV, cooperative perception queries.
- Dependencies: Regulatory approval; integration with broader perception stack; robust field validation.
- Medical imaging and surgical robotics abstention and guidance
- Sectors: healthcare
- What: Systems that abstain under occluded or misleading endoscopic views and recommend specific probe/camera motions or additional modalities.
- Tools/products: OR guidance UIs, NBV for endoscopes/robotic arms, uncertainty logs for quality assurance.
- Dependencies: Clinical validation, regulatory clearance, data governance.
- Autonomous inspection and construction QA
- Sectors: energy, utilities, construction, infrastructure
- What: Drones/rovers that plan paths to minimize occlusions (e.g., behind beams, ducts), abstain on ambiguous readings, and schedule follow-up passes.
- Tools/products: Mission planners with ambiguity costs; fleet scheduling tied to uncertainty metrics.
- Dependencies: Flight safety, GPS-denied navigation, domain adaptation to outdoor/large-scale scenes.
- Consumer AR “reliability overlays”
- Sectors: AR wearables, mobile AR
- What: On-device indicators that highlight when spatial inferences (size, depth) are likely unreliable; real-time prompts to reposition.
- Tools/products: Lightweight uncertainty estimation, on-device NBV hints, UX paradigms for “confidence halos.”
- Dependencies: Efficient on-device models; power constraints; user acceptance.
- Large-scale curricula for observation uncertainty
- Sectors: academia, model providers
- What: Public datasets and training curricula spanning diverse rooms, materials, lighting, dynamic occluders, and outdoor settings to generalize abstention.
- Tools/products: Dataset hubs, curriculum schedulers, leaderboards for AbsViewSel.
- Dependencies: Licensing for assets, compute, community maintenance.
- Liability, logging, and transparency frameworks for abstention
- Sectors: policy, legal, enterprise governance
- What: Policies that require logging abstentions and view-change requests; guidance on when abstention is mandatory vs. optional; auditability in incident reviews.
- Tools/products: Standardized uncertainty logs, evidence provenance, governance dashboards.
- Dependencies: Legal standards, privacy constraints, cross-vendor telemetry norms.
- Market of “uncertainty-aware” MLLM components
- Sectors: AI platforms, integrators
- What: Off-the-shelf SDKs for abstention calibration, NBV planning, and viewpoint scoring, with benchmarks and SLAs.
- Tools/products: Plugins for common MLOps stacks; pre-tuned LoRA adapters for different domains.
- Dependencies: Interoperability standards; sustained evaluation datasets.
- Human–robot collaboration protocols for viewpoint negotiation
- Sectors: manufacturing, fulfillment, home robotics
- What: Natural-language policies where robots ask humans to move objects or cameras to specified positions to resolve ambiguity.
- Tools/products: Dialogue policies grounded in ViewSel; ergonomic instructions; safety-aware constraints.
- Dependencies: HRI design, multilingual support, cultural/operational norms.
Notes on Assumptions and Dependencies Across Applications
- Simulation-to-reality gap: SpatialUncertain is based on indoor simulated scenes; out-of-distribution environments (outdoors, dynamic crowds, specular surfaces) may need new data and validation.
- Candidate-view availability: Viewpoint selection assumes either robot mobility, human cooperation, or a multi-camera rig to provide alternate views.
- Acceptability of abstention: Product and policy contexts must embrace “Cannot determine” as safer than guessing; metrics and UX must reinforce this.
- Compute and licensing: Some applications require access to simulators, 3D assets, and training compute; check asset licenses and privacy constraints.
- Multi-sensor integration: Perspective ambiguity is best mitigated with depth or multi-view sensing; hardware constraints (power/cost/weight) may limit feasibility.
These applications translate the paper’s core contributions—controlled ambiguity evaluation, abstention as a first-class outcome, and data-driven improvement via diverse training—into concrete tools, workflows, and policies for safer, more reliable spatial reasoning in multimodal systems.
Glossary
- AI2-THOR: An interactive 3D simulator used to render controllable, photo-realistic scenes for embodied AI experiments. "All scenes are rendered using AI2-THOR"
- AbstainViewSel: A two-stage evaluation where a model must first abstain on an unreliable view and then select an informative alternative viewpoint. "Abstain-then-ViewSelect (AbstainViewSel): We further introduce a two-stage evaluation that jointly measures abstention and viewpoint selection."
- abstention: The act of choosing not to answer when evidence is insufficient or unreliable. "Structured prompting partially improves abstention but introduces a trade-off with answerable accuracy."
- answerability: Whether a question can be reliably answered from the available observations. "A key design principle is that answerability varies systematically with observation conditions."
- Embodied agents: AI systems that perceive and act within physical or simulated environments. "Such unreliable observations are particularly challenging for embodied agents"
- fine-tuning: Adapting a pretrained model on task-specific data to improve performance on that task. "In contrast, fine-tuning on diverse ambiguity conditions yields more robust observational uncertainty"
- foreshortened: The visually compressed appearance of an object when viewed at an oblique angle. "the object viewed at an oblique angle appears foreshortened, altering its apparent proportions."
- Holodeck: An LLM-driven system that automatically generates diverse 3D indoor scenes and layouts. "We generate 3D indoor scenes using Holodeck"
- LoRA: A parameter-efficient fine-tuning method that adapts models via low-rank updates. "We fine-tune Qwen2.5-VL-7B-Instruct with LoRA"
- micro-averaged accuracy: Overall accuracy computed across all instances, weighting classes by their frequency. "All (micro-averaged accuracy over all questions)."
- MLLMs (Multimodal LLMs): LLMs that process and reason over multiple modalities, such as text and images. "Multimodal LLMs (MLLMs)"
- negative transfer: When training on one type of data degrades performance on another. "indicating negative transfer across ambiguity types."
- occluder: An object that blocks the line of sight to a target object, partially or fully hiding it. "we select an occluder from the remaining objects"
- occlusion: The blocking of an object by another object in the line of sight, making it invisible or partially visible. "occlusion can render objects invisible"
- oracle viewpoint: A camera position verified to provide clear, reliable visual evidence for the task. "selected from the oracle viewpoint: the camera position verified to have clear visibility of the target object."
- perspective ambiguity: Unreliable appearance cues caused by viewpoint-induced distortions, making certain judgments (e.g., size) unanswerable from a single view. "Under perspective ambiguity, visual information is not missing but can become unreliable."
- perpendicular bisector: The geometric locus equidistant from two points; used here to place the camera neutrally between objects. "on the perpendicular bisector of the two objects"
- reference view: A canonical, informative viewpoint designed to provide reliable evidence for spatial reasoning. "The reference view places the camera on the perpendicular bisector of the two objects"
- selective prediction: A learning setup where the model predicts only when confident, abstaining otherwise. "calibration and selective prediction shows that neural networks can be overconfident"
- selective VQA: Visual question answering that allows or encourages models to abstain when the answer is not reliably derivable from the image. "and selective VQA, encouraging models to abstain rather than answer incorrectly"
- sizeâdistance effects: A perceptual phenomenon where nearer objects appear larger than farther ones, even if physically identical. "the nearer object appears larger due to sizeâdistance effects."
- VLMs (Vision-LLMs): Models that combine visual and linguistic inputs to perform joint reasoning. "evaluating observational awareness in VLMs under two projection-induced challenges"
- ViewSel: A single-stage task where the model must pick the most informative viewpoint from several candidates. "ViewSelect (ViewSel): the model is presented with five candidate views"
- viewpoint bias: Systematic distortion in appearance or cues introduced by the camera’s position and angle. "due to viewpoint bias"
- viewpoint-invariant properties: Spatial properties that remain stable under changes in camera viewpoint. "viewpoint-invariant properties (e.g., visibility, relative position)"
- viewpoint selection: The process of choosing a camera view that provides the most reliable evidence for a given question. "viewpoint selection ability in isolation"
- zero-shot setting: Evaluating a model on a task without any task-specific training or fine-tuning. "All models are evaluated in a zero-shot setting"