Open-Vocabulary 3D Ambiguity Detection

• The paper introduces a novel methodology that decouples perception and reasoning to detect ambiguity in 3D instructions, ensuring a unique and safe command interpretation.
• AmbiVer leverages adaptive keyframe selection, ray-based fusion, and multi-view evidence to robustly interpret and disambiguate free-form natural language commands in 3D scenes.
• Empirical evaluations on the Ambi3D benchmark demonstrate that AmbiVer outperforms existing baselines, achieving superior accuracy and F1 scores in safety-critical applications.

Open-vocabulary 3D instruction ambiguity detection is a foundational task in embodied AI, where a model must determine whether a natural language command issued with respect to a 3D environment is unambiguous—i.e., admits a single concrete interpretation given the present scene. This problem is especially critical for safety in high-stakes domains (e.g., surgical robotics), where ambiguous language can induce catastrophic downstream errors. The formalism, dataset resources, algorithmic solutions, and empirical results for this task define a new paradigm in reliability for multimodal AI agents (Ding et al., 9 Jan 2026).

1. Formal Task Definition and Ambiguity Taxonomy

The open-vocabulary 3D instruction ambiguity detection task takes as input a 3D scene representation $S$ and a free-form instruction $T$. The task output is a binary ambiguity label $y\in\{\text{Unambiguous},\ \text{Ambiguous}\}$ or, optionally, a confidence score in $[0,1]$.

Scene and Instruction Representation

• 3D Scene $S$:
  • Egocentric video stream: $V=\{I_t\}_{t=1}^n$, with corresponding camera poses $E=\{E_t\}_{t=1}^n$.
  • Allocentric 3D reconstruction: $P=\mathcal{R}(\{(I_t, E_t)\})$.
  • BEV (bird’s-eye view) projection: $I_{\text{bev}}$.
  • Multi-view keyframes: $\{I_v\}$, selected via pose deviation thresholds.
• Instruction $T$0: Free-form, open-vocabulary natural language.

Output

• A mapping $T$1 or $T$2, with $T$3 as a classifier logit sigmoid.

Ambiguity Types

• Referential Ambiguity:
  • Instance ambiguity: Multiple referents for an object class label (e.g., “the cup” with several cups in view).
  • Attribute ambiguity: Adjectives insufficient to isolate a referent (e.g., “large chair” when two are similarly sized).
  • Spatial ambiguity: Instructional locatives affected by observer viewpoint (e.g., “to the left of the vase”).
• Execution Ambiguity: Verb admits multiple plausible actions (e.g., “handle the bicycle” could mean push, lift, rotate, etc.).
• An instruction is unambiguous only if it uniquely specifies the object(s) and prescribes a single clear action.

2. Ambi3D Benchmark: Dataset Design and Annotation

Ambi3D provides a comprehensive benchmark for evaluating ambiguity detection with 22,081 instructions covering 703 ScanNet-derived 3D scenes.

Data Generation Pipeline

1. Grounded Instructions (37.2%): LLM-converted ScanQA question–answer pairs to executable commands.
2. Synthetic Ambiguities (34.1%): LLM generation via scene metadata and templates for each subtype of ambiguity.
3. Hard Negatives (28.7%): Manually authored (by experts) unambiguous commands that appear ambiguous at first glance.

Annotation Protocol

• 12 trained annotators; three-way annotation for binary ambiguity and subtype.
• Only samples with unanimous agreement are kept for binary labels; sub-type by majority vote.
• Outsamples with <2 votes per type are discarded.

Dataset Statistics

Split	Scenes	Instructions	Unambiguous	Ambiguous	Inst.	Action	Attr.	Spatial
Training	649	19,950	9,407	10,543	~48%	~20%	~19%	~14%
Test	54	2,131	1,073	1,058	~46%	~20%	~19%	~15%
Total	703	22,081	10,480	11,601	5,333	2,302	2,216	1,750

• Instruction lengths: mean 8.08 words; substantial spread ($T$428% short, $T$522% long).

Scene Preprocessing

• Point cloud reconstruction $T$6.
• BEV image creation $T$7.
• Adaptive selection for $T$8 keyframes.
• Multi-view 2D object detection (Grounding DINO) using parsed query $T$9.
• Ray-based clustering to form candidate object groups using proximity, angle, and area thresholds.
• Top-K candidate selection (typically $y\in\{\text{Unambiguous},\ \text{Ambiguous}\}$0) by group confidence scores for downstream reasoning.

3. Baselines and Empirical Evaluation

Multiple large vision-language and 3D LLM architectures are adapted for zero-shot ambiguity detection. Each is assessed via a rule-based extraction of binary ambiguity from natural language outputs.

Compared Methods

• 3D-LLM (NeurIPS’23)
• Chat-Scene (NeurIPS’24)
• Video-3D LLM (CVPR’25)
• LSceneLLM (CVPR’25)
• LLaVA-3D (ICCV’25)
• AmbiVer (proposed)

Quantitative Comparison

Method	Acc	Prec	Rec	F1	Inst.	Attr.	Spatial	Action	Unamb.
3D-LLM	49.16	56.93	13.28	21.54	13.6	14.9	11.7	12.3	88.9
Chat-Scene	47.73	50.75	17.58	26.11	18.9	11.8	17.1	20.3	81.1
Video-3D	50.10	60.65	14.31	23.16	12.9	35.8	7.0	2.4	89.7
LSceneLLM	48.78	58.04	9.07	15.70	9.4	11.0	6.1	8.9	92.7
LLaVA-3D	64.21	63.93	73.17	68.24	59.6	95.6	62.9	91.0	54.3
AmbiVer	81.29	84.23	79.23	81.65	80.1	84.0	75.1	75.6	83.6

Failure analyses indicate off-the-shelf models exhibit pronounced “unambiguous bias” (low recall on ambiguous; inflated precision on clear cases) or “ambiguous bias” (misclassifying clear instructions). No baseline except LLaVA-3D achieves zero-shot F1 > 30%. AmbiVer demonstrates a substantial margin in both zero-shot and supervised settings.

4. AmbiVer: Two-Stage Ambiguity Detection Architecture

AmbiVer operationalizes ambiguity adjudication as a two-stage pipeline: an explicit evidence collection (“Perception Engine”) followed by cross-modal reasoning (“Reasoning Engine”).

Stage 1: Perception Engine (Visual Evidence Collection)

• Instruction Decoupling: Leveraging spaCy, parse $y\in\{\text{Unambiguous},\ \text{Ambiguous}\}$1 into {action, query $y\in\{\text{Unambiguous},\ \text{Ambiguous}\}$2, attributes, relations}.
• Global Feature Acquisition: Reconstruct 3D point cloud and generate BEV image.
• Detailed Feature Acquisition:
  • Adaptive selection of $y\in\{\text{Unambiguous},\ \text{Ambiguous}\}$3 keyframes.
  • Per-keyframe open-vocabulary detection (Grounding DINO) with $y\in\{\text{Unambiguous},\ \text{Ambiguous}\}$4.
  • Ray-based fusion merges multi-view detections into groups $y\in\{\text{Unambiguous},\ \text{Ambiguous}\}$5 by spatial and angular consistency.
  • Reliability scoring:

  $y\in\{\text{Unambiguous},\ \text{Ambiguous}\}$6 - Candidate selection: take top $y\in\{\text{Unambiguous},\ \text{Ambiguous}\}$7 groups, representative views by $y\in\{\text{Unambiguous},\ \text{Ambiguous}\}$8.

Stage 2: Reasoning Engine (Vision–Language Judgment)

• Dossier Construction: Bundle $y\in\{\text{Unambiguous},\ \text{Ambiguous}\}$9, its parse, BEV map, and cropped candidate evidence.
• Zero-shot VLM Adjudication: Use Qwen-3-VL, prompted with the assembled dossier and ambiguity criteria, to return a structured verdict: $S$5
• (Optional) Fine-Tuning: If trained end-to-end on Ambi3D with LoRA, the loss is:

$[0,1]$0

AmbiVer’s full algorithmic flow is as follows: $S$6

5. Experimental Results and Ablation

Evaluation Setup

• Hardware: Tesla V100 / A100 GPUs (4 × 32 GB).
• Key parameters: $[0,1]$1, $[0,1]$2 m, $[0,1]$3, $[0,1]$4, $[0,1]$5, $[0,1]$6 px.
• VLM: Qwen-3-VL with temperature=0.
• Fine-tuning: 3 epochs, AdamW lr=$[0,1]$7, cosine schedule, wd=0.01, warmup=3%, rank $[0,1]$8, $[0,1]$9, dropout=0.1.

Main Results

• AmbiVer zero-shot achieves $S$0 accuracy and $S$1 F1, outperforming all baselines and ablation variants.
• Strong gains of $S$2–$S$3 pts F1 compared to best zero-shot, and $S$4 pts F1 over best supervised baseline.

Ablation Studies

Ablation Case	Acc	F1
w/o parsing	62.06%	65.62%
w/o adaptive keyframes	65.04%	54.56%
w/o 3D fusion	58.59%	38.42%
w/o representative view weights	79.93%	80.31%
Full pipeline (Ours)	81.29%	81.65%

Reasoning Engine Ablations	Acc	F1
w/o BEV map	56.95%	70.57%
w/o local evidence	60.71%	53.56%
w/o all visual info	53.99%	56.91%
Full reasoning (Ours)	81.29%	81.65%

Qualitative Case Studies

• “Pick up the backpack”: Unambiguous, single candidate.
• “Pass me the trash can”: Ambiguous (Instance), two plausible candidates.
• “Please handle the bicycle”: Ambiguous (Action), multiple plausible actions.
• “Adjust the pillow on the couch”: Mixed spatial/action ambiguity, VLM generates clarification queries.

6. Open Problems and Future Directions

Grounded ambiguity detection is established as a critical upstream safety check in embodied agents and robotics, providing an objective, evidence-based adjudication step before interpreting or executing 3D language commands.

Limitations

• Reliance on pre-trained open-vocabulary detectors (e.g., Grounding DINO) is a vulnerability under occlusion and unseen categories.
• Dependence on prompt design exposes reasoning to rare edge-case misparses.
• The framework yields only a binary verdict; it does not currently integrate active clarification dialogue or uncertainty awareness.

Future Directions

• Development of multi-turn clarification dialogue agents, leveraging the VLM verdict explanations.
• Integration of uncertainty-aware planning to mediate between execution and user query.
• Adaptation to dynamic/outdoor scenes, temporal ambiguity.
• Joint or end-to-end training for the perception and reasoning stages, potentially unlocking differentiable evidence pathways.

Open-vocabulary 3D instruction ambiguity detection thus forms a research agenda at the intersection of vision-language understanding, embodied AI safety, and pragmatic language interpretation (Ding et al., 9 Jan 2026).