- The paper introduces a progressive, staged benchmark that evaluates compositional spatial understanding of small objects with precise, reference-based descriptions.
- It decomposes spatial reasoning into tasks: target presence verification, nearest reference identification, fine-grained spatial description, and structured spatial prediction.
- Experimental findings reveal a consistent performance drop across stages, emphasizing the need for improved model fine-tuning for robust spatial localization.
PinpointQA: Benchmarking Small Object-Centric Spatial Understanding in Indoor Videos
Recent advances in multimodal LLMs (MLLMs) have driven progress in video spatial intelligence, spatial-temporal reasoning, and embodied perception; however, a persistent bottleneck is precise localization and spatial expression of small objects in cluttered indoor video—a core requirement for object search, assistive robotics, and diagnostic perception applications. Existing benchmarks insufficiently address the ability to not only localize small objects but also to express their position with reference cues and geometric specificity suitable for downstream reasoning. PinpointQA addresses this gap by introducing a principled benchmark and dataset focused specifically on small object-centric spatial understanding in indoor video sequences (2604.08991).
PinpointQA theorizes spatial understanding as a progressive, compositional capability: (1) determine target presence, (2) identify the nearest reference, (3) compose explanatory descriptions referencing local geometry, and (4) generate structured outputs compatible with machine action. This staged framework exposes model bottlenecks at each step, as opposed to undifferentiated global metrics, and offers a diagnostic lens on how MLLMs handle fine-grained, context-dependent spatial grounding.
Dataset Construction and Task Design
PinpointQA is built upon the ScanNet++ and ScanNet200 3D scene datasets, leveraging dense geometric and semantic annotations for robust intermediate spatial representation. The dataset comprises 1,024 richly annotated indoor scenes and 10,094 QA pairs, each describing scenarios rooted in everyday object search tasks—phones, chargers, keys, etc.—where the targets are typically occluded, weakly salient, and context-dependent, reflecting real-world complexities absent in prior benchmarks.
The benchmark decomposes spatial understanding into four hierarchically ordered tasks:
- Target Presence Verification (TPV): Binary scene-level detection of small object appearance, isolating perception under realistic clutter, occlusion, and viewpoint variation.
- Nearest Reference Identification (NRI): Multiple-choice selection of the non-supporting object most proximal to the target, probing spatial proximity and local context anchoring.
- Fine-Grained Spatial Description (FSD): Generation of natural-language localization including support surface and reference objects with centimeter-level distance, evaluated via a structured LLM-as-a-judge protocol that penalizes loss of referential accuracy and geometric drift.
- Structured Spatial Prediction (SSP): Emission of a standardized JSON schema capturing target, support surface, and key references with explicit relations and distances, enabling downstream pick-and-place or navigation use.
The construction pipeline first selects small object targets from ScanNet semantic categories, filters nearby objects (<1.0 m), computes fine-grained 3D relations, and instantiates task-specific QA formats from the shared intermediate representation. QA pairs undergo both automated filtering (instance uniqueness, answer format validity) and manual spot-checking (referential correctness, plausible grounding), balancing rigor and coverage.
Experimental Findings
Extensive experiments benchmark eight representative MLLMs, including proprietary (GPT-5.4, Kimi K2.5) and open-source (Qwen3-VL-8B-Instruct, LLaVA-OneVision-1.5-8B, InternVL3.5-8B-Instruct, Cambrian-S-7B, SenseNova-SI-1.3-InternVL3-8B, Spatial-MLLM-v1.1) architectures, with additional fine-tuning (LoRA-based) of Qwen3-VL and InternVL backbones. Data splits ensure no scene-level leakage.
Key results:
- Monotonic Degradation Across Task Chain: All models exhibit a consistent and substantial drop in performance as tasks progress from TPV to SSP. Qwen3-VL-8B-Instruct-SFT, for example, achieves 0.83 TPV Micro-F1 but only 0.29 for SSP. Non-finetuned versions and proprietary models show similar trends, indicating a systemic bottleneck in compositional, reference-driven, and structured localization.
- Dataset-Driven Fine-Tuning Gains: LoRA fine-tuning on PinpointQA yields significant improvements, particularly in SSP, but remains substantially lower than TPV. For instance, Qwen3-VL-8B-Instruct improves from 0.39 to 0.48 (Avg-Micro), with gains most pronounced in the hardest task (SSP: +0.17). These improvements underline PinpointQA’s utility both as diagnostic tool and as supervision source for grounded spatial QA training.
- Diagnostic Capability: The progressive task design reveals model-specific failure modes: some architectures fail early at target presence; others maintain plausible global localization yet drift in referential precision or lose structural information, as evidenced by output inconsistencies in both free-form and JSON-based outputs. Such granularity enables ablation studies and error analysis not possible with monolithic metrics.
- Practical Relevance: Human assistance evaluation demonstrates that FSD outputs from MLLMs—when correct—meaningfully accelerate and improve small object localization in video, corroborating the benchmark’s focus on downstream applicability. FSD guidance boosts human localization accuracy from 62% to 79% and halves completion time.
Theoretical and Practical Implications
PinpointQA’s staged framework operationalizes the gap between scene-level perception and actionable, reference-centric spatial reasoning, quantifying model deficiencies with respect to fine-grained localization—an essential precursor to robust embodied AI and assistive robotics. By structurally isolating each stage, it enables systematic study of attention, compositionality, and grounding mechanisms in MLLMs. The significant drop-off from recognition to structured expression underscores the limitations of current architectures, emphasizing the need for enhanced spatial reasoning abilities, local context preservation, and reference-aware output mechanisms.
Practically, the dataset offers a necessary substrate for instruction tuning and benchmarking in agentic scenarios (object search, fetch, assembly), especially in domains with weakly salient targets. The introduction of structured output evaluation protocols also facilitates integration with downstream robotics or spatial computing tasks.
Limitations and Future Directions
PinpointQA is deliberately scoped to indoor scenes and small-object centric queries, forgoing full-scene open-domain reasoning or embodied interaction. Its construction logic, rooted in stable geometric representations, ensures consistency but regularizes natural language output, potentially trading some linguistic diversity for spatial accuracy. Reproduction and extension depend on access to underlying ScanNet assets.
Future work should address: (1) integration of open-set and cross-scene generalization, (2) real-time or continual localization in dynamic environments, (3) unification with broader embodied interaction benchmarks, and (4) exploration of architectures that robustly encode, maintain, and output target-centered spatial context across free-form and structured modalities.
Conclusion
PinpointQA establishes a rigorous, compositional benchmark for evaluating and advancing small object-centric spatial understanding in indoor video by MLLMs. Its progressive task chain and high-quality data curation illuminate persistent weaknesses in reference-based localization, providing both an effective diagnostic testbed and a valuable training resource. These insights lay the groundwork for next-generation multimodal agents requiring robust, actionable spatial intelligence (2604.08991).