---
title: 'PinpointQA: Indoor Video Object Localization'
url: https://www.emergentmind.com/topics/pinpointqa
type: topic
---

# PinpointQA: Indoor Video Object Localization

PinpointQA is a dataset and benchmark for **small object-centric spatial understanding in indoor videos**. It is designed to evaluate whether multimodal large language models can do more than generic video reasoning: specifically, whether they can determine whether a small target object appears in a scene video, ground that object in local context, describe its final location in precise natural language, and encode that location in a structured format suitable for downstream use. Built from **ScanNet++** and **ScanNet200**, PinpointQA contains **1,024 scenes** and **10,094 QA pairs** organized into a **four-task progressive capability chain**: **Target Presence Verification (TPV)**, **Nearest Reference Identification (NRI)**, **Fine-Grained Spatial Description (FSD)**, and **Structured Spatial Prediction (SSP)** [2604.08991].

## 1. Definition and research scope

PinpointQA targets a gap in video-language evaluation that arises when models must localize **small, everyday indoor objects** such as a **phone, key, charger, earbuds, glasses, or remote control** and communicate their positions with enough precision to be useful for **object search** and **assistive applications**. The benchmark is motivated by the claim that prior benchmarks on video spatial intelligence, embodied reasoning, and diagnostic perception do not directly test whether a model can localize a small target in video and express that localization with sufficient precision for downstream use. Its problem formulation therefore centers on **target-centered local spatial understanding**, rather than broad scene captioning or generic video question answering [2604.08991].

The emphasis on small objects is methodologically important. The benchmark describes these objects as especially difficult because they are **less salient**, **more easily occluded**, and often identifiable only through their **local context** rather than global scene understanding. This makes PinpointQA a benchmark for grounded spatial inference under cluttered, multi-view indoor observation, rather than a benchmark for coarse scene semantics. The paper further characterizes it as the first benchmark that simultaneously emphasizes **spatial reasoning**, **continuous scene observation**, **everyday indoor object context**, and **small object localization** [2604.08991].

## 2. Dataset composition and grounded construction

PinpointQA is built from **ScanNet++** and **ScanNet200**, which provide **aligned 3D meshes, instance-level annotations, and segmentation masks**. The benchmark contains **1,024 scenes** and **10,094 QA pairs**, with approximately **73.2% from ScanNet++** and **26.8% from ScanNet200**. The four tasks are broadly balanced: **TPV 26.47%**, **NRI 23.10%**, **FSD 25.08%**, and **SSP 25.34%** [2604.08991].

| Aspect | Value |
|---|---|
| Sources | ScanNet++, ScanNet200 |
| Scenes | 1,024 |
| QA pairs | 10,094 |
| Source mix | 73.2% / 26.8% |
| Task balance | TPV 26.47%, NRI 23.10%, FSD 25.08%, SSP 25.34% |

A defining design choice is the use of an **intermediate spatial representation** rather than direct manual free-form annotation. For each target, the representation includes the **target small object**, its **supporting surface**, nearby **reference objects**, **3D spatial relations** between the target and those nearby objects, **centimeter-level distances**, and directional or vertical relations such as **above**, **below**, **side_above**, and **side_below**. Nearby objects within **1.0 m** of the target are retained by default. The benchmark then derives all four QA tasks from this shared grounded representation, so the tasks are different projections of the same spatial substrate rather than independent annotation pipelines [2604.08991].

The construction pipeline has four stages: **data collection**, **scene curation**, **task-specific QA generation**, and **quality control**. Scene curation selects candidate targets from a predefined small-object vocabulary and combines segmentation with aligned meshes to produce object-level geometric representations. Quality control includes automatic filtering—such as removing invalid labels, excluding background or structural objects, keeping only target instances satisfying the vocabulary constraint, generating QA only for targets with **unique labels within a scene**, ensuring that NRI has **four distinct options**, requiring FSD answers to be **human-readable**, and requiring SSP outputs to be **parsable**—followed by iterative manual review of **QA templates and answer logic**. Each review round spot-checks roughly **100 QA pairs across 10–15 scenes**, and only templates and generation logic are manually refined; the grounded scene components remain unchanged [2604.08991].

## 3. Progressive task hierarchy and spatial formalization

PinpointQA’s four tasks are explicitly organized as a **progressive capability chain** from easier to harder abilities: **target presence verification**, **reference-based grounding**, **fine-grained natural-language description**, and **structured spatial output**. The benchmark’s logic is that detecting a small object, anchoring it relative to nearby objects, verbalizing its final position, and producing a machine-usable spatial representation are distinct capabilities that should not be conflated [2604.08991].

**TPV** asks whether the target appears anywhere in the video. It is a yes/no task and is described as the entry-level ability, since it tests existence but not explicit grounding. Positive questions use targets present in the scene; negative questions use small-object categories absent from the scene.

**NRI** asks the model to identify the **nearest reference object** to the target’s final location while **excluding the supporting surface**. It is a four-choice multiple-choice task. Its function is to test whether the model can anchor the target in its **local spatial context** and reason about **proximity** beyond mere presence.

**FSD** requires a natural-language description of the target’s final location. The benchmark instructions shown in the figure specify: “**Use 1-2 natural sentences. Mention the supporting surface and 1-2 nearest objects with exact distances in cm. No meters or vague approximations.**” This makes FSD a test of whether a model can preserve grounded spatial facts while converting them into readable language.

**SSP** requires the same content as FSD, but in structured JSON. The schema contains `target`, `support_surface`, and `references` with **0–2 items**. Each reference must contain `object`, `relation`, and `distance_cm`, and the allowed relations are:
\[
\{\texttt{on}, \texttt{under}, \texttt{above}, \texttt{below}, \texttt{side\_above}, \texttt{side\_below}, \texttt{next\_to}, \texttt{near}, \texttt{attached\_to}\}.
\]
The benchmark figure specifies: “**Output ONLY a valid parsable JSON object with fields target, support_surface, and references (0-2 items). Each reference must contain object, relation (on, under, above, below, side_above, side_below, next_to, near, attached_to), and float distance_cm. Sort references by distance. No extra text or markdown.**”

An example SSP output is:

```json
{"target": "phone", "support_surface": "desk", "references": [{"object": "laptop", "relation": "next_to", "distance_cm": 3.0}, {"object": "pouch", "relation": "near", "distance_cm": 5.0}]}
```

The benchmark’s spatial formulation is therefore explicitly **target-centered** and **local**: the support surface is the main anchor, one or two nearby references refine localization, and distances are given at centimeter scale. This suggests that PinpointQA is less concerned with exhaustive scene graphs than with compact, operational object localization [2604.08991].

## 4. Evaluation protocol and experimental design

PinpointQA uses task-specific evaluation rather than a single unified metric. **TPV** and **NRI** use **exact-match accuracy**. **FSD** is evaluated by **LLM-as-a-judge** using **GPT-5.4**, because BLEU, ROUGE, and exact string match are considered unsuitable for spatially equivalent but lexically different descriptions. The FSD judge returns structured JSON with scores and error tags such as **wrong main location**, **wrong supporting surface**, **missing key reference**, **wrong spatial relation**, **missing or wrong numeric distance**, **unclear expression**, and **hallucinated reference**. The FSD score uses a 10-point rubric, normalized to \([0,1]\), across five dimensions: main location and supporting surface \(0\!-\!3\), key reference objects \(0\!-\!2\), spatial relations \(0\!-\!2\), centimeter-level distance \(0\!-\!2\), and clarity \(0\!-\!1\), with
\[
\texttt{total\_score} = A + B + C + D + E.
\]
Distance scoring uses range-aware tolerances: about \(\pm 5\) cm for ground-truth distances in \(0\!-\!10\) cm, about \(\pm 10\) cm in \(10\!-\!30\) cm, and about \(\pm 15\) cm above \(30\) cm [2604.08991].

**SSP** is evaluated by parsing and normalizing predicted JSON, aligning predicted references to ground-truth slots, and computing a weighted soft score. The weights are **0.4** for supporting surface, **0.2** for object identity, **0.2** for relation, and **0.2** for distance. Soft relation matching is allowed for semantically close pairs such as `next_to` vs. `near`, `under` vs. `below`, and `on` vs. `attached_to`. Aggregation is reported as **Micro** (average over all QA pairs of a task), **Macro** (average of per-scene means), and **Avg** (arithmetic mean across the four tasks), with all scores normalized to \([0,1]\) [2604.08991].

The benchmark evaluates three groups of MLLMs: **proprietary models** (**GPT-5.4**, **Kimi K2.5**), **open-source models** (**LLaVA-OneVision-1.5-8B**, **Qwen3-VL-8B-Instruct**, **InternVL3.5-8B-Instruct**, **Spatial-MLLM-v1.1-Instruct-820K**, **SenseNova-SI-1.3-InternVL3-8B**, **Cambrian-S-7B**), and **fine-tuned models** (**Qwen3-VL-8B-Instruct-SFT**, **InternVL3.5-8B-Instruct-SFT**). The split is at the **scene level** in an approximate **6:2:2** ratio: **6,121 training**, **1,954 validation**, and **2,019 evaluation** samples. Inference uses **64 uniformly sampled frames** per input video, instruction-guided prompts, mostly consistent templates across models, and **greedy decoding**. Fine-tuning uses **LoRA**, trains on the training split, selects the best checkpoint on validation, and runs for **2 epochs** on two representative backbones. The main text does **not** provide full mathematical task definitions or training loss formulas for the fine-tuned models [2604.08991].

## 5. Empirical findings

The central empirical result is a **steady degradation** in model performance along the progression **TPV \(\rightarrow\) NRI \(\rightarrow\) FSD \(\rightarrow\) SSP**, with **SSP** consistently the hardest stage. The best overall system is **Qwen3-VL-8B-Instruct-SFT**, with **Avg-Micro 0.48** and **Avg-Macro 0.49**; the second-best is **InternVL3.5-8B-Instruct-SFT** at **0.45 / 0.45**. Among proprietary models, **Kimi K2.5** outperforms **GPT-5.4** at **0.42 / 0.44** versus **0.38 / 0.40**. Among non-fine-tuned open-source models, **Qwen3-VL-8B-Instruct** is strongest at **0.39 / 0.40** [2604.08991].

The progressive-capability result is visible in per-task breakdowns. For **Qwen3-VL-8B-Instruct-SFT**, scores are **TPV 0.83 / 0.84**, **NRI 0.44 / 0.45**, **FSD 0.36 / 0.37**, and **SSP 0.29 / 0.29**. For **Kimi K2.5**, they are **0.80 / 0.84**, **0.42 / 0.44**, **0.32 / 0.33**, and **0.15 / 0.15**. For **Qwen3-VL-8B-Instruct**, they are **0.78 / 0.80**, **0.37 / 0.37**, **0.28 / 0.29**, and **0.12 / 0.12**. The interpretation offered by the benchmark is that current MLLMs are better at detecting the target than at grounding it, describing it precisely, or encoding it reliably in structured form.

**SSP** is the weakest stage across almost all models. Reported SSP scores include **0.07 / 0.06** for **LLaVA-OneVision-1.5-8B**, **0.05 / 0.06** for **Cambrian-S-7B**, **0.00 / 0.00** for **Spatial-MLLM-v1.1-Instruct-820K**, **0.15 / 0.16** for **GPT-5.4**, and **0.29 / 0.29** for the best fine-tuned model. The paper’s interpretation is that **executable spatial grounding** remains the primary bottleneck.

Supervised fine-tuning on PinpointQA yields substantial gains, especially on the harder tasks. **Qwen3-VL-8B-Instruct** improves from **0.39 / 0.40** to **0.48 / 0.49** overall and from **0.12 / 0.12** to **0.29 / 0.29** on SSP after fine-tuning. **InternVL3.5-8B-Instruct** improves from **0.34 / 0.36** to **0.45 / 0.45** overall and from **0.09 / 0.10** to **0.23 / 0.24** on SSP. The largest gains therefore appear on the structured grounding task rather than on target presence, which suggests that PinpointQA functions not only as a diagnostic benchmark but also as a useful supervision source [2604.08991].

The qualitative error analysis supports the same interpretation. **TPV** failures show that some models still miss the target entirely under clutter and viewpoint variation. **NRI** failures show that models can identify the target yet choose a more salient distractor instead of the true nearest reference. **FSD** failures are often fluent but spatially wrong, drifting away from the correct target-centered local context. **SSP** failures frequently produce syntactically valid JSON with incorrect support surfaces, references, relations, or distances. The benchmark therefore separates structural validity from grounded correctness.

## 6. Human utility, related work, and limitations

PinpointQA is presented as both a **diagnostic benchmark** and a **training dataset**, and the human-assistance study provides evidence for its practical utility. Using **60 FSD samples** and a web-based click interface, the study compares **Unguided**, **Model-Assisted** using FSD generated by **Qwen3-VL-8B-Instruct-SFT**, and **GT-Assisted** settings. Accuracy is scored by
\[
s = \max \left(0, 1 - \left(\frac{d}{r}\right)^2 \right), \qquad r = 0.12,
\]
where \(d\) is the normalized Euclidean distance between the participant click and the annotated target point, provided the selected frame is correct. Reported accuracies are **62.1%** for Unguided, **79.4%** for Model-Assisted, and **87.8%** for GT-Assisted; average completion times are **29 s**, **15 s**, and **9 s**, respectively. This indicates that fine-grained spatial descriptions are practically useful for small-object search, while still leaving a measurable gap between model output and ground truth [2604.08991].

In the broader literature, PinpointQA occupies a specific niche. “**Point and Ask: Incorporating Pointing into Visual Question Answering**” [2011.13681] studies point-conditioned image VQA, where a user-provided point is required to disambiguate the referent. PinpointQA differs in that it does not assume an explicit point input and instead evaluates whether a model can recover small-object localization directly from an indoor video sequence and then express that localization in natural-language and structured forms. “**Focus, Don’t Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding**” [2603.22815] addresses instruction-relevant region selection for document and infographic QA. A plausible implication is that PinpointQA probes an analogous failure mode in video MLLMs—insufficiently precise allocation of attention to small, localization-critical evidence—but under indoor video and target-object conditions rather than document layouts.

The benchmark is also explicit about its limitations. It is restricted to **indoor scenes**; it focuses on **small object-centric localization and expression**, not full-scene understanding, open-world tracking, dense video captioning, or embodied interaction; and its QA pairs are derived from **intermediate spatial representations**, which improves consistency but may regularize language style relative to unconstrained human descriptions. The repository does **not redistribute original videos or raw scene assets**, so full reproduction depends on access to the underlying source datasets [2604.08991].

PinpointQA’s main contribution is therefore methodological as much as empirical. It formalizes a narrow but consequential question for multimodal systems: whether a model can **detect**, **ground**, **describe**, and **structurally encode** the location of a small indoor object in video. By separating those abilities into TPV, NRI, FSD, and SSP, it makes explicit that target presence is not equivalent to grounded spatial competence, and that structured spatial prediction remains substantially weaker than recognition alone [2604.08991].

Source: https://www.emergentmind.com/topics/pinpointqa