PinpointQA: Indoor Video Object Localization

Updated 5 July 2026

PinpointQA is a benchmark for small object-centric spatial understanding in indoor video scenes, emphasizing detection, grounding, and precise spatial description.
It structures evaluation into four progressive tasks—from target presence verification to structured spatial prediction—using data from ScanNet++ and ScanNet200.
The dataset enables practical assessments of models with challenging centimeter-level accuracy requirements for real-world object search and assistive applications.

PinpointQA is a dataset and benchmark for small object-centric spatial understanding in indoor videos. It is designed to evaluate whether multimodal LLMs can do more than generic video reasoning: specifically, whether they can determine whether a small target object appears in a scene video, ground that object in local context, describe its final location in precise natural language, and encode that location in a structured format suitable for downstream use. Built from ScanNet++ and ScanNet200, PinpointQA contains 1,024 scenes and 10,094 QA pairs organized into a four-task progressive capability chain: Target Presence Verification (TPV), Nearest Reference Identification (NRI), Fine-Grained Spatial Description (FSD), and Structured Spatial Prediction (SSP) (Zhou et al., 10 Apr 2026).

1. Definition and research scope

PinpointQA targets a gap in video-language evaluation that arises when models must localize small, everyday indoor objects such as a phone, key, charger, earbuds, glasses, or remote control and communicate their positions with enough precision to be useful for object search and assistive applications. The benchmark is motivated by the claim that prior benchmarks on video spatial intelligence, embodied reasoning, and diagnostic perception do not directly test whether a model can localize a small target in video and express that localization with sufficient precision for downstream use. Its problem formulation therefore centers on target-centered local spatial understanding, rather than broad scene captioning or generic video question answering (Zhou et al., 10 Apr 2026).

The emphasis on small objects is methodologically important. The benchmark describes these objects as especially difficult because they are less salient, more easily occluded, and often identifiable only through their local context rather than global scene understanding. This makes PinpointQA a benchmark for grounded spatial inference under cluttered, multi-view indoor observation, rather than a benchmark for coarse scene semantics. The paper further characterizes it as the first benchmark that simultaneously emphasizes spatial reasoning, continuous scene observation, everyday indoor object context, and small object localization (Zhou et al., 10 Apr 2026).

2. Dataset composition and grounded construction

PinpointQA is built from ScanNet++ and ScanNet200, which provide aligned 3D meshes, instance-level annotations, and segmentation masks. The benchmark contains 1,024 scenes and 10,094 QA pairs, with approximately 73.2% from ScanNet++ and 26.8% from ScanNet200. The four tasks are broadly balanced: TPV 26.47%, NRI 23.10%, FSD 25.08%, and SSP 25.34% (Zhou et al., 10 Apr 2026).

Aspect	Value
Sources	ScanNet++, ScanNet200
Scenes	1,024
QA pairs	10,094
Source mix	73.2% / 26.8%
Task balance	TPV 26.47%, NRI 23.10%, FSD 25.08%, SSP 25.34%

A defining design choice is the use of an intermediate spatial representation rather than direct manual free-form annotation. For each target, the representation includes the target small object, its supporting surface, nearby reference objects, 3D spatial relations between the target and those nearby objects, centimeter-level distances, and directional or vertical relations such as above, below, side_above, and side_below. Nearby objects within 1.0 m of the target are retained by default. The benchmark then derives all four QA tasks from this shared grounded representation, so the tasks are different projections of the same spatial substrate rather than independent annotation pipelines (Zhou et al., 10 Apr 2026).

The construction pipeline has four stages: data collection, scene curation, task-specific QA generation, and quality control. Scene curation selects candidate targets from a predefined small-object vocabulary and combines segmentation with aligned meshes to produce object-level geometric representations. Quality control includes automatic filtering—such as removing invalid labels, excluding background or structural objects, keeping only target instances satisfying the vocabulary constraint, generating QA only for targets with unique labels within a scene, ensuring that NRI has four distinct options, requiring FSD answers to be human-readable, and requiring SSP outputs to be parsable—followed by iterative manual review of QA templates and answer logic. Each review round spot-checks roughly 100 QA pairs across 10–15 scenes, and only templates and generation logic are manually refined; the grounded scene components remain unchanged (Zhou et al., 10 Apr 2026).

3. Progressive task hierarchy and spatial formalization

PinpointQA’s four tasks are explicitly organized as a progressive capability chain from easier to harder abilities: target presence verification, reference-based grounding, fine-grained natural-language description, and structured spatial output. The benchmark’s logic is that detecting a small object, anchoring it relative to nearby objects, verbalizing its final position, and producing a machine-usable spatial representation are distinct capabilities that should not be conflated (Zhou et al., 10 Apr 2026).

TPV asks whether the target appears anywhere in the video. It is a yes/no task and is described as the entry-level ability, since it tests existence but not explicit grounding. Positive questions use targets present in the scene; negative questions use small-object categories absent from the scene.

NRI asks the model to identify the nearest reference object to the target’s final location while excluding the supporting surface. It is a four-choice multiple-choice task. Its function is to test whether the model can anchor the target in its local spatial context and reason about proximity beyond mere presence.

FSD requires a natural-language description of the target’s final location. The benchmark instructions shown in the figure specify: “Use 1-2 natural sentences. Mention the supporting surface and 1-2 nearest objects with exact distances in cm. No meters or vague approximations.” This makes FSD a test of whether a model can preserve grounded spatial facts while converting them into readable language.

SSP requires the same content as FSD, but in structured JSON. The schema contains target, support_surface, and references with 0–2 items. Each reference must contain object, relation, and distance_cm, and the allowed relations are: $\{\texttt{on}, \texttt{under}, \texttt{above}, \texttt{below}, \texttt{side\_above}, \texttt{side\_below}, \texttt{next\_to}, \texttt{near}, \texttt{attached\_to}\}.$ The benchmark figure specifies: “Output ONLY a valid parsable JSON object with fields target, support_surface, and references (0-2 items). Each reference must contain object, relation (on, under, above, below, side_above, side_below, next_to, near, attached_to), and float distance_cm. Sort references by distance. No extra text or markdown.”

An example SSP output is:

$0\!-\!3$ 0

The benchmark’s spatial formulation is therefore explicitly target-centered and local: the support surface is the main anchor, one or two nearby references refine localization, and distances are given at centimeter scale. This suggests that PinpointQA is less concerned with exhaustive scene graphs than with compact, operational object localization (Zhou et al., 10 Apr 2026).

4. Evaluation protocol and experimental design

PinpointQA uses task-specific evaluation rather than a single unified metric. TPV and NRI use exact-match accuracy. FSD is evaluated by LLM-as-a-judge using GPT-5.4, because BLEU, ROUGE, and exact string match are considered unsuitable for spatially equivalent but lexically different descriptions. The FSD judge returns structured JSON with scores and error tags such as wrong main location, wrong supporting surface, missing key reference, wrong spatial relation, missing or wrong numeric distance, unclear expression, and hallucinated reference. The FSD score uses a 10-point rubric, normalized to $[0,1]$ , across five dimensions: main location and supporting surface $0\!-\!3$ , key reference objects $0\!-\!2$ , spatial relations $0\!-\!2$ , centimeter-level distance $0\!-\!2$ , and clarity $0\!-\!1$ , with

$\texttt{total\_score} = A + B + C + D + E.$

Distance scoring uses range-aware tolerances: about $\pm 5$ cm for ground-truth distances in $0\!-\!10$ cm, about $[0,1]$ 0 cm in $[0,1]$ 1 cm, and about $[0,1]$ 2 cm above $[0,1]$ 3 cm (Zhou et al., 10 Apr 2026).

SSP is evaluated by parsing and normalizing predicted JSON, aligning predicted references to ground-truth slots, and computing a weighted soft score. The weights are 0.4 for supporting surface, 0.2 for object identity, 0.2 for relation, and 0.2 for distance. Soft relation matching is allowed for semantically close pairs such as next_to vs. near, under vs. below, and on vs. attached_to. Aggregation is reported as Micro (average over all QA pairs of a task), Macro (average of per-scene means), and Avg (arithmetic mean across the four tasks), with all scores normalized to $[0,1]$ 4 (Zhou et al., 10 Apr 2026).

The benchmark evaluates three groups of MLLMs: proprietary models (GPT-5.4, Kimi K2.5), open-source models (LLaVA-OneVision-1.5-8B, Qwen3-VL-8B-Instruct, InternVL3.5-8B-Instruct, Spatial-MLLM-v1.1-Instruct-820K, SenseNova-SI-1.3-InternVL3-8B, Cambrian-S-7B), and fine-tuned models (Qwen3-VL-8B-Instruct-SFT, InternVL3.5-8B-Instruct-SFT). The split is at the scene level in an approximate 6:2:2 ratio: 6,121 training, 1,954 validation, and 2,019 evaluation samples. Inference uses 64 uniformly sampled frames per input video, instruction-guided prompts, mostly consistent templates across models, and greedy decoding. Fine-tuning uses LoRA, trains on the training split, selects the best checkpoint on validation, and runs for 2 epochs on two representative backbones. The main text does not provide full mathematical task definitions or training loss formulas for the fine-tuned models (Zhou et al., 10 Apr 2026).

5. Empirical findings

The central empirical result is a steady degradation in model performance along the progression TPV $[0,1]$ 5 NRI $[0,1]$ 6 FSD $[0,1]$ 7 SSP, with SSP consistently the hardest stage. The best overall system is Qwen3-VL-8B-Instruct-SFT, with Avg-Micro 0.48 and Avg-Macro 0.49; the second-best is InternVL3.5-8B-Instruct-SFT at 0.45 / 0.45. Among proprietary models, Kimi K2.5 outperforms GPT-5.4 at 0.42 / 0.44 versus 0.38 / 0.40. Among non-fine-tuned open-source models, Qwen3-VL-8B-Instruct is strongest at 0.39 / 0.40 (Zhou et al., 10 Apr 2026).

The progressive-capability result is visible in per-task breakdowns. For Qwen3-VL-8B-Instruct-SFT, scores are TPV 0.83 / 0.84, NRI 0.44 / 0.45, FSD 0.36 / 0.37, and SSP 0.29 / 0.29. For Kimi K2.5, they are 0.80 / 0.84, 0.42 / 0.44, 0.32 / 0.33, and 0.15 / 0.15. For Qwen3-VL-8B-Instruct, they are 0.78 / 0.80, 0.37 / 0.37, 0.28 / 0.29, and 0.12 / 0.12. The interpretation offered by the benchmark is that current MLLMs are better at detecting the target than at grounding it, describing it precisely, or encoding it reliably in structured form.

SSP is the weakest stage across almost all models. Reported SSP scores include 0.07 / 0.06 for LLaVA-OneVision-1.5-8B, 0.05 / 0.06 for Cambrian-S-7B, 0.00 / 0.00 for Spatial-MLLM-v1.1-Instruct-820K, 0.15 / 0.16 for GPT-5.4, and 0.29 / 0.29 for the best fine-tuned model. The paper’s interpretation is that executable spatial grounding remains the primary bottleneck.

Supervised fine-tuning on PinpointQA yields substantial gains, especially on the harder tasks. Qwen3-VL-8B-Instruct improves from 0.39 / 0.40 to 0.48 / 0.49 overall and from 0.12 / 0.12 to 0.29 / 0.29 on SSP after fine-tuning. InternVL3.5-8B-Instruct improves from 0.34 / 0.36 to 0.45 / 0.45 overall and from 0.09 / 0.10 to 0.23 / 0.24 on SSP. The largest gains therefore appear on the structured grounding task rather than on target presence, which suggests that PinpointQA functions not only as a diagnostic benchmark but also as a useful supervision source (Zhou et al., 10 Apr 2026).

The qualitative error analysis supports the same interpretation. TPV failures show that some models still miss the target entirely under clutter and viewpoint variation. NRI failures show that models can identify the target yet choose a more salient distractor instead of the true nearest reference. FSD failures are often fluent but spatially wrong, drifting away from the correct target-centered local context. SSP failures frequently produce syntactically valid JSON with incorrect support surfaces, references, relations, or distances. The benchmark therefore separates structural validity from grounded correctness.

PinpointQA is presented as both a diagnostic benchmark and a training dataset, and the human-assistance study provides evidence for its practical utility. Using 60 FSD samples and a web-based click interface, the study compares Unguided, Model-Assisted using FSD generated by Qwen3-VL-8B-Instruct-SFT, and GT-Assisted settings. Accuracy is scored by

$[0,1]$ 8

where $[0,1]$ 9 is the normalized Euclidean distance between the participant click and the annotated target point, provided the selected frame is correct. Reported accuracies are 62.1% for Unguided, 79.4% for Model-Assisted, and 87.8% for GT-Assisted; average completion times are 29 s, 15 s, and 9 s, respectively. This indicates that fine-grained spatial descriptions are practically useful for small-object search, while still leaving a measurable gap between model output and ground truth (Zhou et al., 10 Apr 2026).

In the broader literature, PinpointQA occupies a specific niche. “Point and Ask: Incorporating Pointing into Visual Question Answering” (Mani et al., 2020) studies point-conditioned image VQA, where a user-provided point is required to disambiguate the referent. PinpointQA differs in that it does not assume an explicit point input and instead evaluates whether a model can recover small-object localization directly from an indoor video sequence and then express that localization in natural-language and structured forms. “Focus, Don’t Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding” (Kwon et al., 24 Mar 2026) addresses instruction-relevant region selection for document and infographic QA. A plausible implication is that PinpointQA probes an analogous failure mode in video MLLMs—insufficiently precise allocation of attention to small, localization-critical evidence—but under indoor video and target-object conditions rather than document layouts.

The benchmark is also explicit about its limitations. It is restricted to indoor scenes; it focuses on small object-centric localization and expression, not full-scene understanding, open-world tracking, dense video captioning, or embodied interaction; and its QA pairs are derived from intermediate spatial representations, which improves consistency but may regularize language style relative to unconstrained human descriptions. The repository does not redistribute original videos or raw scene assets, so full reproduction depends on access to the underlying source datasets (Zhou et al., 10 Apr 2026).

PinpointQA’s main contribution is therefore methodological as much as empirical. It formalizes a narrow but consequential question for multimodal systems: whether a model can detect, ground, describe, and structurally encode the location of a small indoor object in video. By separating those abilities into TPV, NRI, FSD, and SSP, it makes explicit that target presence is not equivalent to grounded spatial competence, and that structured spatial prediction remains substantially weaker than recognition alone (Zhou et al., 10 Apr 2026).