MV-ScanQA: 3D Multi-View QA Benchmark

• MV-ScanQA is a 3D question answering dataset designed for evaluating multi-view compositional reasoning in indoor scenes.
• It addresses the single-view bottleneck by requiring integration of evidence from multiple egocentric views, with 68% of questions demanding such reasoning.
• The benchmark leverages the TripAlign pre-training corpus and LEGO baseline to transfer 2D vision-language expertise into complex 3D tasks.

Searching arXiv for the specified paper and closely related 3D vision-language benchmarks to ground the article with current citations. Querying the arXiv API for the paper metadata. Attempting an arXiv API lookup via Python. The execution environment does not expose external network access, so the article is grounded in the supplied paper record and metadata for the specified arXiv entry (Mo et al., 14 Aug 2025). MV-ScanQA is a 3D question answering dataset introduced to evaluate multi-view compositional reasoning in indoor scenes, together with the TripAlign pre-training corpus and the LEGO baseline for transferring 2D large vision-LLM knowledge into the 3D domain (Mo et al., 14 Aug 2025). The benchmark is motivated by a reported limitation of prior 3D vision-language datasets: despite their 3D formulation, more than $93\%$ of questions in ScanQA, ScanRefer, and Nr3D are solvable from a single egocentric view under the paper’s witnessing criterion, and existing annotations typically bind language to at most one target object rather than to contextually related object groups (Mo et al., 14 Aug 2025). MV-ScanQA is designed so that most questions require integrating evidence across views, while TripAlign supplies large-scale $\langle$2D view, set of 3D objects, text$\rangle$ supervision for learning such alignments (Mo et al., 14 Aug 2025).

1. Motivation and problem setting

The work situates MV-ScanQA within 3D vision-language learning, specifically the problem of grounding language in 3D scenes when the relevant evidence is spatially distributed and not jointly visible from a single viewpoint (Mo et al., 14 Aug 2025). The central claim is that existing 3D VL benchmarks exhibit a “Single-View Bottleneck”: a solvability analysis over ScanQA, ScanRefer, and Nr3D found that more than $93\%$ of all questions can be answered from a single view if every mentioned object is witnessed in that view (Mo et al., 14 Aug 2025). In parallel, prior corpora are characterized as having sparse single-object annotation, in which each instruction is linked to at most one target object, omitting richer contextual alignments among multiple objects (Mo et al., 14 Aug 2025).

This formulation matters because it distinguishes nominal 3D grounding from genuine multi-view reasoning. A common assumption in 3D VL is that operating on reconstructed scenes or point clouds automatically induces broad scene understanding. The reported solvability analysis challenges that assumption: if benchmark questions can be answered from one egocentric frame, then the task pressure for integrating distant or distributed evidence is weak (Mo et al., 14 Aug 2025). This suggests that benchmark design, not only model architecture, is a limiting factor in the development of models for deeper 3D scene understanding.

MV-ScanQA is therefore constructed to force compositional reasoning over two or more views. The paper states that $68\%$ of its questions explicitly require integrating information from multiple views, compared with less than $7\%$ in existing datasets (Mo et al., 14 Aug 2025). That statistic is the benchmark’s defining property and underlies both its dataset design and its associated training methodology.

2. Formal design principles and question synthesis

The dataset formalizes whether an object is visible enough to support reasoning through the intersection-over-smallest-area measure:

$$\mathrm{IoSA}(A,B)=\frac{\mathrm{Area}(A\cap B)}{\min(\mathrm{Area}(A),\mathrm{Area}(B))}\,.$$

An object is “witnessed” if its IoSA with the view’s image plane exceeds $0.5$ (Mo et al., 14 Aug 2025). Using this criterion, solvability is defined in terms of whether a set of egocentric views jointly witnesses every object mentioned in a question (Mo et al., 14 Aug 2025).

Question construction proceeds by compositional synthesis from existing ScanQA items. Original single-view questions $(q_1,q_2)$ are paired when their related object sets $O_1$ and $\langle$0 satisfy

$\langle$1

These constraints enforce partial overlap without degenerating into full redundancy or strict containment (Mo et al., 14 Aug 2025). The paired questions, together with their ground-truth answers, are then fed to Claude-3.5-Sonnet under prompts that enforce integrative complexity and clear QA verifiability, yielding a new question $\langle$2 and answer $\langle$3 (Mo et al., 14 Aug 2025).

The design objective is not merely to concatenate two questions, but to synthesize a single unambiguous query whose resolution depends on cross-view integration. The paper’s examples include spatial relationship, counting, attribute comparison, and composite multi-view queries such as “What is on the right of the small desk where the wooden chair is in front of?” (Mo et al., 14 Aug 2025). These examples indicate that the dataset targets compositional reference chains spanning multiple objects and viewpoints rather than isolated object recognition.

3. Dataset composition and annotation pipeline

MV-ScanQA reuses the ScanNet training, validation, and test splits used in ScanQA, and contains approximately $\langle$4 newly synthesized QA pairs (Mo et al., 14 Aug 2025). The benchmark’s composition is summarized by the minimum number of egocentric frames required to witness all mentioned objects.

Requirement	Share
$\langle$5 view	32%
$\langle$6 views	55%
$\langle$7 views	13%

These values are measured by finding the minimum number of ScanNet egocentric frames whose union of witnessed objects covers all mentions in the question (Mo et al., 14 Aug 2025). In consequence, $\langle$8 of questions require more than one view, which is the benchmark’s principal differentiator from prior datasets (Mo et al., 14 Aug 2025).

The annotation process has three stages (Mo et al., 14 Aug 2025). First, candidate pair selection automatically finds all ScanQA question pairs satisfying the overlap and non-subset constraints on object sets. Second, LLM-guided composition uses Claude-3.5-Sonnet with system-level instructions to weave each pair into a single unambiguous question and answer. Third, manual verification assesses synthesis quality by randomly sampling 100 synthesized pairs, of which $\langle$9 were judged correct by human annotators (Mo et al., 14 Aug 2025).

The resulting question types, as reported in the paper, include spatial relationship, counting, attribute comparison, and composite multi-view queries (Mo et al., 14 Aug 2025). This typology indicates that MV-ScanQA is not restricted to one reasoning primitive; rather, it mixes relational, numerical, comparative, and compositional reasoning under a multi-view observability constraint.

4. Evaluation protocol and benchmark interpretation

Evaluation follows ScanQA by using Exact Match (EM) between the model’s string answer and the ground-truth answer (Mo et al., 14 Aug 2025). The train/validation/test partition inherits the ScanNet splits and ensures no scene overlap (Mo et al., 14 Aug 2025). Reported baselines are BridgeQA, LL3DA, LEO, and ChatScene, all evaluated against LEGO under the same benchmark definition (Mo et al., 14 Aug 2025).

On MV-ScanQA, the paper reports the following EM results (Mo et al., 14 Aug 2025):

Method	All	$\rangle$0
BridgeQA	24.6	25.6
LL3DA	19.6	23.3
LEO	24.2	20.9
ChatScene	23.3	20.9
LEGO (1 view)	30.0	23.3
LEGO (4 views)	34.1	30.2

The paper emphasizes that a simple multi-view extension of LEGO yields $\rangle$1 percentage points overall and $\rangle$2 points on $\rangle$3 questions relative to the one-view LEGO setting (Mo et al., 14 Aug 2025). The stratification by required view count is significant because it reveals that overall EM alone would understate the importance of viewpoint integration. Performance gaps widen on the more demanding subsets, which suggests that MV-ScanQA is sensitive to the intended reasoning variable rather than merely to generic language modeling quality.

An important interpretive point is that the benchmark does not equate multi-view reasoning with arbitrary frame accumulation. Its minimum-view analysis ties difficulty to witnessed object coverage under the IoSA criterion (Mo et al., 14 Aug 2025). This means that the benchmark operationalizes multi-view necessity at the level of object observability, not simply input length.

5. TripAlign pre-training corpus

TripAlign is a large-scale pre-training dataset of more than 1 million triplets of the form $\rangle$42D view, set of 3D objects, instruction text$\rangle$5 (Mo et al., 14 Aug 2025). Its purpose is to provide 2D-3D-language alignment signals that are richer than prior single-object annotations and directly compatible with transfer from pre-trained 2D VL models (Mo et al., 14 Aug 2025).

The corpus is built through two pipelines (Mo et al., 14 Aug 2025). The first captions egocentric ScanNet frames: egocentric views are sampled, captions are generated via a frozen 2D VL model such as BLIP-2 or GPT-4, high-quality pairs are filtered by a CLIP-style image-text retriever, and the visible 3D objects in the frame are recorded using the IoSA $\rangle$6 visibility rule. The second extends existing 3D VL datasets: for each instruction in ScanQA, ScanRefer, or Nr3D, the method retrieves the single best-matching view either by text-image similarity using BLIP or by maximal visibility for the referent object, then forms triplets by grouping all objects above the visibility threshold in that frame (Mo et al., 14 Aug 2025).

The paper characterizes TripAlign’s advantages in two ways (Mo et al., 14 Aug 2025). First, it provides richer multi-object annotation than single-object datasets such as ScanRefer and Scan2Cap. Second, it yields true 2D-3D-text alignment, which enables cross-modal transfer from 2D VL expert models. A plausible implication is that TripAlign functions both as supervision for view-grounded object grouping and as an interface layer between 2D LVLM priors and 3D scene representations.

The pre-training objective is standard auto-regressive generation over the triplets, using the same loss as LEGO (Mo et al., 14 Aug 2025). In the paper’s framing, this forces the model to ground text simultaneously in 2D appearance and 3D structure rather than in either modality alone.

6. LEGO architecture and training

LEGO is the baseline method proposed for the multi-view reasoning challenge, and it is explicitly designed to transfer knowledge from pre-trained 2D LVLMs to the 3D domain with TripAlign (Mo et al., 14 Aug 2025). The architecture combines a 3D encoder $\rangle$7, which extracts $\rangle$8 object-level features $\rangle$9 from a point cloud $93\%$0, with a 2D encoder $93\%$1, which extracts $93\%$2 patch features $93\%$3 from selected egocentric views (Mo et al., 14 Aug 2025).

A key component is view-dependent multi-object alignment. The method filters 3D proposals so that only objects satisfying

$93\%$4

are retained and aligned with the caption (Mo et al., 14 Aug 2025). This ties the 3D object set to the actual 2D evidence present in a given view, preventing alignment from drifting toward scene-level but view-invisible content.

The multimodal prefix is constructed as

$93\%$5

Training then uses the auto-regressive loss

$93\%$6

where $93\%$7 is the tokenized instruction-plus-answer sequence and $93\%$8 is the answer start (Mo et al., 14 Aug 2025). Efficient fine-tuning is performed with LoRA adapters (Mo et al., 14 Aug 2025).

The ablation results reported in the paper clarify the role of these design choices. Removing the IoSA-based object filter reduces performance by $93\%$9 points on ScanRefer and $68\%$0 points on ScanQA, indicating that view-dependent alignment contributes substantially to dense captioning and measurably to QA (Mo et al., 14 Aug 2025). A separate data-pipeline ablation reports a progression from from-scratch 3D VL pre-training to egocentric view extension and then to full TripAlign, with scores improving from $68\%$1 to $68\%$2 and then $68\%$3 on ScanRefer, Nr3D, and ScanQA EM respectively (Mo et al., 14 Aug 2025). This supports the claim that TripAlign’s tri-modal supervision, rather than simple scale alone, is central to the observed gains.

7. Reported results, impact, and stated future directions

Beyond MV-ScanQA itself, the paper reports transfer gains on 3D dense captioning and conventional 3D question answering (Mo et al., 14 Aug 2025). On ScanRefer and Nr3D under the Scan2Cap protocol, LEGO achieves $68\%$4 CIDEr@$68\%$5 and $68\%$6 on Nr3D@$68\%$7, compared with $68\%$8 and $68\%$9 for Vote2Cap-DETR++, and $7\%$0 for Scene-LLM on ScanRefer (Mo et al., 14 Aug 2025). On ScanQA, the reported results are EM $7\%$1 on validation and EM $7\%$2 on test without object input, with text-similarity $7\%$3 scores of $7\%$4 and $7\%$5, compared with BridgeQA at EM $7\%$6 and $7\%$7, and ChatScene at EM $7\%$8 and $7\%$9 (Mo et al., 14 Aug 2025).

Within the paper’s scope, these results are presented as state of the art on the proposed multi-view QA benchmark, on 3D dense captioning benchmarks, and on ScanQA (Mo et al., 14 Aug 2025). The broader significance lies in the joint formulation of evaluation and pre-training. MV-ScanQA supplies a testbed in which most questions are genuinely multi-view, while TripAlign supplies the dense multi-object supervision that prior 3D VL datasets lacked (Mo et al., 14 Aug 2025). Taken together, they define a research program centered on view-grounded compositional alignment rather than isolated referent prediction.

The paper also specifies several future directions (Mo et al., 14 Aug 2025). These include dynamic view selection policies conditioned on the question, extension to open-ended language generation beyond EM, incorporation of temporal or motion cues for dynamic scenes, and further tri-modal contrastive losses that explicitly align 2D patches, 3D objects, and language tokens. These recommendations indicate that MV-ScanQA is intended not merely as a benchmark release, but as a scaffold for subsequent work on 3D LVLMs that must reason over distributed evidence, structured geometry, and language simultaneously.