---
title: 'MSMU-Bench: Spatial VQA Evaluation'
url: https://www.emergentmind.com/topics/msmu-bench
type: topic
---

# MSMU-Bench: Spatial VQA Evaluation

Searching arXiv for MSMU-Bench and closely related benchmarks to ground the article in the cited papers.
arxiv_search.query({"search_query":"all:\"MSMU-Bench\" OR all:\"Massive Spatial Measuring and Understanding\" OR all:\"SD-VLM\"","start":0,"max_results":10,"sort_by":"submittedDate","sort_order":"descending"})
arxiv_search.search({"query":"MSMU-Bench SD-VLM Massive Spatial Measuring and Understanding", "max_results": 10})
MSMU-Bench is the held-out evaluation benchmark derived from the Massive Spatial Measuring and Understanding (MSMU) dataset introduced in "SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models" [2509.17664]. It is a test-only benchmark sampled from unseen scans and consists of about 1K spatial VQA pairs designed to assess whether a vision-language model can recover metric spatial quantities and discrete spatial relations from 2D images plus depth. In contrast to benchmarks centered on generic visual semantics, MSMU-Bench targets quantitative 3D spatial reasoning with precise, metric-accurate ground truth grounded in real indoor scenes, and it was introduced specifically to address small data volume, noisy labels from model-driven pipelines, and incomplete task coverage in prior spatial benchmarks [2509.17664].

## 1. Definition, scope, and benchmark rationale

MSMU-Bench evaluates spatial measuring and understanding rather than only semantic scene recognition. Its central object of study is a model’s ability to infer physical size, distance, location, and relative spatial relations that depend on 3D structure. The benchmark is not an unrelated corpus; it is an evaluation subset of MSMU built from unseen scans, and it explicitly tests generalization beyond the training portion of the dataset [2509.17664].

The parent MSMU dataset is large-scale and metrically grounded. It is constructed from real, metrically calibrated 3D indoor scenes from ScanNet and ScanNet++, and comprises 2K scenes, 25K images, 75K objects, 700K QA pairs, 2.5M physical numerical annotations, and 10K chain-of-thought augmented samples. MSMU-Bench inherits this construction logic but is reserved for evaluation, making it a stress test for quantitative spatial VQA rather than a mixed train-test resource [2509.17664].

A defining feature of the benchmark is that it treats spatial reasoning as a measurement problem. All physical quantities are expressed in meters, 2D coordinates are in pixels, and labels are derived from 3D scene structure rather than only from textual annotation. This emphasis on metrically accurate supervision distinguishes MSMU-Bench from benchmarks whose spatial labels are weakly specified or inferred through heuristic pipelines [2509.17664].

## 2. Dataset construction and annotation pipeline

MSMU-Bench uses RGB images paired with depth maps and links each image to an underlying 3D scene graph containing objects, 3D bounding boxes, centroids, and dimensions in meters. Camera intrinsics are not directly given to the VLM at inference; instead, the dataset provides metrically accurate labels computed in the 3D scene frame and transferred to 2D via rasterization [2509.17664].

The construction pipeline proceeds through several stages. First, a scene graph is built from the 3D point cloud, with per-object 3D bounding boxes and physical size attributes. Second, 3D instances are rasterized to 2D masks so that image regions can be aligned with object-level spatial labels. Third, images and objects are filtered by visibility, usefulness, and semantic disambiguation: occluded, truncated, too-small, or non-functional elements such as walls and ceilings are omitted, and similar instances are relabeled via Qwen2.5-VL to expressions such as “the white table” or “the wooden table.” Fourth, QA pairs are generated with human-verified templates for each task type, and a subset is augmented with chain-of-thought rationales using Qwen2.5-VL and filtered by DeepSeek-V3 for factuality and coherence [2509.17664].

When 3D points are needed, the dataset follows the standard depth-to-3D mapping
$$
\mathbf{P} = z\,\mathbf{K}^{-1}\mathbf{p}
= \left(\frac{(u-c_x)z}{f_x},\, \frac{(v-c_y)z}{f_y},\, z\right),
$$
where $\mathbf{p} = [u, v, 1]^\top$ is an image-plane pixel, $z$ is depth, and $\mathbf{K}$ is the camera intrinsic matrix. Euclidean distances are then computed as
$$
d = \|\mathbf{P}_1 - \mathbf{P}_2\|_2.
$$
These equations are not merely background notation; they formalize the metric grounding that underlies MSMU-Bench labels [2509.17664].

## 3. Task taxonomy

MSMU-Bench spans eight task categories covering both discrete and numeric spatial reasoning. The categories are designed to probe hallucination resistance, cardinality, metric estimation, grounding, relative relations, and reference-based inference [2509.17664].

| Category | Core target | Output form |
|---|---|---|
| Existence | Detect the absence of a queried object | Binary discrete QA |
| Object counting | Count instances of a class | Numeric integer |
| Scale estimation | Height, width, or size of a target object | Numeric regression in meters |
| Grounding | Return a 2D coordinate or identify an object at a pixel | Coordinates or discrete label |
| Relative position | Left/right, closer, higher/lower | Discrete QA |
| Absolute distance | Pairwise distance between objects | Numeric regression in meters |
| Scale comparison | Bigger/smaller, tallest/lowest | Discrete QA |
| Reference object estimation | Infer target measurements from a known reference measurement | Numeric regression |

The benchmark therefore mixes symbolic and quantitative demands. Existence and relative position emphasize categorical judgment, while scale estimation, absolute distance, grounding, and reference object estimation require the model to output metric values or coordinates with usable numerical precision. The reference object estimation category is particularly diagnostic because it asks the model to combine visual evidence with a supplied measurement constraint, often through chain-of-thought-style reasoning [2509.17664].

Typical items illustrate this structure. A scale estimation question may ask, “What is the size of the dark pillow?” with a ground truth such as “0.8 m × 0.63 m × 0.55 m.” A reference-object question may specify, “The height of the bed is 0.81 m, what is the height of the table and nightstand?” with gold answers “1.02 m and 0.93 m.” An absolute-distance question asks for the distance between two designated objects, with success contingent on numerical accuracy relative to 3D ground truth [2509.17664].

## 4. Evaluation protocol and scoring

MSMU-Bench is evaluated with GPT-4 scoring under separate protocols for qualitative and quantitative tasks. For qualitative tasks such as relative positions, existence, and comparisons, GPT-4 assigns a score between 0 and 1, and benchmark success is the average of these scores. For quantitative tasks such as scales, distances, grounding coordinates, and counts, numerical values are extracted from the model output and assessed against a tolerance criterion [2509.17664].

The key quantitative success criterion is the multiplicative error band
$$
\delta = \max\left(\frac{\hat{d}}{d^*},\, \frac{d^*}{\hat{d}}\right),
\qquad \text{success if } \delta \le 1.25,
$$
where $\hat{d}$ is the model estimate and $d^*$ is the ground truth. The paper states that 1.25 is the threshold used in the MSMU-Bench implementation. When unit conversion is needed, the evaluator applies fixed rules: 1 inch = 0.0254 m, 1 foot = 0.3048 m, and 1 cm = 0.01 m [2509.17664].

For discrete tasks, accuracy can be written as
$$
\mathrm{Acc} = \frac{\#\text{correct}}{\#\text{total}},
$$
and a generic relative error definition is
$$
\mathrm{RE} = \frac{|\hat{y} - y|}{|y|}.
$$
The benchmark’s reported numbers are success rates aggregated across categories rather than a single undifferentiated metric. This is important because MSMU-Bench combines binary judgment, counting, coordinate regression, and metric estimation within one evaluation suite [2509.17664].

A practical implication is that the benchmark rewards usable quantitative calibration rather than approximate verbal plausibility. A response that names the right object category but outputs the wrong scale, distance, or coordinate can fail even if its description appears qualitatively sensible.

## 5. Depth-aware modeling in SD-VLM

MSMU-Bench was introduced together with SD-VLM, a model built on LLaVA-1.5-7B with a CLIP ViT/14 vision encoder and a depth positional encoding (DPE) mechanism [2509.17664]. The benchmark is therefore closely tied to a concrete architectural hypothesis: that adding depth priors to visual tokens improves quantitative spatial reasoning.

Let $D \in \mathbb{R}^{H \times W \times 1}$ be the depth map and $E^{\mathrm{image}} \in \mathbb{R}^{H' \times W' \times d}$ the CLIP feature map. The depth map is patch-pooled to $D' \in \mathbb{R}^{H' \times W' \times 1}$, and DPE applies channel-wise sinusoidal embeddings:
$$
E^{\mathrm{depth}}(i,j,2t) = \sin\!\Big(\frac{D'(i,j)}{10000^{2t/d}}\Big), \qquad
E^{\mathrm{depth}}(i,j,2t+1) = \cos\!\Big(\frac{D'(i,j)}{10000^{2t/d}}\Big).
$$
The resulting depth-aware representation is formed additively:
$$
E^{\mathrm{vision}} = E^{\mathrm{image}} + E^{\mathrm{depth}}.
$$
These features are then flattened and passed to the language model [2509.17664].

When ground-truth depth is unavailable, SD-VLM uses an external estimator, specifically Depth-Anything-V2; UniDepth is also tested. Training follows visual instruction tuning with LoRA on one epoch of MSMU using 8×V100 GPUs, batch size 2 per GPU, and 32 GPU-hours. The CLIP ViT/14 vision encoder is frozen, the learning rate is $2\times10^{-4}$ for the LLM and $2\times10^{-5}$ for the projector, and the training objective is next-token cross-entropy:
$$
\mathcal{L}_{\mathrm{CE}} =
-\frac{1}{T}\sum_{t=1}^{T}
\log p\big(w_t \,\big|\, w_{<t},\, E^{\mathrm{vision}}\big).
$$
Although these are model details rather than benchmark definitions, they are central to the empirical story of MSMU-Bench because the benchmark was designed to reveal whether explicit depth integration improves measurable spatial competence [2509.17664].

## 6. Empirical results, ablations, and observed failure modes

On MSMU-Bench, SD-VLM achieves an average success rate of 56.31%, and the MSMU-CoT variant reaches 59.19% [2509.17664]. The per-category success rates for SD-VLM are 87.23 on existence, 47.92 on object counting, 51.35 on scale estimation, 42.53 on grounding, 75.16 on relative position, 40.00 on absolute distance, 55.66 on scale comparison, and 46.07 on reference object estimation. With MSMU-CoT augmentation, the reported values are 87.23, 42.71, 51.74, 49.43, 73.29, 50.00, 69.81, and 49.32, respectively [2509.17664].

Compared to baseline models, the reported benchmark averages are 32.28 for GPT-4o, 33.63 for Intern-VL3-78B, 35.17 for Gemini-2, 30.82 for Qwen2.5-VL-72B, 27.59 for Qwen2.5-VL-32B, 3.19 for Qwen2.5-VL-7B, 19.45 for LLaVA-1.5-7B, 28.98 for SpatialRGPT, and 29.17 for SpatialBot. The paper highlights that SD-VLM outperforms GPT-4o and Intern-VL3-78B by 26.91% and 25.56%, respectively, on MSMU-Bench [2509.17664].

Ablations on depth integration show that the benchmark is sensitive to how depth is incorporated. The no-depth baseline achieves 46.73%, depth as image 22.64%, depth as prompt 48.78%, depth as token 35.72%, DPE with estimated depth 55.35%, DPE-learnable 56.18%, and the proposed DPE-sincos 56.31% [2509.17664]. These results support the paper’s claim that a lightweight additive encoding is more effective than concatenation, extra tokens, or textualization for this benchmark.

MSMU-Bench also serves as a basis for robustness analysis. Adding zero-mean Gaussian noise to normalized depth with standard deviations 0.1, 0.3, 0.5, and 0.7 reduces performance from 56.3% to 55.1%, 54.0%, 53.3%, and 51.4%, respectively, still above the no-depth baseline of 46.7%. Replacing Depth-Anything-V2 with UniDepth yields nearly identical accuracy, 56.2% versus 56.3% [2509.17664]. This suggests that the benchmark is responsive to depth quality but not narrowly overfit to a single depth estimator.

The paper identifies several recurring failure modes among baselines: numeric hallucination, unit mishandling, over-reliance on 2D cues while ignoring depth, and poor exploitation of reference-object constraints. The strong result on the existence category, where SD-VLM obtains 87.23%, is presented as evidence that the benchmark’s anti-hallucination design is meaningful rather than cosmetic [2509.17664].

## 7. Relation to prior work and naming ambiguities

MSMU-Bench was created in response to limitations in earlier spatial benchmarks, specifically small data volume, noisy labels from model-driven pipelines, and incomplete task coverage [2509.17664]. Its design combines precise 3D-to-2D annotation transfer, quantitative task diversity, and explicit evaluation of both discrete relations and metric estimates. In that sense, it occupies a distinct position within spatial VQA: it is neither a purely semantic benchmark nor a generic multimodal reasoning set.

The name is easily confused with two speech-language benchmarks that appear in contemporaneous literature. "MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark" introduces MMSU and its evaluation suite MMSU_Bench for spoken language understanding, comprising 5,000 audio-question-answer triplets across 47 tasks and focusing on phonetics, prosody, rhetoric, syntactics, semantics, and paralinguistics rather than visual spatial reasoning [2506.04779]. Separately, "MSU-Bench: Towards Understanding the Conversational Multi-talker Scenarios" explicitly states that the formal benchmark name is MSU-Bench, not “MSMU-Bench,” and defines a speaker-centric benchmark for multi-speaker conversational understanding with four progressive tiers [2508.08155].

These distinctions matter because the three benchmarks address different modalities and evaluation regimes. MMSU_Bench is MCQ-only and speech-centric, MSU-Bench is open-ended and multi-speaker conversational, and MSMU-Bench is test-only spatial VQA grounded in RGB-plus-depth perception and metric 3D reasoning. The superficial similarity of their names can obscure substantive differences in task design, annotation pipeline, and scoring methodology.

Within the spatial domain itself, MSMU-Bench also functions as a cross-benchmark reference point. SD-VLM, trained on MSMU, generalizes to Q-Spatial++ with 56.2% and to SpatialRGPT-Bench with 33.3% quantitative and 65.5% qualitative scores, outperforming several proprietary and open-source baselines there as well [2509.17664]. A plausible implication is that the benchmark is not merely measuring dataset-specific memorization, but is at least partially aligned with broader spatial reasoning competence.

Source: https://www.emergentmind.com/topics/msmu-bench