MSMU-Bench: Spatial VQA Evaluation

• MSMU-Bench is a test-only evaluation benchmark for spatial VQA derived from the MSMU dataset, emphasizing metric 3D spatial reasoning with precise RGB and depth data.
• It uses a systematic 3D-to-2D annotation pipeline to generate spatial QA pairs across eight tasks, including object counting, scale estimation, and relative positioning.
• Empirical results demonstrate that depth-encoded models like SD-VLM outperform baselines, validating the benchmark’s design for addressing previous spatial reasoning challenges.

Searching arXiv for MSMU-Bench and closely related benchmarks to ground the article in the cited papers. arxiv_search.query({"^^^^^^^^4search_query4 OR ^^^^^^^^4all:\4 Spatial Measuring and Understanding\" OR ^^^^^^^^4all:\4 arxiv_search.search({"query":"MSMU-Bench SD-VLM Massive Spatial Measuring and Understanding", "max_results": ^^^^4all:\4search_query4^^^^}) MSMU-Bench is the held-out evaluation benchmark derived from the Massive Spatial Measuring and Understanding (MSMU) dataset introduced in "SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-LLMs" (&&&^^^^4search_query4^^^^&&&). It is a test-only benchmark sampled from unseen scans and consists of about ^^^^4all:\4^^^^K spatial VQA pairs designed to assess whether a vision-LLM can recover metric spatial quantities and discrete spatial relations from ^^^^4 OR all:\4^^^^D images plus depth. In contrast to benchmarks centered on generic visual semantics, MSMU-Bench targets quantitative ^^^^4 OR all:\4^^^^D spatial reasoning with precise, metric-accurate ground truth grounded in real indoor scenes, and it was introduced specifically to address small data volume, noisy labels from model-driven pipelines, and incomplete task coverage in prior spatial benchmarks (&&&^^^^4search_query4^^^^&&&).

^^^^4all:\4^^^^. Definition, scope, and benchmark rationale

MSMU-Bench evaluates spatial measuring and understanding rather than only semantic scene recognition. Its central object of study is a model’s ability to infer physical size, distance, location, and relative spatial relations that depend on ^^^^4 OR all:\4^^^^D structure. The benchmark is not an unrelated corpus; it is an evaluation subset of MSMU built from unseen scans, and it explicitly tests generalization beyond the training portion of the dataset (&&&^^^^4search_query4^^^^&&&).

The parent MSMU dataset is large-scale and metrically grounded. It is constructed from real, metrically calibrated ^^^^4 OR all:\4^^^^D indoor scenes from ScanNet and ScanNet++, and comprises ^^^^4 OR all:\4^^^^K scenes, ^^^^4 OR all:\4^^^^5K images, 75K objects, 7^^^^4search_query4search_query4^^^^K QA pairs, ^^^^4 OR all:\4^^^^.5M physical numerical annotations, and ^^^^4all:\4search_query4^^^^K chain-of-thought augmented samples. MSMU-Bench inherits this construction logic but is reserved for evaluation, making it a stress test for quantitative spatial VQA rather than a mixed train-test resource (&&&^^^^4search_query4^^^^&&&).

A defining feature of the benchmark is that it treats spatial reasoning as a measurement problem. All physical quantities are expressed in meters, ^^^^4 OR all:\4^^^^D coordinates are in pixels, and labels are derived from ^^^^4 OR all:\4^^^^D scene structure rather than only from textual annotation. This emphasis on metrically accurate supervision distinguishes MSMU-Bench from benchmarks whose spatial labels are weakly specified or inferred through heuristic pipelines (&&&^^^^4search_query4^^^^&&&).

^^^^4 OR all:\4^^^^. Dataset construction and annotation pipeline

MSMU-Bench uses RGB images paired with depth maps and links each image to an underlying ^^^^4 OR all:\4^^^^D scene graph containing objects, ^^^^4 OR all:\4^^^^D bounding boxes, centroids, and dimensions in meters. Camera intrinsics are not directly given to the VLM at inference; instead, the dataset provides metrically accurate labels computed in the ^^^^4 OR all:\4^^^^D scene frame and transferred to ^^^^4 OR all:\4^^^^D via rasterization (&&&^^^^4search_query4^^^^&&&).

The construction pipeline proceeds through several stages. First, a scene graph is built from the ^^^^4 OR all:\4^^^^D point cloud, with per-object ^^^^4 OR all:\4^^^^D bounding boxes and physical size attributes. Second, ^^^^4 OR all:\4^^^^D instances are rasterized to ^^^^4 OR all:\4^^^^D masks so that image regions can be aligned with object-level spatial labels. Third, images and objects are filtered by visibility, usefulness, and semantic disambiguation: occluded, truncated, too-small, or non-functional elements such as walls and ceilings are omitted, and similar instances are relabeled via Qwen^^^^4 OR all:\4^^^^.5-VL to expressions such as “the white table” or “the wooden table.” Fourth, QA pairs are generated with human-verified templates for each task type, and a subset is augmented with chain-of-thought rationales using Qwen^^^^4 OR all:\4^^^^.5-VL and filtered by DeepSeek-V^^^^4 OR all:\4^^^^ for factuality and coherence (&&&^^^^4search_query4^^^^&&&).

When ^^^^4 OR all:\4^^^^D points are needed, the dataset follows the standard depth-to-^^^^4 OR all:\4^^^^D mapping

PRESERVED_PLACEHOLDER_^^^^4search_query4^^^^

where PRESERVED_PLACEHOLDER_^^^^4all:\4^^^^ is an image-plane pixel, PRESERVED_PLACEHOLDER_^^^^4 OR all:\4^^^^ is depth, and PRESERVED_PLACEHOLDER_^^^^4 OR all:\4^^^^ is the camera intrinsic matrix. Euclidean distances are then computed as

$d = \|\mathbf{P}_1 - \mathbf{P}_2\|_2.$

These equations are not merely background notation; they formalize the metric grounding that underlies MSMU-Bench labels (&&&^^^^4search_query4^^^^&&&).

^^^^4 OR all:\4^^^^. Task taxonomy

MSMU-Bench spans eight task categories covering both discrete and numeric spatial reasoning. The categories are designed to probe hallucination resistance, cardinality, metric estimation, grounding, relative relations, and reference-based inference (&&&^^^^4search_query4^^^^&&&).

Category	Core target	Output form
Existence	Detect the absence of a queried object	Binary discrete QA
Object counting	Count instances of a class	Numeric integer
Scale estimation	Height, width, or size of a target object	Numeric regression in meters
Grounding	Return a ^^^^4 OR all:\4^^^^D coordinate or identify an object at a pixel	Coordinates or discrete label
Relative position	Left/right, closer, higher/lower	Discrete QA
Absolute distance	Pairwise distance between objects	Numeric regression in meters
Scale comparison	Bigger/smaller, tallest/lowest	Discrete QA
Reference object estimation	Infer target measurements from a known reference measurement	Numeric regression

The benchmark therefore mixes symbolic and quantitative demands. Existence and relative position emphasize categorical judgment, while scale estimation, absolute distance, grounding, and reference object estimation require the model to output metric values or coordinates with usable numerical precision. The reference object estimation category is particularly diagnostic because it asks the model to combine visual evidence with a supplied measurement constraint, often through chain-of-thought-style reasoning (&&&^^^^4search_query4^^^^&&&).

Typical items illustrate this structure. A scale estimation question may ask, “What is the size of the dark pillow?” with a ground truth such as “^^^^4search_query4^^^^.8 m × ^^^^4search_query4^^^^.6^^^^4 OR all:\4^^^^ m × ^^^^4search_query4^^^^.55 m.” A reference-object question may specify, “The height of the bed is ^^^^4search_query4^^^^.8^^^^4all:\4^^^^ m, what is the height of the table and nightstand?” with gold answers “^^^^4all:\4^^^^.^^^^4search_query4 OR all:\4^^^^ m and ^^^^4search_query4^^^^.9^^^^4 OR all:\4^^^^ m.” An absolute-distance question asks for the distance between two designated objects, with success contingent on numerical accuracy relative to ^^^^4 OR all:\4^^^^D ground truth (&&&^^^^4search_query4^^^^&&&).

4. Evaluation protocol and scoring

MSMU-Bench is evaluated with GPT-4 scoring under separate protocols for qualitative and quantitative tasks. For qualitative tasks such as relative positions, existence, and comparisons, GPT-4 assigns a score between ^^^^4search_query4^^^^ and ^^^^4all:\4^^^^, and benchmark success is the average of these scores. For quantitative tasks such as scales, distances, grounding coordinates, and counts, numerical values are extracted from the model output and assessed against a tolerance criterion (&&&^^^^4search_query4^^^^&&&).

The key quantitative success criterion is the multiplicative error band

$$\delta = \max\left(\frac{\hat{d}}{d^*},\, \frac{d^*}{\hat{d}}\right), \qquad \text{success if } \delta \le 1.25,$$

where $\hat{d}$ is the model estimate and $d^*$ is the ground truth. The paper states that ^^^^4all:\4^^^^.^^^^4 OR all:\4^^^^5 is the threshold used in the MSMU-Bench implementation. When unit conversion is needed, the evaluator applies fixed rules: ^^^^4all:\4^^^^ inch = ^^^^4search_query4^^^^.^^^^4search_query4 OR all:\4^^^^54 m, ^^^^4all:\4^^^^ foot = ^^^^4search_query4^^^^.^^^^4 OR all:\4search_query44^^^^8 m, and ^^^^4all:\4^^^^ cm = ^^^^4search_query4^^^^.^^^^4search_query4all:\4^^^^ m (&&&^^^^4search_query4^^^^&&&).

For discrete tasks, accuracy can be written as

$$\mathrm{Acc} = \frac{\#\text{correct}}{\#\text{total}},$$

and a generic relative error definition is

$\mathrm{RE} = \frac{|\hat{y} - y|}{|y|}.$

The benchmark’s reported numbers are success rates aggregated across categories rather than a single undifferentiated metric. This is important because MSMU-Bench combines binary judgment, counting, coordinate regression, and metric estimation within one evaluation suite (&&&^^^^4search_query4^^^^&&&).

A practical implication is that the benchmark rewards usable quantitative calibration rather than approximate verbal plausibility. A response that names the right object category but outputs the wrong scale, distance, or coordinate can fail even if its description appears qualitatively sensible.

5. Depth-aware modeling in SD-VLM

MSMU-Bench was introduced together with SD-VLM, a model built on LLaVA-^^^^4all:\4^^^^.5-7B with a CLIP ViT/^^^^4all:\44^^^^ vision encoder and a depth positional encoding (DPE) mechanism (&&&^^^^4search_query4^^^^&&&). The benchmark is therefore closely tied to a concrete architectural hypothesis: that adding depth priors to visual tokens improves quantitative spatial reasoning.

Let PRESERVED_PLACEHOLDER_^^^^4all:\4search_query4^^^^ be the depth map and PRESERVED_PLACEHOLDER_^^^^4all:\4all:\4^^^^ the CLIP feature map. The depth map is patch-pooled to PRESERVED_PLACEHOLDER_^^^^4all:\4 OR all:\4^^^^, and DPE applies channel-wise sinusoidal embeddings:

PRESERVED_PLACEHOLDER_^^^^4all:\4 OR all:\4^^^^

The resulting depth-aware representation is formed additively:

PRESERVED_PLACEHOLDER_^^^^4all:\44^^^^

These features are then flattened and passed to the LLM (&&&^^^^4search_query4^^^^&&&).

When ground-truth depth is unavailable, SD-VLM uses an external estimator, specifically Depth-Anything-V^^^^4 OR all:\4^^^^; UniDepth is also tested. Training follows visual instruction tuning with LoRA on one epoch of MSMU using 8×V^^^^4all:\4search_query4search_query4^^^^ GPUs, batch size ^^^^4 OR all:\4^^^^ per GPU, and ^^^^4 OR all:\4 OR all:\4^^^^ GPU-hours. The CLIP ViT/^^^^4all:\44^^^^ vision encoder is frozen, the learning rate is PRESERVED_PLACEHOLDER_^^^^4all:\4^^^^5 for the LLM and PRESERVED_PLACEHOLDER_^^^^4all:\4^^^^6 for the projector, and the training objective is next-token cross-entropy:

PRESERVED_PLACEHOLDER_^^^^4all:\4^^^^7

Although these are model details rather than benchmark definitions, they are central to the empirical story of MSMU-Bench because the benchmark was designed to reveal whether explicit depth integration improves measurable spatial competence (&&&^^^^4search_query4^^^^&&&).

6. Empirical results, ablations, and observed failure modes

On MSMU-Bench, SD-VLM achieves an average success rate of 56.^^^^4 OR all:\4all:\4^^^^%, and the MSMU-CoT variant reaches 59.^^^^4all:\4^^^^9% (&&&^^^^4search_query4^^^^&&&). The per-category success rates for SD-VLM are 87.^^^^4 OR all:\4 OR all:\4^^^^ on existence, 47.9^^^^4 OR all:\4^^^^ on object counting, 5^^^^4all:\4^^^^.^^^^4 OR all:\4^^^^5 on scale estimation, 4^^^^4 OR all:\4^^^^.5^^^^4 OR all:\4^^^^ on grounding, 75.^^^^4all:\4^^^^6 on relative position, 4^^^^4search_query4^^^^.^^^^4search_query4search_query4^^^^ on absolute distance, 55.66 on scale comparison, and 46.^^^^4search_query4^^^^7 on reference object estimation. With MSMU-CoT augmentation, the reported values are 87.^^^^4 OR all:\4 OR all:\4^^^^, 4^^^^4 OR all:\4^^^^.7^^^^4all:\4^^^^, 5^^^^4all:\4^^^^.74, 49.4^^^^4 OR all:\4^^^^, 7^^^^4 OR all:\4^^^^.^^^^4 OR all:\4^^^^9, 5^^^^4search_query4^^^^.^^^^4search_query4search_query4 69.8^^^^4all:\4^^^^, and 49.^^^^4 OR all:\4 OR all:\4^^^^, respectively (&&&^^^^4search_query4^^^^&&&).

Compared to baseline models, the reported benchmark averages are ^^^^4 OR all:\4 OR all:\4^^^^.^^^^4 OR all:\4^^^^8 for GPT-4o, ^^^^4 OR all:\4 OR all:\4^^^^.6^^^^4 OR all:\4^^^^ for Intern-VL^^^^4 OR all:\4^^^^-78B, ^^^^4 OR all:\4^^^^5.^^^^4all:\4^^^^7 for Gemini-^^^^4 OR all:\4^^^^, ^^^^4 OR all:\4search_query4^^^^.8^^^^4 OR all:\4^^^^ for Qwen^^^^4 OR all:\4^^^^.5-VL-7^^^^4 OR all:\4^^^^B, ^^^^4 OR all:\4^^^^7.59 for Qwen^^^^4 OR all:\4^^^^.5-VL-^^^^4 OR all:\4 OR all:\4^^^^B, ^^^^4 OR all:\4^^^^.^^^^4all:\4^^^^9 for Qwen^^^^4 OR all:\4^^^^.5-VL-7B, ^^^^4all:\4^^^^9.45 for LLaVA-^^^^4all:\4^^^^.5-7B, ^^^^4 OR all:\4^^^^8.98 for SpatialRGPT, and ^^^^4 OR all:\4^^^^9.^^^^4all:\4^^^^7 for SpatialBot. The paper highlights that SD-VLM outperforms GPT-4o and Intern-VL^^^^4 OR all:\4^^^^-78B by ^^^^4 OR all:\4^^^^6.9^^^^4all:\4^^^^% and ^^^^4 OR all:\4^^^^5.56%, respectively, on MSMU-Bench (&&&^^^^4search_query4^^^^&&&).

Ablations on depth integration show that the benchmark is sensitive to how depth is incorporated. The no-depth baseline achieves 46.7^^^^4 OR all:\4^^^^%, depth as image ^^^^4 OR all:\4 OR all:\4^^^^.64%, depth as prompt 48.78%, depth as token ^^^^4 OR all:\4^^^^5.7^^^^4 OR all:\4^^^^%, DPE with estimated depth 55.^^^^4 OR all:\4^^^^5%, DPE-learnable 56.^^^^4all:\4^^^^8%, and the proposed DPE-sincos 56.^^^^4 OR all:\4all:\4^^^^% (&&&^^^^4search_query4^^^^&&&). These results support the paper’s claim that a lightweight additive encoding is more effective than concatenation, extra tokens, or textualization for this benchmark.

MSMU-Bench also serves as a basis for robustness analysis. Adding zero-mean Gaussian noise to normalized depth with standard deviations ^^^^4search_query4^^^^.^^^^4all:\4 ^^^^4search_query4^^^^.^^^^4 OR all:\4^^^^, ^^^^4search_query4^^^^.5, and ^^^^4search_query4^^^^.7 reduces performance from 56.^^^^4 OR all:\4^^^^% to 55.^^^^4all:\4^^^^%, 54.^^^^4search_query4^^^^%, 5^^^^4 OR all:\4^^^^.^^^^4 OR all:\4^^^^%, and 5^^^^4all:\4^^^^.4%, respectively, still above the no-depth baseline of 46.7%. Replacing Depth-Anything-V^^^^4 OR all:\4^^^^ with UniDepth yields nearly identical accuracy, 56.^^^^4 OR all:\4^^^^% versus 56.^^^^4 OR all:\4^^^^% (&&&^^^^4search_query4^^^^&&&). This suggests that the benchmark is responsive to depth quality but not narrowly overfit to a single depth estimator.

The paper identifies several recurring failure modes among baselines: numeric hallucination, unit mishandling, over-reliance on ^^^^4 OR all:\4^^^^D cues while ignoring depth, and poor exploitation of reference-object constraints. The strong result on the existence category, where SD-VLM obtains 87.^^^^4 OR all:\4 OR all:\4^^^^%, is presented as evidence that the benchmark’s anti-hallucination design is meaningful rather than cosmetic (&&&^^^^4search_query4^^^^&&&).

7. Relation to prior work and naming ambiguities

MSMU-Bench was created in response to limitations in earlier spatial benchmarks, specifically small data volume, noisy labels from model-driven pipelines, and incomplete task coverage (&&&^^^^4search_query4^^^^&&&). Its design combines precise ^^^^4 OR all:\4^^^^D-to-^^^^4 OR all:\4^^^^D annotation transfer, quantitative task diversity, and explicit evaluation of both discrete relations and metric estimates. In that sense, it occupies a distinct position within spatial VQA: it is neither a purely semantic benchmark nor a generic multimodal reasoning set.

The name is easily confused with two speech-language benchmarks that appear in contemporaneous literature. "MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark" introduces MMSU and its evaluation suite MMSU_Bench for spoken language understanding, comprising 5,^^^^4search_query4search_query4search_query4^^^^ audio-question-answer triplets across 47 tasks and focusing on phonetics, prosody, rhetoric, syntactics, semantics, and paralinguistics rather than visual spatial reasoning (&&&^^^^4 OR all:\44^^^^&&&). Separately, "MSU-Bench: Towards Understanding the Conversational Multi-talker Scenarios" explicitly states that the formal benchmark name is MSU-Bench, not “MSMU-Bench,” and defines a speaker-centric benchmark for multi-speaker conversational understanding with four progressive tiers (&&&^^^^4 OR all:\4^^^^5&&&).

These distinctions matter because the three benchmarks address different modalities and evaluation regimes. MMSU_Bench is MCQ-only and speech-centric, MSU-Bench is open-ended and multi-speaker conversational, and MSMU-Bench is test-only spatial VQA grounded in RGB-plus-depth perception and metric ^^^^4 OR all:\4^^^^D reasoning. The superficial similarity of their names can obscure substantive differences in task design, annotation pipeline, and scoring methodology.

Within the spatial domain itself, MSMU-Bench also functions as a cross-benchmark reference point. SD-VLM, trained on MSMU, generalizes to Q-Spatial++ with 56.^^^^4 OR all:\4^^^^% and to SpatialRGPT-Bench with ^^^^4 OR all:\4 OR all:\4^^^^.^^^^4 OR all:\4^^^^% quantitative and 65.5% qualitative scores, outperforming several proprietary and open-source baselines there as well (&&&^^^^4search_query4^^^^&&&). A plausible implication is that the benchmark is not merely measuring dataset-specific memorization, but is at least partially aligned with broader spatial reasoning competence.