Massive Spatial Measuring and Understanding
- MSMU is a research program that acquires quantitative spatial measurements and structured 3D understanding from visual data.
- It integrates metric grounding, structural analysis, and operational reasoning to support tasks from VQA to dynamic planning.
- Benchmarking in MSMU leverages diverse datasets and model architectures to assess spatial intelligence across multiple scales and modalities.
Searching arXiv for the cited MSMU-related papers and benchmarks to ground the article. Massive Spatial Measuring and Understanding (MSMU) denotes a line of research on spatial intelligence centered on the joint acquisition of quantitative spatial measurements and structured spatial understanding from visual data. In the narrow sense introduced by "SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-LLMs" (Chen et al., 22 Sep 2025), MSMU is the name of a large-scale, physically grounded spatial VQA dataset and its held-out benchmark. In the broader sense that emerges across later work, MSMU refers to a research program that combines metric estimation, spatial relations, grounding, cross-view consistency, planning, and temporal or 4D reasoning from RGB, RGB-D, multi-image, or video inputs. This broader usage is explicit in work that argues spatial intelligence requires both dense metric scale estimation and holistic 3D structural understanding (Zheng et al., 25 May 2026), extends spatial reasoning from millimeters to kilometers (Sun et al., 10 Oct 2025), or treats dynamic 4D cognition as a core benchmark target (Wang et al., 31 Dec 2025). A related domain transfer appears in Materials Spatial Intelligence, which applies spatial-intelligence principles to aligned EBSD and HR-DIC fields for mechanism discovery in structural alloys (Calvat et al., 19 Jun 2026).
1. Origins and conceptual scope
The original MSMU formulation is motivated by a gap between ordinary 2D semantic visual understanding and quantitative 3D reasoning. In "SD-VLM" (Chen et al., 22 Sep 2025), the central claim is that image-only VLMs are weak at estimating physically meaningful quantities such as size, depth, and distance because a 2D image does not preserve the underlying 3D structure. The paper formalizes this with the standard camera equation
making explicit that real-world coordinates require both depth and camera intrinsics. MSMU, in that formulation, is therefore not merely relation classification; it is the attempt to recover measurable 3D structure from visual input.
Subsequent work broadens that definition rather than replacing it. "Dual-Pathway Geometry-Aware MLLM for Spatial Intelligence" argues that robust spatial intelligence from RGB alone requires two complementary competencies: dense, unit-aware metric estimation and sparse, relational 3D structural understanding (Zheng et al., 25 May 2026). "SpaceVista: All-Scale Visual Spatial Reasoning from mm to km" extends the scope from indoor scenes to tiny tabletop, tabletop, indoor, outdoor, and drone-view regimes, making physical scale itself a conditioning variable for spatial reasoning (Sun et al., 10 Oct 2025). "Spatial4D-Bench" extends the scope again by treating motion, state change, route planning, action prediction, and physical plausibility as part of 4D spatial intelligence rather than as auxiliary tasks (Wang et al., 31 Dec 2025).
A useful conceptual distinction emerges from these works. MSMU combines at least three layers. The first is metric grounding, which covers quantities such as object size, room size, distance, and depth. The second is structural understanding, which covers relations among objects, cameras, and regions, together with viewpoint change and scene layout. The third is operational reasoning, which covers counting across views, grounding, planning, prediction, memory, and cross-video or multi-view integration. This suggests that MSMU is best understood not as a single benchmark category but as a family of tasks unified by the requirement that spatial claims be both semantically interpretable and geometrically grounded.
2. Task structure and benchmark landscape
The benchmark ecosystem makes clear that MSMU is not reducible to single-image relation recognition. The original MSMU dataset defines eight categories: Scale Estimation, Grounding, Absolute Distance, Counting, Reference Object Estimation, Existence, Relative Position, and Scale Comparison (Chen et al., 22 Sep 2025). Later benchmarks widen that space in different directions. "MMSI-Bench" operationalizes multi-image spatial intelligence and requires that every question use multiple images by construction (Yang et al., 29 May 2025). "MMSI-Video-Bench" organizes video spatial intelligence into Perception, Planning, Prediction, and Cross-Video Reasoning, while releasing five main categories and 13 subtypes (Lin et al., 11 Dec 2025). "Spatial4D-Bench" further systematizes 18 tasks into six cognitive categories, spanning object understanding through spatiotemporal reasoning (Wang et al., 31 Dec 2025). "SOLIDGEO" adds a mathematically structured variant of MSMU focused on solid geometry, projection, unfolding, spatial measurement, and vector reasoning (Wang et al., 27 May 2025).
The resulting benchmark landscape can be summarized as follows.
| Resource | Scale / modality | Task structure |
|---|---|---|
| MSMU-Bench | About 1K VQA pairs from unseen scans | 8 categories including scale, distance, grounding, counting |
| MMSI-Bench | 1,000 MCQ, avg 2.55 images per question | 11 categories around camera, object, region, motion, multi-step |
| MMSI-Video-Bench | 1,106 questions over 1,278 clips | 5 main categories, 13 subtypes, including planning and cross-video |
| Spatial4D-Bench | 39,305 QA pairs | 18 tasks in 6 cognitive categories |
| SolidGeo | 3,113 multimodal problems | 8 solid-geometry categories |
This landscape corrects a common simplification. A frequent misconception is that spatial intelligence benchmarks mostly test local left-right or above-below judgments. The benchmark record does not support that view. MMSI-Bench requires overlap matching, scene reconstruction, situation transformation, and spatial logic across multiple images (Yang et al., 29 May 2025). MMSI-Video-Bench adds memory update, multi-view integration, navigation, manipulation, and temporal localization (Lin et al., 11 Dec 2025). Spatial4D-Bench makes route planning, egocentric reasoning, state change, and physical plausibility central rather than peripheral (Wang et al., 31 Dec 2025). SolidGeo shows that even diagram-grounded mathematics benchmarks decompose spatial competence into projection, unfolding, composite volumetric reasoning, and 3D coordinate or vector reasoning (Wang et al., 27 May 2025).
A second important pattern is the coexistence of qualitative and quantitative outputs. MSMU-Bench mixes left-right and larger-smaller judgments with explicit numerical estimation of size and distance (Chen et al., 22 Sep 2025). VSI-style and 4D benchmarks evaluate both multiple-choice reasoning and numerical answers, often with tolerance-based scoring for continuous quantities (Wang et al., 31 Dec 2025). This suggests that MSMU evaluation has converged on a hybrid regime in which language, measurement, and structured geometry are all first-class outputs.
3. Data engines, datasets, and supervision pipelines
The data infrastructure behind MSMU has diversified from single datasets into a set of distinct supervision strategies. The original MSMU dataset in "SD-VLM" contains 2K scenes, 25K images, 75K objects, 700K QA pairs, 2.5M numerical values, and 10K chain-of-thought augmented samples, all built from physically scaled 3D scenes from ScanNet and ScanNet++ (Chen et al., 22 Sep 2025). The key design choice is that metric annotations are transferred from physically scaled 3D scene data rather than estimated through a stack of perception models. That choice makes the dataset unusually explicit about metric fidelity.
Later work shifts from one dataset to reusable generation engines. "OpenSpatial" proposes an open-source data engine centered on 3D oriented bounding boxes as the fundamental primitive, with a task hierarchy spanning Spatial Measurement, Spatial Relationship, Camera Perception, Multi-view Consistency, and Scene-Aware Reasoning (Liu et al., 8 Apr 2026). The released OpenSpatial-3M contains 3 million high-fidelity samples. The engine projects scene-level 3D boxes into frame space, applies frustum filtering and occupancy-based occlusion filtering, refines masks with SAM, and skips measurement QA when the metric flag is false. This creates a direct data-engine interpretation of MSMU: large-scale spatial understanding becomes a byproduct of a stable 3D box-centric representation.
"SpaceVista-1M" moves in a different direction by using a specialist-driven automated pipeline over video scenes across five spatial scales, yielding approximately 1M spatial QA pairs over about 38K video scenes and 19 task types (Sun et al., 10 Oct 2025). Its main novelty is scale diversity: tiny tabletop, tabletop, indoor, outdoor, and drone-view scenes are handled within a common training resource. "MM-Spatial" contributes CA-VQA, a generated QA corpus over CA-1M and ARKitScenes with about 10M training QA pairs and a benchmark of about 62K QA pairs, combining RGB, metric depth, multi-view images, camera intrinsics, relative camera pose, and high-quality oriented 3D boxes (Daxberger et al., 17 Mar 2025). "SpatialMosaic" contributes a multi-view pipeline specialized for partial visibility, low-overlap, and occlusion, with Table 1 reporting 2M training QAs and 1M evaluation QAs, though the main experiments use reduced 200K and 100K subsets for compute reasons (Lee et al., 29 Dec 2025).
These resources can be summarized compactly.
| Resource | Reported scale | Core supervision |
|---|---|---|
| MSMU | 700K QA, 2.5M numerical values, 10K CoT | Physically scaled 3D scenes, metric VQA |
| OpenSpatial-3M | 3M samples | OBB-centric SM, SR, CP, MC, SAR |
| SpaceVista-1M | Approx. 1M QA over about 38K video scenes | 19 tasks across 5 scales |
| CA-VQA | About 10M train QA, about 62K benchmark QA | Oriented 3D boxes, depth, multi-view |
| SpatialMosaic | 2M train QA, 1M eval QA in table report | Low-overlap, partial visibility, occlusion |
A recurring tension in this literature concerns automatic scale versus benchmark fidelity. OpenSpatial and SpaceVista argue for scalable programmatic generation over geometry-aware annotations (Liu et al., 8 Apr 2026, Sun et al., 10 Oct 2025). SpatialMosaic shows that realism under partial visibility can also be generated automatically when camera calibration, scene meshes, and visibility computations are available (Lee et al., 29 Dec 2025). By contrast, MMSI-Bench and MMSI-Video-Bench rely heavily on expert manual curation to reduce ambiguity, underscoring that automatically generated spatial supervision and high-confidence evaluation sets remain different objects (Yang et al., 29 May 2025, Lin et al., 11 Dec 2025). This suggests that MSMU has developed two complementary infrastructures: massive automatic supervision for training, and smaller but stricter human-audited benchmarks for evaluation.
4. Model architectures and learning paradigms
The model side of MSMU is heterogeneous, but several architectural patterns recur. The first is explicit metric priors. "SD-VLM" introduces Depth Positional Encoding, in which pooled depth values are transformed into sinusoidal depth embeddings and added to image features,
thereby attempting to upgrade a VLM’s spatial awareness from 2D to 3D space (Chen et al., 22 Sep 2025). On MSMU-Bench, this design outperforms alternative depth-injection methods such as depth-as-image, depth-as-prompt, and depth-as-token.
The second pattern is decoupled geometry pathways. GAMSI augments Qwen3-VL-8B-Instruct with two sets of learnable geometry queries, one metric and one structural, so that the transformer input becomes
with queries in each group (Zheng et al., 25 May 2026). A task-decoupled attention mask blocks leakage between the metric and structural query streams, while Expert-Guided Visual Grounding aligns the aggregated cues to Depth Anything V2 and VGGT features during training only. This makes the model RGB-only at inference time while still internalizing geometry priors.
The third pattern is object-centric or scene-centric externalization of geometry rather than dense end-to-end geometric prediction. "MM-Spatial" finds that metric depth is most effective when exposed to the model through object-centric chain-of-thought or tool-use, where the model predicts 2D boxes and then reasons from median depth values, rather than trying to encode the full depth map directly (Daxberger et al., 17 Mar 2025). "OpenSpatial" similarly uses 3D oriented bounding boxes parameterized as
as the minimum sufficient geometry for scalable measurement, relation reasoning, and cross-view consistency (Liu et al., 8 Apr 2026). "SpatialMosaicVLM" goes further by treating a reconstruction model, VGGT, as a frozen geometry encoder whose features are fused with CLIP tokens through cross-attention, rather than as a standalone explicit reconstruction pipeline (Lee et al., 29 Dec 2025).
The fourth pattern is self-supervised or weakly supervised geometry learning. "Spatial-SSRL" abandons expensive spatial QA supervision and instead constructs five verifiable pretext tasks from RGB or RGB-D images: shuffled patch reordering, flipped patch recognition, cropped patch inpainting, regional depth ordering, and relative 3D position prediction (Liu et al., 31 Oct 2025). Reinforcement learning then uses the simple reward
where correctness is exact-match against a deterministically known answer. This is a different answer to the MSMU data problem: rather than generating large labeled QA corpora, it turns intrinsic geometric structure into automatically verifiable supervision.
Across these systems, a plausible synthesis is that MSMU architectures are converging on a hybrid regime. Dense geometry, symbolic reasoning, and language modeling are not merged into a single homogeneous latent space; instead, they are linked through metric priors, box-centric abstractions, query separation, frozen geometry encoders, or verifiable self-supervision. This suggests that MSMU is becoming less about a single monolithic world model and more about controlled interfaces between geometry and language.
5. Empirical state of the field
The empirical record shows clear progress, but it also shows that MSMU remains far from solved. On the original MSMU-Bench, SD-VLM reaches 56.31%, and 59.19% with MSMU-CoT, compared with 35.17% for Gemini-2, 33.63% for Intern-VL3-78B, and 32.28% for GPT-4o (Chen et al., 22 Sep 2025). On seven spatial intelligence benchmarks, GAMSI achieves a macro-average of 75.8%, surpassing SenseNova-SI-InternVL3-8B at 64.0% by +11.8% absolute (Zheng et al., 25 May 2026). On specialized spatial benchmarks, OpenSpatial-trained Qwen3-VL-8B reaches a 3D-Avg of 62.1 versus 56.7 for the Qwen3-VL-8B baseline, while OpenSpatial-trained Qwen2.5-VL-7B improves from 50.0 to 59.5 (Liu et al., 8 Apr 2026). These are substantial gains, but they remain benchmark-local gains within a field whose evaluation suite is itself expanding.
The broader benchmark picture is more sobering. Human-model gaps remain large on multi-image, video, 4D, and geometry-heavy tasks. The following headline comparisons are representative.
| Benchmark | Best reported model | Human |
|---|---|---|
| MMSI-Bench | o3: 41.0 | 97.2 |
| MMSI-Video-Bench | Gemini 3 Pro: 38.0 | 96.4 |
| Spatial4D-Bench | GPT-5: 60.90 | 78.02 |
| SolidGeo | OpenAI-o1: 49.5 | 77.5 |
These gaps are not confined to one modality. MMSI-Bench shows that current MLLMs remain weak at overlap matching, scene reconstruction, and situation transformation across multiple images (Yang et al., 29 May 2025). MMSI-Video-Bench finds that many video models remain near chance on reasoning-intensive spatial questions, and that more frames, 3D cue augmentation, or chain-of-thought prompting do not reliably fix the problem (Lin et al., 11 Dec 2025). Spatial4D-Bench reports that current MLLMs are stronger at some static metric tasks than at route planning, egocentric reasoning, spatial memory, and physical plausibility (Wang et al., 31 Dec 2025). SolidGeo shows that even strong reasoning models remain far below human performance on projection, unfolding, and multimodal solid-geometry measurement (Wang et al., 27 May 2025).
A second empirical pattern is that improvements are often asymmetric across capability types. In GAMSI, structural priors particularly help relational and viewpoint-heavy tasks such as MindCube-Tiny and SPAR-Bench, while the metric pathway is needed to recover performance on scale-sensitive benchmarks such as CV-Bench (Zheng et al., 25 May 2026). In MM-Spatial, multi-view inputs and tool-mediated depth improve metric tasks more than dense depth-map encoding does (Daxberger et al., 17 Mar 2025). In Spatial-SSRL, RGB-only tasks improve 2D and general visual reasoning, while RGB-D tasks contribute more strongly to 3D reasoning (Liu et al., 31 Oct 2025). This suggests that MSMU is not a single axis of progress; gains on one spatial subproblem do not guarantee gains on others.
A third pattern is the persistence of difficult failure modes. MMSI-Bench identifies grounding errors, overlap-matching and scene-reconstruction errors, situation-transformation reasoning errors, and spatial-logic errors (Yang et al., 29 May 2025). MMSI-Video-Bench highlights geometric reasoning errors, detailed grounding errors, prompt-alignment errors, and latent logical inference errors (Lin et al., 11 Dec 2025). SolidGeo reports that visual perception errors and reasoning errors together account for over 70% of failures in analyzed samples (Wang et al., 27 May 2025). The empirical implication is that current models still lack stable internal spatial representations, especially when observation is partial, viewpoint varies, or the task requires multi-step transformation rather than direct recognition.
6. Limitations, methodological tensions, and future directions
Several limitations recur across the MSMU literature. The first is domain narrowness. The original MSMU dataset concentrates on indoor settings and objects typical of domestic environments (Chen et al., 22 Sep 2025). MM-Spatial is similarly focused on indoor ARKitScenes-based data (Daxberger et al., 17 Mar 2025). SpaceVista deliberately broadens scale and scenario coverage, but its own appendix reveals small inconsistencies around the exact number of scales and task accounting, and its training labels are filtered for perceptual correctness rather than strict physical correctness (Sun et al., 10 Oct 2025). OpenSpatial acknowledges weaker gains in some desktop-level and outdoor settings and uses a metric flag precisely because absolute metric scale is not always available (Liu et al., 8 Apr 2026). These facts suggest that domain transfer remains an open problem, especially for outdoor, mixed-scale, and dynamic environments.
The second limitation is evaluation format. Many benchmarks are multiple-choice or short-answer QA. This is computationally convenient and yields clean comparisons, but it does not test closed-loop control, persistent scene memory under deployment constraints, or calibrated metric outputs. MMSI-Video-Bench explicitly notes that it is not a benchmark for coordinate-accurate localization, SLAM, dense mapping, or explicit world-model construction (Lin et al., 11 Dec 2025). Spatial4D-Bench is stronger on coverage than on interactive embodiment, and its heavy reliance on 64-frame sampling means benchmark difficulty partly reflects context compression as well as spatial cognition (Wang et al., 31 Dec 2025). A plausible implication is that current MSMU benchmarks are best interpreted as diagnostic probes of spatial competence rather than complete tests of deployable spatial systems.
The third limitation is supervision dependence and scalability tension. GAMSI removes geometry experts from inference time, but it still depends heavily on Depth Anything V2, VGGT, SenseNova-SI-800K, and the 152,776-sample MTS corpus during training (Zheng et al., 25 May 2026). MMSI-Bench and MMSI-Video-Bench achieve high benchmark fidelity through manual expert curation, yet both identify annotation scalability as a core limitation (Yang et al., 29 May 2025, Lin et al., 11 Dec 2025). OpenSpatial and SpaceVista solve the scale problem with automated geometry-grounded generation, but this raises persistent questions about label noise, metric faithfulness, and specialist bias (Liu et al., 8 Apr 2026, Sun et al., 10 Oct 2025). Spatial-SSRL offers an alternative by avoiding human spatial annotation altogether, but its pretext tasks remain proxies and partly depend on RGB-D availability (Liu et al., 31 Oct 2025). The field therefore has not resolved the trade-off between benchmark fidelity and supervision scale; it has only developed several workable positions along that axis.
The fourth limitation is representation mismatch between measurement and reasoning. Measurement-heavy tasks require calibrated quantities, while relational and planning tasks require abstract, viewpoint-aware structure. GAMSI treats that mismatch explicitly through decoupled metric and structural pathways (Zheng et al., 25 May 2026). SolidGeo exposes an analogous split between symbolic 3D reasoning and visual-spatial grounding, and notes the absence of a widely adopted formal language for solid geometry (Wang et al., 27 May 2025). This suggests that a fully unified MSMU representation remains elusive. A plausible implication is that future progress will depend less on forcing all spatial abilities into one undifferentiated latent space and more on principled interfaces among metric estimators, structural scene abstractions, memory, and language.
Future directions are already visible in the current literature. One direction is broader scale and modality coverage, from indoor scans to all-scale video, materials microstructures, and 4D scene evolution (Sun et al., 10 Oct 2025, Calvat et al., 19 Jun 2026, Wang et al., 31 Dec 2025). A second is better data engines, especially open and auditable ones that can expose how visibility filtering, task diversity, or 3D lifting affect performance (Liu et al., 8 Apr 2026). A third is training-time geometry with inference-time efficiency, as seen in GAMSI and MM-Spatial (Zheng et al., 25 May 2026, Daxberger et al., 17 Mar 2025). A fourth is intrinsic supervision at scale, as seen in Spatial-SSRL (Liu et al., 31 Oct 2025). Taken together, these directions suggest that MSMU is evolving from a narrow benchmark theme into a general research program on how to measure, represent, and reason about space at scale, under incomplete observation, with both geometric fidelity and semantic interpretability.