---
title: 'SpatialMosaic: Multi-View 3D Spatial Reasoning'
url: https://www.emergentmind.com/topics/spatialmosaic-dataset
type: topic
---

# SpatialMosaic: Multi-View 3D Spatial Reasoning

SpatialMosaic is a large-scale multi-view instruction-tuning dataset and evaluation benchmark specifically constructed to advance spatial reasoning and 3D scene understanding in Vision-Language Models (VLMs), particularly under partial visibility, occlusion, and low-overlap conditions. Building upon high-fidelity real-world indoor scenes from ScanNet++ and employing a scalable QA generation and annotation pipeline, SpatialMosaic introduces 2 million multimodal QA training pairs and a dedicated 1 million-sample benchmark (“SpatialMosaic-Bench”) covering six spatial reasoning tasks. The accompanying SpatialMosaicVLM architecture leverages both visual and explicit 3D geometry encoders, demonstrating substantial improvements over prior state-of-the-art methods in accuracy and transfer robustness [2512.23365].

## 1. Data Sources and Multi-View Sampling Strategy

SpatialMosaic is derived from 849 indoor scenes from ScanNet++ (Yeshwanth et al. 2023), each featuring dense RGB-D imagery and 3D semantic meshes. The dataset design enforces fragmented and complementary multi-view scenarios by sampling 2–5 view combinations per scene with strictly constrained 3D-point overlap. Specifically, the overlap between any two views $(i,j)$ is defined by:
\[
\text{Overlap}(i, j) = \frac{|V^i \cap V^j|}{|V^i \cup V^j|},
\]
where $V^i$ aggregates all visible 3D points across object instances in view $i$. Only view pairs with $\text{Overlap} < 0.3$ are retained to discourage trivial redundancy and elicit genuine multi-view reasoning.

Occlusion and visibility are rigorously annotated:
- **Object-level occlusion ratio** $r_{n,\text{obj}}$ quantifies the fraction of object $n$'s points physically occluded by other scene geometry within a view.
- **Field-of-view occlusion ratio** $r_{n,\text{FoV}}$ captures the proportion of an object lying beyond the observed image boundary, computed via extended rendering in a $2H \times 2W$ canvas.

A filtering protocol excludes objects fully visible in all selected views (no partial cues) or with $>90\%$ occlusion in all views (excessive difficulty), ensuring that resulting samples reflect realistic fragmented observations.

## 2. QA Generation Pipeline and Task Formulation

SpatialMosaic’s QA generation consists of a four-stage pipeline:
1. **Combination Filtering:** Candidate view sets (2–5 per sample) are selected based on occlusion/overlap constraints.
2. **Object Instance Selection:** For each view combination, all object instances appearing in at least one but not all views are considered.
3. **3D Bounding-Box Transformation:** Object locations are computed in the camera frame of a “query view” using $v^{(w)} \rightarrow v^{(c)} = R_{wc}(v-t_{wc})$, enabling spatial relation extraction.
4. **Task-Specific QA Construction:** Templates cover six distinct task types, each with systematic distractor generation through axis inversion or orthogonal contradiction:

| Task Name                                        | Format                      | Description                                     |
|--------------------------------------------------|-----------------------------|-------------------------------------------------|
| Object Count                                     | 4-way multiple choice       | “How many [category] across these frames?”      |
| Best-View Selection                             | 4-way multiple choice       | “Which frame gives the most of [category]?”     |
| Object Localization                             | Binary + bbox coordinates   | “Is there an [object] in Frame 1? Where?”       |
| Occlusion-Aware Object Existence                | Binary                     | Relation-based existence between objects         |
| Occlusion-Aware Attribute                       | 4-way single-answer         | “Which [object] is to the left of the lamp?”    |
| Occlusion-Aware Spatial Relation                | 4-way multiple choice       | “Where is X relative to Y?”                     |

Uniform sampling ensures balanced coverage across the six tasks, with ~333K QA pairs for each in the train partition.

## 3. Dataset Composition and Diagnostic Annotations

The SpatialMosaic dataset comprises approximately 8 million images (training split), 2 million multi-view QA training pairs, and 1 million QA pairs in the benchmark (SpatialMosaic-Bench), uniformly sampled across 849 scenes (679 for training, 170 for evaluation). Each sample involves 2–5 frames with diagnostic annotation per QA specifying:
- **Visibility Scenario:** Distinguishes fully versus partially visible targets.
- **Ground Truth Coverage:** Differentiates full versus partial scene coverage for category instances.

This annotation regime facilitates difficulty analysis and supports evaluation of spatial reasoning under authentic partial observability.

## 4. Benchmarking Protocol and Evaluation Metrics

SpatialMosaic-Bench operationalizes six rigorous evaluation tasks replicating the train set structure. The main metrics are:
- **Accuracy:** Percentage of correct responses for multiple-choice or binary outputs.
- **(Optional) F1 Score:** Used for binary tasks where relevant; class imbalance was not observed but fine-grained token-level F1 can be reported.
- **Spatial Consistency:** Alignment between predicted relations and ground-truth computed from 3D bounding-boxes.

Evaluations employ the aforementioned occlusion, visibility, and overlap measures to calibrate difficulty and interpret model performance.

## 5. SpatialMosaicVLM Model Architecture

SpatialMosaicVLM introduces a hybrid multi-view framework integrating explicit geometry modeling:
- **Visual Encoder ($E_{vis}$):** CLIP ViT backbone generating per-view patch tokens $F_{vis}^{(v)} \in \mathbb{R}^{T_{vis} \times d}$.
- **Geometry Encoder ($E_{geo}$):** VGGT 2025 architecture producing spatial tokens $F_{spa}$ and camera tokens $z$, concatenated to $F_{geo} \in \mathbb{R}^{(T_{spa}+V)\times d}$ across views.
- **Fusion:** Cross-attention mechanism combines $F_{vis}$ and $F_{geo}$:
  \[
  F_{fuse} = \text{softmax}\big((F_{vis} W_q)(F_{geo} W_k)^T / \sqrt{d_k}\big)(F_{geo} W_v)
  \]
  with $W_q, W_k, W_v$ as trainable projections.
- **Language Model Integration:** $F_{fuse}$ undergoes two-layer MLP projection, concatenation with tokenized question, and is passed to an LLM (e.g., LLaVA Next).
- **Training Objective:** Cross-entropy loss $L_{CE} = -\sum_{c \in \text{choices}} y_c \log p_c$ for all QA tasks; geometry-consistency loss is omitted, as encoders are frozen and only fusion plus QA head are fine-tuned.

A plausible implication is that explicit geometry encoding, via VGGT, provides significant gains in occlusion- and relation-sensitive settings where naïve vision transformers alone underperform.

## 6. Experimental Analysis and Performance Insights

Comprehensive results validate the utility of both dataset and architecture:
- **SpatialMosaic-Bench:** SpatialMosaicVLM (7B, fine-tuned) achieves 81.8% average accuracy; the best open-source baseline (LLaVA-NeXT-Video-7B) attains 47.8%, while VLM-3R (re-implemented, fine-tuned) reaches 81.7%. Fine-tuning on SpatialMosaic yields ≈34 percentage-point improvement over the strongest zero-shot baseline.
- **Zero-Shot Transfer (VSTI-Bench):** SpatialMosaicVLM attains 46.8% without further fine-tuning, outperforming best baseline (44.0%), indicating robust generalization to camera-centric and temporal tasks.
- **Ablation Studies:** Removal of geometry encoder (VGGT) reduces accuracy by ≈12 points, underscoring the necessity of explicit 3D cues. Performance sharply decreases with increasing occlusion/view-overlap, confirming the metrics’ correlation with task difficulty. In extreme “Partially Visible + Partial Coverage” scenarios, SpatialMosaicVLM maintains >65% accuracy, compared to >20-point drops in all competing methods.

This suggests that carefully annotated, occlusion-aware multi-view QA construction in SpatialMosaic, combined with hybrid vision-geometry models, substantially advances spatial reasoning capabilities in VLMs [2512.23365].

Source: https://www.emergentmind.com/topics/spatialmosaic-dataset