SpatialMosaicVLM: Hybrid Vision-Language Model

• SpatialMosaicVLM is a hybrid vision-language model that integrates CLIP-ViT appearance tokens with geometry-aware tokens from 3D reconstructions for robust spatial reasoning.
• The model employs a dual-branch framework with cross-attention fusion, effectively handling challenges like occlusion and limited field-of-view overlap.
• Evaluated on the SpatialMosaic dataset, it demonstrates competitive performance in tasks such as object localization and counting under complex visual conditions.

SpatialMosaicVLM refers to a class of hybrid Vision-LLMs (VLMs) designed to perform robust spatial reasoning across multiview input images, particularly under conditions of partial visibility, heavy occlusion, and low field-of-view overlap. The SpatialMosaicVLM architecture explicitly fuses appearance features from conventional visual encoders with geometry-aware tokens derived from 3D reconstruction models, enabling human-level scene understanding from fragmented visual cues. This approach is instantiated and evaluated in the context of the SpatialMosaic dataset and benchmark, which specifically target realistic multiview QA tasks reflecting the complexities of real-world environments (Lee et al., 29 Dec 2025).

1. Architectural Foundations of SpatialMosaicVLM

SpatialMosaicVLM is structured as a dual-branch hybrid framework:

• Visual Branch: Utilizes a standard visual encoder, specifically CLIP-ViT, to extract dense appearance tokens from each input RGB image.
• Geometry Branch: Employs a pretrained 3D reconstruction model (VGGT) that processes the set of images jointly, generating multi-view spatial tokens representing the fused 3D geometry and per-camera pose tokens.

The core data flow is as follows:

• For $V$ RGB images $\{I_v\}_{v=1}^V$ and a natural-language query $Q$:
  • Visual features: $$F_{\text{vis}}^{(v)} = E_{\text{vis}}(I_v) \in \mathbb{R}^{T_{\text{vis}} \times d}$$, concatenated across views.
  • Geometry features: $$(F_{\text{spa}}, z) = E_{\text{geo}}(\{I_v\}_{v=1}^V)$$, where $F_{\text{spa}}$ contains multi-view spatial tokens and $z$ per-camera pose tokens.
  • Fusion via cross-attention: $$F_{\text{fuse}} = \mathrm{softmax}\left(\frac{F_{\text{vis}} W_q (F_{\text{geo}} W_k)^T}{\sqrt{d_k}}\right)\cdot(F_{\text{geo}}W_v)$$, then projected to a fixed dimension for downstream language modeling.

Each QA instance is processed by aligning fused visual-geometry features and input question tokens before predicting the answer.

Branch	Encoder	Output Tokens	Role
Visual (CLIP-ViT)	$E_{\text{vis}}$	$F_{\text{vis}}$	2D appearance
Geometry (VGGT)	$E_{\text{geo}}$	$F_{\text{spa}}, z$	3D spatial reasoning

This explicit fusion mechanism models both object appearance and geometric context, a configuration shown to outperform purely visual or geometry-unaware approaches (Lee et al., 29 Dec 2025).

2. Mathematical Framework: Occlusion, Overlap, Fusion

SpatialMosaicVLM formalizes several critical geometric quantities:

• Visibility & Occlusion Ratios: Given per-instance ($D_n$) and global ($D$) depth maps, let $P_n$ be instance $n$'s 3D points. Then:
  • Occluded: $O_n = \{p \in P_n \;\mid\; 0 < D(p) < D_n(p)\}$
  • Visible: $$V_n = \{p \in P_n \;\mid\; D_n(p) \leq D(p) < \infty\}$$
  • Occlusion ratio: $$r_{n,\text{obj}} = \frac{|O_n|}{|O_n| + |V_n|} \in [0,1]$$
• Field-of-View Truncation: By rendering an extended image plane ($2W\times 2H$), the coverage of points inside and outside the original FOV is computed, yielding $r_{n,\text{FOV}}$.
• Multi-view Overlap: For two views $i,j$, overlap is defined as:

$$\mathrm{Overlap}(i,j) = \frac{|V^i \cap V^j|}{|V^i \cup V^j|}$$

Image sets are selected for training with $\mathrm{Overlap} < 0.3$, enforcing low redundancy and challenging reasoning.

• Feature Fusion Objective: Cross-attention-based fusion projects appearance and geometry tokens into a shared latent, with the composite passed through an MLP before concatenation with tokenized queries.

The LLM’s head minimizes standard cross-entropy loss over the fused features and labeled answers.

3. Dataset Generation and Training Paradigm

Data Sourcing and Annotation: The SpatialMosaic pipeline synthesizes instruction-tuning data from ScanNet++ scans ($\sim$850 scenes, high-density 3D mesh + semantics):

• Scenes are partitioned into train/test splits (679/170).
• For each query, 2–5 views are sampled with $<30\%$ pairwise overlap; targets are filtered to guarantee partial visibility and moderate occlusion.
• Each candidate object or instance is annotated with per-view occlusion, bounding box projections, and relative positions/directions in $\mathbb{R}^3$.

Question-Answer (QA) Families: Six task templates are used:

1. Object Count (MCQ)
2. Best-View Selection (MCQ)
3. Object Localization (MCQ or binary + coordinates)
4. Occlusion-Aware Existence (yes/no)
5. Occlusion-Aware Attribute (MCQ)
6. Occlusion-Aware Spatial Relation (MCQ)

Distractor options are programmatically generated to challenge both geometric and semantic reasoning.

Training Protocol:

• Model: LLaVA-based backbone (7B) with frozen CLIP-ViT (visual) and VGGT (geometry).
• Optimization: Cross-attention fusion weights, projector, and LM head are trained using DeepSpeed ZeRO-2 over 5 epochs.
• Regular data augmentation includes random view selection and instance composition under occlusion/visibility constraints.

4. Empirical Performance and Benchmark Results

The SpatialMosaic-Bench provides a suite of 1M multiview QA pairs for comprehensive evaluation, partitioned by task and scenario:

Model	Avg.	Count	Best-View	Exist.	Attr.	Rel.	Loc.
LLaVA-OneVis 0.5B	37.7	29.5	44.4	55.3	20.7	37.7	38.3
InternVL2 8B	46.0	61.6	49.0	54.4	38.6	39.8	43.3
VLM-3R (7B)	81.7	89.8	73.0	81.2	74.2	83.6	100
SpatialMosaicVLM (7B)	81.8	89.9	72.9	81.5	74.3	84.0	100

SpatialMosaicVLM matches or slightly exceeds the performance of VLM-3R, particularly in object counting and localization. In zero-shot temporal transfer (VSTI-Bench), SpatialMosaicVLM (7B) attains 46.8% average accuracy, surpassing all open-source and proprietary baselines not exposed to temporally grounded queries during training (Lee et al., 29 Dec 2025).

5. Modular Extensions and Application-Specific Adaptations

SpatialMosaicVLM has been instantiated in diverse settings:

• Atlas Urban Index (AUI): The AUI applies a SpatialMosaicVLM-inspired pipeline to monitoring urban development from temporally calibrated satellite mosaics. Core steps include:
  • Constructing cloud-minimized Sentinel-2 mosaics per region and time window.
  • Prompting a VLM with current, reference, and temporally anchored images.
  • Producing scalar development scores with robust temporal smoothness and inter-region calibratability.

Resulting AUI values demonstrate higher temporal correlation and lower score variance (cloud noise) than pixel-based indices such as NDBI, as evidenced by $0.94$ temporal correlation with ground-truth change in urbanizing test regions (Chander et al., 26 Oct 2025).

• MAPWise Benchmark Adaptation: MAPWise identifies architectural and training requirements for adapting SpatialMosaicVLMs to map-based queries:
  • Integration of adjacency graph encoding, grid-based positional embeddings, and pattern-disambiguation subnetworks are proposed.
  • Empirical results show that current VLM architectures underperform on spatial mosaic reasoning tasks—average accuracy on relative spatial queries is $25$–$30\%$, revealing a significant performance gap and motivating further advances in geometry-aware fusion (Mukhopadhyay et al., 2024).

6. Key Insights, Contributions, and Limitations

Notable Insights

• Geometry priors, injected via pretrained 3D reconstructors, are crucial for robust spatial reasoning in sparse, occluded, or low-overlap visual regimes.
• Off-the-shelf, vision-only VLMs exhibit systematic failure modes in environments with heavy occlusion or limited view coverage.

Major Contributions

1. Introduction of the SpatialMosaic dataset and benchmark: Over 3M QA pairs focusing on partial visibility and occlusion.
2. Hybrid VLM architecture fusing CLIP-ViT and VGGT features through cross-attention for geometric grounding.
3. Scalable data generation pipeline leveraging high-fidelity 3D mesh scan annotation for robust instruction tuning.

Limitations

• The geometry encoder (VGGT) is heavyweight and operates in a frozen (non-finetuned) mode, which may entrench mis-reconstruction artifacts.
• The system is restricted to indoor, static scenes with no dynamic elements or outdoor generalizability.
• Performance is susceptible to any failure in upstream geometry inference.

7. Implications and Prospects

SpatialMosaicVLM’s hybrid approach positions it as an enabling technology for embodied AI applications—search-and-rescue robotics, warehouse automation, AR/VR scene comprehension, and assistive devices where robust, human-like 3D spatial reasoning is imperative. The architecture’s modularity allows for extension to other domains requiring joint visual/geometric processing, as evidenced by its adaptation in remote sensing (AUI) and cartographic QA (MAPWise). A plausible implication is that future systems will incorporate learnable, fine-tuned geometry modules as lighter and more generalizable 3D encoders emerge.

Ongoing work includes generalizing the paradigm to dynamic scenes, outdoor environments, and more diverse modalities, as well as integrating self-supervised learning strategies and external spatial knowledge for open-world deployment scenarios (Lee et al., 29 Dec 2025, Chander et al., 26 Oct 2025, Mukhopadhyay et al., 2024, Fan et al., 26 May 2025).