Spa3-VLM: Self-Supervised 3D VLM

• Spa3-VLM is a vision-language model that augments traditional 2D VLMs with self-supervised, view-invariant 3D representations derived solely from unposed multi-view images.
• It employs a Spa3R encoder and a residual cross-attention adapter to efficiently fuse 2D visual tokens with compact 3D spatial latents while preserving baseline 2D features.
• The system leverages predictive spatial field modeling with L1 and cosine loss supervision, achieving up to a 7.5% improvement in spatial reasoning benchmarks over simpler methods.

Spa3-VLM is a vision-LLM architecture that augments traditional 2D vision-LLMs (VLMs) with self-supervised, view-invariant 3D spatial representations derived solely from unposed multi-view images. This integration is achieved via the Spa3R (Predictive Spatial Field Modeling, PSFM) encoder and a lightweight cross-attention adapter, producing a system with state-of-the-art performance on 3D visual reasoning benchmarks such as VSI-Bench, while avoiding explicit 3D instruction tuning or reliance on external geometric modalities like depth sensors or pre-built 3D maps (Jiang et al., 24 Feb 2026).

1. Model Architecture

The Spa3-VLM framework consists of two primary modules: the Spa3R encoder and a VLM backbone with an inserted residual cross-attention adapter.

• Spa3R Encoder: Receives as input a set of $N_c$ context image-view pairs $\{(v_c, F_c)\}_{c=1}^{N_c}$, where $v_c \in \mathbb{R}^6$ denotes camera pose (extrinsics plus intrinsics) and $F_c \in \mathbb{R}^{H \times W \times D_{vis}}$ is a spatially aligned, coordinate-framed feature map from a VGGT (Vision Geometry GPT) backbone. The encoder $E_\phi$ (6-layer Transformer) processes learnable query embeddings ($q \in \mathbb{R}^{N_q \times D}$ with $N_q=256, D=768$) and context features, outputting a unified 3D spatial latent $z \in \mathbb{R}^{N_q \times D}$. A decoder $D_\theta$ enables synthesis of target features for arbitrary novel camera poses.
• Adapter-augmented VLM: The core vision-LLM (Qwen2.5-VL in the reference implementation) uses a residual cross-attention adapter at each cross-modal block. The 2D vision tokens $F_V$ query the Spa3R latent $z$ to yield fused tokens $F_V' = F_V + \mathrm{MLP}_\mathbf{0}(F_{fused})$, with the MLP zero-initialized to preserve original model identity before instruction-tuning. The fused visual tokens are concatenated with text tokens and fed into the LLM.
• Asymmetric View Aggregator: Spatial context features $F_c$ are produced independently of target views but share a common spatial coordinate frame, via attention masking in VGGT: context views $C$ attend only to $C$, target views $T$ attend to both $C$ and $T$, using a masking matrix

$$M_{ij} = \begin{cases} 0 & \text{if } i \in T \lor j \in C, \ -\infty & \text{otherwise.} \end{cases}$$

This design enforces a strong spatial bottleneck, ensuring that all view-specific predictions must be synthesized from a single, compact global representation.

2. Predictive Spatial Field Modeling (PSFM)

Spa3R is trained under the PSFM paradigm: a 3D scene is treated as a continuous feature field $f: \mathcal{V} \rightarrow \mathcal{J}$, mapping any camera pose $v$ to a feature map $F$. At each iteration, context and target views are randomly partitioned. The encoder $E_\phi$ summarizes context into the latent $z$, and the decoder $D_\theta$ predicts each target feature map $\hat F_t$: $$z = E_\phi(C); \quad \hat F_t = D_\theta(v_t \mid z)$$ Supervision combines L1 and cosine loss over geometric (VGGT) and semantic (DINOv3) targets: $$\mathcal{L}(\hat F_t, F_t) = \|\hat F_t - F_t\|_1 + \Bigl(1 - \tfrac{\hat F_t \cdot F_t}{\|\hat F_t\|_2 \|F_t\|_2}\Bigr)$$ The total loss is summed over all target views and both target types. This self-supervised training leverages multi-view RGB-D datasets (ScanNet, ScanNet++).

3. Integration with Vision-LLMs

During instruction tuning, the Spa3R encoder and vision encoder remain frozen. Only the residual cross-attention adapter and LLM are fine-tuned, ensuring the original 2D and spatial priors are retained. Within each cross-modal block:

• Queries: 2D vision token matrix $F_V \in \mathbb{R}^{N_V \times D_V}$
• Keys/values: Spa3R spatial latent $z \in \mathbb{R}^{N_q \times D}$
• Output: Cross-attention results fused via a zero-initialized MLP, so $F_V' \approx F_V$ at the beginning of tuning.

Gradient flow into only the adapter/LLM enables efficient transfer of spatial reasoning without overwriting baseline 2D perceptual abilities.

4. Training Regimen and Datasets

Spa3R is pre-trained on 2000 multi-view indoor scenes, with $M\in[4,12]$ sampled views per scene per iteration. Half are assigned to context, half as target. The VGGT aggregator and DINOv3 backbone providing geometric/semantic features are frozen. Pre-training uses AdamW (lr$=1e^{-3}$, 80K steps, 8 NVIDIA 5090 GPUs).

Instruction-tuning employs datasets:

• VSI-590K: multi-view video spatial instruction-following
• SPAR-234K, LLaVA-Hound, VLM3R: image-based and 3D spatial reasoning tasks

The base VLM is Qwen2.5-VL-3B, with only adapters and LM unfrozen.

5. Experimental Performance

Spa3-VLM achieves state-of-the-art accuracy on challenging spatial reasoning tasks. On the VSI-Bench 3D VQA benchmark:

Model	Avg. Accuracy
Spa3-VLM-4B	58.6
Cambrian-S-3B	57.3
VG-LLM-8B	50.7
Qwen2.5VL-7B	33.0
Gemini-1.5-Pro	45.4
GPT-4o	34.0

Notable accuracy on sub-tasks includes Object Count (69.0%), Object Size (70.6%), Relative Direction (57.9%), and Appearance Order (73.6%) (Jiang et al., 24 Feb 2026).

Spa3-VLM also demonstrates superior results across other spatial benchmarks (SPAR, ViewSpa). Ablation studies confirm that the full Spa3R spatial representation provides a 3.5% absolute gain over using VGGT features alone, and the cross-attention integration yields a 7.5% improvement over simpler sequence appending.

6. Discussion, Limitations, and Future Directions

The Spa3-VLM approach, by enforcing a predictive spatial representation bottleneck, circumvents the need for explicit 3D geometry supervision or instruction rewriting. The system learns global scene semantics and geometry directly by synthesizing arbitrary novel views, providing a coherent internal model of scene structure that is accessible to the LLM through adapter-based grounding.

Key strengths include:

• Spatial reasoning grounded in global, view-invariant embeddings
• Preserved or enhanced 2D vision-language performance after integration
• Efficient, modular adapter-based augmentation (no base model retraining required)

Current limitations arise primarily from the scope of pre-training (indoor, static scenes; limited outdoor/dynamic scene coverage) and the implicit nature of metric depth (not directly regressed). Performance may degrade in highly reflective or textureless environments or under very sparse view sampling. Future research may incorporate explicit photometric/rendering losses, temporal motion cues, or learned viewpoint priors to extend spatial coverage and robustness.

Overall, Spa3-VLM establishes that comprehensive spatial intelligence for 3D vision-language reasoning can emerge from self-supervised, view-predictive modeling of multi-view 2D images, without reliance on explicit 3D instruction-tuning, expensive depth sensors, or pre-built maps (Jiang et al., 24 Feb 2026).