GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models

Abstract: Multimodal LLMs (MLLMs) demonstrate exceptional semantic reasoning but struggle with 3D spatial perception when restricted to pure RGB inputs. Despite leveraging implicit geometric priors from 3D reconstruction models, image-based methods still exhibit a notable performance gap compared to methods using explicit 3D data. We argue that this gap does not arise from insufficient geometric priors, but from a misalignment in the training paradigm: text-dominated fine-tuning fails to activate geometric representations within MLLMs. Existing approaches typically resort to naive feature concatenation and optimize directly for downstream tasks without geometry-specific supervision, leading to suboptimal structural utilization. To address this limitation, we propose GAP-MLLM, a Geometry-Aligned Pre-training paradigm that explicitly activates structural perception before downstream adaptation. Specifically, we introduce a visual-prompted joint task that compels the MLLMs to predict sparse pointmaps alongside semantic labels, thereby enforcing geometric awareness. Furthermore, we design a multi-level progressive fusion module with a token-level gating mechanism, enabling adaptive integration of geometric priors without suppressing semantic reasoning. Extensive experiments demonstrate that GAP-MLLM significantly enhances geometric feature fusion and consistently enhances performance across 3D visual grounding, 3D dense captioning, and 3D video object detection tasks.

• The paper introduces geometry-aligned pre-training with token-level, multi-layer gated fusion to activate implicit 3D representations in RGB-only multimodal language models.
• GAP-MLLM-3B improves ScanRefer [email protected] from 36.4 to 53.1, Scan2Cap CIDEr from 78.6 to 84.7, and four-frame detection [email protected] from 38.2 to 50.6 over VG-LLM-4B.
• Ablations show that joint geometry-semantic pre-training and gated fusion are complementary, while strict metric reconstruction results indicate genuinely metric-consistent spatial understanding.

Motivation and central claim

GAP-MLLM addresses a specific deficiency in image-only 3D scene perception with multimodal LLMs (MLLMs): although feed-forward reconstruction models such as VGGT supply pixel-aligned implicit geometric priors from RGB inputs, methods built on these priors consistently trail approaches that consume explicit 3D data. The paper's central argument is that this gap originates not from insufficient geometric signal but from a training-paradigm misalignment: downstream fine-tuning is dominated by language objectives, so geometric representations injected via naive feature fusion remain weakly activated. This diagnosis is supported by an attention-map analysis showing that the same geometric-encoder token exhibits substantially different attention patterns across layers, meaning last-layer tokens—the default choice in prior work—carry only a biased subset of the structural information.

Architecture

The model pairs Qwen3-VL-2B (visual branch and LLM decoder) with VGGT-1B (geometric branch), both with $L=24$ layers. After applying identical $2\times2$ token merging to both branches to guarantee spatial alignment, each layer receives an independent gating mechanism:

$$g_{i,j} = \sigma\big(\text{MLP}\big([\mathcal{T}_{i,j}^{V'}, \mathcal{T}_{i,j}^{G'}]\big)\big), \qquad \mathcal{T}_{i,j}^{S} = g_{i,j} \odot \mathcal{T}_{i,j}^{V'} + (1-g_{i,j}) \odot \mathcal{T}_{i,j}^{G'}$$

This token-level gating allows geometry–semantics weighting to vary by layer and spatial position. Following DeepStack-style hierarchical injection, final-layer fused tokens serve as the primary decoder input, while intermediate-layer fused tokens ($L_1{=}5$, $L_2{=}11$, $L_3{=}17$) are added into early decoder blocks. The visual encoder and geometric encoder are frozen; only the LLM backbone and fusion module are optimized.

Gating-weight visualization confirms complementary behavior: intermediate layers weight geometry more heavily while the final layer shifts toward semantic features, empirically justifying multi-level rather than single-layer fusion.

Training paradigm

All outputs share a consistent metric coordinate system anchored at the first frame, following VGGT conventions. Training proceeds in two stages:

Sparse geometry–semantics joint pre-training. Using roughly 500K samples from ScanNet and EmbodiedScan, a red-cross visual prompt marks one pixel; the model must predict that pixel's 3D coordinate and semantic label in JSON form. Notably, because supervision is applied only at prompted pixels, the effective pixel-level supervision is equivalent to approximately two $680\times480$ images—yet this sparse signal suffices to activate structural perception. Joint semantic supervision measurably stabilizes metric structure relative to pointmap-only training.

Object-level fine-tuning. Downstream tasks include two-stage 3D video grounding (frame selection followed by anchor-conditioned box prediction of center, size, and rotation), coordinate-conditioned dense captioning using Mask3D proposals, and 3D video detection emitting all objects under the first-frame frame. The supplementary analysis shows the two-stage grounding formulation alone lifts ScanRefer [email protected] from 37.5 to 45.4 over the single-pass VG-LLM formulation, an independent contribution worth noting.

Results

The headline comparisons on RGB-only benchmarks:

Task	Metric	VG-LLM-4B	GAP-MLLM-3B
ScanRefer	[email protected] / @0.5	36.4 / 11.8	53.1 / 26.0
Scan2Cap	CIDEr / BLEU-4 @0.5	78.6 / 40.9	84.7 / 42.1
Detection (4-frame)	[email protected]	38.2	50.6
Detection (6-frame)	[email protected]	36.4	48.5

On grounding, a 3B-parameter model improves [email protected]/0.5 by more than 11 points over the larger VG-LLM-4B while approaching explicit-3D-input methods; on captioning it surpasses several explicit-input baselines including LLaVA-3D and Video-3D LLM. On detection it exceeds even VG-LLM-7B by a wide margin.

Ablations establish complementarity: on GAP-MLLM-3B, joint pre-training alone raises detection F1 from 44.7 to 48.7, gated fusion alone to 47.5, and their combination to 50.6. Applying the full paradigm to VG-LLM yields consistent gains across backbones, supporting architectural generality. A pure Qwen3-VL backbone without any geometric encoder also gains from joint pre-training (F1 39.7 → 42.4), indicating the pre-training strengthens intrinsic spatial perception rather than merely calibrating the fusion module. Among fusion strategies, gated fusion (F1 50.6) outperforms cross-attention (49.1), addition (47.9), and weighted averaging (47.8).

Metric reconstruction serves as a probe of learned metric awareness. Under Sim(3)-aligned evaluation, CUT3R remains slightly better, but under strict metric evaluation without alignment, GAP-MLLM achieves the lowest error (Overall mean 0.0616 vs. CUT3R's 0.1122), evidencing genuinely metric-consistent geometry—a property the authors attribute partly to semantic co-supervision.

Limitations and open questions

The paper concedes three limitations directly. First, performance depends on the quality of the external geometric encoder; the framework activates priors but does not improve reconstruction itself, so stronger backbones could shift results. Second, sparse supervision, though sufficient for activation, yields coarse fine-grained geometry and blurry semantic boundaries compared with dense supervision. Third, whether MLLMs can internalize geometric perception without any external geometric prior remains open, as does scaling the pre-training objective toward denser or more precise supervision. Evaluation breadth is also bounded: all quantitative benchmarks are indoor scenes (ScanNet/EmbodiedScan), leaving outdoor and dynamic-scene transfer unverified.

Conclusion

GAP-MLLM reframes RGB-only 3D perception as a training problem rather than a representation problem: a sparse pixel-level geometry–semantics pre-training stage plus per-layer gated fusion reliably activates implicit geometric priors, yielding large cross-task gains with fewer parameters than prior implicit-prior methods and, in some cases, surpassing explicit-3D baselines. The evidence supports the paper's claim that activation-oriented pre-training, not richer geometric input alone, is a decisive factor for spatial perception in MLLMs.