Camera-Guided Modality Fusion

• Camera-Guided Modality Fusion is a paradigm that integrates camera pose and geometry into VLMs to enhance spatial reasoning from RGB inputs.
• The CGMF module employs dual encoders and structured fusion steps, including camera-conditioned biasing and token reliability weighting, to align visual and geometric features.
• Empirical results demonstrate that CGMF achieves state-of-the-art performance on spatial reasoning benchmarks, providing data-efficient 3D awareness without external depth data.

Camera-Guided Modality Fusion (CGMF) is a paradigm for integrating camera pose and geometry information directly into the fusion process of vision-LLMs (VLMs), enabling improved spatial reasoning from purely monocular (RGB) vision inputs. It introduces an explicit, structured mechanism that conditions the multimodal fusion stage on the global and local geometric context provided by the camera, moving beyond conventional shallow feature fusion to enable genuinely spatially grounded language understanding (Zhao et al., 28 Nov 2025).

1. Foundations and Motivation

Camera-Guided Modality Fusion was formalized in the context of the SpaceMind architecture, targeting the challenge that contemporary VLMs, even those trained on large-scale multimodal data, lack true 3D spatial awareness. Existing 3D-aware methods are dependent either on external 3D data or employ shallow, parameter-efficient fusion layers atop geometry encoders. These approaches are limited in their ability to infer physical relationships such as distance estimation, cross-view consistency, and spatial navigation based exclusively on RGB images. CGMF addresses this by making the camera representation an active, gating modality within the token fusion process, aligning all reasoning with the actual scene geometry and observer viewpoint. This is especially critical in applications where only monocular video is available and no depth or multi-view triangulation can be performed (Zhao et al., 28 Nov 2025).

2. CGMF Module: Architectural Overview

The CGMF module is integrated between the dual visual encoders and the LLM backbone. SpaceMind employs a dual-encoder setup:

• 2D Visual Encoder (InternViT-300M): Processes RGB frames into patchwise semantic tokens $f_v \in \mathbb{R}^{N \times M_v \times d_v}$, optimized for high-level object recognition.
• Spatial Understanding Encoder (VGGT): Generates geometry-rich spatial tokens $f_s \in \mathbb{R}^{N \times M_s \times d_s}$ and per-frame camera tokens $f_c \in \mathbb{R}^{N \times 1 \times d_s}$ summarizing pose and scene structure.

The CGMF module takes as input $(f_v, f_s, f_c)$, aligning and fusing them via a sequence of projection, camera-conditioned biasing, token reliability weighting, cross-attention, and final camera-gated fusion. The fused output $$f_{\mathrm{fused}} \in \mathbb{R}^{N \times M_v \times d_v}$$ is then passed to the LLM (InternVL3-8B + LoRA adapters).

3. Technical Details of Fusion

The CGMF fusion pipeline consists of the following steps (Zhao et al., 28 Nov 2025):

1. Linear Projection to Shared Space: All token streams are layer-normalized and projected to a joint attention space of dimension $d_a$. This yields:

$$Q = P_Q(\mathrm{LN}(f_v)),\; K = P_K(\mathrm{LN}(f_s)),\; V = P_V(\mathrm{LN}(f_s)),\; C = P_C(f_c)$$

2. Camera-Conditioned Biasing: Each spatial token is concatenated with its corresponding camera token. An MLP computes an additive bias $B_g$:

$B_g = \mathrm{MLP}([f_s, f_c])$

This bias is added to both key and value tokens:

$K \leftarrow K + B_g,\quad V \leftarrow V + B_g$

By introducing such bias, region semantics in the spatial tokens become viewpoint-aware, which is essential for disambiguating symmetric structures or occlusions.

3. Query-Independent Geometry Weighting: A separate MLP produces a scalar reliability weight $W_t = \sigma(\mathrm{MLP}(f_s))$ per spatial token, which modulates the value tensor:

$V \leftarrow V \odot W_t$

This mechanism prioritizes high-confidence, unoccluded, or structurally salient regions irrespective of the query's attention.

4. Cross-Attention with Camera Insertion: The camera token $C$ is prepended to both key and value sequences. Cross-attention is performed using:

$K' = [C; K],\quad V' = [C; V]$

$\hat{f} = \mathrm{Attn}(Q, K', V')$

This ensures all patchwise associations between semantic and geometry tokens are referenced to the observed viewpoint.

5. Camera-Conditioned SwiGLU Gating: The cross-attended representations are projected back to the visual feature dimension, and a gated fusion is implemented using the Swish-Gated Linear Unit (SwiGLU) mechanism parameterized from the global camera embedding:

$$u = P_{g,1}(\bar{C}),\quad v = P_{g,2}(\bar{C}),\quad g = \mathrm{Swish}(u) \odot v$$

The final fusion is a residual-gated add to the original visual features:

$$f_\mathrm{fused} = P_L(f_\mathrm{proj}) \odot g[:,\mathrm{None},:] + f_v$$

These steps realize three key inductive biases: explicit camera-biasing, token-level geometry reliability, and camera-gated fusion, all acting prior to VLM decoding.

4. Training Scheme and Empirical Results

The entire SpaceMind stack, including CGMF, is fine-tuned end-to-end on question-answering and spatial reasoning datasets (VLM−3R-data, ViCA-322K, SQA3D). The only loss imposed is cross-entropy over target language tokens; encoders are kept frozen, and LoRA adapters of rank 256 are applied to the LLM for parameter-efficient tuning. No auxiliary contrastive or explicit geometric supervision is imposed within CGMF.

Empirically, SpaceMind surpasses both open and proprietary benchmarks on VSI-Bench and SPBench and achieves state-of-the-art results on SQA3D, providing evidence that the CGMF module is an effective inductive bias for spatially grounded intelligence in VLMs (Zhao et al., 28 Nov 2025).

5. Relationship to Other Spatial Fusion Paradigms

Prior approaches for fusing visual and structural information in multimodal models fall into two main categories: (1) shallow feature fusion (concatenation, sum) of geometry from depth or pose encoders with RGB-based features, or (2) reliance on explicit 3D data representations. CGMF distinguishes itself by conditioning the fusion not only on local patch tokens, but also on the global camera context, and by introducing query-independent weighting to handle unreliable or occluded geometry regions. This represents a principled architectural advance, permitting strong 3D spatial reasoning without access to direct depth or multi-view data (Zhao et al., 28 Nov 2025).

Integration of mid-level geometry abstractions—such as those produced by GAN-based semantic segmentation pipelines for architectural scene structure—can be naturally accomodated within the SpaceMind framework. These mid-level representations, when paired with camera-conditioned fusion, support downstream applications in 3D reconstruction, virtual navigation, and spatial design (Tas et al., 2023).

6. Implications for Spatial Intelligence and Applications

Camera-Guided Modality Fusion enables VLMs to perform physically meaningful spatial inference tasks—such as distance estimation, volumetric reasoning, and viewpoint-aware grounding—entirely from RGB streams. This unlocks new capabilities in fields ranging from autonomous navigation in interior architectural environments to question answering about spatial configurations, all without the need for external 3D data or sensor fusion. The paradigm can be generalized to domains where viewpoint context is critical for semantic grounding.

A plausible implication is that explicit camera-guided fusion mechanisms will become a normative design pattern in next-generation VLMs tasked with spatial reasoning, as they provide data-efficient, geometry-aware inductive biases supporting robust generalization from visual context (Zhao et al., 28 Nov 2025, Tas et al., 2023).