Spatial-Grounded Fusion in Multimodal Systems

• Spatial-Grounded Fusion is a multimodal design strategy that constrains feature aggregation by enforcing explicit spatial relationships to maintain modality-specific alignment.
• It integrates diverse grounding mechanisms such as co-registration, token alignment, and spatial-frequency decomposition to improve model interpretability and performance.
• Empirical studies show that spatial-grounded fusion enhances performance in 3D vision-language, autonomous driving, and image restoration by mitigating feature misalignment and error accumulation.

Spatial-Grounded Fusion is a multimodal design principle in which fusion is constrained by explicit spatial structure rather than performed as unconstrained feature mixing. Across recent work, this structure may be metric geometry, co-registered tissue coordinates, frame-consistent token layouts, image gradients, wavelet sub-bands mapped back to spatial grids, or map cells endowed with probabilistic semantics. In that sense, Spatial-Grounded Fusion is not a single architecture but a recurring strategy for making cross-modal aggregation respect where information originates, how it aligns, and which spatial relations are valid (Zhang et al., 28 Mar 2026, Liu et al., 12 Feb 2026, Sitdhipol et al., 26 Jul 2025, Li et al., 5 Feb 2026).

1. Conceptual basis

A common thread in the literature is the claim that naive concatenation, late fusion, or globally unconstrained attention often discards or corrupts spatial correspondence. In 3D vision-language modeling, deep-only geometry fusion is reported to lose fine-grained cues because shallow geometry layers preserve local spatial precision whereas deeper layers capture broader structure; fusing only the deepest geometry features therefore creates a bottleneck for spatial understanding (Zhang et al., 28 Mar 2026). In active geometry integration, passive global mixing is explicitly criticized for inducing semantic-geometry misalignment and redundant signals, motivating selective retrieval of geometry conditioned on the model’s internal reasoning state (Li et al., 5 Feb 2026).

The same logic appears in other domains. In RGB-thermal semantic segmentation, indiscriminate dual-modality fusion is described as neglecting modality differences under varying lighting conditions, leading to a design in which spatial weights and confidence gates modulate cross-spectral residual exchange (Li et al., 2023). In infrared-visible fusion under degraded conditions, decoupled “pre-enhancement → fusion” pipelines are described as causing feature mismatch and error accumulation, which motivates end-to-end spatial-frequency joint optimization guided by degradation-aware prompts (Zhang et al., 5 Sep 2025). In pathology and spatial transcriptomics, hard co-location between spots and histology patches is treated as the basis for coherent multimodal prognosis modeling, because morphology and molecular programs must be fused at matched tissue positions rather than at slide level only (Liu et al., 12 Feb 2026).

This suggests that Spatial-Grounded Fusion is best understood as a constraint class: the fusion rule is conditioned by spatial validity, not merely by feature similarity.

2. Grounding mechanisms and mathematical forms

The grounding signal varies by problem, but several recurrent mechanisms appear. One family uses explicit co-registration or projection. PathoSpatial builds bags of paired instances by extracting a whole-slide patch centered at the same coordinates as each spatial transcriptomics spot, then models within-modality spatial dependencies before prototype-based cross-modal aggregation (Liu et al., 12 Feb 2026). Fusion-Poly uses 3D polyhedral boxes projected into camera views, then refines matched 3D–2D detections by minimizing an image-projection objective based on $1-\mathrm{IoU}$ between projected and observed 2D boxes (Wu et al., 9 Mar 2026). FP-LGN defines a grounded likelihood $p_\theta(r\mid x,\ell,M)$ over map locations, turning spatial language into a probabilistic field over a discretized environment (Sitdhipol et al., 26 Jul 2025).

A second family grounds fusion through token alignment and constrained attention. SpatialStack extracts geometry features from multiple internal layers of VGGT, aligns them to merged visual tokens, and injects them into early LLM layers by masked additive fusion on the vision-token slice only (Zhang et al., 28 Mar 2026). SpaceMind uses a camera token as an active guiding modality: it biases spatial keys and values, predicts query-independent token importance, and gates the fused representation before passing it to the LLM (Zhao et al., 28 Nov 2025). GeoThinker goes further by enforcing frame-strict cross-attention, so semantic tokens can only query geometry from the corresponding frame, while Importance Gating adds a learned per-frame attention bias (Li et al., 5 Feb 2026).

A third family grounds fusion in spatial structure represented by local neighborhoods, gradients, or spatially indexed sub-bands. AngularFuse constrains fusion by both gradient magnitude and direction, using a reference gradient field and an angle-aware loss so that fused edges are not only strong but correctly oriented (Liu et al., 14 Oct 2025). SFAFNet learns spatially variant low-pass filters per row, decomposes features into low- and high-frequency components, and then uses statistically grounded gating plus cross-attention to fuse spatial and frequency-domain features (Gao et al., 20 Feb 2025). ISFM similarly couples multi-scale frequency fusion with frequency-guided spatial interaction rather than treating spatial and frequency pathways as independent streams (Zhu et al., 4 Feb 2026).

Across these variants, the mathematical forms differ—projection operators, masked attention, wavelet decompositions, likelihood maps, spatial masks, or residual gates—but each imposes an explicit admissibility structure on multimodal exchange.

3. Architectural patterns across domains

One major pattern is hierarchical geometry-language fusion. SpatialStack progressively aligns multi-level geometric features with the language backbone and reports that shallow geometry injection improves low-level perception while deeper geometry injection benefits higher-level spatial reasoning (Zhang et al., 28 Mar 2026). SpaceMind preserves the visual token shape expected by the multimodal LLM, but conditions fusion on camera embeddings via biasing, weighting, and gating (Zhao et al., 28 Nov 2025). SpatialGeo takes a different route, adding a geometry-centric encoder to a CLIP-based MLLM and interleaving geometry-aware tokens with semantic tokens after hierarchical adapter aggregation; random feature dropping is used to prevent trivial reliance on CLIP alone (Guo et al., 21 Nov 2025). GeoThinker departs from passive mixing entirely by turning geometry into selectively retrievable evidence inside the LLM (Li et al., 5 Feb 2026).

A second pattern is modality-specific preprocessing followed by spatially grounded aggregation. In audio-visual navigation, the Audio Spatial feature Encoder computes an intensity-weighted latent audio spatial state $e_t$, and the Audio Spatial State Guided Fusion module then uses $e_t$ both as a query over audio-visual memory and as a gate on the fused output (Zhou et al., 2 Apr 2026). In audio-based 3D visual grounding, Audio-3DVG detects object mentions from speech and then applies audio-guided self- and cross-attention over point-cloud object embeddings that already include centers and sizes, thereby making relational speech cues act on spatial object structure (Cao-Dinh et al., 1 Jul 2025). RSFNet follows a comparable principle for RGB-thermal segmentation: one modality produces spatial weights for the other, while image-level confidence scores regulate how much residual cross-modal content should be injected (Li et al., 2023).

A third pattern is dual-domain spatial-frequency fusion. FUSION processes each RGB channel with wavelength-aware spatial kernels, CBAM attention, FFT-based frequency attention, and a Frequency Guided Fusion module, then performs inter-channel fusion and adaptive channel recalibration (Walia et al., 1 Apr 2025). SFDFusion runs a spatial branch and a frequency branch in parallel, uses a Dual-Modality Refinement Module to extract complementary spatial cues, a Frequency Domain Fusion Module to fuse amplitude and phase, and a frequency-domain loss tied to saliency regions (Hu et al., 2024). W-DUALMINE uses dense reliability maps, a global-context spatial expert, a wavelet-domain frequency expert, and a soft gradient-based arbitration mechanism, before reconstructing the fused image as an average-based base plus a bounded residual (Islam, 13 Jan 2026). GD$^2$Fusion combines prompt-guided frequency extraction with Guided Spatial Modality-Aggregated Fusion, explicitly coupling degradation perception, frequency suppression, and spatial aggregation (Zhang et al., 5 Sep 2025).

4. Empirical performance and application domains

In autonomous driving, a camera-LiDAR fusion method using Transformer modules at multiple resolutions reports higher driving and infraction scores on challenging adversarial benchmarks, with 8% and 19% improvement in driving scores over TransFuser on the Longest6 and Town05 Long benchmarks, respectively (Lai-Dang et al., 2023). In 3D multi-object tracking, Fusion-Poly achieves 76.5% AMOTA on the nuScenes test set by combining synchronized and asynchronous LiDAR-camera observations within a spatial-temporal fusion pipeline (Wu et al., 9 Mar 2026).

In semantic perception, RSFNet reports 75.9/56.2 mAcc/mIoU on MFNet and 80.5 mIoU on PST900 with its RGB-thermal design (Li et al., 2023). In audio-visual embodied navigation, ASGF-Nav reaches 63.3/76.5/36.9 SPL/SR/SNA on unheard tasks in Replica and 52.2/66.4/29.9 on unheard tasks in Matterport3D, outperforming prior baselines in the generalization setting emphasized by the paper (Zhou et al., 2 Apr 2026).

In image fusion and restoration, FUSION reports a highest PSNR of 23.717 dB and SSIM of 0.883 on UIEB, together with LPIPS of 0.112 and UIQM of 3.414, while using 0.28M parameters (Walia et al., 1 Apr 2025). AngularFuse reports EN 7.122, SD 47.429, SF 12.596, and AG 4.300 on MSRS, and attributes gains to the combination of complementary masking, fine-grained reference synthesis, and angle-aware supervision (Liu et al., 14 Oct 2025). SFDFusion reports 0.14M parameters, 42.81 GFLOPs, about 0.003 s per image, and an M3FD downstream detection result of mAP@0.5 = 0.795 (Hu et al., 2024). SFAFNet reports 34.25 dB / 0.971 on GoPro and 31.92 dB / 0.949 on HIDE for its larger variant, while also reducing FLOPs relative to a frequency-only baseline (Gao et al., 20 Feb 2025).

In 3D multimodal reasoning, SpatialStack reports 67.5 average on VSI-Bench for its 5B model and 69.14 as a cross-benchmark average over VSI-Bench, SPAR-Bench, BLINK-Spatial, and CV-Bench (Zhang et al., 28 Mar 2026). SpaceMind reports 69.6 on VSI-Bench, 61.1 EM@1 on SQA3D, and 67.3 overall on SPBench (Zhao et al., 28 Nov 2025). GeoThinker reports a peak score of 72.6 on VSI-Bench (Li et al., 5 Feb 2026). SpatialGeo reports 52.49 average accuracy on SpatialRGPT-Bench and states an improvement of at least 8.0% over state-of-the-art models with approximately 50% less memory cost during inference (Guo et al., 21 Nov 2025).

5. Interpretability, uncertainty, and reliability weighting

A distinctive feature of this research area is that spatial grounding is often tied to interpretability and uncertainty rather than to alignment alone. PathoSpatial uses prototype banks for pathology and spatial transcriptomics, performs gated attention over prototype-conditioned features, and then decomposes risk at the prototype level; histology prototypes align with interpretable tissue regions, while spatial transcriptomics prototypes are annotated by differential expression and over-representation analysis (Liu et al., 12 Feb 2026). FP-LGN takes an explicitly probabilistic route by estimating $p_\theta(r\mid x,\ell,M)$ and then combining multiple human language observations and robot sensor likelihoods in a Bayesian posterior; it reports mean NLL 0.384, close to expert-designed rules at 0.387, but with lower standard deviation, and reduces mean search steps to 1054 versus 2021 for robot-only sensing (Sitdhipol et al., 26 Jul 2025).

Reliability weighting is another common motif. W-DUALMINE learns dense spatial reliability maps $w_1^s,w_2^s$ that modulate each modality before expert fusion, then uses a residual-to-average formulation to preserve global correlation while enhancing local structure (Islam, 13 Jan 2026). RSFNet supervises modality confidence scores with saliency-derived pseudo-labels and uses those scores to scale residual fusion strength (Li et al., 2023). Audio-3DVG introduces Object Mention Detection as a multi-label problem so that relational anchors named in speech are made explicit before grounding, rather than left implicit in a monolithic audio embedding (Cao-Dinh et al., 1 Jul 2025).

Taken together, these works show that Spatial-Grounded Fusion is frequently coupled to quantitative notions of trust: geometric reliability, linguistic uncertainty, prototype salience, or modality confidence.

6. Limitations, misconceptions, and directions of development

A common misconception is that any multimodal fusion layer qualifies as spatially grounded. The literature argues otherwise. SpatialStack shows that naive multi-layer fusion in the vision pathway underperforms carefully synchronized geometry-language fusion (Zhang et al., 28 Mar 2026). GeoThinker reports that indiscriminate fusion creates redundancy and that fusing every layer can collapse decoding behavior (Li et al., 5 Feb 2026). GD$^2$Fusion argues that manually chained pre-enhancement and fusion pipelines accumulate errors because they do not optimize degradation handling and fusion jointly (Zhang et al., 5 Sep 2025). RSFNet shows that simple sum or concatenation is weaker than confidence-gated residual spatial fusion (Li et al., 2023).

Another recurrent limitation is dependence on alignment quality. PathoSpatial explicitly notes co-registration errors as a risk because the method assumes accurate WSI-ST alignment (Liu et al., 12 Feb 2026). Fusion-Poly remains sensitive to calibration and projection quality despite its robustness to asynchronous sensing (Wu et al., 9 Mar 2026). SpaceMind still faces monocular scale ambiguity because it reasons from RGB only (Zhao et al., 28 Nov 2025). GeoThinker depends on geometry encoder quality and token-grid alignment (Li et al., 5 Feb 2026).

Future directions in the literature mostly strengthen or soften the grounding constraint rather than abandoning it. PathoSpatial proposes soft spatial kernels for nearby but non-identical positions (Liu et al., 12 Feb 2026). Audio-visual navigation work points toward explicit spatial priors such as learned DoA, depth, occupancy, or SLAM-derived maps (Zhou et al., 2 Apr 2026). SpatialStack suggests alternative geometry backbones and learnable fusion operators beyond masked addition (Zhang et al., 28 Mar 2026). SpaceMind points to explicit camera parameter encoders and temporal motion cues (Zhao et al., 28 Nov 2025). GD$^2$Fusion suggests denser VLM localization and uncertainty-aware guidance for degraded multimodal fusion (Zhang et al., 5 Sep 2025).

The broad trajectory is therefore not toward generic larger fusion blocks, but toward more selective, better aligned, and more uncertainty-aware interaction rules. Spatial-Grounded Fusion, in this sense, has become a general framework for deciding not only how modalities should be combined, but where, when, and under what spatial constraints they are allowed to influence one another.