RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation

Abstract: Multimodal semantic segmentation has emerged as a powerful paradigm for enhancing scene understanding by leveraging complementary information from multiple sensing modalities (e.g., RGB, depth, and thermal). However, existing cross-modal fusion methods often implicitly assume that all modalities are equally reliable, which can lead to feature degradation when auxiliary modalities are noisy, misaligned, or incomplete. In this paper, we revisit cross-modal fusion from the perspective of modality reliability and propose a novel framework termed the Reliability-aware Self-Gated State Space Model (RSGMamba). At the core of our method is the Reliability-aware Self-Gated Mamba Block (RSGMB), which explicitly models modality reliability and dynamically regulates cross-modal interactions through a self-gating mechanism. Unlike conventional fusion strategies that indiscriminately exchange information across modalities, RSGMB enables reliability-aware feature selection and enhancing informative feature aggregation. In addition, a lightweight Local Cross-Gated Modulation (LCGM) is incorporated to refine fine-grained spatial details, complementing the global modeling capability of RSGMB. Extensive experiments demonstrate that RSGMamba achieves state-of-the-art performance on both RGB-D and RGB-T semantic segmentation benchmarks, resulting 58.8% / 54.0% mIoU on NYUDepth V2 and SUN-RGBD (+0.4% / +0.7% over prior best), and 61.1% / 88.9% mIoU on MFNet and PST900 (up to +1.6%), with only 48.6M parameters, thereby validating the effectiveness and superiority of the proposed approach.

• The paper presents a state space model that integrates uncertainty and consistency gating to enable reliability-aware fusion for multimodal semantic segmentation.
• It employs dual fusion streams—global RSGMB for long-range dependencies and local LCGM for fine spatial refinement—enhancing both accuracy and efficiency.
• Empirical results on datasets like NYUDepth V2 and SUN-RGBD demonstrate improved mIoU and reduced computational complexity compared to conventional methods.

RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation

Motivation and Context

Multimodal semantic segmentation leverages signal complementarity among RGB, depth, and thermal modalities to improve dense prediction under adverse or ambiguous visual conditions. However, existing fusion strategies typically treat all modalities as equally reliable, leading to feature degradation in the presence of noise or misalignment in auxiliary modalities. This work directly addresses that challenge by proposing reliability-aware fusion for multimodal segmentation, incorporating uncertainty and consistency estimation into a state space modeling framework.

Visualization of input modality distributions (Figure 1) reveals strong statistical heterogeneity, with auxiliary signals exhibiting high unreliability across many spatial regions. This motivates an adaptive, reliability-conditioned fusion paradigm.

(Figure 1)

Figure 1: Cross-modal reliability visualization for RGB, depth, and thermal inputs; regions with unreliable cues are prominent in auxiliary signals.

Architecture Overview

The proposed RSGMamba architecture (Figure 2) is structured as a U-shaped segmentation network with parallel weight-sharing encoders for RGB and auxiliary modalities, followed by a decoder. The core innovation resides in the dual-branch fusion mechanism at each semantic level, comprising:

• Global fusion via the Reliability-aware Self-Gated Mamba Block (RSGMB), responsible for long-range dependency modeling and reliability-aware cross-modal state fusion.
• Local fusion via the Local Cross-Gated Modulation (LCGM), refining spatial granularity and preserving boundary structures.

Outputs from both branches are aggregated and supplied to the decoder for semantic mask generation.

Figure 2: Overall architecture of RSGMamba; cross-modal fusion is decomposed into global (RSGMB) and local (LCGM) branches.

Reliability-Aware Self-Gated Mamba Block (RSGMB)

The RSGMB module (Figure 3) leverages state space modeling (Mamba), augmenting it with token-wise reliability and cross-modal consistency gating to regulate the flow of cross-modal information.

Key design elements:

• Low-rank projections constrain fusion capacity and reduce parameter overhead, serving as an implicit regularizer to mitigate overfitting.
• Uncertainty-based gating: Each modality is equipped with an uncertainty-aware gate, outputting a reliability mask via an MLP over normalized local responses.
• Consistency gating: A parallel gate estimates region-wise cross-modal consistency, facilitating dynamic selection of trustworthy auxiliary cues.
• Reliability-controlled cross-state readout: The effective state space output matrices are computed as weighted combinations modulated by reliability and consistency, enabling selective, token-level cross-modal feature aggregation.

When auxiliary data are noisy or inconsistent, fusion is suppressed, falling back to unimodal operation.

Figure 3: (a) Parallel processing and reliability-controlled fusion; (b) RSG-SS module; (c) Internal logic of the reliability gating mechanism.

Local Cross-Gated Modulation (LCGM)

LCGM (Figure 4) addresses local contextual refinement lost in global models. It consists of linear projections and shallow convolutions to produce gating masks for each modality, which are then used to modulate features via elementwise multiplication. Depthwise convolution augments local pattern extraction. Cross-modal interaction is realized by inter-modality multiplication and residual aggregation, yielding a spatially enhanced local feature.

Figure 4: LCGM structure, aligning and modulating RGB and auxiliary features, followed by depthwise convolution and cross-modality fusion.

Empirical Evaluation

Datasets

Experiments span both RGB-D (NYUDepth V2, SUN-RGBD) and RGB-Thermal (MFNet, PST900) datasets, covering a mixture of indoor and outdoor, low-illumination, and challenging scenes.

Main Results

RSGMamba sets a new state-of-the-art on all four benchmarks:

• NYUDepth V2: 58.8% mIoU (+0.4% over prior best) with only 48.6M parameters.
• SUN-RGBD: 54.0% mIoU (+0.7%).
• MFNet: 61.1% mIoU (+1.6%).
• PST900: 88.9% mIoU (+0.3%).

Importantly, this is achieved with significantly lower computational complexity compared to attention-based and large-scale pretrained models.

Qualitative segmentation comparisons (Figure 5, Figure 6) highlight superior structure preservation, cleaner boundaries, and robust performance in noisy and ambiguous regions.

(Figure 5)

Figure 5: Qualitative results on NYUDepth V2 and SUN-RGBD, showing improved boundary sharpness and fine-structure preservation.

(Figure 6)

Figure 6: MFNet and PST900 results; superior object delineation and reduced spurious detections under low-illumination and thermal ambiguity.

Ablation Analysis

Detailed ablations dissect the contributions of:

• RSGMB: Provides +1.3% mIoU (global reliability-aware fusion).
• LCGM: Adds +1.7% mIoU (local detail refinement).
• Reliability and consistency gating: Each component further improves mIoU, with a combined gain of +1.5%.
• Low-rank projections (LoRA): Slight accuracy gain with parameter and FLOP reduction.

Comparisons to Cross-Attention and Cross-Mamba demonstrate RSGMB’s dynamic, reliability-aware design as the decisive factor in performance gains.

Visualization of feature maps (Figure 7) confirms that RSGMB yields focused, structured activations, successfully suppressing regions with unreliable auxiliary signals.

(Figure 7)

Figure 7: Feature responses under various fusion schemes; RSGMB yields more coherent and semantically distinct activations.

Implications and Future Perspectives

• Theoretical: The explicit integration of uncertainty and cross-modal consistency in SSM-based models sets a precedent for more robust multimodal representation learning beyond the pixel fusion paradigm.
• Practical: The linear complexity and token-adaptive behavior of RSGMamba render it suitable for real-time and edge deployment scenarios where auxiliary modalities routinely suffer from sensor noise and misalignment.
• Future Directions: This reliability-aware control could be extended to multi-modal transformers, video segmentation (temporal consistency), and prompt-based multimodal adaptation. Furthermore, task-level uncertainty estimation for downstream active perception and decision-making could directly leverage the output reliability maps.

Conclusion

RSGMamba represents a substantial advance in the design of efficient, robust, and adaptive multimodal semantic segmentation systems. By incorporating reliability- and consistency-driven modulation into state space fusion, it both advances segmentation accuracy and mitigates the negative transfer endemic to noisy auxiliary modalities. The architecture’s parameter efficiency, competitive computational profile, and strong empirical results confirm the merits of explicitly modeling reliability in multimodal fusion pipelines.

---

Reference:

"RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation" (2604.12319)