DGSSM: Diffusion guided state-space models for multimodal salient object detection

Abstract: Salient object detection (SOD) requires modeling both long-range contextual dependencies and fine-grained structural details, which remains challenging for convolutional, transformer-based, and Mamba-based state space models. While recent Mamba-based state space approaches enable efficient global reasoning, they often struggle to recover precise object boundaries. In contrast, diffusion models capture strong structural priors through iterative denoising, but their use in discriminative dense prediction is still limited due to computational cost and integration challenges. In this work, we propose DGSSM, a diffusion-guided state space (Mamba) framework that formulates multimodal salient object detection as a progressive denoising process. The framework integrates diffusion structural priors with multi-scale state space encoding, adaptive saliency prompting, and an iterative Mamba diffusion refinement mechanism to improve boundary accuracy. A boundary-aware refinement head and self-distillation strategy further enhance spatial coherence and feature consistency. Extensive experiments on 13 public benchmarks across RGB, RGB-D, and RGB-T settings demonstrate that DGSSM consistently outperforms state-of-the-art methods across multiple evaluation metrics while maintaining a compact model size. These results suggest that diffusion-guided state space modeling is an effective and generalizable paradigm for multimodal dense prediction tasks.

• The paper introduces a unified DGSSM framework that fuses state-space encoders with diffusion-guided priors for efficient and accurate multimodal salient object detection.
• The method incorporates adaptive saliency prompting, multi-scale selective scanning, and iterative diffusion refinement to enhance boundary precision and global structure.
• Experimental results show state-of-the-art performance across RGB, RGB-D, and RGB-T benchmarks while maintaining a computationally efficient and compact model architecture.

DGSSM: Diffusion-Guided State-Space Models for Multimodal Salient Object Detection

Problem Formulation and Motivation

Salient object detection (SOD) demands capturing both long-range dependencies and fine-grained structural details, attributes that conventional CNN and transformer-based networks often struggle to balance due to their limited receptive fields or computational inefficiencies. Recent advancements with Mamba-based state-space models (SSMs) offer efficient global modeling through linear-complexity selective state transitions, yet these approaches are prone to producing imprecise saliency boundaries and fragmented object localization. In parallel, diffusion models have demonstrated robust structural priors via iterative denoising, effectively yielding strong spatial coherence and sharp boundaries in generative tasks, but their direct use for discriminative, multimodal dense prediction tasks incurs significant computational cost and detachment from efficient feature pipelines.

The paper "DGSSM: Diffusion guided state-space models for multimodal salient object detection" (2604.17585) proposes a unified framework that fuses Mamba-type state-space encoders with diffusion-based priors, reformulating multimodal SOD as a progressive denoising process. DGSSM explicitly integrates compact diffusion-guided structural cues at multiple levels of state-space encoding and introduces a series of architectural innovations—diffusion structural priors, adaptive saliency prompting, hierarchical multi-scale selective scanning, boundary-aware refinement, and iterative diffusion-based correction—to address core SOD challenges across RGB, RGB-D, and RGB-T benchmarks.

Architecture and Technical Contributions

DGSSM operates within an encoder-decoder paradigm where each component is functionally driven to leverage both deterministic and stochastic priors for dense multimodal predictions. The overall system can be summarized as follows:

• Diffusion Structural Prior (DSP): A compact latent encoding is extracted from a pretrained diffusion model and partially denoised for a limited number of steps to yield a frozen prior embedding $z_{t^*}$. This prior captures global layout and structural cues, which are injected into the state-space encoder at each hierarchical level through feature concatenation. This hybridization improves the initialization of sequential feature extraction and biases the encoder toward global object consistency.
• Adaptive Saliency Prompting (ASP): At every encoder layer, a prompt vector is derived via global average pooling and lightweight projection, conditioning state transitions adaptively with saliency priors and context sensitivity.
• Multi-Scale Selective State Space Scan (MS-SS): Encoding utilizes directional, scale-selective scans across spatial axes, capturing long-range dependencies while permitting aggregation across scales, improving context modeling of both coarse and fine structures.
• Deterministic-Stochastic Interaction: Features from the MS-SS encoder are fused at each stage with projected diffusion priors using a learnable mixing coefficient, allowing controllable transfer of global structure from the stochastic diffusion process into the deterministic backbone.
• Decoder and Boundary-Aware Refinement Head (BARH): The decoder upsamples multiscale features to a coarse saliency prediction, post-processed by a Sobel-based edge extraction and fusion module. This BARH explicitly enhances boundary localization.
• Iterative Mamba Diffusion Refinement (IMDR): The output of the BARH is further iteratively refined. At each step, a lightweight diffusion-guided correction is applied, regulated by progressive losses at each iteration to stabilize training and improve prediction granularity and boundary adherence.
• Self-Distillation: The deepest encoder features serve as a teacher for intermediate representations, promoting hierarchical consistency via an auxiliary knowledge distillation loss.

  Figure 1: Overall architecture of DGSSM illustrating the interaction of diffusion priors with hierarchical state space encoding, adaptive prompting, multi-scale scanning, and iterative boundary refinement.

Experimental Evaluation

DGSSM is extensively evaluated on thirteen benchmarks encompassing RGB, RGB-D, and RGB-T SOD scenarios. Quantitative results show that DGSSM achieves superior performance compared to CNN, Transformer, and SOTA SSM-based approaches across multiple metrics, including structure measure ($S_m$), mean F-measure ($F_m$), and mean E-measure ($E_m$).

Noteworthy findings include:

• On RGB SOD tasks, DGSSM reports best-in-class results with clear improvements in both object completeness and boundary precision, outperforming transformer-based methods like VSCode-S and Samba while using substantially fewer parameters. The improvement in $E_m$ and $S_m$ on datasets such as DUTS and ECSSD signals stronger structural and spatial consistency.
• In the RGB-D setting, DGSSM demonstrates resilience to noisy, incomplete, or modality-shifted depth cues, delivering state-of-the-art scores on all principal datasets (e.g., NJUD, NLPR, SIP, DUTLF-D). Its parameter efficiency is emphasized in comparison with transformer backbones, due largely to the deployment of diffusion refinement in compact representational spaces.
• For RGB-T tasks, DGSSM dominates low-illumination and thermal-dominant scenes, outperforming even recent multimodal transformer and SSM competitors on all major metrics and datasets (VT821, VT1000, and VT5000).

  Figure 2: DGSSM yields Pareto-superior accuracy vs. computational cost compared to state-of-the-art SOD models, with bubble size proportional to model size.

Qualitative results show that DGSSM minimizes fragmented activation, consistently yields sharper object edges, and is robust to background clutter and adverse modal distortions.

Figure 3: Visual comparison reveals DGSSM's superior boundary definition and spatial coherence relative to competing methods across diverse scenes.

Ablation and Component Analysis

Systematic ablations confirm that each architectural innovation contributes independently toward improved performance. The greatest gain comes from iterative diffusion refinement, but ablations of DSP, ASP, MS-SS, and BARH all show non-trivial additive effects. Specifically, the addition of the DSP provides an early structural advantage, while BARH is particularly impactful for boundary quality.

Theoretical and Practical Implications

DGSSM exemplifies a hybrid approach to dense prediction: combining the efficiency of state-space global reasoning (Mamba-based SSMs) with the structural expressivity of diffusion-generated priors results in a model that is both computationally lean and capable of preserving boundary accuracy, a long-standing challenge in SOD. The iterative correction mechanism suggests a new direction for plugging stochastic generative priors into deterministic dense prediction pipelines.

On the practical side, DGSSM’s performance and efficiency profile enables deployment in resource-constrained or real-time applications—such as robotics, autonomous surveillance, or multi-modal interactive systems—where accurate salient region detection under real-world, multimodal conditions is key.

Future Directions

Prospective avenues include extending DGSSM to video SOD with temporal diffusion priors, instance-level segmentation, and other multimodal tasks where global structure and local coherence are equally vital. Further research on dynamic allocation of denoising steps and on-task adaptive prior learning could generalize the framework’s application to a broader range of dense prediction problems.

Conclusion

DGSSM introduces a formally principled integration of diffusion-guided priors and state-space visual modeling, providing robust, scalable, and computationally efficient multimodal SOD. Quantitative and qualitative evidence supports its outperformance of CNN-, Transformer-, and SSM-based baselines with a smaller parameter footprint and stronger saliency boundary localization, positioning the framework as an effective paradigm for subsequent research in dense multimodal vision tasks.

(2604.17585)