RGB-D Video Salient Object Detection

Updated 16 May 2026

RGB-D Video Salient Object Detection is a computational task that integrates RGB, depth, and flow cues to produce temporally coherent salient masks across video frames.
It leverages advanced fusion methodologies, including selective cross-modal attention and transformer-based architectures, to improve detection precision as evidenced by high S-measure and F-measure scores.
Practical implementations benefit from real-time efficiency and robustness on diverse benchmarks like RDVS and ViDSOD-100, though challenges remain in depth quality variability and computational demands.

RGB-D Video Salient Object Detection (VSOD) is a computational problem that targets the extraction of temporally and spatially coherent salient object masks from videos that include both visual (RGB) and depth (D) modalities. This task leverages appearance, geometry, and motion to localize objects that attract human attention over sequences. RGB-D VSOD has quickly emerged as a significant research area, driven by advances in multi-modal sensor technology, the proliferation of depth-enriched video datasets, and the demonstrated accuracy advantages attainable by integrating depth and motion cues. Recent work establishes that depth and flow modalities contribute complementary information for saliency—but their utility is context- and pixel-dependent, and optimal fusion demands sophisticated attentive frameworks and robust, generalizable architectures (He et al., 29 Jul 2025, Mou et al., 2023, Lin et al., 2024, Lu et al., 2022, Lin et al., 13 Nov 2025, Liu et al., 12 May 2026).

1. Task Formulation and Datasets

RGB-D VSOD aims to produce a binary mask sequence $\{M_t\}_{t=1}^T$ for a video, where each $M_t$ segments the salient object(s) in frame $t$ , integrating RGB frames $I^{(r)}_t$ , depth maps $I^{(d)}_t$ , and motion (often as optical flow $I^{(f)}_t$ ). Unlike RGB image or video-only SOD, this domain exploits 3D geometry per frame and dynamic temporal cues jointly. The inclusion of authentic, sensor-derived depth is critical, as synthetic depth (e.g., monocular prediction) underperforms on difficult scenes (Mou et al., 2023, Lin et al., 2024).

Several modern datasets have been introduced:

RDVS: 57 videos (4,087 frames) from a mix of synthetic and real depth sensors, with exhaustive per-frame object annotation based on eye-tracking (Mou et al., 2023).
ViDSOD-100: 100 videos (9,362 frames), real depth, dense binary masks, and broad category diversity; frames are thoroughly annotated, supporting both algorithm development and comprehensive evaluation (Lin et al., 2024).
DViSal: 237 videos, public RGB-D VSOD benchmark (Lin et al., 13 Nov 2025, Liu et al., 12 May 2026).

These datasets standardize split protocols, emphasize object- and attribute-level balance, and support comparative benchmarking with consistent metrics: S-measure ( $S_\alpha$ ), E-measure ( $E_\xi$ ), maximum F-measure ( $F_\beta$ ), and mean absolute error ( $M$ ).

2. Core Methodologies and Fusion Paradigms

RGB-D VSOD architectures are universally multi-branch or fusion-centric, employing modality-specific encoders (typically sharing ResNet-34/50, Res2Net, or transformer backbones) and dedicated fusion modules. Key fusion approaches include:

Selective Cross-Modal Fusion: SMFNet integrates a Pixel-level Selective Fusion (PSF) block, producing a spatial weight map $M_t$ 0 that selectively weights depth and flow features per pixel, with pseudo-supervision generated via competitive evaluation against saliency outputs of unimodal streams. This enables per-pixel selection based on modality reliability (He et al., 29 Jul 2025).
Attentive Triple-Fusion: ATF-Net maintains parallel RGB, depth, and flow U-Nets, with MEA (Encoder Aggregation) and MDA (Decoder Aggregation) modules effecting cross-modal channel and spatial attention-fusion at every encoder/decoder hierarchy. MEA computes channel-reduced fusions and deep cross-modal attention between depth-flow, while MDA applies multi-head spatial matching and progressive fusion (Lin et al., 2024).
Multi-Modal Attention and Refinement: DCTNet+ and its earlier version DCTNet prioritize RGB as main modality and introduce a Multi-Modal Attention Module (MAM)—a non-local block operating independently across RGB-depth and RGB-flow pairs, enhancing long-range interaction—and a Refinement Fusion Module (RFM) for progressive, attention-guided merging that suppresses modality-specific noise (Mou et al., 2023, Lu et al., 2022).
Quality-Aware Manipulation: DFM-Net for RGB-D and VSOD employs a Depth Quality-Inspired Feature Manipulation (DQFM) module, estimating depth/flow reliability using boundary alignment with the RGB stream and modulating cross-modal fusion weights to suppress low-quality input regions. Lightweight backbones and spatial attention yield real-time performance on resource-constrained devices (Zhang et al., 2022).

For transformer-based architectures built atop foundation models (SAM2):

SAM-DAQ: Freezes the SAM2 encoder, introducing depth-guided parallel adapters (DPAs) and a query-driven temporal memory (QTM) module where learnable frame- and video-level queries prompt the model; this unifies prompt-dependent VSOD under a prompt-free, temporally-coherent mask prediction regime (Lin et al., 13 Nov 2025).
M $M_t$ 1-SAM: Re-engineers LoRA adapters into a modality-aware, convolutional mixture-of-experts (MoE) within SAM2, with hierarchical feature aggregation and pseudo-guided initialization to approximate prompt-based memory without external cues. A cross-attention decoder enforces short-term temporal consistency (Liu et al., 12 May 2026).

3. Architecture Details and Training Paradigms

While model specifics vary, state-of-the-art RGB-D VSOD systems typically share a selection of the following components:

Component	Function	Example Models
Modality Encoders	Parallel streams for RGB, depth, and flow	SMFNet, DCTNet+, ATF-Net
Fusion Module	Cross-modal selective/pixel-wise/attention fusion	PSF (SMFNet), MAM+RFM (DCTNet+)
Attention Blocks	Multi-dim. or channel/spatial, non-local, gating	MSAM (SMFNet), MEA/MDA (ATF-Net)
Decoding Structure	U-Net/progressive upsampling, multi-level outputs	SMFNet, DCTNet+, ATF-Net
Temporal Modeling	Flow input, query memory, or attention-based	QTM (SAM-DAQ), memory (M $M_t$ 2-SAM)
Supervision	BCE + IoU hybrid losses, auxiliary side supervision	All leading models

Training is executed in multi-stage regimes: pretraining of unimodal or cross-modal branches, followed by joint fine-tuning. Batch size, learning rate, and augmentation strategies are largely conventional (SGD/Adam, multi-resolution crops/flips). In foundation model adaptation, only adapters or memory/query parameters are updated; backbone weights remain frozen (Lin et al., 13 Nov 2025, Liu et al., 12 May 2026). Auxiliary tasks—such as edge prediction (M $M_t$ 3-SAM) or coarse mask estimation (ATF-Net)—regularize mask sharpness and accelerate convergence.

4. Benchmark Results and Comparative Analysis

On public datasets, current methods demonstrate strong performance improvements relative to former RGB-only or image-based SOD techniques. Key results include:

SMFNet: On RDVS, $M_t$ 4, $M_t$ 5, $M_t$ 6; on DViSal, $M_t$ 7, $M_t$ 8, $M_t$ 9, with consistent superiority over 19 benchmarks (He et al., 29 Jul 2025).
DCTNet+: $t$ 0 on RDVS up to 0.836, $t$ 1, $t$ 2 post-fine tuning; best or top-3 across all five pseudo RGB-D VSOD sets (Mou et al., 2023).
ATF-Net: On ViDSOD-100, $t$ 3, $t$ 4, $t$ 5, outperforming all competitors from RGB-D SOD, VSOD, and VOS domains (Lin et al., 2024).
SAM-based models: M $t$ 6-SAM and SAM-DAQ achieve $t$ 7 (ViDSOD-100), $t$ 8 (DViSal), $t$ 9, with gains of $I^{(r)}_t$ 0– $I^{(r)}_t$ 1 and marked improvement over prior art, including prompt-free VSOD (Liu et al., 12 May 2026, Lin et al., 13 Nov 2025).
Efficiency models: DFM-Net reaches S_α = 0.898 at 20FPS (CPU) with only 8.5MB size (Zhang et al., 2022).

Ablation studies on module design (e.g., presence of PSF/MSAM, MEA/MDA, query memory, or DQFM) consistently show that targeted attention-fusion, modality-quality control, and temporally-aware adapters drive the bulk of performance improvements. Notably, depth stream ablation always induces significant metric drop, particularly on challenging real-depth datasets (Mou et al., 2023, Lin et al., 2024).

5. Analysis of Fusion and Attention Mechanisms

Effective cross-modal fusion in RGB-D VSOD is intrinsically context-sensitive, as the contributions of depth and motion vary spatially and temporally. Key findings regarding fusion and attention strategies include:

Pixel-Wise Selectivity: Pixel-level weighting (e.g., PSF in SMFNet) enables the network to default to flow or depth at each spatial location, outperforming equal or naive concatenation strategies (He et al., 29 Jul 2025).
Multi-Dimensional and Progressive Attention: Multi-dimensional attention (MSAM) and hierarchical integration (MEA/MDA in ATF-Net, MAM/RFM in DCTNet+) facilitate information exchange not just within but also across channels, spatial positions, and modalities (He et al., 29 Jul 2025, Lin et al., 2024).
Reliability-Weighted Fusion: Explicit quality estimation and suppression of unreliable regions (DFM-Net’s DQW/DHA) improves both accuracy and efficiency, especially on noisy or heterogeneous depth/flow sources (Zhang et al., 2022).
Prompt-Free Temporal Memory: Modern transformer-based pipelines with learned queries and temporal context memory demonstrate practical prompt elimination, allowing for effective zero-shot or “automatic” first-frame adaptation (Lin et al., 13 Nov 2025, Liu et al., 12 May 2026).
Ablation Insights: All top models demonstrate in controlled studies that introducing self- or cross-modal non-local attention, edge or boundary regularization, and multi-scale fusion consistently boosts both structure and pixel-wise metrics.

6. Limitations, Open Issues, and Future Research Directions

The field has progressed rapidly, but several challenges remain:

Depth Quality Variability: Model accuracy remains sensitive to the quality and consistency of input depth; consumer sensors and monocular estimators can produce noisy or misaligned data, leading to performance degradation (Zhang et al., 2022, Lin et al., 2024).
Dataset Size and Diversity: Most RGB-D VSOD datasets remain modest in scope compared to large-scale RGB corpora. There is a call for larger, more diverse, and standardized benchmarks, especially with ground-truth real depth (Mou et al., 2023, Lin et al., 2024).
Temporal Consistency: Present fusion strategies are mainly frame-level or exploit implicit temporal cues via flow. Stronger temporal modeling—e.g., transformer sequence modeling or explicit memory refinement—remains an open avenue (Lin et al., 13 Nov 2025, Liu et al., 12 May 2026).
Boundary Sharpness and Fine Structure: Many approaches observe boundary blurring when all modalities are degraded. Edge-refinement modules and hybrid edge/saliency objectives show promise but are not yet ubiquitous (He et al., 29 Jul 2025).
Computational Efficiency: While lightweight models (DFM-Net, MobileNet-based) exist, adaptation of large vision foundation models (SAM2) for VSOD remains computationally intensive. Further work is required on distillation, quantization, and architectural pruning for edge or real-time deployment (Zhang et al., 2022, Liu et al., 12 May 2026).
Generalization and Self-/Unsupervised Transfer: Most models are fully supervised and require dense annotation. Research opportunities include semi-/self-supervised learning, unsupervised domain adaptation, and joint optimization for depth saliency and object segmentation (Lin et al., 2024, Mou et al., 2023).

7. Context and Impact

RGB-D VSOD stands at the confluence of video understanding, 3D perception, and multi-modal learning. The integration of depth and motion in saliency prediction reflects broader trends toward multi-sensory, cross-domain fusion in visual AI. The rapid evolution from handcrafted fusion schemes to attentive, prompt-free, transformer-based architectures demonstrates the field’s rapid cycle of methodological absorption and re-invention. The latest advances set the stage for RGB-D VSOD’s adoption in robotics, augmented reality, video editing, and human-computer interaction systems, where robust and explainable salient object detection under naturally complex, moving, and cluttered scenes is critical.

Further developments—such as broad deployment of foundation model-based VSOD, large-scale dataset creation, efficient model compression, and deeper integration of semantic and motion cues—are likely to define the next research phase in this expanding domain (He et al., 29 Jul 2025, Mou et al., 2023, Lu et al., 2022, Lin et al., 2024, Liu et al., 12 May 2026, Lin et al., 13 Nov 2025).