HiDA-Net: Hierarchical Depth-Aware Saliency

• The paper introduces a hierarchical RGB-D model that integrates fine-grained local detail with semantic cues for improved salient object detection.
• It employs granularity-based attention by partitioning depth maps into adaptive regions, preserving spatial structure and enhancing foreground/background separation.
• The model uses cross dual-attention and a shared decoder to fuse multi-modal and multi-scale features, resulting in state-of-the-art detection performance.

HiDAnet is an RGB-D salient object detection architecture introduced in “HiDAnet: RGB-D Salient Object Detection via Hierarchical Depth Awareness” (Wu et al., 2023). It is designed to fuse RGB appearance cues and depth/geometric cues to localize salient regions, with particular emphasis on cases in which foreground and background have similar visual appearance but occupy distinct camera distances. The model combines a granularity-based attention scheme, a unified cross dual-attention module, a shared decoder, and multi-scale loss. The expression “High-Resolution Detail-Aggregation Network” may be used as an Editor’s term for this model family, because the method explicitly emphasizes fine-grained local detail, hierarchical feature aggregation, and boundary-sensitive RGB-D fusion; however, the paper’s formal name is HiDAnet and its core concept is hierarchical depth awareness (Wu et al., 2023).

1. Problem formulation and motivation

RGB-D salient object detection aims to identify the most visually important object regions by combining RGB appearance cues, such as color, texture, contrast, and semantics, with depth/geometric cues, such as distance, shape, spatial layout, and foreground-background separation (Wu et al., 2023). Within this setting, the paper positions depth as an auxiliary modality that can disambiguate scenes in which RGB evidence alone is insufficient, including low-contrast scenes, cluttered backgrounds, occlusion, and visually similar foreground/background configurations.

The central motivation is that existing works often adopt attention modules for feature modeling, but few methods explicitly leverage fine-grained details to merge with semantic cues. The consequence, as stated in the paper, is that even with auxiliary depth information, it remains challenging to distinguish objects with similar appearances but at distinct camera distances. The proposed remedy is not merely stronger global fusion, but a hierarchical use of depth information that preserves local structure and aligns it with network hierarchy (Wu et al., 2023).

A key conceptual claim is that the multi-granularity properties of geometric priors correlate well with neural network hierarchies. Shallow layers preserve local detail, whereas deeper layers capture semantic abstraction. This suggests that depth should be injected at multiple granularities and multiple levels, rather than treated as a single global side input. In the paper’s examples, this is relevant to objects and scenes such as a motorbike and the street, cups and a table, a sculpture with hollow regions, and humans with complex boundaries. The intended effect is improved discriminability, more accurate foreground/background separation, and sharper object boundaries.

2. Architectural organization

HiDAnet adopts a U-Net-like encoder-decoder architecture with two parallel encoders, one for RGB and one for depth, followed by multi-level enhancement, multi-modal and multi-level fusion, a shared decoder, and deep supervision via multi-scale loss (Wu et al., 2023). The paper evaluates multiple backbones—VGG16, ResNet50, and Res2Net50—and in each configuration RGB and depth are processed by separate encoders that extract hierarchical features at multiple scales.

After each level, a Receptive Field Block (RFB) is used for accurate object detection. The network produces five levels of outputs corresponding to the hierarchical branches. The overall structure is organized around three named modules: Granularity-Based Attention (GBA), Cross Dual-Attention (CDA), and Efficient Multi-Input Fusion (EMI). RGB features provide appearance and semantic cues, while depth features provide geometric cues; both streams are enhanced by GBA, fused by CDA, and then decoded through a shared branch.

The architecture is explicitly coarse-to-fine. Deeper layers contribute semantic or global information, shallower layers contribute finer details, and the decoder progressively aggregates these representations into saliency predictions. In this sense, HiDAnet is “hierarchical” in two simultaneous ways: it uses hierarchical feature extraction within each modality, and it uses hierarchical fusion across modalities and across encoder-decoder stages. A plausible implication is that the model’s “detail-aggregation” character derives from the interaction between low-level locality and high-level semantics, rather than from a single specialized high-resolution head.

3. Granularity-based attention and depth-aware local detail

The granularity-based attention scheme is motivated by a limitation of conventional channel attention based on global average pooling: spatial information is collapsed into a single descriptor, foreground and background contribute equally, and local geometric structure is lost (Wu et al., 2023). HiDAnet addresses this by constructing local, granularity-aware attention from the depth map.

The depth map is partitioned into multiple Otsu regions by multi-level thresholding. The thresholds are defined as

$$\{d_1^*, d_2^*, ..., d_T^*\} = \arg\max \{\sigma_w^2(d_1, d_2, ..., d_T)\},$$

where

$\sigma_w^2 = \sum_{i=1}^{T+1} w_i \sigma_i^2.$

This maximizes inter-class variance and yields adaptive regions intended to reflect scene depth structure. For each region mask $m_i$, the input feature map $f_{in}$ is masked as $f_{in} \otimes m_i$, after which local channel attention is computed with local average pooling rather than global average pooling:

$$L\text{-}ECA(x) = \sigma(Conv_{1d}(L\text{-}AP(x))) \otimes x.$$

The module output is then

$$f_{out} = \sum_{i=1}^{T+1} L\text{-}ECA(f_{in} \otimes m_i) + f_{in}.$$

This mechanism is applied separately to RGB and depth streams. According to the paper, it strengthens fine-grained local features, improves foreground/background separation, preserves sharp edges and local detail, and enables the model to exploit depth-specific regions more effectively than vanilla channel attention (Wu et al., 2023). The paper compares three variants—global average pooling, global pooling with local masks, and local pooling with local masks—and identifies local pooling aligned with depth regions as the most discriminative and stable.

The paper’s ablations further examine the number of Otsu thresholds. It tests $T=0$, $T=1$, $T=2$, and $\sigma_w^2 = \sum_{i=1}^{T+1} w_i \sigma_i^2.$0, and reports that $\sigma_w^2 = \sum_{i=1}^{T+1} w_i \sigma_i^2.$1 gives the best overall performance. The interpretation offered is that $\sigma_w^2 = \sum_{i=1}^{T+1} w_i \sigma_i^2.$2 is too coarse, whereas $\sigma_w^2 = \sum_{i=1}^{T+1} w_i \sigma_i^2.$3 over-discretizes and becomes less effective. The paper interprets $\sigma_w^2 = \sum_{i=1}^{T+1} w_i \sigma_i^2.$4 as a practical split into roughly close, middle, and far regions.

4. Cross dual-attention and hierarchical fusion

CDA is the core fusion module for both multi-modal interaction and multi-level interaction (Wu et al., 2023). It fuses RGB with depth and also merges encoder and decoder features across hierarchy levels. The design premise is that both cases can be treated as feature interaction problems in which mutual guidance is more effective than isolated self-enhancement.

Given paired features $\sigma_w^2 = \sum_{i=1}^{T+1} w_i \sigma_i^2.$5 and $\sigma_w^2 = \sum_{i=1}^{T+1} w_i \sigma_i^2.$6, the module first applies a transformation

$\sigma_w^2 = \sum_{i=1}^{T+1} w_i \sigma_i^2.$7

$\sigma_w^2 = \sum_{i=1}^{T+1} w_i \sigma_i^2.$8

For each transformed feature map, CDA computes channel attention

$\sigma_w^2 = \sum_{i=1}^{T+1} w_i \sigma_i^2.$9

and spatial attention

$m_i$0

The attention estimated from one feature is then used to enhance the other:

$m_i$1

$m_i$2

$m_i$3

This is “cross dual-attention” because it combines channel and spatial attention, and because the learned attention is cross-applied between paired features rather than only self-applied. The intended effect is that RGB helps refine depth and depth helps refine RGB. The same principle is used between encoder and decoder features, so CDA serves as a unifying mechanism for cross-modal and cross-level fusion.

The paper argues that this matters because depth may be noisy or incomplete, while RGB appearance may be ambiguous. Cross-guidance can therefore reduce single-modality errors. In the encoder fusion ablations, HiDAnet compares addition, concatenation plus convolution, self-attention plus addition, self-attention plus concatenation plus convolution, cross attention plus addition, and the full fusion design. The full cross-attention design is reported as best, supporting the claim that cross-domain interaction is superior to self-only enhancement (Wu et al., 2023).

5. Shared decoder, progressive aggregation, and hierarchical supervision

The shared decoder aggregates three inputs: RGB decoded features $m_i$4, depth decoded features $m_i$5, and previous shared features $m_i$6. The paper considers naive concatenation too crude because the channel dimension becomes large and not all features are equally useful. It therefore introduces Efficient Multi-Input Fusion:

$m_i$7

Here, global ECA is used for channel reweighting, and the residual addition is used to preserve prior shared information (Wu et al., 2023).

The decoder is thus built by concatenating multi-input features, convolving them, reweighting channels using ECA, and adding the previous shared feature as a residual. This yields progressively refined shared representations for final saliency prediction. The paper attributes to this design improved semantic consistency, preservation of details via skip-like aggregation, and gradual coarse-to-fine refinement.

HiDAnet also applies multi-scale loss across its hierarchical outputs. For each output level $m_i$8,

$m_i$9

where BCE is pixel-level binary cross-entropy and IoU is intersection-over-union loss. The total multi-level loss is

$f_{in}$0

with

$f_{in}$1

The paper states that this supervision is intended to encourage discriminative representations, preserve short- and long-range spatial relationships, exploit the entire hierarchy, and maintain consistency across scales (Wu et al., 2023).

An interpretive point requires some care. The source description says that early layers are supervised more strongly than deeper layers by larger weights. Since the loss is indexed across five outputs and not verbally tied to explicit layer depth in the formula itself, that statement should be read as the paper’s characterization of its weighting scheme rather than as an independently re-derived architectural fact.

6. Empirical results, ablations, and methodological significance

The paper evaluates HiDAnet on DES, NLPR-test, NJU2K-test, STERE, SIP, and the difficult COME15K set, using Mean Absolute Error $f_{in}$2, max F-measure $f_{in}$3, S-measure $f_{in}$4, and max E-measure $f_{in}$5 (Wu et al., 2023). It reports that HiDAnet performs favorably over state-of-the-art methods on all major benchmarks.

For the VGG16 backbone, HiDAnet is described as especially competitive on NLPR and SIP, with a reported model size of 269 MB and around 6 FPS. For the ResNet50 backbone, it is reported to set new state of the art on DES, NLPR, and NJU2K, with a model size of 523 MB and around 12 FPS. For the Res2Net50 backbone, it performs favorably versus SPNet while using 525 MB versus 702 MB for SPNet and running at around 11 FPS. On the difficult COME15K set, the paper reports $f_{in}$6 and best or tied-best $f_{in}$7.

Qualitative comparisons are said to show improved boundary precision, hole preservation, foreground-background separation, and robustness to clutter and low contrast. The model is described as particularly effective when target and background have similar appearance but different depth granularity. The paper also studies noisy depth maps and reports that HiDAnet is more stable than methods such as CMINet and SPNet, which it takes as evidence that GBA and cross-fusion improve resistance to depth corruption.

The ablation studies attribute incremental gains to GBA, CDA, skip connection with cross attention, EMI, and multi-level loss. Additional ablations indicate that applying GBA first only to RGB and then to both RGB and depth improves performance, supporting the claim that GBA helps RGB by injecting depth-aware locality and helps depth by self-enhancement (Wu et al., 2023). Taken together, these experiments define the methodological significance of HiDAnet: it is a hierarchical RGB-D fusion framework in which local depth granularity is not an auxiliary afterthought but a primary organizing principle for attention, feature interaction, and decoder aggregation.

A common misconception in RGB-D saliency detection is that stronger performance derives mainly from deeper backbones or more aggressive global attention. HiDAnet argues against that simplification. Its reported gains are tied instead to the coordinated use of depth-derived granularity masks, local pooling, cross-modal dual attention, and hierarchical supervision. A plausible implication is that, within RGB-D SOD, preserving structured local geometry can be as important as increasing semantic capacity.