Multi-Dimensional Selective Attention Module (MSAM)

• MSAM is a multi-dimensional attention module that refines fused RGB and motion-depth features by selectively focusing on width, height, spatial, and channel dimensions.
• It employs a lightweight four-branch design with modality-specific gates to enhance cross-modal perceptual interactions, integrated sequentially after pixel-level fusion.
• Empirical evaluations demonstrate that MSAM improves key metrics in RGB-D salient object detection, validating its effectiveness over conventional fusion methods.

Searching arXiv for the primary MSAM paper and closely related attention/fusion references. [arXivSearch] query="(He et al., 29 Jul 2025) OR \"multi-dimensional selective attention module\" OR SMFNet RGB-D VSOD" The Multi-Dimensional Selective Attention Module (MSAM) is a cross-modal fusion component introduced in SMFNet for RGB-D video salient object detection. In that setting, MSAM follows a prior Pixel-level Selective Fusion (PSF) stage, receives RGB features together with fused motion-depth features, and refines them by conducting selective attention on four dimensions—width, height, spatial, and channel—at every encoder scale. Its role is not generic self-attention over arbitrary tensors, but dimension-specific, modality-aware recalibration for integrating appearance cues with selectively fused optical-flow and depth information (He et al., 29 Jul 2025).

1. Definition and conceptual scope

In SMFNet, MSAM is the second major fusion component after PSF. PSF first resolves the unequal-contribution problem between optical flow and depth by producing fused features $f_i^{df}$. MSAM then integrates those fused motion-depth features with RGB features $f_i^r$ and outputs refined cross-modal features $f_i^{out}$. The defining claim is that simple concatenation of $f_i^{df}$ and $f_i^r$ remains too coarse, because the interactions between RGB appearance and fused motion-depth cues vary across different descriptive dimensions. MSAM is therefore designed to promote what the paper terms “cross-modal perceptual interactions” through selective enhancement (He et al., 29 Jul 2025).

Within this formulation, “multi-dimensional” has a precise and limited meaning. The module conducts selective attention on width, height, spatial, and channel dimensions, respectively. It does not define time as an attention axis inside MSAM itself, and it does not introduce modality as a separate explicit attention dimension. Instead, modality interaction is embodied by pairing RGB features with the already fused motion-depth representation. A common misconception is to read MSAM here as a transformer-style global attention mechanism; the paper instead presents a lightweight four-branch attention block built from pooled descriptors, $1\times1$ convolutions, sigmoid gates, multiplicative recalibration, and additive fusion.

MSAM is also multi-scale. It is applied at each encoder stage $i=1,\dots,5$, so the network constructs a hierarchy of refined features $\{f_i^{out}\}_{i=1}^5$ rather than a single fused representation. This multi-scale deployment is central to its role inside the U-Net-like decoder that follows.

2. Placement within SMFNet

The full SMFNet pipeline uses three inputs per frame: RGB, depth, and optical flow. Optical flow is computed externally by RAFT. RGB, depth, and flow are then processed by parallel encoders sharing the same design, namely a ResNet-34 backbone with ASPP attached to the last layer. This yields five levels of features $f_i^m$ with $m\in\{r,f,d\}$ and $f_i^r$0. A channel compression module reduces every feature map to 64 channels via $f_i^r$1 convolution (He et al., 29 Jul 2025).

The network does not fuse all three modalities symmetrically. Instead, it first applies PSF only to optical-flow and depth features. At each scale,

$f_i^r$2

where $f_i^r$3 is the resized spatial weight map. This stage decides, pixel by pixel, whether motion or depth should be trusted more.

MSAM begins only after that step. At each scale $f_i^r$4, it receives the RGB feature $f_i^r$5 and the PSF-produced fused feature $f_i^r$6. Its output is a refined feature $f_i^r$7. The set $f_i^r$8 is then sent to a U-Net-like decoder that aggregates hierarchical information and predicts five saliency maps, with $f_i^r$9 used at inference. Functionally, PSF answers “between motion and depth, which cue should dominate where,” whereas MSAM answers “given RGB and the selected motion-depth cue, how should they be integrated most effectively.”

3. Internal mechanism

MSAM has four parallel branches, one for each attended dimension: width, height, spatial, and channel. For each branch, the module first pools along the corresponding reduction path to obtain what the paper calls “dual feature vectors” from the RGB stream and the fused motion-depth stream. The paper provides the width branch explicitly and states that the same design is used for the other dimensions (He et al., 29 Jul 2025).

Branch	Dimension emphasized	Branch output
Width branch	Horizontal structural dependencies	$f_i^{out}$0
Height branch	Vertical structural dependencies	$f_i^{out}$1
Spatial branch	Location-wise saliency over the 2D plane	$f_i^{out}$2
Channel branch	Semantic or feature-channel importance	$f_i^{out}$3

For the width branch, the equations are: $f_i^{out}$4

$f_i^{out}$5

$f_i^{out}$6

These equations define the weight perception module (WPM). First, pooled descriptors from the two modalities are concatenated and mixed by a $f_i^{out}$7 convolution. The mixed representation is then split into two parts, one for the fused motion-depth stream and one for RGB. Each part passes through another $f_i^{out}$8 convolution and a sigmoid to produce modality-specific gates $f_i^{out}$9 and $f_i^{df}$0. The branch output is a weighted sum of the two input features rather than a concatenation.

The same pattern is used for height, spatial, and channel branches, producing $f_i^{df}$1, $f_i^{df}$2, and $f_i^{df}$3. The final MSAM output is

$f_i^{df}$4

Two structural properties distinguish this design. First, the module computes separate enhancement weights for RGB and fused motion-depth after a joint interaction step, rather than learning one common attention map over a concatenated tensor. Second, the final integration is additive across the four branches. The paper does not mention softmax normalization, residual shortcuts, explicit affinity matrices, reduction ratios, or multi-head decomposition inside MSAM. It also does not algebraically specify the exact tensor layouts of the pooled descriptors for the width and height branches, so any more detailed account of those axes would be an inference from the figure labels rather than an explicit derivation.

4. Training, supervision, and implementation

MSAM is trained as part of the full RGB-D VSOD system rather than through a separate auxiliary objective. After MSAM refines the feature hierarchy, the U-Net decoder predicts five saliency maps $f_i^{df}$5, and the decoder is supervised hierarchically by

$f_i^{df}$6

with $f_i^{df}$7 used as the final prediction in inference (He et al., 29 Jul 2025).

The implementation details reported for reproducing the full MSAM-equipped network are specific but limited. The encoder backbone is ResNet-34 with ASPP on the last layer; all encoded features are compressed to 64 channels by $f_i^{df}$8 convolution; MSAM is applied on all five levels; and WPM uses $f_i^{df}$9 convolutions and sigmoid activations. Input images are resized to $f_i^r$0. Training uses SGD, batch size 8, learning rates $f_i^r$1 for backbones and $f_i^r$2 for other parts, and convergence after 70 epochs.

The paper does not report reduction ratios, hidden-channel sizes inside WPM, normalization layers inside MSAM, FLOPs or parameter overhead attributable specifically to MSAM, or latency attributable specifically to the module. This omission matters because it limits fine-grained implementation equivalence. A further technical caveat is that the exact pooling axes for width and height attention are described through figure labels such as “pool(H)” and “pool(W)” rather than through full tensor equations. This suggests the intended mechanism clearly, but leaves low-level tensor layout under-specified.

5. Empirical evidence and ablations

The principal ablation isolates the effect of replacing naive concatenation-and-convolution fusion with MSAM. On RDVS, the baseline gives $f_i^r$3, $f_i^r$4, and $f_i^r$5. Adding only MSAM improves performance to $f_i^r$6. On DVisal, the baseline is $f_i^r$7, and adding only MSAM improves it to $f_i^r$8. The full model with both PSF and MSAM reaches $f_i^r$9 on RDVS and $1\times1$0 on DVisal (He et al., 29 Jul 2025).

Configuration	RDVS $1\times1$1	DVisal $1\times1$2
Baseline	0.861 / 0.810 / 0.033	0.837 / 0.839 / 0.044
Baseline + MSAM	0.866 / 0.815 / 0.031	0.848 / 0.846 / 0.040
Full model (PSF + MSAM)	0.874 / 0.823 / 0.028	0.854 / 0.851 / 0.038

A second ablation progressively activates the four branches of MSAM. Starting from “baseline (without MSAM)” at $1\times1$3 on RDVS and $1\times1$4 on DVisal, the branch accumulation is reported as follows: $1\times1$5W gives $1\times1$6 and $1\times1$7; $1\times1$8W+H gives $1\times1$9 and $i=1,\dots,5$0; $i=1,\dots,5$1W+H+S gives $i=1,\dots,5$2 and $i=1,\dots,5$3; and $i=1,\dots,5$4W+H+S+C gives $i=1,\dots,5$5 and $i=1,\dots,5$6. The width branch alone is therefore not sufficient and may slightly hurt some metrics, but the cumulative four-dimensional design yields the best final result. This supports the paper’s central thesis that richer cross-modal interactions emerge when fusion is not restricted to one attention dimension.

The module is also compared against alternative fusion designs while keeping the rest of SMFNet unchanged. MSAM achieves RDVS $i=1,\dots,5$7 and DVisal $i=1,\dots,5$8, compared with MFA $i=1,\dots,5$9 and $\{f_i^{out}\}_{i=1}^5$0, MGA $\{f_i^{out}\}_{i=1}^5$1 and $\{f_i^{out}\}_{i=1}^5$2, RFM $\{f_i^{out}\}_{i=1}^5$3 and $\{f_i^{out}\}_{i=1}^5$4, and CAM $\{f_i^{out}\}_{i=1}^5$5 and $\{f_i^{out}\}_{i=1}^5$6. The paper also reports that SMFNet is evaluated against 19 state-of-the-art models on both RDVS and DVisal and on five video benchmark datasets incorporating synthetic depth.

6. Relation to adjacent attention modules and acronym ambiguity

The acronym “MSAM” is not unique in the literature, and distinguishing these usages is essential. In SMFNet, MSAM denotes Multi-Dimensional Selective Attention Module. In “BAM: A Balanced Attention Mechanism for Single Image Super Resolution,” by contrast, MSAM explicitly means Maxpool Spatial Attention Module, the spatial branch of BAM rather than a multi-axis fusion block (Wang et al., 2021). In “Multimodal Spatial Attention Module for Targeting Multimodal PET-CT Lung Tumor Segmentation,” MSAM denotes Multimodal Spatial Attention Module, a PET-driven spatial gating mechanism for CT features rather than a width-height-spatial-channel decomposition (Fu et al., 2020). Other unrelated expansions include Multi-scale Spectral Attention Module for hyperspectral segmentation (Shah et al., 23 Jun 2025) and Multi-Semantic Adaptive Mining for drone video-text retrieval (Huang et al., 17 Oct 2025).

As an architectural reference point, CBAM factorizes attention over channel and spatial dimensions of the same intermediate feature map and applies them sequentially (Woo et al., 2018). Relative to CBAM, the SMFNet formulation is distinguished by two properties: it attends over four dimensions rather than two, and it generates modality-specific gates for RGB and fused motion-depth features after cross-modal interaction rather than computing attention solely from a single tensor. A broader adjacent line of work, such as SDA-xNet, introduces depth as another attention dimension and shows that attention axes can be treated as composable rather than exclusive (Guo et al., 2022). This suggests a larger design space in which “multi-dimensional” attention may refer to different axis selections in different domains.

Accordingly, the precise encyclopedia usage of MSAM should remain context-dependent. In the SMFNet literature, it refers specifically to a four-branch, cross-modal fusion module operating over width, height, spatial, and channel dimensions. It should not be conflated with generic “MSAM” blocks in other papers that use the same acronym for fundamentally different mechanisms.