SMFNet: Selective Cross-Modal Fusion

• The paper demonstrates that selective, pixel-level fusion of optical flow and depth outperforms symmetric aggregation in RGB-D video saliency detection.
• SMFNet employs a pseudo-supervised mechanism to adaptively weight modality contributions, reducing noise from weaker signals.
• The architecture integrates a multi-dimensional selective attention module to merge motion-depth features with RGB, leading to enhanced accuracy.

Searching arXiv for the cited SMFNet and related selective fusion papers to ground the article in current literature. Selective Cross-Modal Fusion Framework (SMFNet) is a trimodal RGB-D video salient object detection network that is designed to exploit the complementary strengths of RGB appearance, optical flow motion, and depth geometry by making fusion explicitly selective rather than symmetric (He et al., 29 Jul 2025). Its central premise is that optical flow and depth do not contribute equally across scenes or pixels, so they should not be fused with an implicit equal-weight assumption. In SMFNet, selectivity is implemented first through a pixel-level selective fusion strategy (PSF), which adaptively fuses optical flow and depth according to their estimated actual contributions, and then through a multi-dimensional selective attention module (MSAM), which integrates the resulting motion-depth representation with RGB features across width, height, spatial, and channel dimensions (He et al., 29 Jul 2025). Within the broader literature on multimodal fusion, SMFNet belongs to a class of methods that replace uniform aggregation with contribution-aware gating, a design tendency that also appears in infrared-visible fusion and multimodal MRI segmentation under different formulations (Sun et al., 14 Sep 2025, Chen et al., 20 Mar 2025).

1. Conceptual basis and problem formulation

SMFNet was proposed for RGB-D video salient object detection (RGB-D VSOD), an emerging task in which salient object detection is extended to videos with RGB and depth inputs, while also incorporating motion cues through optical flow (He et al., 29 Jul 2025). The framework addresses a specific limitation identified in previous RGB-D VSOD models: existing methods make different attempts to derive motion cues, yet previous models always treat optical flow and depth equally with respect to model designs, without explicitly considering their unequal contributions in individual scenarios (He et al., 29 Jul 2025).

The resulting problem formulation is selective rather than symmetric. Optical flow may be strong where motion is informative, depth may be strong where geometry is clear, and one modality can be noisy or misleading in some regions. SMFNet therefore treats fusion as a pixel-dependent decision process. This means that the network does not simply aggregate auxiliary modalities and hand them to an RGB decoder; instead, it attempts to infer where optical flow should dominate and where depth should dominate before these cues are integrated with RGB.

This design places SMFNet within a broader research pattern in selective cross-modal fusion. A closely related formulation appears in infrared-visible image fusion, where infrared and visible features are allowed to compete and cooperate selectively through feature-level modality-aware attention and pixel-level alpha blending (Sun et al., 14 Sep 2025). A similar principle also appears in multimodal brain tumor segmentation, where selective complementary feature fusion is used to adaptively fuse MRI modality pairs by complementary soft selection weights (Chen et al., 20 Mar 2025). This suggests that SMFNet exemplifies a wider methodological shift from unconditional multimodal aggregation toward learned, content-aware modality selection.

2. Architecture and data flow

SMFNet uses three input branches: an RGB branch for appearance and semantic detail, an optical flow branch for explicit motion cues, and a depth branch for geometric and boundary cues (He et al., 29 Jul 2025). The paper uses ResNet-34 as the backbone for all streams, with an ASPP module attached to the last layer, and the encoder produces five-level hierarchical features for each modality: $$f_{i}^{m}=\left\{ f_{i}^{m}, m\in \left[ r,f,d \right], i=1,2,...,5 \right\}$$ where $f_i^r$ denotes RGB features, $f_i^f$ optical flow features, and $f_i^d$ depth features at level $i$ (He et al., 29 Jul 2025).

For efficiency, all features are compressed to 64 channels using a $1\times1$ convolution-based compression module. The decoding and fusion pipeline is split into two stages. First, PSF selectively fuses optical flow and depth, producing motion-depth fused features $f_i^{df}$. Second, MSAM integrates $f_i^{df}$ with RGB features using multi-dimensional selective attention. A U-Net-like decoder then aggregates the hierarchically fused features and predicts saliency maps (He et al., 29 Jul 2025).

The ordering of these operations is architecturally important. SMFNet does not fuse RGB with flow and depth simultaneously. Instead, it first distills flow and depth into a more reliable motion-depth representation and only then lets that representation interact with RGB. The paper states that this reduces unnecessary noise before cross-modal fusion with RGB (He et al., 29 Jul 2025). A plausible implication is that the framework encodes an explicit hierarchy of trust: auxiliary modalities are first reconciled against each other, after which the resulting representation is merged with the primary appearance stream.

3. Pixel-level Selective Fusion Strategy

PSF is the core idea of SMFNet. It fuses optical flow and depth at the pixel level, based on their estimated actual contributions, rather than through a fixed fusion rule (He et al., 29 Jul 2025). The depth and flow features from all levels are first interacted and aggregated: corresponding levels are concatenated along the channel dimension, higher-level features are upsampled by $\times 2$ bilinear interpolation before concatenation with lower-level features, and this process is repeated to collect cross-level information from both modalities. A final convolution followed by Sigmoid produces a normalized spatial weight map $SW\in[0,1]$, which is resized into level-specific maps $f_i^r$0 (He et al., 29 Jul 2025).

Fusion is then defined as

$f_i^r$1

where $f_i^r$2 is the per-pixel weight for optical flow and $f_i^r$3 is the complementary weight for depth (He et al., 29 Jul 2025). If $f_i^r$4, the model selects optical flow at pixel $f_i^r$5; if $f_i^r$6, it selects depth. The paper explicitly interprets PSF as per-pixel modality selection rather than generic fusion.

A central complication is that an unsupervised spatial weight map can be arbitrary and may not reflect real contributions. To address this, SMFNet introduces a pseudo-supervisory algorithm. The optical flow stream and depth stream are pretrained separately to produce coarse modality-specific saliency maps $f_i^r$7 and $f_i^r$8, both supervised by the true saliency ground truth $f_i^r$9, and all three are normalized to $f_i^f$0 (He et al., 29 Jul 2025). For salient pixels, the modality with the larger predicted saliency value is deemed more contributive: $f_i^f$1 For non-salient pixels, the modality with the smaller saliency response is preferred: $f_i^f$2 and the final pseudo ground truth is

$f_i^f$3

so that $f_i^f$4 indicates optical flow should be selected and $f_i^f$5 indicates depth should be selected (He et al., 29 Jul 2025).

During training, $f_i^f$6 and $f_i^f$7 are supervised by $f_i^f$8, while $f_i^f$9 is supervised by $f_i^d$0; during inference, only the $f_i^d$1 generation branch is needed (He et al., 29 Jul 2025). This makes PSF a learned selective mechanism grounded in modality-specific contribution estimates. The paper’s interpretation is that symmetric fusion can dilute the stronger modality or preserve noise from the weaker one, while PSF learns where each auxiliary modality should be used.

4. Multi-Dimensional Selective Attention Module

After PSF has produced motion-depth fused features $f_i^d$2, SMFNet uses the Multi-Dimensional Selective Attention Module to combine these features with RGB features $f_i^d$3 (He et al., 29 Jul 2025). The stated motivation is that simple concatenation and convolution are insufficient for strong cross-modal interaction. MSAM therefore performs selective attention across width, height, spatial, and channel dimensions, with the aim of capturing structural interactions along multiple axes rather than through a single fusion operation (He et al., 29 Jul 2025).

For a given dimension, MSAM first uses average pooling to compress the relevant dimension to length 1, producing one feature vector from $f_i^d$4 and one from $f_i^d$5. These are processed by a Weight Perception Module. The paper gives the width dimension as an example: $f_i^d$6

$f_i^d$7

$f_i^d$8

where $f_i^d$9 and $i$0 are pooled width-wise descriptors, $i$1 is channel concatenation, $i$2 is a $i$3 convolution, $i$4 separates the mixed vector, and $i$5 applies Sigmoid gating (He et al., 29 Jul 2025). The same logic is independently applied to the height, spatial, and channel dimensions, yielding $i$6, $i$7, and $i$8, and the final output is

$i$9

(He et al., 29 Jul 2025).

MSAM can be interpreted as a multi-view selective cross-modal attention mechanism in which RGB and fused motion-depth features influence each other across multiple structural dimensions. The paper argues that this is more expressive than plain concatenation or a single attention block because it explicitly captures interactions along several axes (He et al., 29 Jul 2025). In comparative context, this resembles the use of spatial and channel information for learned soft selection in SCFF for MRI segmentation, although SCFF operates on pairwise modality fusion and not on a post-fusion RGB integration stage (Chen et al., 20 Mar 2025). The commonality is the treatment of modality relevance as dimension-sensitive rather than globally uniform.

5. Supervision, optimization, and empirical evidence

SMFNet uses a saliency training objective based on BCE and IoU: $1\times1$0 (He et al., 29 Jul 2025). The PSF-related loss contains three supervised outputs: $1\times1$1 while the decoder predicts five saliency maps $1\times1$2, $1\times1$3, with hierarchy-weighted losses

$1\times1$4

and the total objective is

$1\times1$5

(He et al., 29 Jul 2025).

The paper evaluates on two RGB-D video datasets, RDVS and DVisal. RDVS contains 4,030 frames and DVisal contains 7,117 frames. Joint training uses 2,176 RDVS training frames and 3,551 DVisal training frames. Testing excludes the last frame of each sequence because no corresponding optical flow exists (He et al., 29 Jul 2025). The framework is also evaluated on five standard VSOD benchmarks with synthetic depth generated using DPT: DAVIS, DAVSOD, FBMS, SegTrack-V2, and VOS. The model is trained on 7,683 frames from DAVIS, DAVSOD, and FBMS and tested on 9,502 frames across the five benchmarks (He et al., 29 Jul 2025).

Implementation uses PyTorch, an NVIDIA 4090 GPU, SGD optimizer, batch size 8, input resized to $1\times1$6, data augmentation via flipping, cropping, and rotation, initial learning rates of $1\times1$7 for the backbone and $1\times1$8 for other layers, and training convergence after 70 epochs (He et al., 29 Jul 2025). Evaluation uses $1\times1$9 for S-measure, $f_i^{df}$0 for maximum F-measure, and $f_i^{df}$1 for MAE (He et al., 29 Jul 2025).

The principal quantitative findings reported for the full model are as follows:

Dataset	$f_i^{df}$2	$f_i^{df}$3 / $f_i^{df}$4
RDVS	0.874	0.823 / 0.028
DVisal	0.854	0.851 / 0.038

On RDVS and DVisal, the paper reports that SMFNet outperforms all compared methods and surpasses DCTNet+ and all other RGB-D VSOD, RGB-D SOD, and VSOD baselines (He et al., 29 Jul 2025). On DAVSOD with synthetic depth, it improves over DCTNet+ by +1.5% $f_i^{df}$5, +2.7% $f_i^{df}$6, and +1% MAE improvement (He et al., 29 Jul 2025). Ablation findings further support the architecture: on RDVS/DVisal, baseline, baseline+PSF, baseline+MSAM, and full SMFNet are reported respectively as 0.861 / 0.810 / 0.033 and 0.837 / 0.839 / 0.044; 0.869 / 0.817 / 0.030 and 0.849 / 0.850 / 0.039; 0.866 / 0.815 / 0.031 and 0.848 / 0.846 / 0.040; and 0.874 / 0.823 / 0.028 and 0.854 / 0.851 / 0.038 (He et al., 29 Jul 2025). The paper also states that using $f_i^{df}$7 alone helps little, while adding $f_i^{df}$8 significantly improves results, indicating that supervised selection is critical.

6. Position within selective multimodal fusion research

SMFNet’s methodological identity is defined by the claim that motion and depth should not be fused as if they contribute equally (He et al., 29 Jul 2025). In that respect, it stands in contrast to earlier RGB-D VSOD models that fuse auxiliary modalities symmetrically. Its design can be summarized as a sequence of three commitments: pixel-level selection between optical flow and depth, pseudo-supervised estimation of modality contribution, and multi-dimensional selective interaction between the resulting motion-depth representation and RGB.

This logic has direct parallels in other multimodal domains. In infrared and visible image fusion, FusionNet uses modality-aware attention to compute

$f_i^{df}$9

and a pixel-wise alpha blending equation

$f_i^{df}$0

to let infrared and visible features compete and cooperate selectively, while a target-aware loss injects weak semantic supervision through ROI annotations (Sun et al., 14 Sep 2025). In multimodal brain tumor segmentation, CFCI-Net uses Selective Complementary Feature Fusion to adaptively fuse the modality pairs $f_i^{df}$1 and $f_i^{df}$2 through complementary soft selection weights, followed by Modal Feature Compression Interaction to reduce redundancy and restore interactive learning (Chen et al., 20 Mar 2025).

These comparisons do not imply architectural equivalence. SMFNet is a trimodal video saliency model with ResNet-34 backbones, a PSF stage supervised by pseudo labels, and an MSAM stage integrating motion-depth with RGB (He et al., 29 Jul 2025). FusionNet is an end-to-end dual-branch fusion network for aligned infrared-visible image pairs with weak ROI supervision (Sun et al., 14 Sep 2025). CFCI-Net is a 4-branch multimodal 3D segmentation network for MRI with selective pairwise fusion and a transformer-style compression-interaction block (Chen et al., 20 Mar 2025). What links them is a shared premise: multimodal signals are complementary but not equally reliable everywhere, so effective fusion requires learned selection, not uniform mixing. This suggests that “selective cross-modal fusion” is best understood as a design philosophy spanning multiple tasks rather than a single architecture family.

A recurrent misconception in multimodal model design is that adding modalities guarantees improved representation if they are simply concatenated or averaged. The SMFNet formulation directly rejects that assumption. Its argument is that equal fusion can preserve noise from the weaker modality or dilute the stronger one, particularly when depth quality is poor or motion cues are ambiguous (He et al., 29 Jul 2025). Another possible misconception is that selectivity can be obtained by attention alone without explicit supervision. SMFNet’s PSF ablation indicates that the spatial weight map becomes much more effective when guided by pseudo labels, not merely learned implicitly (He et al., 29 Jul 2025). Within the evidence reported, the framework’s contribution is therefore not only architectural but also supervisory: it turns modality selection into a trainable, contribution-aware process.

In the literature represented by these 2025 papers, SMFNet occupies a distinct position as a selective fusion framework for RGB-D VSOD in which optical flow and depth are first reconciled by pixel-level competition and only then integrated with RGB through multi-dimensional attention (He et al., 29 Jul 2025). Its broader significance lies in demonstrating how selective gating, pseudo supervision, and structurally decomposed attention can be combined to model unequal modality utility in dynamic multimodal perception.