Plug-and-Play RGB-D Fusion Module

• Plug-and-play RGB-D fusion modules are reusable components that fuse aligned RGB and depth features while preserving modality-specific cues.
• Multiple fusion strategies such as channel-wise gating, residual cross-modality injection, and saliency-guided modulation are explored to enhance feature complementarity.
• These modules can be integrated at different network stages, improving performance in tasks like semantic segmentation and saliency detection with adaptable fusion approaches.

Searching arXiv for the cited fusion papers to ground the article in current records. arxiv_search(query="Deep feature selection-and-fusion for RGB-D semantic segmentation (Su et al., 2021)", max_results=5) arxiv_search(query="Plug-and-play RGB-D fusion module RGB-D fusion arXiv semantic segmentation saliency detection", max_results=10) A plug-and-play RGB-D fusion module is a reusable component that is inserted into an existing RGB-D network to combine aligned RGB and depth features under minimal architectural assumptions, typically requiring matched spatial size and channel count and returning a fused tensor that can be consumed by an unchanged decoder or task head. In the literature, this concept spans encoder-side feature fusion blocks, decoder-side detail reinjection modules, neck-level fusion units, prediction-level gating schemes, and even training-free inference-time feature updates. Across these variants, the recurring objective is to avoid indiscriminate fusion, preserve modality-specific cues, and adapt the contribution of RGB and depth according to complementarity, reliability, scale, or task guidance (Su et al., 2021).

1. Conceptual scope and recurring interface

In the architectural sense, a plug-and-play RGB-D fusion module is typically defined by a simple interface of the form

$$(F_{\text{rgb}}, F_{\text{d}}) \mapsto F_{\text{fused}}$$

or, when task guidance is used,

$(F_1, F_2, S) \mapsto F,$

with $$F_{\text{rgb}}, F_{\text{d}}, F_1, F_2 \in \mathbb{R}^{C \times H \times W}$$ and $S$ a guidance map or saliency tensor. The common precondition is spatial alignment between RGB and depth and, in most formulations, matching channel numbers or a preceding $1\times1$ projection (Huang et al., 2024).

This notion is broader than a single architecture. In FSFNet, the plug-in is an encoder-side multi-scale fusion block plus a decoder-side detail-aware skip fusion; in SEFFSal, the same fusion operator is reused for both RGB-depth fusion and cross-scale decoder fusion; in Weighted Feature Fusion, the module is a lightweight channel-wise gate that can be “seamlessly integrated with various deep neural network backbones as a plug-and-play solution”; in RXFOOD, the plug-in sits between encoder and decoder and returns enhanced multi-scale features for both branches (Su et al., 2021, Huang et al., 2024, Lin et al., 3 Aug 2025, Ma et al., 2023).

The phrase “plug-and-play” is also used in a stricter, inference-time sense. The PnP depth module updates an intermediate feature map $z$ of a frozen depth network so that predictions become consistent with sparse depth measurements, without retraining the original model. In that formulation, the plug-in is not a learned architectural block inserted during training, but a training-free optimization over latent features at test time (Wang et al., 2018). This suggests that the term covers both reusable fusion layers and reusable fusion procedures.

2. Architectural placement and signal flow

Most plug-and-play RGB-D modules are inserted at one of three locations. The first is the encoder, where RGB and depth features at corresponding stages are fused before deeper processing. FSFNet exemplifies this pattern with a two-stream RGB-HHA encoder and a third fusion branch, where each encoder stage $j$ produces a fused feature $F_{\text{fuse}}^j$ from $F_{\text{rgb}}^j$, $F_{\text{hha}}^j$, and optionally the previous fused feature $(F_1, F_2, S) \mapsto F,$0 (Su et al., 2021). Middle-level fusion for lightweight RGB-D saliency detection follows a related strategy: low- and middle-level unimodal features are extracted separately, then a shared multimodal encoder processes the fused representation from the middle stage onward (Huang et al., 2021).

The second location is the decoder or neck, where already extracted modality-specific or fused features are merged across scales and reweighted before final prediction. SEFF uses one operator $(F_1, F_2, S) \mapsto F,$1 both for encoder RGB-depth fusion and for cross-scale decoder fusion; DCTNet+ uses RFM and UIM to refine and fuse RGB, depth, and flow features at every hierarchy; the Parkinsonian gait framework replaces the original YOLO neck with MLGE plus Cross-Spatial Neck Fusion, receiving aligned multi-scale RGB and depth backbone features and producing fused neck outputs $(F_1, F_2, S) \mapsto F,$2 and $(F_1, F_2, S) \mapsto F,$3 (Huang et al., 2024, Mou et al., 2023, Alnaasan et al., 4 Dec 2025).

The third location is the prediction layer. Adaptive Fusion for RGB-D Salient Object Detection computes a switch map from fine-scale RGB and depth features, then fuses scalar saliency predictions via a convex combination

$(F_1, F_2, S) \mapsto F,$4

This is a plug-in in the sense that it only requires aligned feature maps and two unimodal score maps, and can therefore be attached to another RGB-D predictor without changing the unimodal encoders (1901.01369).

A recurring design decision is whether fusion yields one shared stream or preserves modality-specific streams after interaction. FSFNet and WFF output a single fused feature per stage, whereas LEAF-Mamba’s AFM first outputs interacted features $(F_1, F_2, S) \mapsto F,$5 for continued per-modality processing and only then emits a fused feature $(F_1, F_2, S) \mapsto F,$6 for the decoder (Wu et al., 23 Sep 2025). This distinction is central to how much modality-specific structure is retained after each plug-in step.

3. Core fusion mechanisms

The simplest family uses explicit weighted combination. WFF formulates fusion as

$(F_1, F_2, S) \mapsto F,$7

with channel-wise weights obtained from global average pooled statistics of $(F_1, F_2, S) \mapsto F,$8, a shared FC reduction, two modality-specific FC expansions, and a two-way Softmax (Lin et al., 3 Aug 2025). This yields a lightweight competitive gate: each channel allocates its mass between RGB and depth rather than scaling both independently.

A second family uses residual cross-modality injection. FSFNet’s SCRF first selects complementary subsets

$(F_1, F_2, S) \mapsto F,$9

then forms cross-modality residual sums

$$F_{\text{rgb}}, F_{\text{d}}, F_1, F_2 \in \mathbb{R}^{C \times H \times W}$$0

and finally concatenates them with downsampled previous fused features. The residual term always comes from the other modality, which makes the fusion explicitly bidirectional and symmetric (Su et al., 2021).

A third family uses task-guided or saliency-guided modulation. SEFF concatenates the same saliency tensor $$F_{\text{rgb}}, F_{\text{d}}, F_1, F_2 \in \mathbb{R}^{C \times H \times W}$$1 to both inputs, refines them with convolutional blocks, sums the refined features, and then computes a combined local-channel and global-channel context: $$F_{\text{rgb}}, F_{\text{d}}, F_1, F_2 \in \mathbb{R}^{C \times H \times W}$$2 Its key property is that saliency guidance is external to the two features being fused, so the same operator can be reused across modalities and scales as long as a guidance map is available (Huang et al., 2024).

A fourth family uses learned spatially varying modality weights. MHENet’s ADFM first applies global cross-guidance,

$$F_{\text{rgb}}, F_{\text{d}}, F_1, F_2 \in \mathbb{R}^{C \times H \times W}$$3

then predicts two spatial weight maps through a Softmax split,

$$F_{\text{rgb}}, F_{\text{d}}, F_1, F_2 \in \mathbb{R}^{C \times H \times W}$$4

and fuses them point-wise as

$$F_{\text{rgb}}, F_{\text{d}}, F_1, F_2 \in \mathbb{R}^{C \times H \times W}$$5

Unlike WFF, the competition between modalities is now spatially varying rather than purely channel-wise (Niu et al., 3 Apr 2026).

A fifth family operates on relations rather than raw features. RXFOOD computes spatial energy matrices

$$F_{\text{rgb}}, F_{\text{d}}, F_1, F_2 \in \mathbb{R}^{C \times H \times W}$$6

and channel energy matrices

$$F_{\text{rgb}}, F_{\text{d}}, F_1, F_2 \in \mathbb{R}^{C \times H \times W}$$7

exchanges these energies across scales and modalities via SEEM and CEEM, and only then reconstructs attention-refined features. This makes the plug-in a cross-scale, cross-branch energy mixer rather than a direct feature combiner (Ma et al., 2023).

A sixth family fuses at the level of state-space dynamics and second-order similarity. LEAF-Mamba’s AFM uses Cross-modality Second-order Pooling to build similarity maps $$F_{\text{rgb}}, F_{\text{d}}, F_1, F_2 \in \mathbb{R}^{C \times H \times W}$$8 and distance maps $$F_{\text{rgb}}, F_{\text{d}}, F_1, F_2 \in \mathbb{R}^{C \times H \times W}$$9, then applies a Selective Interaction Module in which RGB and depth swap SSM output matrices $S$0 and modulate them by distance maps, followed by a Selective Enhancement Module that uses similarity-weighted SSM outputs for reliable fusion (Wu et al., 23 Sep 2025). This suggests a more recent trend: replacing explicit attention with SSM-based interaction while keeping the plug-in interface unchanged.

4. Representative module families

The major families can be summarized as follows.

Module family	Core operation	Plug-and-play role
SCRF + DFP	Cross-modality residual sums, cascading, detail propagation	Encoder fusion plus decoder detail reinjection
SEFF	Saliency-guided feature conditioning and context reweighting	Unified operator for RGB-depth and cross-scale fusion
Switch-map fusion	Per-pixel convex combination of unimodal predictions	Prediction-level fusion block
WFF	Channel-wise Softmax competition between RGB and depth	Lightweight feature-level gate
RXFOOD	Spatial/channel energy exchange across scales and modalities	Encoder–decoder bridge for dual-branch networks
ADFM / AFM	Spatially varying dynamic weights or SSM-based selective interaction	Adaptive feature fusion with explicit reliability structure

FSFNet is often taken as a direct blueprint for a reusable RGB-D block because its SCRF interface is generic: it accepts pairs of aligned feature maps, optionally a previous fused feature, and outputs a fused tensor; DFP separately handles detail recovery at intermediate decoder scales (Su et al., 2021). This separation of roles—cross-modality fusion versus detail recovery—has influenced later modular designs.

SEFF generalizes the plug-in notion by making the same $S$1 operator valid for both cross-modal and cross-scale fusion, provided a saliency map or compressed saliency tensor is available. This is one of the clearest examples of a single fusion primitive reused in multiple network locations (Huang et al., 2024).

Adaptive Fusion with a switch map is narrower in interface but highly portable. Because it consumes any aligned RGB/depth features plus two scalar-valued prediction maps, it can be attached to many existing two-branch predictors without redesigning feature extraction (1901.01369). A plausible implication is that prediction-level plug-ins are particularly attractive when the backbone must remain untouched.

WFF and ADFM represent two related but distinct directions in adaptive weighting. WFF is globally conditioned and channel-wise, whereas ADFM is spatially varying and preceded by modality-specific enhancement. This suggests a useful design distinction: lightweight modules often estimate reliability at channel level, while stronger modules allocate reliability per location (Lin et al., 3 Aug 2025, Niu et al., 3 Apr 2026).

RXFOOD and LEAF-Mamba move the plug-in abstraction into relation space. RXFOOD exchanges energy matrices across scales and modalities; AFM in LEAF-Mamba uses second-order pooling and selective state-space interaction to separate complementary regions from reliable regions (Ma et al., 2023, Wu et al., 23 Sep 2025). This suggests that plug-and-play fusion is no longer limited to sum/concat/gate templates, but increasingly includes modular relation modeling.

5. Supervision, training regimes, and evaluation

Plug-and-play RGB-D fusion modules are not tied to a single supervision strategy. In FSFNet, the fusion blocks are trained with weighted cross-entropy and pyramid supervision; in SEFFSal, the module depends on deeply supervised saliency maps from three scales and four decoder layers, trained with Adaptive Pixel Intensity Loss

$S$2

with $S$3, $S$4, $S$5 (Su et al., 2021, Huang et al., 2024). In the switch-map method, supervision includes saliency BCE on all outputs, a switch-map BCE using a pseudo target derived from RGB correctness, and an edge-preserving gradient loss (1901.01369).

Several modules are trained purely through task loss, without any dedicated fusion objective. WFF is optimized end-to-end only with binary cross-entropy for glass segmentation, despite its explicit gating structure (Lin et al., 3 Aug 2025). DSA²F similarly trains its NAS-searched fusion cells under ordinary pixel-wise cross-entropy on the final saliency map (Sun et al., 2021). This corrects a common misconception that an RGB-D fusion plug-in must always require fusion-specific supervision.

At the same time, some modules do exploit explicit auxiliary signals. SEFF relies on saliency guidance; DCTNet+ supervises multi-scale predictions and a coarse map $S$6 with BCE+IoU; Fusion4CA attaches a camera auxiliary branch and a contrastive alignment loss during training, although these are removed at inference (Mou et al., 2023, Luo et al., 5 Mar 2026). This suggests that “plug-and-play” refers to runtime interface and transplantability, not necessarily to minimalist training.

Evaluation also varies with task. FSFNet reports mIoU and Pixel Accuracy on NYUDv2 and SUN RGB-D, with ablations showing that SCRF alone improves NYUDv2 mIoU from $S$7 to $S$8, DFP alone to $S$9, and the full model to $1\times1$0 (Su et al., 2021). SEFFSal reports average $1\times1$1, $1\times1$2, $1\times1$3, and $1\times1$4 for Ours-m across five benchmark datasets, and shows that replacing SEFF by same-parameter CBR blocks degrades all metrics (Huang et al., 2024). WFF highlights boundary sensitivity, improving PSPNet bIoU from $1\times1$5 to $1\times1$6 on the full split and from $1\times1$7 to $1\times1$8 on the Difficult split (Lin et al., 3 Aug 2025). These results indicate that different plug-ins are often best understood through the metrics most aligned with the task: region overlap for segmentation, boundary IoU for glass surfaces, or BEV detection scores for 3D detection.

6. Limitations, ambiguities, and extensions

A recurrent limitation is the assumption of aligned and sufficiently reliable depth. FSFNet explicitly assumes spatially aligned modalities and notes no special handling for missing or noisy depth; WFF is motivated by glass scenes precisely because depth can be missing or distorted; DSA²F assumes that depth histograms contain meaningful modes for DSAM; ADFM improves robustness but still presupposes image-space alignment between enhanced RGB and depth features (Su et al., 2021, Lin et al., 3 Aug 2025, Sun et al., 2021, Niu et al., 3 Apr 2026). A plausible implication is that plug-and-play modules differ substantially in how gracefully they degrade when depth quality collapses.

Another ambiguity concerns the meaning of “plug-and-play.” In most RGB-D feature-fusion papers, it means a reusable architectural block or neck that can be dropped into an encoder–decoder with limited assumptions. In the sparse-depth PnP formulation, however, it means a training-free inference-time optimizer over intermediate features (Wang et al., 2018). These are related but not identical uses: one modifies architecture, the other modifies latent activations.

A further misconception is that plug-and-play fusion must be symmetric. Some modules are explicitly symmetric, such as SCRF or JL-DCF’s cooperative addition-and-multiplication fusion; others are intentionally asymmetric, treating RGB as main and depth as auxiliary, as in DCTNet+ or several RGB-D salient object detection systems (Su et al., 2021, Fu et al., 2020, Mou et al., 2023). This suggests that symmetry is a design choice rather than a defining property.

Extensions are already visible in adjacent modalities and tasks. AFM in LEAF-Mamba is used for RGB-T as well as RGB-D; SEFF is formulated for any two feature maps plus a guidance map; the Parkinsonian gait framework frames MLGE plus neck fusion as reusable for action recognition, pose estimation, and general gait analysis; Fusion4CA provides a plug-and-play blueprint for camera–LiDAR BEV fusion with training-only auxiliary modules and lightweight inference-time add-ons (Wu et al., 23 Sep 2025, Huang et al., 2024, Alnaasan et al., 4 Dec 2025, Luo et al., 5 Mar 2026). This suggests that the most durable abstraction is not “RGB-D saliency block” or “RGB-D segmentation block,” but a modular operator that accepts aligned multimodal features and returns fused features with preserved interface contracts.

In that broader sense, the plug-and-play RGB-D fusion module is less a single mechanism than a design pattern: modality-specific feature extraction, explicit complementary cue selection, adaptive weighting or relation modeling, optional cross-scale propagation, and interface preservation for integration into arbitrary backbones. The historical trajectory from residual sums and switch maps to energy exchange, saliency-conditioned fusion, and SSM-based adaptive modules indicates that the field has progressively moved from implicit fusion toward explicit, learnable, and structurally modular fusion operators (Su et al., 2021, Ma et al., 2023, Wu et al., 23 Sep 2025).