SalientFusion: Saliency-Based Fusion Strategy

• SalientFusion is a saliency-centered fusion approach that integrates complementary input streams to emphasize perceptually relevant content.
• It employs methods like weighted feature selection, hyper-dense connectivity, and guided filtering to fuse multi-modal data effectively.
• Applications range from RGB-D salient object detection to infrared-visible fusion and zero-shot food recognition, demonstrating robust performance.

SalientFusion denotes a family of saliency-centered fusion formulations in which complementary inputs, feature streams, or semantic representations are integrated so that perceptually or task-relevant content dominates the fused output. In the literature considered here, the term appears both as a general design principle—covering salient object detection, RGB-D saliency, infrared-visible image and video fusion, multispectral aerial fusion, and general multi-modal SOD—and as the specific name of a context-aware framework for compositional zero-shot food recognition (Zhang et al., 2018, Qu et al., 2016, Ma et al., 2024, Li et al., 2 Apr 2026, Youssef et al., 2023, Song et al., 4 Sep 2025). Across these settings, the central operation is not mere aggregation, but saliency-aware selection: fusion weights, gates, masks, or interaction operators are constructed so that salient targets, informative regions, or composition-relevant tokens are preferentially preserved.

1. Terminological scope and research trajectory

The surveyed literature indicates that “SalientFusion” is not a single canonical architecture. Instead, it recurs as a unifying principle in multiple research lines. An early RGB-D formulation fused seven handcrafted saliency cues by CNN-based hierarchical feature fusion and then refined the output by Laplacian propagation, explicitly arguing that feature-level fusion is more effective than saliency map-level merging (Qu et al., 2016). HyperFusion-Net then realized SalientFusion in RGB salient object detection by decomposing an RGB image into complementary reflective views and hyper-densely fusing multi-scale, multi-branch features (Zhang et al., 2018). A separate line of work used the term for zero-learning image fusion, where low-frequency base layers are fused using visual saliency and high-frequency detail layers are fused using pre-trained deep features, without task-specific training (Lahoud et al., 2019).

Subsequent work broadened the concept substantially. Lightweight RGB-D salient object detection revisited where to fuse modalities and proposed middle-level fusion with an Information-aware Multi-Modal Feature Fusion module and a Lightweight Feature-level and Decision-level Fusion module (Huang et al., 2021). Infrared-visible image fusion and saliency detection were coupled in an interactively reinforced multi-task paradigm, where the fused image acts as a third modality for downstream saliency prediction (Wang et al., 2023). Multispectral aerial fusion combined conventional aerial images with synthetic-aperture integral images so that salient unoccluded targets from integral imagery are injected into a basis channel that preserves spatial references (Youssef et al., 2023).

More recent works extended SalientFusion to state-space and video settings. S4Fusion used a Saliency-aware Selective State Space Fusion Model with a Cross-Modal Spatial Awareness Module and predictive-entropy minimization to highlight salient targets in infrared-visible fusion (Ma et al., 2024). LFNet introduced liquid fusion to harmonize CNN and SSM representations for general salient object detection across RGB, RGB-D, RGB-T, VSOD, and VDT tasks (Chen et al., 25 Jun 2026). MAVFusion specialized saliency-driven fusion to infrared-visible video by allocating sparse strong interaction to motion regions and weak interaction to static regions (Li et al., 2 Apr 2026). In parallel, the explicitly titled “SalientFusion” framework for compositional zero-shot food recognition fused RGB, foreground, and depth tokens and debiased text prompts through cross-attention (Song et al., 4 Sep 2025).

A common misconception is that SalientFusion is synonymous with attention-only fusion. The literature does not support that view. SalientFusion appears in hyper-dense connectivity, Laplacian propagation, guided filtering, selective state-space modeling, motion-aware sparse interaction, and vision-language cross-attention, depending on task requirements (Zhang et al., 2018, Qu et al., 2016, Ma et al., 2024, Li et al., 2 Apr 2026, Song et al., 4 Sep 2025).

2. Core principles of saliency-centered fusion

At the methodological level, SalientFusion repeatedly relies on complementarity between inputs or representations. In RGB-D saliency, color contrast, depth contrast, background priors, and color compactness are treated as mutually corrective cues; the CNN receives cue vectors rather than raw pixels so that learning is concentrated on saliency-relevant dimensions (Qu et al., 2016). In HyperFusion-Net, the complementary pair is created internally by content-preserving reflection separation, producing a transmitted view $X_T = X_O - E$ and a reflected view $X_R^k = -k(X_O - E)$, with $k = 1$ in experiments (Zhang et al., 2018). In infrared-visible fusion, the complementarity is typically thermal saliency versus visible texture (Ma et al., 2024, Li et al., 2 Apr 2026, Wang et al., 2023). In food recognition, foreground masking suppresses background redundancy while depth cues reduce staple/side role confusion (Song et al., 4 Sep 2025).

A second recurring principle is that saliency-aware fusion is usually performed before hard commitment to a final decision. The RGB-D deep fusion model explicitly contrasts feature-level fusion with map-level fusion and reports that map-level alternatives such as linear fusion, CRF fusion, MCA, and CNN-F perform notably worse (Qu et al., 2016). HyperFusion-Net similarly avoids post hoc fusion of predictions and instead aggregates complementary streams across many scales through hierarchical hyper-dense links (Zhang et al., 2018). Middle-level RGB-D fusion makes the same argument in architectural terms: early mixing loses modality-specific nuances, whereas repeated two-stream fusion is parameter-heavy; a single information-aware fusion point at the third level is presented as a compromise that preserves modality differences while reducing parameters (Huang et al., 2021).

A third principle is the explicit construction of saliency weights, gates, or masks. These may be analytic, as in histogram-based saliency for base-layer fusion and guided-filtered per-pixel weights in zero-learning image fusion (Lahoud et al., 2019); graph-based, as in seed selection plus Laplacian propagation in RGB-D saliency (Qu et al., 2016); connectivity-driven, as in HyperFusion-Net’s cross-branch and top-down dense aggregation (Zhang et al., 2018); or dynamically learned, as in LFNet’s content-aware permeability gate, MAVFusion’s motion-guided Top-K selection, and DebiasAT’s cross-attention-based prompt refinement (Chen et al., 25 Jun 2026, Li et al., 2 Apr 2026, Song et al., 4 Sep 2025).

This suggests that SalientFusion is better understood as a control strategy over information flow than as a fixed fusion operator. The operator may be concatenation, attention, state-space recurrence, additive reconstruction, or guided weighting; what remains invariant is the attempt to privilege salient content while suppressing distractors or redundant signals.

3. Architectural archetypes

Several architectural archetypes recur in the literature.

Hyper-dense multi-branch fusion is exemplified by HyperFusion-Net. Its Interweaved CNN has two weight-stitching branches that share convolutional weights but maintain domain-specific batch-normalization statistics through AdaBN, producing complementary transmitted and reflected features $\{f_T^l, f_R^l\}$. These are fused by a hyper-dense branch in which each scale combines current transmitted features, current reflected features, and top-down fused features from the next layer. The fusion at layer $l$ within a block takes the form $$\hat f^l = g(\{f_T^l, \hat f^{l+1}, f_R^l\}; \theta_{hf})$$, and multi-resolution fused features are then integrated by $$h(\{\hat f^{\underline L_1}, \hat f^{\underline L_2}, \dots\}; \theta_{hf})$$ before final pixel-wise prediction (Zhang et al., 2018).

Cue-vector hierarchical fusion characterizes the RGB-D deep fusion model. Each superpixel is described by seven cue vectors: local/global color contrast, local/global depth contrast, color/depth background priors, and color compactness. Five $N$-length vectors are reshaped to five $32 \times 32$ channels, while the background-prior vectors are zero-padded, concatenated, and reshaped into a sixth channel, yielding a $32 \times 32 \times 6$ input for a shallow CNN. The CNN outputs a regional saliency confidence, and Laplacian propagation then enforces spatial consistency over a color-depth graph (Qu et al., 2016).

Two-scale saliency/detail decomposition defines the zero-learning formulation. Each source image is decomposed into a base layer $X_R^k = -k(X_O - E)$0 and a detail layer $X_R^k = -k(X_O - E)$1. Base fusion uses histogram-based saliency maps $X_R^k = -k(X_O - E)$2, normalized across inputs and refined by guided filtering; detail fusion uses pre-trained CNN features, channel-wise $X_R^k = -k(X_O - E)$3 activity maps, softmax weights, and guided filtering, followed by reconstruction $X_R^k = -k(X_O - E)$4 (Lahoud et al., 2019).

Intermediate modality-aware fusion appears in lightweight RGB-D SOD. The IMFF module projects third-level RGB and depth features into an information-aware space, computes total, shared, and difference descriptors, then produces modality-specific channel-selection weights $X_R^k = -k(X_O - E)$5 and fuses by $X_R^k = -k(X_O - E)$6. LFDF then performs bidirectional aggregation of feature-level and decision-level saliency information across levels $X_R^k = -k(X_O - E)$7 (Huang et al., 2021).

State-space and dynamic-gating formulations dominate later work. S4Fusion uses VMamba-based Visual State Space blocks and a Cross-Modal Spatial Awareness Module that interleaves infrared and visible sequences along four scan directions while sharing hidden states and employing modality-selective $X_R^k = -k(X_O - E)$8, $X_R^k = -k(X_O - E)$9, and $k = 1$0 parameters in the discrete recurrence $k = 1$1, $k = 1$2 (Ma et al., 2024). LFNet casts VMamba features as state and ConvNeXt features as stimulus, then fuses them by a closed-form liquid gate,

$k = 1$3

where $k = 1$4 is a learned spatial permeability (Chen et al., 25 Jun 2026). MAVFusion applies a related logic to video: motion masks derived from optical flow identify dynamic patches, Top-K selection restricts heavy cross-modal attention to those regions, and a weak local-interaction branch preserves static backgrounds (Li et al., 2 Apr 2026).

Vision-language SalientFusion, as defined for CZSFR, uses SalientFormer and DebiasAT. SalientFormer encodes the original image, its foreground segmentation, and its depth map with a CLIP ViT-L/14 image encoder. Queries and keys are convex combinations of depth and foreground tokens, while values are original-image tokens:

$k = 1$5

The fused tokens are then used both for visual recognition and for cross-attentive refinement of attribute, object, and composition text embeddings in DebiasAT (Song et al., 4 Sep 2025).

4. Objectives, supervision, and optimization regimes

The loss design in SalientFusion systems typically mirrors the target notion of saliency. In salient object detection, pixel-wise classification remains central. HyperFusion-Net uses a per-image balanced BCE,

$k = 1$6

with $k = 1$7, and augments it with a structure-perceptual loss

$k = 1$8

yielding $k = 1$9 with $\{f_T^l, f_R^l\}$0 (Zhang et al., 2018). LFNet instead uses multi-scale deep supervision across four decoder stages with $\{f_T^l, f_R^l\}$1 (Chen et al., 25 Jun 2026). Lightweight RGB-D SOD combines cross-entropy with an edge refinement term based on Sobel gradients and applies deep supervision to intermediate forward and backward predictions (Huang et al., 2021).

In fusion tasks, losses are frequently designed to preserve both saliency and structural fidelity. S4Fusion minimizes predictive entropy from a pretrained ResNet-50,

$\{f_T^l, f_R^l\}$2

and balances it with $\{f_T^l, f_R^l\}$3, $\{f_T^l, f_R^l\}$4, and a Sobel-based gradient term through

$\{f_T^l, f_R^l\}$5

with $\{f_T^l, f_R^l\}$6 (Ma et al., 2024). IRFS formalizes the bridge between fusion and detection by

$\{f_T^l, f_R^l\}$7

where $\{f_T^l, f_R^l\}$8 combines saliency-weighted intensity and gradient terms, and $\{f_T^l, f_R^l\}$9 combines weighted BCE and weighted IoU at coarse and precise stages (Wang et al., 2023). MAVFusion adds an explicit temporal term,

$l$0

where $l$1 compares each fused frame to warped adjacent fused frames under validity masks (Li et al., 2 Apr 2026).

A notable contrast is provided by training-free or zero-learning variants. The two-scale image-fusion method uses off-the-shelf pretrained CNNs purely as fixed feature extractors and computes all fusion rules analytically from saliency and feature responses, with no task-specific training or fine-tuning (Lahoud et al., 2019). The aerial multispectral method likewise uses pretrained VGG-19 features and a model-based unified filter without training the fusion pipeline (Youssef et al., 2023). By contrast, the food-recognition SalientFusion is fully trainable and optimizes branch-wise CLIP-style cross-entropies for attributes, objects, and compositions, combined as $l$2 (Song et al., 4 Sep 2025).

This diversity of supervision regimes complicates any narrow definition of SalientFusion as purely deep or purely end-to-end. Some variants are explicitly zero-learning, some are multi-task, and some are prompt-based vision-language systems (Lahoud et al., 2019, Youssef et al., 2023, Song et al., 4 Sep 2025).

5. Empirical performance and practical characteristics

The empirical record reported in these papers is consistently framed in terms of improved saliency preservation, structural coherence, or downstream utility.

System	Setting	Representative result
HyperFusion-Net (Zhang et al., 2018)	RGB salient object detection	ECSSD: $l$3, MAE $l$4, $l$5
RGB-D SalientFusion (Qu et al., 2016)	RGB-D salient object detection	LFSD: $l$6
Middle-level fusion (Huang et al., 2021)	Lightweight RGB-D SOD	3.9M parameters, $l$7 FPS
S4Fusion (Ma et al., 2024)	Infrared-visible image fusion	M3FD detection: mAP@50 $l$8
MAVFusion (Li et al., 2 Apr 2026)	Infrared-visible video fusion	$l$9 FPS at $$\hat f^l = g(\{f_T^l, \hat f^{l+1}, f_R^l\}; \theta_{hf})$$0, 9.90M parameters
LFNet (Chen et al., 25 Jun 2026)	General SOD across five tasks	RGB DUTS: Sm/Fm/Em $$\hat f^l = g(\{f_T^l, \hat f^{l+1}, f_R^l\}; \theta_{hf})$$1; 43.23M parameters
SalientFusion for CZSFR (Song et al., 4 Sep 2025)	Compositional zero-shot food recognition	CZSFood-164 closed-world: HM $$\hat f^l = g(\{f_T^l, \hat f^{l+1}, f_R^l\}; \theta_{hf})$$2, AUC $$\hat f^l = g(\{f_T^l, \hat f^{l+1}, f_R^l\}; \theta_{hf})$$3

In HyperFusion-Net, the reported advantage is explicitly tied to reflective decomposition and hyper-dense fusion. On ECSSD ablations, the full transmitted/reflected model with $$\hat f^l = g(\{f_T^l, \hat f^{l+1}, f_R^l\}; \theta_{hf})$$4 reached $$\hat f^l = g(\{f_T^l, \hat f^{l+1}, f_R^l\}; \theta_{hf})$$5, MAE $$\hat f^l = g(\{f_T^l, \hat f^{l+1}, f_R^l\}; \theta_{hf})$$6, and $$\hat f^l = g(\{f_T^l, \hat f^{l+1}, f_R^l\}; \theta_{hf})$$7, outperforming RGB-only, input-fusion, early-fusion, late-fusion, and ad-hoc fusion variants (Zhang et al., 2018). The RGB-D deep-fusion model similarly reports that CNN-only performance on NLPR, NJUD, and LFSD improves from $$\hat f^l = g(\{f_T^l, \hat f^{l+1}, f_R^l\}; \theta_{hf})$$8, $$\hat f^l = g(\{f_T^l, \hat f^{l+1}, f_R^l\}; \theta_{hf})$$9, and $$h(\{\hat f^{\underline L_1}, \hat f^{\underline L_2}, \dots\}; \theta_{hf})$$0 to $$h(\{\hat f^{\underline L_1}, \hat f^{\underline L_2}, \dots\}; \theta_{hf})$$1, $$h(\{\hat f^{\underline L_1}, \hat f^{\underline L_2}, \dots\}; \theta_{hf})$$2, and $$h(\{\hat f^{\underline L_1}, \hat f^{\underline L_2}, \dots\}; \theta_{hf})$$3 after Laplacian propagation, supporting the claim that reliable seeds plus graph propagation improve spatial consistency (Qu et al., 2016).

Efficiency is a recurrent design constraint. HyperFusion-Net reports inference speed of approximately $$h(\{\hat f^{\underline L_1}, \hat f^{\underline L_2}, \dots\}; \theta_{hf})$$4 fps on a TITAN 1070 GPU, with shared convolutional weights and $$h(\{\hat f^{\underline L_1}, \hat f^{\underline L_2}, \dots\}; \theta_{hf})$$5 convolutions used to limit cost (Zhang et al., 2018). The lightweight RGB-D model reports approximately 3.9M parameters and $$h(\{\hat f^{\underline L_1}, \hat f^{\underline L_2}, \dots\}; \theta_{hf})$$6 FPS, with performance on NJU2000 of MAE $$h(\{\hat f^{\underline L_1}, \hat f^{\underline L_2}, \dots\}; \theta_{hf})$$7, $$h(\{\hat f^{\underline L_1}, \hat f^{\underline L_2}, \dots\}; \theta_{hf})$$8, $$h(\{\hat f^{\underline L_1}, \hat f^{\underline L_2}, \dots\}; \theta_{hf})$$9, and $N$0 (Huang et al., 2021). MAVFusion reports 123.37G FLOPs, 9.90M parameters, and 14.16 FPS at $N$1, while claiming approximately 5.7% of UniVF’s computation at 480P (Li et al., 2 Apr 2026). LFNet emphasizes a parameter advantage over Samba, for example 43.23M versus 49.59M on RGB SOD and 46.07M versus 60.28M on VDT, while improving metrics across the five task families (Chen et al., 25 Jun 2026).

Downstream utility is another repeated theme. S4Fusion reports the best mAP@50 on M3FD multimodal detection at $N$2, compared with $N$3 for IR-only and $N$4 for VIS-only (Ma et al., 2024). IRFS reports that the coupled fusion-and-SOD framework improves both fusion metrics and saliency accuracy on VT5000, and that using the FSFNet fused image as the third modality yields the best saliency results relative to using infrared, visible, or rough average fusion as the auxiliary input (Wang et al., 2023). The aerial multispectral method reports higher MI, VIF, and PSNR than two-channel and multi-channel baselines in search and rescue, wildfire, and wildlife scenarios, with average two-channel results of VIF $N$5, MI $N$6, and PSNR $N$7 (Youssef et al., 2023). In food recognition, SalientFusion improves over Troika by $N$8 and $N$9 in HM on CZSFood-90 and CZSFood-164 closed-world testing, and by $32 \times 32$0 and $32 \times 32$1 in AUC, while also improving real-world unseen-composition results (Song et al., 4 Sep 2025).

A plausible implication is that saliency-aware fusion is valuable not only when the output itself is a saliency map, but also when saliency is an intermediate organizing principle for fusion, recognition, or detection.

6. Limitations, failure modes, and open directions

The limitations reported across the literature are highly task-dependent, but several recurring failure modes are evident. When saliency cues are weak or misleading in all modalities, fusion quality deteriorates. The RGB-D deep-fusion model states that detection may fail or be incomplete if the salient object and background share highly similar color and depth (Qu et al., 2016). HyperFusion-Net notes that scattered fine details can lead to highlighting bulk regions instead of entire objects, that large variance in intra-object saliency may cause missed regions, and that cluttered illumination can confuse reflection-based cues (Zhang et al., 2018). The food-recognition SalientFusion identifies attribute confusion among visually similar cooking methods, such as “fried pork” being misinterpreted as “braised” or “stewed,” even when the ingredient is correctly recognized (Song et al., 4 Sep 2025).

Several formulations also depend heavily on reliable auxiliary signals. MAVFusion explicitly identifies front-end optical-flow cost and flow reliability under severe sensor noise or extreme degradation as open challenges (Li et al., 2 Apr 2026). LFNet notes that noisy depth, thermal, or motion cues may mislead the stimulus stream, and that FFT-based Saliency-Guided Upsampling may be sensitive to distribution shifts (Chen et al., 25 Jun 2026). The aerial multispectral method is sensitive to registration errors, rolling shutter, sparse synthetic-aperture sampling, and spectral mismatch, all of which can leak artifacts into the salient masks or fused result (Youssef et al., 2023). The zero-learning fusion method assumes pre-registered inputs, although guided filtering can mitigate small residual misregistrations (Lahoud et al., 2019).

Open directions are stated explicitly in multiple papers. RGB-D deep fusion proposes end-to-end multi-branch CNNs that ingest raw pixels alongside cue branches and multi-task learning such as joint edge detection or segmentation (Qu et al., 2016). The aerial method suggests adaptive focal-surface selection and multi-depth sectioning (Youssef et al., 2023). LFNet proposes lightweight edge-oriented variants, extension to other dense tasks, and SSM-specific frequency shaping (Chen et al., 25 Jun 2026). MAVFusion points toward more robust motion estimation and better handling of dynamic scenes (Li et al., 2 Apr 2026). The food-recognition framework suggests adding recipe text or other modalities to resolve intrinsically ambiguous cuisine primitives (Song et al., 4 Sep 2025).

Another misconception addressed by the surveyed work is that saliency-aware fusion necessarily implies improved universality. The papers are more circumspect. Most report strong benchmark performance, but also document sensitivity to modality quality, alignment, depth estimation, flow estimation, or prompt semantics (Chen et al., 25 Jun 2026, Li et al., 2 Apr 2026, Song et al., 4 Sep 2025). SalientFusion therefore functions less as a guarantee of robustness than as a systematic attempt to bias fusion toward informative content under specific assumptions about the input signals.

In aggregate, the literature presents SalientFusion as a broad methodological doctrine: identify salient or task-relevant evidence, encode complementarity between heterogeneous inputs, and regulate fusion so that salient information survives aggregation with minimal structural degradation. The particular mechanisms vary—from reflective decomposition and hyper-dense links, to handcrafted cue vectors, guided filters, state-space recurrences, motion-aware sparse interaction, and vision-language prompt debiasing—but the organizing objective remains the same across domains (Zhang et al., 2018, Qu et al., 2016, Ma et al., 2024, Li et al., 2 Apr 2026, Song et al., 4 Sep 2025).