Fusion-ResNet Architectures

• Fusion-ResNet is a deep learning architecture that integrates standard ResNet backbones with auxiliary feature streams to combine diverse signals from different modalities.
• These models employ explicit fusion modules using techniques such as ZCA whitening, PCA/ICA normalization, and adaptive gating to effectively aggregate heterogeneous features.
• Applications span infrared-visible image fusion, biomedical image classification, and multi-label time series analysis, demonstrating enhanced performance over conventional methods.

Fusion-ResNet refers broadly to a family of deep learning architectures and frameworks that combine residual neural networks (ResNets) with explicit feature fusion strategies. These models are characterized by the integration of deep convolutional features with complementary signals—ranging from handcrafted descriptors and multiscale representations to statistical features or external model embeddings—using mathematically principled fusion techniques. Fusion-ResNet has been applied across diverse domains, including image fusion, biomedical image classification, and multi-label time series analysis, with the specific design dictated by the properties of the input modalities and task requirements.

1. Core Methodologies and Architectural Components

Fusion-ResNet approaches share several architectural elements:

• A standard or modified ResNet backbone (typically ResNet-50), responsible for hierarchical feature extraction.
• One or more auxiliary feature streams, often derived from other image processing, statistical analysis, or neural network modules.
• A fusion module or operator, designed to aggregate features from multiple sources or hierarchies; this can involve explicit weighting (via learned or heuristic maps), normalization (e.g., ZCA, PCA/ICA), or adaptive gating.
• A prediction or reconstruction head, tailored to the end task (classification, regression, fusion).

A prototypical pipeline in the context of two-image fusion is as follows:

1. Each input is separately encoded using a pretrained ResNet to yield deep feature maps at a chosen layer (commonly conv4 or conv5 outputs).
2. Features are standardized, for example using zero-phase component analysis (ZCA) whitening, giving scale-invariant, decorrelated activations.
3. Local or global feature saliency is computed via norm-based pooling within spatial neighborhoods.
4. Pixel-wise or regional importance weights are refined and normalized using softmax or similar mechanisms.
5. The fused image is reconstructed as a weighted average of the input pixels, with weights derived from the previous stage (Li et al., 2018).

This general framework is adapted in various contexts with domain-specific modifications, such as adaptive learned gating (Liu et al., 4 Oct 2025), statistical fusion via PCA/ICA (Hoosh et al., 15 Nov 2025), or late fusion of hand-engineered descriptors and transformer embeddings (Tschuchnig et al., 26 Jul 2025).

2. Notable Fusion-ResNet Variants by Application Domain

Fusion-ResNet design is application-dependent, with several distinct instantiations:

A. Infrared and Visible Image Fusion

Li et al. introduced a Fusion-ResNet based framework for combining infrared and visible images:

• Feature Extraction: Employs a fixed ResNet-50 (pretrained on ImageNet) as a pure feature extractor; neither network training nor fine-tuning is performed.
• ZCA Whitening: Each feature channel undergoes ZCA transformation to enforce channel decorrelation and scale normalization; this is achieved by computing the covariance of each channel and applying an eigen-decomposition to extract decorrelated whitening matrices.
• Weight Map Computation: Pixel-wise activity maps are obtained by aggregating the local $l_1$-norm of ZCA-processed features within a sliding window, promoting local saliency.
• Softmax Fusion & Reconstruction: Activity maps from each modality are softmax-normalized to create smooth, spatially adaptive fusion weights, which are then used for per-pixel weighted averaging of the source images (Li et al., 2018).

This approach achieves strong performance on objective metrics such as $SSIM_a$, $EPI_a$, and $N_{abf}$, demonstrating salient foreground and texture preservation with low artifact introduction.

B. Biomedical Image Classification

Recent variants employ deep feature fusion within classification pipelines for medical imaging tasks:

• In skin lesion classification, a dual-branch Fusion-ResNet extracts mid-level (conv4) and high-level (conv5) features. These are spatially aligned and concatenated, then fused by an Adaptive Spatial Feature Fusion (ASFF) module. The ASFF uses trainable fully connected layers to generate sample-adaptive fusion weights via softmax, balancing detail and semantic cues and enabling end-to-end gradient optimization. The fused features are used for the final binary classification head, achieving superior accuracy (93.18%), F1 (93.13%), and AUC ($\approx$0.97) versus standard CNN baselines (Liu et al., 4 Oct 2025).
• In mammographic cancer detection, Fusion-ResNet designates a “hybrid” feature stack, injecting handcrafted edge/texture maps at input (early fusion with the ResNet pipeline) and optionally concatenating DINOv2 Visual Transformer embeddings at the feature level (late fusion). The combination of local structural priors with deep representations enhances AUC and recall on the CBIS-DDSM dataset. The best configuration raises AUC from 78.1% (ResNet-only) to 79.6%, with a peak recall of 80.5% (Tschuchnig et al., 26 Jul 2025).

C. Multi-Label Classification of Time Series

In non-intrusive load monitoring (NILM), Fusion-ResNet leverages statistical feature fusion:

• Feature Extraction: Input signals are projected onto principal directions (PCA) and separated statistically independent directions (ICA), with sub-Gaussian ICA components selected by kurtosis.
• Fusion and Normalization: PCA and ICA features are concatenated and z-score normalized.
• Shallow Residual Network: The fused embedding is forwarded through an 18-block feed-forward residual network (no convolutions, $~$65k parameters), culminating in sigmoid-activated outputs for multi-appliance detection. The design demonstrates higher mean F₁ and lower latency than CNN or LSTM alternatives, retaining robustness with up to 15 concurrent signatures (Hoosh et al., 15 Nov 2025).

3. Mathematical Formulation of Fusion Techniques

Fusion-ResNet models employ mathematically explicit mechanisms to combine features from different domains, layers, or modalities. Two canonical formulations are:

A. Softmax-Weighted Pixel Fusion (Image Fusion):

$$w_k^i(x,y) = \frac{F_k^{\,i,*}(x,y)}{F_1^{\,i,*}(x,y) + F_2^{\,i,*}(x,y)},$$

$$\mathrm{Fused}(x,y) = \sum_{k=1}^2 w_k^i(x,y) \mathrm{Source}_k(x,y)$$

where $F_k^{i,*}(x,y)$ is the local saliency for image $k$ and $w_k^i(x,y)$ is the normalized per-pixel weight (Li et al., 2018).

B. Adaptive Fusion via Learned Weights (Classification):

$$z = \mathrm{GAP}([F_s; F_m]),\quad a = \mathrm{ReLU}(W_1 z + b_1),\quad w = \mathrm{softmax}(W_2 a + b_2)$$

$$F_{\mathrm{fused}} = \alpha F_s + \beta F_m,\quad (\alpha + \beta = 1)$$

where $F_s$ and $F_m$ are mid-level and high-level feature maps, and $w = [\alpha, \beta]^T$ are trainable attention weights (Liu et al., 4 Oct 2025).

Statistical fusion in NILM tasks involves concatenation of optimal projections (PCA/ICA) and explicit normalization, with fusion logic based on component kurtosis and z-scoring (Hoosh et al., 15 Nov 2025).

4. Evaluation Metrics and Empirical Performance

Fusion-ResNet models report quantitative gains across several domains using standard task-specific metrics:

Task/Domain	Key Metrics	Fusion-ResNet Result	Reference
Image Fusion	$SSIM_a$, $EPI_a$, $N_{abf}$, FMI	$SSIM_a\sim0.78$, $EPI_a\sim0.94$, $N_{abf}\sim0.0001$	(Li et al., 2018)
Skin Lesion	Accuracy, F1, AUC	Accuracy 93.18%, F1 93.13%, AUC $\sim$0.97	(Liu et al., 4 Oct 2025)
Mammography	AUC, Recall, F1	AUC 79.6%, Recall 80.5%, F1 67.4%	(Tschuchnig et al., 26 Jul 2025)
NILM (Appliance)	Mean F1, Latency	Mean F1 0.77, Latency 0.0017 ms/sample	(Hoosh et al., 15 Nov 2025)

Fusion-ResNet consistently outperforms or matches state-of-the-art baselines, particularly in modality fusion or high-noise scenarios. Improvements are attributed to information-selective fusion, scale normalization, and the explicit integration of complementary domain priors.

5. Implementation Considerations and Limitations

Reported implementations utilize standard deep learning toolkits (PyTorch, Tensorflow, MATLAB/NumPy) and pretrained ResNet-50 backbones. Typical training/inference is lightweight unless per-channel statistical transforms (as in ZCA) are applied to large feature maps. Public code availability and open-source reproducibility are observed (e.g., [https://github.com/hli1221/imagefusion_resnet50]).

Key limitations include:

• Many variants depend on fixed, pretrained backbones without end-to-end retraining, possibly reducing adaptability to unseen modalities (Li et al., 2018).
• ZCA or other channel-wise whitening operations introduce computational overhead for large spatial feature maps.
• Several implementations are limited to two input modalities natively, with extension to $K>2$ requiring explicit code generalization.
• Classification-oriented uses are sometimes restricted to binary tasks; few works test generalization on external datasets or multi-class problems (Liu et al., 4 Oct 2025).
• No end-to-end optimization is available in pipelines where all fusion weights are heuristically derived rather than learned (Li et al., 2018, Hoosh et al., 15 Nov 2025).

6. Related Methods and Comparative Context

Fusion-ResNet fits within a broader context of modality fusion, multi-stream, and attention-based architectures:

• Related architectures include Res2Net-based fusion (e.g., Res2NetFuse with multi-scale block aggregation and spatial attention) (Song et al., 2021), hierarchical/parallel fusion in encoder-decoder models, and GAN-based approaches.
• This suggests that explicit feature fusion, whether learned or heuristic, offers advantages where single-stream CNNs are insufficient for heterogeneous data integration, or where domain priors can be distilled into auxiliary features.
• While transformer-based embeddings (e.g., DINOv2) have been incorporated in some hybrid frameworks, empirical results indicate that simple edge or texture priors may outperform such global self-attention features in limited-data regimes (Tschuchnig et al., 26 Jul 2025).
• Statistical fusion of PCA/ICA features (NILM-ICPC) provides robustness in time-series multi-label settings, with significantly lower latency than LSTM or CNN baselines (Hoosh et al., 15 Nov 2025).

7. Outlook and Extensions

Ongoing research trends include:

• Extending adaptive fusion weights to multi-class and multi-modal problems with more general attention mechanisms.
• Reducing computational costs of statistical transforms (e.g., ZCA, ICA) for high-resolution inputs.
• Integrating end-to-end learnable fusion, including differentiable normalization/fusion steps, to optimize performance across diverse data distributions.
• Benchmarking generalization on external, cross-domain datasets and evaluating under distribution shift.

A plausible implication is that Fusion-ResNet architectures, by decoupling backbone feature extraction from domain-informed fusion modules, offer a flexible, modular approach adaptable to a wide range of fused learning settings, especially where data heterogeneity, noise, and task-specific priors are significant.