Squeeze-and-Excitation Fusion

• Squeeze-and-Excitation Fusion is a neural mechanism that adaptively reweights feature channels and fuses multi-modal, multi-scale inputs using SE blocks.
• It integrates squeeze, excitation, and scaling processes to enhance representational efficiency in CNNs and other architectures across tasks like gaze tracking and MRI reconstruction.
• Empirical evidence demonstrates that SE fusion methods reduce errors and improve accuracy in applications including image classification, multi-modal learning, and speech emotion recognition.

Squeeze-and-Excitation Fusion refers to a class of neural mechanisms and architectural modules that integrate the principles of Squeeze-and-Excitation (SE) attention with various feature fusion strategies across channels, modalities, spatial axes, or tasks. Originally developed for convolutional neural networks to perform adaptive channel reweighting, SE-based fusion has evolved into a general paradigm for both intra-modal and cross-modal attention, multi-scale integration, residual network recalibration, and multimodal representation learning.

1. Foundations of Squeeze-and-Excitation Fusion

At the core of Squeeze-and-Excitation Fusion is the SE block, introduced for channel-wise attention in CNNs. The SE block operates in three stages: (1) squeeze, which globally pools each feature map to form a channel descriptor; (2) excitation, in which the descriptor is transformed by a compact, nonlinear, often two-layer bottleneck multilayer perceptron (MLP) with a sigmoid gate; and (3) channel-wise scaling of the input tensor by the resulting activations. Formally, for an input $U\in\mathbb{R}^{H\times W\times C}$, the outputs are: $$z_c = \frac{1}{H W} \sum_{i=1}^H \sum_{j=1}^W u_c(i,j)$$

$s = \sigma\bigl(W_2\,\delta(W_1\,z)\bigr)$

$\widetilde{u}_c = s_c u_c$

where $W_1\in \mathbb{R}^{C/r \times C}$, $W_2\in\mathbb{R}^{C\times C/r}$, $\delta$ is ReLU, $\sigma$ is sigmoid, and $r$ is the reduction ratio (Hu et al., 2017).

SE-based fusion generalizes this structure to aggregate, weight, and combine information from different feature sources—such as dual eyes in gaze tracking, multiple sensor modalities, multi-scale streams, or spatial axes—based on their global or local agreement, confidence, or complementarity (Bao et al., 2021, Guo et al., 2023, Li et al., 2023).

2. SE Fusion in CNN Architectures

SE fusion is integrated at critical points of neural network architectures, enhancing representational efficiency and fusing information adaptively. In the Adaptive Feature Fusion Network (AFF-Net) for gaze estimation, SE blocks are deployed both after sequential convolutional stages in each eye’s tower and at the fusion step where left and right eye feature maps are stacked and fused. This preserves both local spatial structure and adaptively weights joint channels according to their appearance similarity:

• Feature maps from each eye are stacked to form a 128-dimensional representation.
• An SE block pools across all channels (including both eyes), applies a two-layer bottleneck MLP (with reduction ratio $r=16$), and rescales channels.
• Channels signifying mutual agreement are amplified; channels dominated by noise or occlusion are down-weighted (Bao et al., 2021).

Empirically, SE-based fusion outperforms naïve concatenation of feature vectors, yielding consistent reductions in gaze estimation error—even if the absolute improvement can be modest (3% relative error drop). This demonstrates the functional importance of dynamic, attention-style fusion in vision backbones.

3. Cross-Modal and Multi-Modal Fusion Strategies

Squeeze-and-Excitation Fusion extends naturally to multimodal learning. In SEFusion for meme emotion analysis, independently extracted text and image features are compressed to single scalars per modality, concatenated and passed through a bottleneck excitation MLP to learn per-modality weights, then used to linearly combine the full original embeddings:

• For $$z_c = \frac{1}{H W} \sum_{i=1}^H \sum_{j=1}^W u_c(i,j)$$0 modalities ($$z_c = \frac{1}{H W} \sum_{i=1}^H \sum_{j=1}^W u_c(i,j)$$1 in the canonical case), each feature $$z_c = \frac{1}{H W} \sum_{i=1}^H \sum_{j=1}^W u_c(i,j)$$2 is linearly projected (squeeze), then the stack $$z_c = \frac{1}{H W} \sum_{i=1}^H \sum_{j=1}^W u_c(i,j)$$3 is passed through an excitation subnetwork.
• The excitation outputs $$z_c = \frac{1}{H W} \sum_{i=1}^H \sum_{j=1}^W u_c(i,j)$$4, interpreted as modality importance weights, are multiplied with the reshaped input feature matrix $$z_c = \frac{1}{H W} \sum_{i=1}^H \sum_{j=1}^W u_c(i,j)$$5 to produce a fused vector.
• The approach is fully modality-agnostic and supports extension to multiple modalities by appropriate modifications of the squeeze and fusion stages (Guo et al., 2023).

The Expansion-Squeeze-Excitation Fusion Network (ESE-FN) for activity recognition applies an ESE block that first locally expands modal or channel features via convolution before global pooling and two-layer excitation, allowing adaptive fusion at both the modal and channel levels. This multi-stage attention achieves state-of-the-art multi-modal fusion performance (Shu et al., 2021).

The Dual-Branch Squeeze-Fusion-Excitation module (DuSFE) addresses cross-modality alignment (e.g., SPECT/CT registration) by recalibrating both channel and spatial distributions for each modality. Channel vectors are globally pooled, fused, and split; spatial maps are extracted, fused via convolution, and both types of gating are applied to each modality's tensor. Embedding such fusion modules at multiple network depths reliably reduces registration and reconstruction errors compared to early or late concatenation (Chen et al., 2022).

4. Extensions: Multi-Scale, Residual, and Advanced SE Fusions

SE fusion is actively adapted for various deep learning problems. Prominent research directions include:

• Multi-Scale Squeeze-and-Excitation Fusion: In MS-SENet for speech emotion recognition, multi-branch CNNs extract time-frequency features at different scales, which are concatenated then globally recalibrated by an SE block. This setup allows dynamic weighting of frequency bands and timesteps, yielding pronounced improvements under both clean and low-resource constraints (Li et al., 2023).
• Competitive and Inner-Imaging SE Fusion: The Competitive Inner-Imaging SE (CMPE-SE) module lets the identity and residual branches in a ResNet jointly determine channel scaling via separate embeddings and a fused excitation. This enforces competition between skip and learned flows, yielding more efficient channel utilization and improved error rates versus standard SE-ResNets (Hu et al., 2018).
• Triplet SE Fusion: Triplet Squeeze-and-Excitation (TripSE) blocks combine triplet-attention—three-way attention over channel, width, and height axes—with SE-based channel weighting (permuted appropriately per branch), and fuse attention maps for spatial and channel recalibration. Multiple variants (SE at branch start, parallel SE+TA, affine-fusion) all increase classification accuracy, with the most expressive variant achieving new state-of-the-art results for facial expression recognition (Alhazmi et al., 9 May 2025).
• SE Fusion in Dual-Domain and Cross-Iteration Pipelines: In dual-domain MRI reconstruction, SE blocks deployed within both image and k-space subnetworks, along with cross-iteration residuals, enable iterative fusing and reweighting of domain-specific features. This configuration achieves superior normalized mean-square error and structural similarity compared to single-domain and non-attentive fusion baselines (Chen et al., 2023, Chen et al., 2022).

5. Neural Architecture Search and Advanced SE Fusion Blocks

User-driven or automated search for optimal SE fusion structures is an active research area. SASE (Searching Architecture for Squeeze and Excitation operations) employs differentiable neural architecture search (NAS) to explore fine-grained search spaces within the squeeze, excitation, channel, and spatial domains:

• Each edge in the SASE super-network chooses among multiple squeeze operations (e.g. GAP, GMP, GSoP), excitation functions (MLPs, various convolutions, affine transforms), and their spatial-channel analogs.
• The final block averages the results of channel and spatial attention branches, using learned or searched operators for each stage.
• This methodology yields attention modules surpassing both hand-crafted SE/CBAM/ECA and prior NAS-derived blocks, e.g., achieving $$z_c = \frac{1}{H W} \sum_{i=1}^H \sum_{j=1}^W u_c(i,j)$$6 Top-1 accuracy on ImageNet-1K with ResNet-50 (Wang et al., 2024).

In parallel, the Global Context (GC) block unifies attention frameworks by showing both SE and non-local blocks are instances of a three-step model (context modeling, excitation, fusion), and implements a hybrid block that matches SE in cost while approaching the context modeling power of non-local attention (Cao et al., 2019).

6. Quantitative Impact and Empirical Evidence

SE fusion modules consistently improve metrics across tasks, as demonstrated in extensive ablations:

• In gaze tracking, SE fusion reduces 2D estimation error relative to simple concatenation (Bao et al., 2021).
• On large-scale image classification (ImageNet, CIFAR), SE-ResNets outperform plain ResNets by $$z_c = \frac{1}{H W} \sum_{i=1}^H \sum_{j=1}^W u_c(i,j)$$7 top-1 accuracy with minor parameter and FLOP increases (Hu et al., 2017, Narayanan, 2023).
• For multi-modal emotion and activity recognition, SE-based fusion methods achieve top rankings in competitive challenges and exceed non-attentive baselines by significant margins (Guo et al., 2023, Shu et al., 2021).
• Dual-domain SE fusion in MRI reconstruction reduces NMSE by over $$z_c = \frac{1}{H W} \sum_{i=1}^H \sum_{j=1}^W u_c(i,j)$$8 versus single-domain networks and statistically outperforms all baselines (p$$z_c = \frac{1}{H W} \sum_{i=1}^H \sum_{j=1}^W u_c(i,j)$$90.001) (Chen et al., 2023, Chen et al., 2022).
• Advanced SE fusions (e.g. SASE, TripSE, DuSFE, ESE-FN) reliably outperform classic SE and other attention designs, especially when integrated at multiple depths and combined with spatial, modal, or temporal attention.

Table: Selected Quantitative Gains from SE Fusion Modules

Task/Domain	SE Fusion Method	Improved Metric	Reference
Gaze tracking (tablets)	AFF-Net (SE-fusion)	2D error: $s = \sigma\bigl(W_2\,\delta(W_1\,z)\bigr)$00.05cm	(Bao et al., 2021)
Meme emotion (multimodal)	SEFusion	1st place (F1 score)	(Guo et al., 2023)
ImageNet classification	SE-ResNet-50	Top-1 acc: $s = \sigma\bigl(W_2\,\delta(W_1\,z)\bigr)$11.5%	(Hu et al., 2017)
Speech emotion recognition	MS-SENet (SE-multiscale)	UAR$s = \sigma\bigl(W_2\,\delta(W_1\,z)\bigr)$20.96%, WAR$s = \sigma\bigl(W_2\,\delta(W_1\,z)\bigr)$30.96%	(Li et al., 2023)
MRI reconstruction	DD-CISENet (dual SE)	NMSE: $s = \sigma\bigl(W_2\,\delta(W_1\,z)\bigr)$41.77%	(Chen et al., 2023)
ImageNet (NAS)	SASE	Top-1 acc: $s = \sigma\bigl(W_2\,\delta(W_1\,z)\bigr)$52.49%	(Wang et al., 2024)
FER2013 (expression)	TripSE (full-affine)	Top-1 acc: $s = \sigma\bigl(W_2\,\delta(W_1\,z)\bigr)$61.08%	(Alhazmi et al., 9 May 2025)

7. Limitations, Generalization, and Future Directions

Despite their efficiency and task-specific gains, SE fusion modules have limitations:

• Optimal reduction ratios, fusion sites, and attention branch widths remain highly task- and architecture-dependent (Hu et al., 2017).
• Some approaches induce cross-contamination when modalities are reshaped without projection, as observed in SEFusion for multi-modal emotion analysis (Guo et al., 2023).
• Fusing cross-modal or spatial hierarchies may require more nuanced designs, such as hierarchical stacks of SE/attention blocks, integration of cross-modal attention, or use of NAS to select optimal sub-blocks dynamically (Wang et al., 2024).
• Extensive search spaces and NAS-based methods can introduce significant training complexity, but yield strictly better modules across benchmarks (Wang et al., 2024).

Potential avenues include:

• Expansion to non-visual domains (audio, multimodal, neuroscience) using variant squeeze/excitation operations.
• Incorporation into transformer and hybrid architectures with learnable global context pooling (e.g., GCNet) (Cao et al., 2019).
• Exploration of hierarchical and recurrent SE fusion, domain-adaptive blocks, or integration with other attention paradigms (e.g., triplet attention, inner-imaging) (Alhazmi et al., 9 May 2025, Hu et al., 2018).
• Further theoretical formalization, as suggested by the three-step formulation unifying SE, NL, and GC blocks (Cao et al., 2019).

Squeeze-and-Excitation Fusion represents a unifying approach for adaptive feature integration across neural architectures, generalized for modality, scale, space, residual structure, and long-range context, with empirical effectiveness across vision, speech, multi-modal, and medical applications.