Dual-Branch Squeeze-Fusion-Excitation (DuSFE)

• The module's main contribution is enabling progressive, bidirectional fusion of modality-specific features through integrated channel and spatial attention for improved SPECT-CT registration.
• DuSFE interleaves attention modules at multiple down-sampling stages in dual-stream CNN architectures to recalibrate paired feature tensors dynamically.
• Empirical results demonstrate that DuSFE reduces mean translation and rotation errors while modestly increasing parameters and training time compared to traditional fusion methods.

The Dual-Branch Squeeze-Fusion-Excitation (DuSFE) module is a hybrid cross-modality feature recalibration unit designed for integration within two-stream convolutional neural network (CNN) architectures. DuSFE enables progressive, bidirectional fusion and recalibration of paired feature tensors originating from separate imaging modalities—specifically demonstrated on cardiac single-photon emission computed tomography (SPECT) and computed tomography (CT)-derived attenuation maps (μ-maps). By jointly leveraging channel-wise and spatial information, DuSFE systematically addresses limitations of early and late fusion, fostering multi-stage, context-aware interaction between modality branches and achieving superior registration performance and attenuation-corrected image quality in clinical SPECT-CT workflows (Chen et al., 2022).

1. Architectural Principles

DuSFE is architecturally situated within a dual-stream CNN backbone, each branch dedicated to a distinct modality (e.g., SPECT and CT). Rather than performing feature fusion exclusively at the input (early fusion) or output (late fusion), DuSFE modules are interleaved at multiple depths—specifically at each major down-sampling (dense block) stage—allowing for multi-scale, cross-modality recalibration throughout the feature hierarchy. At each such location, the paired feature maps $$F_1, F_2 \in \mathbb{R}^{H \times W \times D \times C}$$ are processed in parallel by two dedicated attention branches:

• Channel-Squeeze-Fusion-Excitation (cSFE): Learns modality-dependent channel attention coefficients via feature squeezing, fusion, and excitation.
• Spatial-Squeeze-Fusion-Excitation (sSFE): Computes voxel-wise attention via channel-reduction, cross-branch spatial fusion, and locality-aware recalibration.

Upon conclusion of cSFE and sSFE processing, the recalibrated outputs are combined by element-wise multiplication and residual addition, updating the dual-branch features prior to forward propagation. At the final layer, recalibrated features from both branches are concatenated and used for downstream regression of rigid registration parameters.

2. Mathematical Operations and Data Flow

The DuSFE module decomposes into sequential sub-operations for both channel-wise and spatial-wise recalibration:

2.1 Channel-wise Squeeze and Fusion:

• Squeeze: For each branch, perform global average pooling:

$$V_1(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_1(i,j,k,c), \quad V_2(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_2(i,j,k,c)$$

• Fusion: Concatenate $V_1, V_2$ and project via a fully connected (FC) layer:

$$V_{fuse} = W_f [V_1; V_2] + b_f, \quad W_f \in \mathbb{R}^{C \times 2C}$$

• Excitation: Two separate FC layers generate attention vectors:

$$R_1 = W_1 V_{fuse} + b_1, \quad R_2 = W_2 V_{fuse} + b_2$$

Channel attention maps are then obtained by applying a sigmoid and broadcast-multiplying:

$$F_{1\_cSFE}(i,j,k,c) = \sigma(R_1(c)) \cdot F_1(i,j,k,c)$$

$$F_{2\_cSFE}(i,j,k,c) = \sigma(R_2(c)) \cdot F_2(i,j,k,c)$$

2.2 Spatial Squeeze and Fusion:

• Channel Reduction: Project each tensor to a single channel via $1\times1\times1$ convolutions:

$M_1 = K_{in1} * F_1, \quad M_2 = K_{in2} * F_2$

• Fusion: Concatenate $M_1, M_2$ along the channel axis; apply another $$V_1(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_1(i,j,k,c), \quad V_2(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_2(i,j,k,c)$$0 convolution:

$$V_1(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_1(i,j,k,c), \quad V_2(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_2(i,j,k,c)$$1

• Excitation: Output two separate spatial attention maps via $$V_1(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_1(i,j,k,c), \quad V_2(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_2(i,j,k,c)$$2 convolutions:

$$V_1(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_1(i,j,k,c), \quad V_2(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_2(i,j,k,c)$$3

After passing through a sigmoid, these are broadcast to all channels:

$$V_1(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_1(i,j,k,c), \quad V_2(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_2(i,j,k,c)$$4

$$V_1(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_1(i,j,k,c), \quad V_2(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_2(i,j,k,c)$$5

2.3 Residual Recombination:

• For each branch, the final output is a sum of input and both recalibrated maps:

$$V_1(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_1(i,j,k,c), \quad V_2(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_2(i,j,k,c)$$6

$$V_1(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_1(i,j,k,c), \quad V_2(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_2(i,j,k,c)$$7

This dual-branch design allows both channel- and spatial-level cross-talk, addressing heterogeneous modality-specific cues at multiple scales (Chen et al., 2022).

3. Comparison with Fusion Baselines

Traditional CNN-based cross-modality registration strategies are typified by two paradigms:

• Early fusion: Simply concatenates raw modalities at the input, forfeiting the ability for dynamic, modality-specific recalibration.
• Late fusion: Extracts features via independent streams and merges them only at the output, precluding mutual influence during feature extraction.

DuSFE advances beyond these by facilitating recurrent, hierarchical feature fusion and recalibration at several stages within the network. Multi-scale attention ensures that both modalities dynamically influence representation refinement, promoting richer cross-modal correlations.

Empirical results demonstrate that DenseNet-structured networks with DuSFE outperform both early and late fusion as well as more recent non-local attention baselines in rigid SPECT-CT registration. DuSFE achieves lower mean registration errors and improved image-space metrics, such as normalized MSE and MAE on μ-maps and attenuation-corrected SPECT images (all $$V_1(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_1(i,j,k,c), \quad V_2(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_2(i,j,k,c)$$8 vs. alternatives) (Chen et al., 2022).

4. Computational Complexity and Implementation

DuSFE introduces modest parameter and computational overhead. Integration into a dual-stream DenseNet network increases total parameters by approximately $$V_1(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_1(i,j,k,c), \quad V_2(c) = \frac{1}{HWD} \sum_{i=1}^H \sum_{j=1}^W \sum_{k=1}^D F_2(i,j,k,c)$$9 (from $V_1, V_2$0M to $V_1, V_2$1M, a $V_1, V_2$2 uplift). Training time per batch rises from $V_1, V_2$3s to $V_1, V_2$4s, while inference time per case remains below $V_1, V_2$5s. Each DuSFE instance comprises several convolutions and FC layers per fusion branch. To maintain compatibility and computational efficiency, stride-$V_1, V_2$6 convolutions with padding preserve spatial resolution at all stages, and sigmoid activations are used exclusively for attention maps (Chen et al., 2022).

Table: Parameter and Runtime Comparison

Configuration	Parameters (M)	Training Time / Batch (s)	Inference Time / Case (s)
Baseline DenseNet	11.22	1.08	<0.05
+ Non-Local Attention	11.42	–	<0.05
+ DuSFE	11.73	1.25	<0.05

Placement of DuSFE modules is at each of three major down-sampling stages in both branches, for a total of six modules per network.

5. Quantitative Performance and Ablation Studies

In clinical SPECT-CT registration, DuSFE yields the lowest mean translation and rotation errors among tested methods (translation error $V_1, V_2$7: $V_1, V_2$8 mm; rotation error $V_1, V_2$9: $$V_{fuse} = W_f [V_1; V_2] + b_f, \quad W_f \in \mathbb{R}^{C \times 2C}$$0). This surpasses classical mutual information, DVNet, MSReg, and baseline DenseNet with or without non-local attention. Downstream, DuSFE also results in superior normalized MSE and MAE on registered μ-maps and attenuation-corrected SPECT images (Chen et al., 2022).

6. Limitations and Future Directions

The initial DuSFE framework is limited to rigid registration scenarios. Non-rigid intra-modality deformations (e.g., respiration-induced) are not addressed. The module introduces a slight increase in training complexity and parameter count. Future research directions include:

• Development of unsupervised or self-supervised DuSFE variants to obviate reliance on synthetic ground-truth transforms.
• Extension and evaluation within deformable-registration architectures for broader anatomical applicability.
• Exploration of dynamic weighting schemes to adaptively blend or select between cSFE and sSFE pathways on the basis of local cross-modal informativeness.

A plausible implication is that similar dual-attention, dual-branch recalibration mechanisms may have value in multimodal image analysis domains beyond rigid cardiac SPECT-CT registration, especially when early or late fusion proves suboptimal (Chen et al., 2022).

7. Relationship to Squeeze-Excitation and Multimodal Fusion Approaches

DuSFE draws conceptual lineage from channel-wise squeeze-excitation (SE) modules and multimodal attention units as exemplified by the Multimodal Transfer Module (MMTM). While MMTM applies global squeeze and excitation at the channel level for cross-modal fusion and can be inserted at various CNN depths, DuSFE extends this strategy by adding a parallel spatial attention fusion branch. Unlike MMTM, DuSFE explicitly recalibrates both spatial and channel features, and merges these recalibrations with a residual scheme:

• MMTM produces channel-wise gates via bottlenecked, modality-shared embeddings and is optimal when spatial alignment is not needed (Joze et al., 2019).
• DuSFE introduces spatially resolved recalibration and summative integration, which may provide advantages in volume registration tasks where voxel-wise alignment is critical (Chen et al., 2022).

Empirical results substantiate DuSFE’s efficacy in high-fidelity, multimodal medical image registration, demonstrating best-in-class accuracy and parameter efficiency within its application domain.