MMCAF-Net: Multiscale Fusion for 3D Imaging & EHR

• The paper presents a novel architecture combining multi-scale and bidirectional cross-attention fusion to enhance small lesion detection in lung disease classification.
• It uses an Efficient 3D Multi-Scale Convolutional Attention module along with a Kolmogorov–Arnold Network for robust feature extraction from 3D images and EHR data.
• Quantitative evaluations reveal improved performance with AUROC = 0.786, Accuracy = 0.791, and F1 = 0.690 compared to existing multimodal fusion approaches.

The Multimodal Multiscale Cross-Attention Fusion Network (MMCAF-Net) is a deep learning architecture designed to address the dimensional misalignment and information fusion challenges inherent in the integration of 3D medical imaging (CT/PET) and electronic health record (EHR) tabular data, with a specific focus on small lesion-aware lung disease classification. MMCAF-Net leverages a feature pyramid encoding strategy, a 3D multi-scale convolutional attention mechanism, and a novel bidirectional multi-scale cross-attention fusion module to maximize lesion-level discriminability while enhancing multimodal semantic alignment (Yu et al., 6 Aug 2025).

1. Architectural Overview and Motivations

MMCAF-Net is structured in a four-stage pipeline: (1) a 3D image encoder employing a feature pyramid and Efficient 3D Multi-Scale Convolutional Attention (E3D-MSCA); (2) a Kolmogorov–Arnold Network (KAN) encoder for EHR tabular information; (3) a multi-scale cross-attention fusion module with bidirectional scale fusion (BSF) for integrating multimodal features at all pyramid levels; and (4) a classification head composed of a fully connected layer followed by a sigmoid activation.

The architectural emphasis on a feature pyramid allows simultaneous extraction of features across small, medium, and large lesion scales, thereby reducing the risk of missing tiny nodules. The bidirectional design enables information exchange from images to EHR and vice versa at multiple scales, mitigating single-direction fusion biases and facilitating semantic alignment across modalities.

2. Efficient 3D Multi-Scale Convolutional Attention (E3D-MSCA) Module

At the core of the image encoder is the E3D-MSCA, which processes each pyramid scale output $P_s$ to produce rich, fused feature representations $F'_s$. The E3D-MSCA sequentially applies:

• 3D Channel Attention Block (CAB): Computes channel-wise importance by

$$M_c = \sigma(W_2(\operatorname{ReLU}(W_1(\operatorname{AvgPool}(F)))))$$

and then $F_c = M_c \odot F$.

• 3D Spatial Attention Block (SAB): Operates across the spatial domain,

$$M_s = \sigma(\operatorname{Conv}^{7 \times 7 \times 7}([\operatorname{AvgPool}_c(F_c);\operatorname{MaxPool}_c(F_c)]))$$

followed by $F_s = M_s \odot F_c$.

• 3D Depth-Wise Convolution Fusion Block (DCFB): Applies dilated depthwise convolutions with rates $d \in \{1,2,3\}$:

$$G_d = \operatorname{DWConv}(F_s; \text{kernel}=3\times3\times3,\, \text{dilation}=d)$$

Results are concatenated $F' = \operatorname{Concat}(G_1, G_2, G_3)$.

To further emphasize small lesion voxels, a lightweight scoring function $\phi$ yields per-voxel logits $F'_s$0, and output features are reweighted $F'_s$1, where high-contrast (i.e., small lesion) regions are amplified.

3. Multi-Scale Cross-Attention Fusion and Bidirectional Scale Fusion

The fusion module harmonizes image and EHR feature representations at all relevant scales.

• Dimension Alignment: Each image scale feature $F'_s$2 and EHR feature $F'_s$3 is projected into a common dimension $F'_s$4 via inverted-pyramid linear layers.
• Multi-Scale Cross-Attention (MSCA): For each scale $F'_s$5:

$F'_s$6

Partitioned into $F'_s$7 heads of dimension $F'_s$8 ($F'_s$9), multi-head attention is computed as:

$$M_c = \sigma(W_2(\operatorname{ReLU}(W_1(\operatorname{AvgPool}(F)))))$$0

Concatenated outputs $$M_c = \sigma(W_2(\operatorname{ReLU}(W_1(\operatorname{AvgPool}(F)))))$$1 are produced for each scale.

• Bidirectional Scale Fusion (BSF): For each adjacent pair $$M_c = \sigma(W_2(\operatorname{ReLU}(W_1(\operatorname{AvgPool}(F)))))$$2, BSF computes softmax importance across channel dimensions, reweights the outputs, and sums them to yield the final fused representation $$M_c = \sigma(W_2(\operatorname{ReLU}(W_1(\operatorname{AvgPool}(F)))))$$3.

The merging of adjacent scales also utilizes the Bidirectional Feedback Propagation Unit (BFPU) defined as: $$M_c = \sigma(W_2(\operatorname{ReLU}(W_1(\operatorname{AvgPool}(F)))))$$4 where $$M_c = \sigma(W_2(\operatorname{ReLU}(W_1(\operatorname{AvgPool}(F)))))$$5 is the sigmoid activation, $$M_c = \sigma(W_2(\operatorname{ReLU}(W_1(\operatorname{AvgPool}(F)))))$$6 denotes elementwise multiplication, and $$M_c = \sigma(W_2(\operatorname{ReLU}(W_1(\operatorname{AvgPool}(F)))))$$7 is channel concatenation, promoting robust inter-scale semantic propagation.

4. Data Handling, Training Protocols, and Implementation

MMCAF-Net is evaluated on the Lung-PET-CT-Dx dataset (355 patients) which includes CT/PET DICOM images and EHR records (sex, age, TNM stage, etc.). Class imbalance is addressed by oversampling the minority class (squamous carcinoma 34→198 in train).

Preprocessing includes extraction of 12 lesion-representative slices per case, resizing to $$M_c = \sigma(W_2(\operatorname{ReLU}(W_1(\operatorname{AvgPool}(F)))))$$8, intensity normalization to $$M_c = \sigma(W_2(\operatorname{ReLU}(W_1(\operatorname{AvgPool}(F)))))$$9, and augmentations such as rotation (±15°), sharpening, and Gaussian noise.

Model hyperparameters comprise $F_c = M_c \odot F$0 feature pyramid levels, $F_c = M_c \odot F$1 attention heads with head dimension $F_c = M_c \odot F$2 ($F_c = M_c \odot F$3), and E3D-MSCA convolutions at dilations $F_c = M_c \odot F$4. Optimization uses binary cross-entropy loss

$F_c = M_c \odot F$5

with SGD (momentum $F_c = M_c \odot F$6, weight decay $F_c = M_c \odot F$7). Learning rate starts at $F_c = M_c \odot F$8, reduced by $F_c = M_c \odot F$9 at epochs 30 and 40. Batch size is 4, with training for 50 epochs using torch-2.4.1 on a Tesla V100.

5. Quantitative Performance and Ablation Analyses

On the Lung-PET-CT-Dx test set, MMCAF-Net achieves:

• AUROC = 0.786
• Accuracy = 0.791
• F1 = 0.690
• Specificity = 0.857
• Sensitivity = 0.667
• PPV = 0.714
• NPV = 0.828

In comparison, the leading alternative (MMTM) reports AUROC = 0.802, Accuracy = 0.698; all other baselines (PECon, MedFuse, DrFuse, PEfusion, daft) fall below MMCAF-Net in both accuracy and F1. Image encoder ablation demonstrates substantial gains with the E3D-MSCA (+dropout): AUROC = 0.712, Accuracy = 0.767, F1 = 0.545, compared to PENet alone (AUROC = 0.560). Fusion module ablation shows MSCA (Accuracy = 0.791) outperforms Cross-Attn (0.744), clip_Fusion (0.698), and Late_Fusion (0.674).

6. Algorithmic Pipeline and Pseudocode Specification

The algorithmic workflow, as formalized in the provided pseudocode, proceeds as follows:

1. Image Encoding: Input images are passed through a PENet backbone and feature pyramid, with E3D-MSCA + BFPU applied at each scale.
2. EHR Encoding: EHR data is encoded using a Kolmogorov–Arnold Network.
3. Multi-Scale Fusion: At each of three scales, cross-attention fuses image and EHR features, followed by bidirectional scale fusion.
4. Classification: The resulting fused features are classified with a linear layer and sigmoid.
5. Training Loop: A standard batch-iterative procedure computes predictions and binary cross-entropy loss, with parameter updates via SGD, and learning rate scheduling.

The full modular structure is summarized in the workflow table:

Component	Processing Block(s)	Output Dimensions
3D Image Encoder	PENet + Feature Pyramid + E3D-MSCA + BFPU	$$M_s = \sigma(\operatorname{Conv}^{7 \times 7 \times 7}([\operatorname{AvgPool}_c(F_c);\operatorname{MaxPool}_c(F_c)]))$$0
EHR Encoder	Kolmogorov–Arnold Network	$$M_s = \sigma(\operatorname{Conv}^{7 \times 7 \times 7}([\operatorname{AvgPool}_c(F_c);\operatorname{MaxPool}_c(F_c)]))$$1
Fusion Module	Multi-Scale Cross-Attention + Bidirectional Scale Fusion	$$M_s = \sigma(\operatorname{Conv}^{7 \times 7 \times 7}([\operatorname{AvgPool}_c(F_c);\operatorname{MaxPool}_c(F_c)]))$$2
Classification Head	FC → Sigmoid	Probabilities

7. Significance, Limitations, and Research Context

MMCAF-Net demonstrates substantial improvement in small lesion detection and multimodal fusion for lung disease classification, particularly reflected in accuracy and F1 metrics compared to prior methods. The combined use of multi-scale pyramid features, channel and spatial attentions, and explicit cross-modal alignment addresses longstanding challenges in medical image-EHR fusion. Ablation results reinforce the critical role of each architectural component. A plausible implication is that such bidirectional, multiscale, and attention-based fusion modules could be generalized to other multimodal, multi-resolution diagnostic tasks involving high-dimensional 3D data and heterogeneous non-image sources (Yu et al., 6 Aug 2025).