---
title: 'MMCAF-Net: Multiscale Fusion for 3D Imaging & EHR'
url: https://www.emergentmind.com/topics/multimodal-multiscale-cross-attention-fusion-network-mmcaf-net
type: topic
---

# MMCAF-Net: Multiscale Fusion for 3D Imaging & EHR

The Multimodal Multiscale Cross-Attention Fusion Network (MMCAF-Net) is a deep learning architecture designed to address the dimensional misalignment and information fusion challenges inherent in the integration of 3D medical imaging (CT/PET) and electronic health record (EHR) tabular data, with a specific focus on small lesion-aware lung disease classification. MMCAF-Net leverages a feature pyramid encoding strategy, a 3D multi-scale convolutional attention mechanism, and a novel bidirectional multi-scale cross-attention fusion module to maximize lesion-level discriminability while enhancing multimodal semantic alignment [2508.04205].

## 1. Architectural Overview and Motivations

MMCAF-Net is structured in a four-stage pipeline: (1) a 3D image encoder employing a feature pyramid and Efficient 3D Multi-Scale Convolutional Attention (E3D-MSCA); (2) a Kolmogorov–Arnold Network (KAN) encoder for EHR tabular information; (3) a multi-scale cross-attention fusion module with bidirectional scale fusion (BSF) for integrating multimodal features at all pyramid levels; and (4) a classification head composed of a fully connected layer followed by a sigmoid activation.

The architectural emphasis on a feature pyramid allows simultaneous extraction of features across small, medium, and large lesion scales, thereby reducing the risk of missing tiny nodules. The bidirectional design enables information exchange from images to EHR and vice versa at multiple scales, mitigating single-direction fusion biases and facilitating semantic alignment across modalities.

## 2. Efficient 3D Multi-Scale Convolutional Attention (E3D-MSCA) Module

At the core of the image encoder is the E3D-MSCA, which processes each pyramid scale output $P_s$ to produce rich, fused feature representations $F'_s$. The E3D-MSCA sequentially applies:

- **3D Channel Attention Block (CAB):** Computes channel-wise importance by
  $$
  M_c = \sigma(W_2(\operatorname{ReLU}(W_1(\operatorname{AvgPool}(F)))))
  $$
  and then $F_c = M_c \odot F$.
- **3D Spatial Attention Block (SAB):** Operates across the spatial domain,
  $$
  M_s = \sigma(\operatorname{Conv}^{7 \times 7 \times 7}([\operatorname{AvgPool}_c(F_c);\operatorname{MaxPool}_c(F_c)]))
  $$
  followed by $F_s = M_s \odot F_c$.
- **3D Depth-Wise Convolution Fusion Block (DCFB):** Applies dilated depthwise convolutions with rates $d \in \{1,2,3\}$:
  $$
  G_d = \operatorname{DWConv}(F_s; \text{kernel}=3\times3\times3,\, \text{dilation}=d)
  $$
  Results are concatenated $F' = \operatorname{Concat}(G_1, G_2, G_3)$.

To further emphasize small lesion voxels, a lightweight scoring function $\phi$ yields per-voxel logits $A_{i,j,k} = \operatorname{softmax}(\phi(F_{i,j,k}))$, and output features are reweighted $F' = A \odot F$, where high-contrast (i.e., small lesion) regions are amplified.

## 3. Multi-Scale Cross-Attention Fusion and Bidirectional Scale Fusion

The fusion module harmonizes image and EHR feature representations at all relevant scales.

- **Dimension Alignment:** Each image scale feature $F_s^{img} \in \mathbb{R}^{B,N,D_{img}}$ and EHR feature $F_s^{ehr} \in \mathbb{R}^{B,M,D_{ehr}}$ is projected into a common dimension $D$ via inverted-pyramid linear layers.
- **Multi-Scale Cross-Attention (MSCA):** For each scale $s$:
  $$
  Q_s = F_s^{img} W_q\,,\quad K_s = F_s^{ehr} W_k\,,\quad V_s = F_s^{ehr} W_v
  $$
  Partitioned into $H$ heads of dimension $C$ ($D=H\cdot C$), multi-head attention is computed as:
  $$
  \mathrm{Attn}_{s,h} = \operatorname{softmax}\left(\frac{Q_{s,h}K_{s,h}^\top}{\sqrt{C}}\right)\!,\quad
  O_{s,h} = \mathrm{Attn}_{s,h} V_{s,h}
  $$
  Concatenated outputs $O_s$ are produced for each scale.

- **Bidirectional Scale Fusion (BSF):** For each adjacent pair $(O_s, O_{s+1})$, BSF computes softmax importance across channel dimensions, reweights the outputs, and sums them to yield the final fused representation $F_{fused}$.

The merging of adjacent scales also utilizes the Bidirectional Feedback Propagation Unit (BFPU) defined as:
\[
F_{mid} = \sigma(\operatorname{Conv}_{3\times3}(F_a)\otimes \operatorname{Conv}_{3\times3}(F_b)),\quad 
F_{out} = [F_a + F_{mid}\otimes F_a,\, F_b + F_{mid}\otimes F_b]
\]
where $\sigma$ is the sigmoid activation, $\otimes$ denotes elementwise multiplication, and $[\cdot]$ is channel concatenation, promoting robust inter-scale semantic propagation.

## 4. Data Handling, Training Protocols, and Implementation

MMCAF-Net is evaluated on the Lung-PET-CT-Dx dataset (355 patients) which includes CT/PET DICOM images and EHR records (sex, age, TNM stage, etc.). Class imbalance is addressed by oversampling the minority class (squamous carcinoma 34→198 in train).

Preprocessing includes extraction of 12 lesion-representative slices per case, resizing to $192 \times 192$, intensity normalization to $[0,1]$, and augmentations such as rotation (±15°), sharpening, and Gaussian noise.

Model hyperparameters comprise $S=3$ feature pyramid levels, $H=4$ attention heads with head dimension $C=32$ ($D=128$), and E3D-MSCA convolutions at dilations $\{1,2,3\}$. Optimization uses binary cross-entropy loss
\[
\mathcal{L}_{cls}=-\frac{1}{N}\sum_i y_i\log\hat y_i + (1-y_i)\log(1-\hat y_i)
\]
with SGD (momentum $0.9$, weight decay $1\times10^{-2}$). Learning rate starts at $1\times10^{-4}$, reduced by $0.1$ at epochs 30 and 40. Batch size is 4, with training for 50 epochs using torch-2.4.1 on a Tesla V100.

## 5. Quantitative Performance and Ablation Analyses

On the Lung-PET-CT-Dx test set, MMCAF-Net achieves:
- AUROC = 0.786
- Accuracy = 0.791
- F1 = 0.690
- Specificity = 0.857
- Sensitivity = 0.667
- PPV = 0.714
- NPV = 0.828

In comparison, the leading alternative (MMTM) reports AUROC = 0.802, Accuracy = 0.698; all other baselines (PECon, MedFuse, DrFuse, PEfusion, daft) fall below MMCAF-Net in both accuracy and F1. Image encoder ablation demonstrates substantial gains with the E3D-MSCA (+dropout): AUROC = 0.712, Accuracy = 0.767, F1 = 0.545, compared to PENet alone (AUROC = 0.560). Fusion module ablation shows MSCA (Accuracy = 0.791) outperforms Cross-Attn (0.744), clip_Fusion (0.698), and Late_Fusion (0.674).

## 6. Algorithmic Pipeline and Pseudocode Specification

The algorithmic workflow, as formalized in the provided pseudocode, proceeds as follows:
1. **Image Encoding:** Input images are passed through a PENet backbone and feature pyramid, with E3D-MSCA + BFPU applied at each scale.
2. **EHR Encoding:** EHR data is encoded using a Kolmogorov–Arnold Network.
3. **Multi-Scale Fusion:** At each of three scales, cross-attention fuses image and EHR features, followed by bidirectional scale fusion.
4. **Classification:** The resulting fused features are classified with a linear layer and sigmoid.
5. **Training Loop:** A standard batch-iterative procedure computes predictions and binary cross-entropy loss, with parameter updates via SGD, and learning rate scheduling.

The full modular structure is summarized in the workflow table:

| Component            | Processing Block(s)                                     | Output Dimensions    |
|----------------------|--------------------------------------------------------|---------------------|
| 3D Image Encoder     | PENet + Feature Pyramid + E3D-MSCA + BFPU              | $\{P_3, P_4, P_5\}$ |
| EHR Encoder          | Kolmogorov–Arnold Network                              | $E_s^{ehr}$         |
| Fusion Module        | Multi-Scale Cross-Attention + Bidirectional Scale Fusion| $F_{fused}$         |
| Classification Head  | FC → Sigmoid                                           | Probabilities       |

## 7. Significance, Limitations, and Research Context

MMCAF-Net demonstrates substantial improvement in small lesion detection and multimodal fusion for lung disease classification, particularly reflected in accuracy and F1 metrics compared to prior methods. The combined use of multi-scale pyramid features, channel and spatial attentions, and explicit cross-modal alignment addresses longstanding challenges in medical image-EHR fusion. Ablation results reinforce the critical role of each architectural component. A plausible implication is that such bidirectional, multiscale, and attention-based fusion modules could be generalized to other multimodal, multi-resolution diagnostic tasks involving high-dimensional 3D data and heterogeneous non-image sources [2508.04205].

Source: https://www.emergentmind.com/topics/multimodal-multiscale-cross-attention-fusion-network-mmcaf-net