MultiModNet: Adaptive Multimodal Fusion

• MultiModNet is an adaptive neural network architecture that fuses heterogeneous imaging modalities and anatomical ROI priors for precise Parkinson's disease diagnosis.
• It employs hierarchical modality-specific attention and channel-wise gating to integrate volumetric QSM, T1-weighted MRI, and ROI masks, enhancing feature fusion.
• Comparative evaluations demonstrate that gated ROI fusion in MultiModNet achieves superior performance, with accuracy up to 85% and improved clinical interpretability.

MultiModNet refers to adaptive neural network architectures designed for multimodal fusion, particularly in neuroimaging-based disease diagnosis. A canonical example is GateFuseNet, an adaptive 3D multimodal fusion network that integrates Quantitative Susceptibility Mapping (QSM), T1-weighted MRI, and anatomical region-of-interest (ROI) priors for the diagnosis of Parkinson's disease (PD) (Jin et al., 26 Oct 2025). The architecture employs hierarchical modality-specific attention and channel-wise gating mechanisms to achieve anatomically informed, ROI-guided feature fusion, resulting in improved diagnostic accuracy and interpretability. MultiModNet systems are characterized by their capacity to exploit heterogeneous imaging sources and anatomical priors, calibrated through spatial and channel-level gating.

1. Architectural Principles

MultiModNet architectures are constructed to process and fuse multiple volumetric modalities. In GateFuseNet, three inputs are used per subject: volumetric QSM (sensitive to iron deposition in deep gray matter nuclei), T1-weighted MRI (providing high-resolution anatomical structure), and binary ROI masks (defining nuclei such as the substantia nigra, putamen, caudate, globus pallidus, and subthalamic nucleus) (Jin et al., 26 Oct 2025).

Each input is preprocessed to a uniform spatial resolution and shape, then passed through a modality-specific Stem Module (stacked 3×3×3 convolutions with ELU and batch-norm, followed by 2×2×2 max-pooling). After initial feature extraction, a Gated Fusion (GF) block merges low-level cues. Three repeated Fusion Modules then perform deeper feature extraction: each comprises three parallel, CBAM-augmented bottleneck branches (separately for QSM, T1w, and ROI input), with mid-level fusion achieved through additional GF blocks. The Decision Module aggregates fused features via dilated bottleneck blocks and global average pooling, outputting binary classification.

2. Gated Fusion Mechanism

The distinguishing feature of MultiModNet, as implemented in GateFuseNet, is the adaptive multimodal fusion (AMF) block, enhanced by a channel-wise gating (CWG) mechanism. For each modality $m \in \{\text{ROI}, \text{QSM}, \text{T1}\}$, let $x_m \in \mathbb{R}^{C \times D \times H \times W}$ denote its feature map. The multimodal fusion proceeds as follows:

• Modality-specific Attention: Concatenate input feature maps and apply parallel 3×3×3 grouped convolutions, batch-norm, and sigmoid activations to produce voxel-wise attention maps $a_m \in \mathbb{R}^{1 \times D \times H \times W}$, normalized such that $\sum_m a_m(i,j,k) = 1$ for all spatial indices.
• Voxel-wise Fusion: Compute the fused feature tensor by weighted sum:

$$f = \sum_{m \in \{\text{ROI}, \text{QSM}, \text{T1}\}} a_m \odot x_m,$$

where $\odot$ denotes element-wise multiplication with attention broadcast over channels.

• Channel-wise Gating: A learnable gate vector $g \in \mathbb{R}^C$ produces $\sigma(g) \in [0,1]^C$ after sigmoid, modulating each channel of $f$: $f' = \sigma(g) \odot f$.
• Residual Injection and Hierarchical Fusion: $f'$ is added via residual connection to the ROI branch: $x_{\text{ROI}}^{\text{new}} = x_{\text{ROI}} + f'$. This process is applied at multiple network depths, allowing hierarchical recalibration of multimodal contributions (Jin et al., 26 Oct 2025).

3. Anatomical ROI Guidance

MultiModNet leverages explicit anatomical priors through ROI masks registered into QSM space using standard non-linear registration pipelines (e.g., ANTs and the MuSus-100 template, with atlas derived from AAL3). The binary masks for key deep gray matter (DGM) nuclei are incorporated as a separate input modality, serving a dual function: they inform feature extraction in one network branch and directly steer fusion via the channel-wise gating mechanism. Notably, the ROI pathway is not subject to an auxiliary loss; exploitation of ROI signal is instead promoted by the global focal-loss objective and the network’s gating structure (Jin et al., 26 Oct 2025).

4. Training Regimen and Data

The architecture was evaluated on a dataset of 316 subjects (161 PD, 155 healthy controls), with 64 held out for independent testing and the remainder split by five-fold cross-validation for training/validation. All imaging volumes are resampled to isotropic 1 mm³ and cropped/padded to 128³. Data augmentation includes random affine transformations, bias-field corruption, and Gaussian noise (applied with stated probabilities and parameter ranges). Optimization is conducted with AdamW (initial learning rate $2\times10^{-4}$, cosine annealed over 30 epochs, batch size 8), and binary focal loss with focus parameter $\gamma=2.0$ and class balance $\alpha=0.5$:

$$\mathcal{L}_\mathrm{focal} = -\alpha(1-p)^\gamma y\log(p) - (1-\alpha) p^\gamma (1-y) \log(1-p)$$

where $p = \sigma(z)$, the predicted probability, and $y \in \{0,1\}$ is the ground truth label. Checkpoint selection is determined by combined validation AUC and F1 (Jin et al., 26 Oct 2025).

5. Performance Benchmarks and Ablation

Quantitative evaluation demonstrates that GateFuseNet, and by extension MultiModNet architectures with adaptive gating, achieve superior performance over baseline and alternative multi-input models. On the held-out test set, GateFuseNet attains 85.00% accuracy, 0.9206 AUC, and 0.9227 AUPR, significantly outperforming ResNeXt (76.56% accuracy, AUC 0.8594), AG-SE-ResNeXt, and DenseFormer-MoE. Ablation experiments underscore the importance of both the fusion strategy and fusion block placement:

Model/Fusion Strategy	Accuracy (%)	AUC
Weighted Sum Fusion	76.68	0.8642
Simple Concatenation	78.17	0.8823
Gated Fusion (MultiModNet)	85.00	0.9206

Insertion of the fusion block into the ROI branch yields the largest performance gain (accuracy 85.00%, AUC 0.9206), compared to the T1 (77.06%, 0.8761) or QSM (78.43%, 0.8920) branches. This outcome supports the effectiveness of ROI-guided fusion as implemented in MultiModNet (Jin et al., 26 Oct 2025).

6. Interpretability and Clinical Relevance

Qualitative analysis via Grad-CAM reveals that MultiModNet models with hierarchical fusion consistently attend to basal ganglia substructures—most notably the substantia nigra and globus pallidus—corresponding with established PD pathology. The visualizations corroborate the model's reliance on physiologically meaningful image regions, as opposed to spurious cues, and thus enhance interpretability in a clinical context. A plausible implication is that the integration of anatomical priors through adaptive fusion not only boosts diagnostic accuracy but also yields more trustworthy decision-making in medical imaging workflows (Jin et al., 26 Oct 2025).