HyPCA-Net: Advancing Multimodal Fusion in Medical Image Analysis

Abstract: Multimodal fusion frameworks, which integrate diverse medical imaging modalities (e.g., MRI, CT), have shown great potential in applications such as skin cancer detection, dementia diagnosis, and brain tumor prediction. However, existing multimodal fusion methods face significant challenges. First, they often rely on computationally expensive models, limiting their applicability in low-resource environments. Second, they often employ cascaded attention modules, which potentially increase risk of information loss during inter-module transitions and hinder their capacity to effectively capture robust shared representations across modalities. This restricts their generalization in multi-disease analysis tasks. To address these limitations, we propose a Hybrid Parallel-Fusion Cascaded Attention Network (HyPCA-Net), composed of two core novel blocks: (a) a computationally efficient residual adaptive learning attention block for capturing refined modality-specific representations, and (b) a dual-view cascaded attention block aimed at learning robust shared representations across diverse modalities. Extensive experiments on ten publicly available datasets exhibit that HyPCA-Net significantly outperforms existing leading methods, with improvements of up to 5.2% in performance and reductions of up to 73.1% in computational cost. Code: https://github.com/misti1203/HyPCA-Net.

• The paper introduces HyPCA-Net, a multimodal fusion framework combining parallel spatial–channel attention with cascaded spatial–frequency modeling to preserve information while reducing computational demands.
• HyPCA-Net achieves up to 5.2% higher performance than leading baselines, cuts parameters by up to 92% and FLOPs by up to 81.47%, and improves segmentation DSC to 90.3% on COVID-19 CT and 93.8% on ISIC2018.
• Ablation results support the value of RALA, Hy-SFA, and MMMUA, although near-perfect benchmark scores, limited 2D evaluation, and absent uncertainty testing leave generalization under clinical distribution shifts unresolved.

Overview

HyPCA-Net is a multimodal fusion framework for medical image analysis that targets two shortcomings of existing fusion architectures: high computational cost and progressive information loss in cascaded attention designs (2602.16245). The framework couples parallel spatial–channel fusion attention with cascaded dual-domain (spatial–frequency) attention, and is evaluated on ten public medical imaging benchmarks for both classification and segmentation. The authors report performance gains of up to 5.2% over state-of-the-art baselines alongside reductions of up to 73.1% in computational cost.

Motivation and problem formulation

Existing attention-driven fusion methods—such as HAMLET, MuMu, M³Att, MTTU-Net, and DRIFA-Net—rely on standard convolutions and multi-stage sequential attention, which the authors argue causes two problems: (1) computational expense that limits deployment in low-resource clinical settings, and (2) information loss at inter-module transitions in cascaded pipelines, which degrades shared cross-modal representations and multi-disease generalization.

The framework is formulated as a mapping $\mathcal{F}: \mathcal{X} \rightarrow \mathcal{Y}$ from $m$ heterogeneous modalities to $t$ task labels, structured in two phases: Robust Multimodal Information Learning (RMIL), which produces shared representations $X^s$ via modality-specific branches of cascaded HyPCA blocks, and Multimodal Multitask Learning (MML), which performs downstream classification through a weighted sum of task-modality-specific losses $$\mathcal{L}_{MML} = \sum_t \sum_i \lambda_t^i \mathcal{L}_t^i$$.

Architecture

RALA block: parallel spatial–channel refinement

The Residual Adaptive Learning Attention block addresses three limitations the authors identify in EMCAD's Multi-Scale Convolution Block: single-stage multi-scale processing, homogeneous branch design, and absent spatial–channel fusion. RALA stacks two SCALA modules for progressive refinement. Each SCALA combines:

• MSHC: heterogeneous convolution branches (group point-wise, dilated depth-wise, and depth-wise convolutions at $1\times1$, $3\times3$, $5\times5$ scales) followed by channel shuffle and channel-restoring layers.
• SCPFA: a parallel fusion of Channel Holistic Information-Learning Attention (CHIA) and Spatial Holistic Information-Learning Attention (SHIA), each aggregating four pooling statistics—global average, max, min, and standard-deviation pooling—to produce a joint spatial–channel attention map applied via Hadamard product.

Because CHIA and SHIA operate in parallel rather than sequentially, early discriminative information is preserved before the subsequent cascaded stage.

DVCA block: cascaded hybrid-space, dual-domain modeling

The Dual-View Cascaded Attention block learns shared representations through two modules. Hy-SFA co-attends token-space and feature-space information within a single module—a combination the authors claim had not previously been unified in multimodal medical fusion pipelines. Its TFSI submodule jointly processes spatial tokens and frequency tokens obtained via normalized 2D DCT over shifted windows, while its FDCA submodule treats features as the initial state of a continuous-depth flow in the neural-ODE formulation, applying Euler and Runge–Kutta second-order solvers in parallel to capture coarse- and fine-grained context, fused adaptively and refined by heterogeneous channel attention.

MMMUA then performs mutual cross-modal updates across scales and frequency bands. FCIF decomposes inputs into wavelet sub-bands (LL, HL, LH, HH) and fuses pooled contexts hierarchically; SMIF reuses MSHC + SCPFA to derive spatial-channel descriptors; MCBI applies bidirectional asymmetric interactions between frequency-domain ($C_f$) and spatial-domain ($C_{SP}$) contexts of different modalities, yielding per-modality spatial–frequency attention maps $m$0 used to recalibrate features into $m$1.

Empirical results

On eight classification benchmarks (brain tumor MRI, lung cancer CT/CXR, tuberculosis, pediatric leukemia, HAM10000 dermoscopy, SIPaKMeD cytology, colorectal histology, CBIS-DDSM mammography), HyPCA-Net instantiated with four backbones (ResNet-18, ResNet-50, Inception-v3, ViT-Ti) consistently ranks first or second. Representative results include:

Model	Params (M)	GFLOPs	Notable results
DRIFA-Net	53.8	4.83	Prior best fusion baseline
MuMu	56.6	2.97	—
M³Att	183	12.14	Largest baseline
HyPCA-Net18	14.47	2.25	98.8% acc (D1), 99.8% (D2)
HyPCA-Net-ViT	22.5	3.42	100% acc/F1/AUC on HAM10000

Notably, HyPCA-Net-ViT achieves perfect 100% accuracy, F1, and AUC on HAM10000—a result that should be interpreted cautiously given dataset saturation—and reaches 97.8% accuracy on CBIS-DDSM. Relative to baselines, improvements span 0.1%–43.9%, with parameter reductions up to 92% and FLOP reductions up to 81.47%.

For segmentation on COVID-19 lung CT and ISIC2018, HyPCA-Net-EMCAD attains 90.3/82.5 DSC/IoU on COVID-19 and 93.8/86.4 on ISIC, exceeding EMCAD (85.8/78.6; 90.9/84.1) with fewer parameters (18.6M vs. 26.8M).

Ablations confirm each component contributes: removing any of RALA, Hy-SFA, or MMMUA reduces F1/DSC by up to 7.9%, and component-level ablations show TFSI and FDCA individually account for substantial gains. A direct comparison of cascaded versus hybrid (parallel) CHIA+SHIA configurations shows the hybrid variant yields a 0.4–0.5% F1 gain at identical parameter count and FLOPs, supporting the paper's central claim that parallel fusion mitigates cascaded information loss. GRAD-CAM visualizations on two datasets qualitatively support localization of discriminative regions.

Limitations and open questions

Several caveats warrant note. The near-ceiling scores on several benchmarks (e.g., 99.8–100%) suggest dataset saturation, where headroom for demonstrating generalization is limited and differences among top methods may not be statistically meaningful; the paper does not report variance, confidence intervals, or significance tests. The claimed efficiency advantages are measured against specific backbones, and the 43.9% upper bound on improvement appears driven by weak baselines such as UNet. Evaluation is confined to 2D imaging tasks resized to fixed resolutions; behavior on volumetric data, class-imbalanced clinical distributions, and external validation sets is untested. The conclusion identifies adversarial robustness and domain-shift adaptation as unaddressed, leaving open whether HyPCA-Net's shared representations generalize under distribution shift—a question particularly relevant to the paper's stated motivation of scalable healthcare AI.

Conclusion

HyPCA-Net integrates parallel spatial–channel fusion attention (RALA/SCPFA) with cascaded token–frequency, neural-ODE-based dual-domain attention (DVCA/Hy-SFA/MMMUA) to learn robust shared multimodal representations at reduced computational cost. Extensive evaluation across ten datasets supports the paper's claims of improved accuracy and substantially lower parameter and FLOP counts relative to existing fusion frameworks, though saturated benchmark scores and the absence of uncertainty reporting temper the strength of some comparisons.