Foundation Model Fusion in Cancer Staging

• Foundation model fusion for cancer staging is the integration of large-scale pretrained models from diverse modalities to improve diagnostic accuracy.
• Fusion methods leverage concatenation, cross-attention, and pruning techniques to combine modality-specific features and enhance system robustness.
• Robust pretraining and federated learning pipelines enable scalable, privacy-preserving staging solutions that outperform traditional single-modality approaches.

Foundation model fusion for cancer staging refers to the integration of large, pretrained models—capable of extracting generalizable features from diverse biomedical imaging and clinical modalities—into composite systems that improve the accuracy, robustness, and interpretability of cancer staging tasks. Modern cancer staging requires synthesizing heterogeneous data, including radiology (CT, PET, MRI), pathology (whole-slide images, WSIs), and text (clinical metadata), each benefiting from specialized foundation models (FMs). Fusion approaches aim to combine strengths across these data streams, mitigate modality/account-level bias, and increase resilience to data scarcity, privacy constraints, and distributional shifts.

1. Foundation Models in Cancer Staging: Modalities and Encoders

Foundation models for cancer staging are typically instantiated as large-scale, pretrained encoders applied to major medical imaging modalities and, in some cases, text-based clinical metadata.

• Imaging Modalities: Commonly used inputs include whole-body PET/CT for staging and lesion identification (Oh et al., 4 Mar 2025), whole-slide H&E stained histopathology (WSI) for grade/stage classification (Flannery et al., 11 Dec 2025, Yang et al., 31 Oct 2025), MRI for rectal cancer risk markers (Zhang et al., 23 May 2025), and cross-modality fusion such as integrating CT and WSI for personalized recurrence risk (Boeke et al., 29 Aug 2025).
• Encoder Architectures:
  • Vision Transformer backbones (ViT, Swin-Transformer, LongNet) are dominant for both 2D and 3D imaging.
  • Universal biomedical transformers (e.g. UMedPT) leverage large multi-modal pretraining (Zhang et al., 23 May 2025).
  • Text encoders, usually CLIP-style ViTs, capture structured clinical prompts (e.g., tracer type, diagnosis, demographics) (Oh et al., 4 Mar 2025).
• Representational goals: Encoders are trained to capture both local morphological signals (e.g., lesion shape, tissue architecture) and global semantic patterns (e.g., metabolic uptake, organ-specific risk factors, molecular markers).

2. Fusion Mechanisms: Algorithms and Mathematical Formulations

Fusion addresses the challenge of combining multi-source FM representations into a unified decision framework. Major strategies fall into three families:

1. Feature Concatenation and MLP Heads: Concatenate modality-specific embeddings (e.g., z_CT, z_PET, z_WSI) and train an MLP for downstream prediction (e.g., cancer stage, risk score) (Boeke et al., 29 Aug 2025, Zhang et al., 23 May 2025). This “intermediate fusion” is mathematically expressed as

$$z_{\text{joint}} = [z_{\text{CT}} ; z_{\text{WSI}}] \in \mathbb{R}^{2d}$$

followed by

$$R = g(z_{\text{joint}}) = W_2 \sigma(W_1 z_{\text{joint}} + b_1) + b_2$$

where $g(\cdot)$ is an MLP classifier/regressor.

1. Cross-Attention Decoders: Cross-modal masked autoencoders (e.g., FratMAE) reconstruct masked regions in one modality using context from another, with cross-attention blocks fusing PET and CT embeddings (Oh et al., 4 Mar 2025):
   • Query = masked modality embedding $Z_S$
   • Key/Value = unmasked modality embedding $Z_{\lnot S}$
   • Loss is the sum of MAE reconstruction and InfoNCE contrastive alignment.
2. Correlation-Driven and Clustered Feature Pruning:
   • Intelligent fusion methods employ correlation-guided pruning to reduce redundancy and select complementary, discriminative FM features before downstream classification (Flannery et al., 11 Dec 2025). Retained features are selected to have low pairwise correlation $|\rho_{fg}| \leq \theta$ and high univariate discriminative score, resulting in compact, task-optimized signatures.
   • Cluster-level re-embedding (as in FuseCPath) pools patch-level outputs across FMs using self-attention mechanisms constrained to local clusters, then distills heterogeneous slide-level models into a unified student via KL divergence (Yang et al., 31 Oct 2025).

3. Pretraining, Data, and Training Pipelines

Successful FM fusion models depend critically on the breadth of their pretraining data and the rigor of their training pipelines.

• PET/CT: FratMAE is pretrained on 1,292 whole-body scans (AutoPET III) across four cancer types and three radiotracers, using 3D coronal patch subsampling and simultaneous textual prompt ingestion (Oh et al., 4 Mar 2025).
• Pathology WSIs: Preprocessing pipelines segment tissue, tile WSIs, and standardize intensity. FMs (e.g., CONCH, Virchow2, H-Optimus1) are pretrained on up to 1.3 billion patches, with slide-level aggregation by transformer or graph models (Flannery et al., 11 Dec 2025, Yang et al., 31 Oct 2025).
• MRI: Frequency-domain harmonization is applied before encoding to mitigate scanner/protocol heterogeneity. Foundation models are pretrained via masked image modeling and contrastive learning (Zhang et al., 23 May 2025).
• Cross-institutional Learning: MedForge enables decentralized LoRA adaptation with dataset distillation, so that updates are communicated as small adapter weights plus highly compressed pseudo-datasets, never raw patient data (Tan et al., 22 Feb 2025).

Fine-tuning for staging is typically performed only on the encoder’s final layers or via a lightweight head (e.g., UNETR for segmentation, logistic regression for risk, CLAM for attention-based MIL) (Oh et al., 4 Mar 2025, Flannery et al., 11 Dec 2025, Zhang et al., 23 May 2025).

4. Quantitative Performance in Cancer Staging Applications

Foundation model fusion has achieved or exceeded state-of-the-art performance on diverse staging-related tasks:

• Lesion and Stage Classification (PET/CT): FratMAE achieves lesion Dice=0.640 and IoU=0.496 on lymphoma segmentation—outperforming both scratch and unimodal pretraining baselines; Ann Arbor staging sensitivity is raised to 0.648, accuracy 0.654 (Oh et al., 4 Mar 2025).
• Patch- and Slide-level Pathology: Intelligent fusion outperforms the best single FM and naive concatenation for tile-level staging/grade classification (e.g., F1=0.89 for rectal cancer, F1=0.94 for prostate), with all improvements statistically significant (Wilcoxon p<0.01) (Flannery et al., 11 Dec 2025).
• Multicenter MRI (Rectal Cancer EVI/MFI Detection): Multi-view fusion (axial+sagittal) of UMedPT features yields AUC=0.82 (sens 0.75, F1=0.73) for EVI, outperforming the previous challenge winner by ΔAUC>0.08 (Zhang et al., 23 May 2025).
• Multimodal Renal Cancer Prognostics: Intermediate fusion of CT and WSI encoders achieves C-index=0.775, approaching the clinical Leibovich score (C=0.805 baseline, but C=0.749 after random tie-breaking) (Boeke et al., 29 Aug 2025).
• Heterogeneous FM Ensembles (FuseCPath): Cluster-based patch selection plus multi-head self-attention and slide-level distillation achieve gains up to +12 AUROC points in colorectal TCGA biomarker classification versus leading baselines (Yang et al., 31 Oct 2025).
• Collaborative Federated Fusion (MedForge): On multi-center binary/multiclass histopathology, progressive LoRA-merged models attain average accuracy up to 0.963 and AUC 0.972, with asynchronous updates and strict data localization (Tan et al., 22 Feb 2025).

5. Interpretability, Embedding Space Analysis, and Clinical Relevance

• Embedding Complementarity: Global similarity (Centered Kernel Alignment, SVCCA) between FMs is high (CKA~0.7–0.9), but local neighborhood agreement is low (k-NN Jaccard ~0.12–0.28), establishing that FMs capture complementary fine-grained features (Flannery et al., 11 Dec 2025).
• Attention Focus: Intelligent tile-level fusion increases the attention of downstream MIL models on tumor regions (90% tumor coverage at 25th percentile), while reducing spurious attention on benign tissue; naive concatenation dilutes this effect (Flannery et al., 11 Dec 2025).
• Cluster Analysis: Compact, pruned feature signatures lead to more compact, well-separated unsupervised clusters of tumor versus benign tiles (silhouette ~0.48 vs <0.4 for single FMs), supporting the biological plausibility of fused representations.
• Clinical Integration: Foundation model fusion enables more reproducible, high-sensitivity, and site-generalizable staging scores suitable for automated decision support, as demonstrated in rectal cancer EVI/MFI and renal cancer risk stratification (Zhang et al., 23 May 2025, Boeke et al., 29 Aug 2025).

6. Privacy, Federated Collaboration, and Scaling Considerations

• Federated Model Development: MedForge exemplifies asynchronous, decentralized FM fusion, where LoRA-adapted modules and distilled pseudo-data are merged incrementally into composite plugins, sidestepping data-silo barriers (Tan et al., 22 Feb 2025).
• Asynchronicity and Robustness: Model integration can proceed with no global client synchronization; old or noisy adapters are down-weighted, and hyperparameters (e.g., merging coefficients) are tuned via gradient-free optimizers.
• Parameter Efficiency: LoRA adapters and dataset distilled sets are orders of magnitude smaller than full model checkpoints or raw data, promoting practical multi-institution FM collaboration.

This suggests that privacy-preserving, progressively refined FM fusion can aggregate staging knowledge across institutions, improving robustness to distribution shift and enabling continuous improvement as new data sources join.

7. Outlook: Limitations, Extensions, and Future Directions

While fusion of foundation models for cancer staging has demonstrated robust improvements, several directions remain open:

• Multi-class and Multi-center Staging: Most published results focus on binary or dichotomized staging. Extension to fine-grained AJCC stage groups, response prediction, and multi-center harmonization—potentially via cross-modal transformers—remains a key research area (Oh et al., 4 Mar 2025, Flannery et al., 11 Dec 2025).
• Improved Radiology Encoders: Current radiology FMs underperform relative to pathology FMs, suggesting the need for large-scale self-supervised pretraining in radiology, especially for underrepresented cancer types (Boeke et al., 29 Aug 2025).
• Advanced Fusion Architectures: Cross-attentional, hierarchical, or graph-based fusion—moving beyond concatenation or majority voting—could further enhance integration of spatial and semantic information (Yang et al., 31 Oct 2025).
• Interpretability at Clinical Scale: Systematic adoption of attention visualization and embedding clustering can provide clinicians with transparent rationale for stage assignment and risk stratification.
• Regulatory, Standardization, and Deployment: The move toward foundation model fusion necessitates new standards for versioning, validation, and continued learning in clinical practice, as well as transparent reporting of data provenance and harmonization protocols (Tan et al., 22 Feb 2025).

In summary, foundation model fusion for cancer staging unites the complementary strengths of large-scale encoders across diverse medical data modalities. By leveraging advanced fusion algorithms—cross-attention, correlation pruning, federated LoRA merging—these systems routinely outperform unimodal or non-fused baselines, enable cross-silo collaboration, and deliver increasingly interpretable, robust, and individualized staging for oncological decision support (Oh et al., 4 Mar 2025, Flannery et al., 11 Dec 2025, Yang et al., 31 Oct 2025, Boeke et al., 29 Aug 2025, Zhang et al., 23 May 2025, Tan et al., 22 Feb 2025).