Multimodal Oncology Agent (MOA)

• Multimodal Oncology Agents (MOA) are computational frameworks that integrate heterogeneous biomedical data to improve diagnostic accuracy and therapeutic decision-making.
• They employ advanced deep learning models with modality-specific encoders and attention-based fusion techniques to handle data heterogeneity in oncology.
• Recent MOA designs achieve state-of-the-art performance through interpretable reasoning, multi-agent orchestration, and retrieval-augmented strategies in precision oncology.

A Multimodal Oncology Agent (MOA) is a computational framework that integrates heterogeneous biomedical data and advanced reasoning tools to assist, automate, or augment complex oncology tasks, including diagnosis, prognosis, therapeutic decision-making, and mechanism-of-action discovery. MOA architectures combine deep learning systems for multimodal data embedding and fusion (e.g., histopathology, genomics, radiology, clinical text) with agentic or multi-agent orchestration strategies, leveraging retrieval-augmented knowledge, chain-of-thought (CoT) reasoning, and tool-based workflows to deliver clinical decision support in precision oncology settings. Recent MOA designs achieve state-of-the-art results in clinical benchmarks by unifying multimodal feature extraction, attention-guided information alignment, interpretable reasoning traces, and scalable multi-agent collaboration (Yi et al., 11 Jun 2024, Akebli et al., 5 Dec 2025, Vasilev et al., 25 Nov 2025, Pang et al., 10 Jul 2025, Zhang et al., 9 Dec 2025, Huang et al., 20 Nov 2025, Waqas et al., 2023).

1. Foundations and Motivation

MOA development is driven by the recognition that cancer is a fundamentally multimodal disease. Relevant data modalities for oncologic decision-making span digital pathology (whole-slide images), radiology (CT, MRI), molecular assays (genomics, transcriptomics, proteomics), structured tabular data (biomarkers, labs), and unstructured clinical text. Each modality contributes unique and complementary information, and "multimodal learning" (MML) aims to harness these synergies to improve clinical endpoints such as diagnostic accuracy, survival prediction, and treatment response (Waqas et al., 2023). Initial MML frameworks emphasized early (feature-level), intermediate (cross-modal embedding), and late (decision-level) fusion, but recent advances leverage joint attention, graph-based integration, and hybrid neural-symbolic approaches to mitigate alignment challenges and data heterogeneity.

2. Core Architectures and Mathematical Formulations

Modern MOA architectures are typically structured around modality-specific encoders coupled with deep multimodal fusion modules. The "Unified Modeling Enhanced Multimodal Learning" (UMEML) blueprint exemplifies this paradigm (Yi et al., 11 Jun 2024):

• Modality Encoders: Separate Transformer-based or graph neural network encoders extract high-dimensional features from each input domain (e.g., patch tokens for histology $P \in \mathbb{R}^{M \times d}$, gene-group tokens for genomics $G \in \mathbb{R}^{N \times d}$).
• Prototype Clustering: Each modality's encoder reduces raw features to a small set of latent "prototypes" ($K$ for pathology, $N$ for genomics), using cross-attention mechanisms:

$Q = C^{l-1} W_q,\quad K = P W_k,\quad V = P W_v$

$$\bar C^l = \mathrm{Softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V,\quad C^l = C^{l-1} + \bar C^l$$

• Alignment and Modularity: Cross-modal assignment matrices $S^p, S^g$ are computed by

$$S^p_{i,j} = \max(0, \cos(C_i, P_j)),\quad S^g_{i,j} = \max(0, \cos(G_i, P_j))$$

Affinity graphs $A$ and modularity losses regularize the alignment:

$$\mathcal{L}_{\mathrm{mod}} = -\frac{1}{2e}\left(\alpha\,\mathrm{Tr}(W S^{pT}S^p) + \beta\,\mathrm{Tr}(W S^{gT}S^g)\right)$$

• Unified Decoder and Register Tokens: Fused multi-head self-attention attends over both prototype sets, with intervening registration tokens to facilitate information flow and cross-modal disentangling:

$\hat U = [\hat C^L\, ; R\, ;\, \hat G^L]$

$$U^\mathrm{out} = \mathrm{MSA}(\mathrm{LayerNorm}(\hat U)) + \hat U$$

Alternative MOA blueprints employ multi-agent frameworks (e.g., parallel expert agents for text, image, lab modalities and a core orchestrator (Zhang et al., 9 Dec 2025)) and retrieval-augmented generation, hierarchical CoT reasoning, and dichotomy-based multi-expert inference (Huang et al., 20 Nov 2025).

3. Training, Optimization, and Fusion Strategies

MOA models are optimized by supervised (or semi-supervised) losses aligned with their downstream tasks:

• Classification (Diagnosis/Grading): Cross-entropy loss over class predictions.
• Survival Prediction: Negative log-likelihood (Cox model), or dichotomized interval refinement.
• Mechanism-of-Action (MoA) Recognition: Cross-modal contrastive losses (e.g., CLIP-style for molecular + video features (Pang et al., 10 Jul 2025)) and metric learning (hard triplet, center loss).

Fusion strategies vary, comprising simple concatenation, gating networks, attention-based fusion, or domain-specific joint embeddings. Weighted regularization (e.g., hyperparameter $\gamma$ on modality alignment loss) prevents unimodal bias, balancing contributions from each source (Yi et al., 11 Jun 2024).

Key optimization details include:

• Batch size: often constrained by high-resolution images, e.g., batch size 1 for WSI (Yi et al., 11 Jun 2024).
• Regularization: weight decay (e.g., $1\times10^{-5}$), stratified cross-validation, and normalization per fold.
• Losses: multimodal and modularity-based regularizers, auxiliary tool-specific targets, multi-task loss aggregation (Zhang et al., 9 Dec 2025).

4. Representative Implementations and Empirical Performance

Several state-of-the-art MOA instances have been benchmarked in diverse oncology datasets and settings:

System / Benchmark	Key Modalities	Task	Major Metrics	Outcome Highlights
UMEML (Yi et al., 11 Jun 2024)	Histology, genomics	Glioma grading/class/surv	Acc, AUC, c-index	Acc↑3.8pp vs MCAT, c-index 0.8396
TITAN MOA (Akebli et al., 5 Dec 2025)	Histology, clinical, text	IDH1 mutation pred (LGG)	F1, Acc, AUROC	F1=0.912, histology baseline 0.894
MTBBench (Vasilev et al., 25 Nov 2025)	H&E/IHC/case history	Sequential oncology QA	Accuracy, file access	Tool-aug +9% multimodal acc.
SurvAgent (Huang et al., 20 Nov 2025)	WSI, genomics	Survival prediction	c-index, ablation	c-index 0.713, interpretable CoT
Multi-Agent MDT (GI) (Zhang et al., 9 Dec 2025)	Endoscopy, lab, rad, text	MDT diagnosis (GI cancer)	7-dim composite expert score	4.60/5.00 composite, logic↑1.17
MolCLIP (Pang et al., 10 Jul 2025)	Cell video, molecule	Drug MoA ID, recognition	mAP, class accuracy	MoA mAP↑20.5pp vs baseline

All top-performing MOAs exhibit performance gains over single-modality or monolithic baselines, often in the range of 4–15 percentage points absolute on relevant metrics, and substantial increases in reasoning logic and diagnostic comprehensiveness (Yi et al., 11 Jun 2024, Zhang et al., 9 Dec 2025, Vasilev et al., 25 Nov 2025, Huang et al., 20 Nov 2025, Pang et al., 10 Jul 2025).

5. Multi-Agent and Retrieval-Augmented Reasoning Paradigms

Recent MOA architectures increasingly adopt multi-agent and retrieval-augmented frameworks to address the complexity of modern clinical workflows. In "Multi-Agent Intelligence for Multidisciplinary Decision-Making in Gastrointestinal Oncology," autonomous agents specializing in unstructured clinical notes, endoscopic imaging, radiology, and laboratory data ingest and encode modality-specific representations, while a central core agent fuses these via nonlinear gating or attention and orchestrates final report generation, with explicit conflict detection routines to mimic the cross-validation processes of real-world multidisciplinary teams (Zhang et al., 9 Dec 2025).

"SurvAgent" extends this approach by combining hierarchical CoT-enhanced case banks, cross-modal patch mining, and dichotomy-based, multi-expert agent inference. Through recurrence to structured reasoning on earlier historical analogs (retrieval-augmented generation) and progressive interval refinement, SurvAgent achieves C-index gains over both unimodal and prior multimodal survival predictors (Huang et al., 20 Nov 2025).

The agentic paradigm is further reinforced in MTBBench (Vasilev et al., 25 Nov 2025), which simulates multistep, sequential clinical encounters, with file-access tracking as a proxy for groundedness and tool-invocation for both interpretation (e.g., H&E diagnosis) and external knowledge retrieval.

6. Challenges, Limitations, and Future Extensions

Despite significant progress, MOA systems contend with challenges such as data heterogeneity, scale mismatch, missing modalities, modality collapse during joint training, and the need for interpretability and uncertainty quantification (Waqas et al., 2023). Addressed strategies include:

• Modality-specific encoders and dynamic weighting to accommodate data variability and missingness.
• Cross-attention, registration tokens, and modularity loss to prevent domination by a single modality (Yi et al., 11 Jun 2024).
• Incorporation of external tools, self-supervised pretraining, and retrieval-augmented reasoning for robustness.
• Multi-agent architectures for scalable, audit-friendly, and clinically congruent integration (Zhang et al., 9 Dec 2025).
• Use of interpretable reasoning traces (e.g., CoT), explicit tracing for validation, and audit logs (Huang et al., 20 Nov 2025).

Future directions include expanding to new modalities (radiology, proteomics, EHRs) by appending specialized encoders and appropriately extending fusion and regularization losses (Yi et al., 11 Jun 2024); end-to-end fine-tuning for agent orchestration; and adoption of federated learning and foundation model pretraining for improved generalizability and data privacy (Waqas et al., 2023).

7. Clinical Significance and Outlook

MOAs deliver clinically relevant gains in diagnostic accuracy, comprehensiveness, and explainability, as validated in expert-reviewed composite scores and real-world benchmarks (Zhang et al., 9 Dec 2025, Akebli et al., 5 Dec 2025, Yi et al., 11 Jun 2024). Their interpretability, traceability, and auditable inference pathways position them for integration into clinical decision support systems such as Molecular Tumor Boards and MDT workflows. Limitations remain in reconciling longitudinal evidence, handling ambiguous or contradictory findings, and extending to dynamic decision-making settings, motivating ongoing research in sequential agentic reasoning and domain-specialized foundation models (Vasilev et al., 25 Nov 2025). By combining deep multimodal neural networks, multi-agent orchestration, and robust retrievable knowledge, the MOA paradigm is establishing the blueprint for next-generation, high-reliability AI systems in precision oncology.