SurvAgent: Multimodal Survival Prediction

• SurvAgent is a hierarchical chain-of-thought multi-agent system that integrates multimodal clinical data for survival prediction using detailed pathology image analysis and gene-level stratification.
• It employs a two-stage pipeline to construct structured case banks and uses dichotomy-based multi-expert agent inference for calibrated and interpretable survival estimates.
• Extensive evaluations on TCGA cancer cohorts demonstrate its state-of-the-art performance and significant improvements in risk stratification, offering a promising approach for explainable precision oncology.

SurvAgent is a hierarchical chain-of-thought (CoT)-enhanced multi-agent system designed for multimodal survival prediction, addressing critical limitations in existing clinical AI approaches such as lack of transparent reasoning, insufficient multimodal integration, and ineffective experiential learning from historical cases. Specifically, SurvAgent synergistically combines detailed pathology image (WSI) analysis and gene-level stratification to create structured, interpretable case banks, and then leverages a dichotomy-based multi-expert agent reasoning protocol for transparent, interval-refined survival estimation. Extensive evaluation on five TCGA cancer cohorts demonstrates that SurvAgent achieves state-of-the-art concordance indices and robust patient risk stratification, presenting a new paradigm for explainable precision oncology applications (Huang et al., 20 Nov 2025).

1. System Architecture and Workflow

SurvAgent follows a two-stage pipeline:

• WSI–Gene CoT-Enhanced Case Bank Construction: Historical patient cases are processed to yield two multimodal retrieval databases (“case banks”)—one for whole-slide pathology images (WSI), and one for gene functional groups. Each entry comprises a concise report, a structured CoT reasoning trace, and observed survival time.
  • Hierarchical WSI Analysis: Operates at three magnifications (2.5×, 10×, 20×). Initial low-magnification global reports are obtained via a PathAgent, followed by Cross-Modal Similarity-Aware Patch Mining (CoSMining) at 10× using visual (φ) and text (ψ) encoders:

  $$S^v_{ij} = \cos(\phi(\mathbf p_i),\phi(\mathbf p_j)), \quad S^t_{ij} = \cos(\psi(\mathcal R_i),\psi(\mathcal R_j))$$

  Patches are retained if they are both visually and semantically unique. Confidence-aware mining at 20× further targets low-confidence regions. - Gene-Stratified Analysis: Genomic input is partitioned into six functional groups (Tumor Suppressors, Oncogenes, Kinases, Differentiation, Transcription Factors, Cytokines). Group-level statistics $\mathbf s_l = [\mu_l, m_l, r_{\mathrm{mut}}^l]$ are computed and summarized. GenAgent selects prognostic gene subsets and produces stratified reports.

• Dichotomy-Based Multi-Expert Agent Inference: For a test case, reports are generated for both modalities. Nearest neighbors are retrieved from both case banks using cosine similarity on report embeddings. The system further incorporates predictions from M pre-trained survival models, and applies progressive interval refinement (“dichotomy”) at each decision step ($y_d$), repeatedly splitting the risk group:

$$y_d = \mathcal A_{\mathrm{infer}}(\mathcal B_{\mathrm{retr}}, \mathcal R_{\mathrm{test}}^{\mathrm{WSI}}, \mathcal R_{\mathrm{test}}^{\mathrm{gene}}, \{\hat t_m\}, y_{d-1}; \text{level}=d)$$

This process ultimately delivers a calibrated survival estimate, the full multimodal reasoning trace, and citation of influential historical cases.

2. Hierarchical CoT-Enhanced Case Bank Construction

WSI CoT Case Bank

• Magnification Hierarchy: Slides are examined at 2.5× for global features, 10× for local diversity via CoSMining, and 20× for confidence-driven deep zoom.
• Patch Selection: Out of N candidates, selection via dual-thresholding ($\tau_v, \tau_t$) on self-patch matrices ensures that only representative, unique, and diagnostically diverse regions contribute.
• Report Aggregation: Structured global and local reports are compiled, deduplicated, and then passed to the reasoning agent. The system generates stepwise CoT explanations and refines them via a Qwen2.5-32B validator when quality is insufficient.

Gene-Stratified CoT Case Bank

• Functional Grouping: Six gene categories are handled independently, with feature selection based on mean, median, and mutation statistics.
• Report Generation: GenAgent processes each group to extract key prognostic markers and produces summary reports.
• Self-Critique Loop: Analogous to the WSI pipeline, initial CoT explanations are iteratively reviewed and refined for validity before banking.

Both banks store tuples: summarized report, refined CoT reasoning trace, and survival ground truth.

3. Dichotomy-Based Multi-Expert Agent Inference

• Test Case Processing: For each new patient, both WSI and gene analyses are carried out as above to yield corresponding reports.
• Retrieval-Augmented Generation (RAG): Cosine similarity on report embeddings identifies K analogous cases from both WSI and gene banks.
• Ensemble Prediction Integration: M expert survival models produce preliminary predictions ($\{\hat t_m\}$).
• Hierarchical Dichotomy Reasoning: At each dichotomy level, $y_1$ (early vs. late survival) is refined by $y_2$ and beyond, mirroring clinical risk group stratification. The final prediction $\hat t_{\mathrm{final}}$ is determined by progressively narrowing the interval.
• Full Trace Output: The system returns the predicted survival time, full multimodal reports, and explicit CoT documentation for transparency.

4. Experimental Evaluation and Results

Extensive empirical validation was performed on five TCGA cancer cohorts (BLCA, BRCA, GBMLGG, LUAD, UCEC).

• Metrics: Concordance index (C-index) and risk separation via Kaplan–Meier analysis (log-rank $p$-value).
• Comparative Performance: SurvAgent outperforms both conventional survival predictors (SNN, MaxMIL, M3IF, MOTCat, MCAT, CCL), proprietary MLLMs (Gemini-2.5-Pro, Claude-4.5, GPT-5), and agent-based systems (MDAgent, MedAgent).

$\begin{array}{l|ccccc|c} \text{Model} & \text{BLCA} & \text{BRCA} & \text{GBMLGG} & \text{LUAD} & \text{UCEC} & \text{Overall} \ \hline \text{MOTCat*} & 0.674 & 0.684 & 0.831 & 0.674 & 0.667 & 0.706 \ \text{Gemini-2.5-Pro}& 0.572 & 0.555 & 0.551 & 0.531 & 0.498 & 0.541 \ \text{Claude-4.5} & 0.545 & 0.555 & 0.505 & 0.509 & 0.479 & 0.519 \ \text{GPT-5} & 0.576 & 0.434 & 0.493 & 0.510 & 0.495 & 0.502 \ \text{MDAgent} & 0.558 & 0.482 & 0.495 & 0.524 & 0.509 & 0.514 \ \text{MedAgent} & 0.515 & 0.510 & 0.483 & 0.485 & 0.551 & 0.509 \ \hline \textbf{SurvAgent} & \mathbf{0.683} & \mathbf{0.695} & \mathbf{0.833} & \mathbf{0.676} & \mathbf{0.676} & \mathbf{0.713} \end{array}$

• Ablation Analysis: Each subsystem (WSI bank, gene bank, dichotomy reasoning) confers substantial improvement; joint deployment produces synergistic effects with final overall C-index 0.713.

5. Interpretability and Clinical Implications

• Transparency: Every SurvAgent inference is accompanied by a clinician-style chain-of-thought reasoning trace, with explicit documentation of dichotomy steps and citation of relevant historical cases and features.
• Patient Stratification: SurvAgent consistently yields statistically significant risk separation ($p<0.05$) in Kaplan–Meier analysis—a frequently unmet requirement by other MLLMs and agent systems.
• Workflow Integration: The structured outputs facilitate tumor board review and enable oncologists to incorporate transparent AI rationale into treatment planning.

6. Limitations and Future Directions

• Resource Intensiveness: Case bank construction, especially CoT curation and validation, remains computationally and labor intensive.
• Reliance on CoT Quality: System reliability is contingent on robust self-critique; propagated errors are possible if validation fails.
• Prospective Validation: Clinical deployment studies are needed for further substantiation.
• Proposed Extensions: Automation of CoT validators, generalization to additional modalities (radiology, immunostains), integration of prospective real-world follow-up data, and lightweight inference variants for resource-constrained settings.

A plausible implication is that SurvAgent’s combination of multimodal case banking and interpretable dichotomy-based reasoning addresses longstanding demands for explainable clinical AI, but its full impact will depend on continued validation and refinement in diverse healthcare environments (Huang et al., 20 Nov 2025).