---
title: 'MTBBench: Molecular Tumor Board Benchmark'
url: https://www.emergentmind.com/topics/mtbbench
type: topic
---

# MTBBench: Molecular Tumor Board Benchmark

MTBBench (Molecular Tumor Board Benchmark) is a clinically grounded, agentic evaluation suite designed to assess how well large language and vision-language models can simulate the multidisciplinary decision-making workflow of a molecular tumor board (MTB) in oncology. Unlike earlier biomedical benchmarks that focus on static, unimodal question-answering, MTBBench embeds models within a multi-turn "doctor-agent" protocol that requires active data retrieval, temporally unfolding patient histories, and complex reasoning over heterogeneous, multimodal inputs including digital pathology, lab results, genomics, and longitudinal clinical timelines [2511.20490].

## 1. Benchmark Architecture and Modalities

MTBBench is organized to emulate MTB workflows where diagnostic, prognostic, and therapeutic decisions arise from longitudinal, multimodal information integration. The data sources cover:

- **Digital histopathology:** Hematoxylin & eosin (H&E) and immunohistochemistry (IHC) whole-slide images annotated for regions of interest.
- **Laboratory Data:** Preoperative hematology (CRP, MPV, leukocyte differentials, creatinine), structured in CSV with reference ranges.
- **Genomics & Pathology Reports:** Somatic mutation calls, copy-number alterations, and pathology narratives.
- **Clinical Timelines:** Structured event sequences logging diagnoses, procedures, treatments, and outcomes.

The case corpus consists of two tracks:
- **MTBBench-Multimodal:** 26 head-and-neck cancer patients (from the Hancock dataset), each with ≈40 modality-specific files and ≈15 QA pairs per patient (390 total).
- **MTBBench-Longitudinal:** 40 "MSK" clinicogenomic cohort patients, each with ≈5 structured files and ≈4.6 QA pairs per patient (183 total).

## 2. Agentic Workflow and Interaction Protocol

In the agentic setting, the benchmark frames model evaluation as interactive, multi-turn dialogues. At each round $t$, the agent receives a question $q_t$ and a list of available files $\mathcal{F}_t$; it may request a subset $\mathcal{R}_t \subseteq \mathcal{F}_t$, retrieve the corresponding files (images, tables, reports), update its internal state, and respond. Files are not persistent across turns, simulating the episodic nature of EMR access; they may be re-requested if needed for future inference. For the longitudinal track, an evolving timeline $\mathcal{T}_t$ accumulates all previous events, forming the substrate for temporal reasoning.

Canonical clinical task categories include:

- **Diagnostic Classification (Digital Pathology)**
- **Spatial Biomarker Interpretation (IHC)**
- **Preoperative Hematology Reasoning**
- **Prognostic Deliberation (Outcome & Recurrence)**
- **Longitudinal Outcome & Progression Forecasting**

Illustrative interactions require the agent to identify tumor subtypes based on H&E, interpret IHC spatial patterns, estimate bleeding risk from lab values, predict survival, and infer progression risk after therapy and new molecular findings.

## 3. Ground Truth Annotation and Validation

All QA pairs within MTBBench were co-developed and validated with domain clinicians using a Streamlit-based companion application. Clinicians could:

- Explore patient demographics, grouped images, lab data, and genomics.
- Review each multiple-choice or true/false question with options to approve, reword, or flag as inappropriate.

For inter-rater reliability, ten external oncology experts reviewed overlapping question subsets (45 QA pairs), establishing high validation rates: mean pairwise accuracy 0.94, Cohen's $\kappa$ 0.81, Fleiss' $\kappa$ 0.79, PABAK 0.91, Gwet's AC1 0.91, Krippendorff's $\alpha$ 0.79.

## 4. Evaluation Metrics, Model Baselines, and Tool-Augmentation

Performance is measured using accuracy:
\[
\mathrm{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\{\hat y_i=y_i\}
\]
and, when relevant, binary F1-score:
\[
\mathrm{F}_1 = 2 \cdot \frac{\mathrm{Precision}\,\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}
\]
and percentage improvement for tool-augmented versus baseline conditions:
\[
\Delta\% = \frac{\mathrm{Score}_\text{tool} - \mathrm{Score}_\text{baseline}}{\mathrm{Score}_\text{baseline}}\times 100\%
\]
File-access count per question is also reported, serving as a proxy for information-seeking depth, with higher counts correlating with more comprehensive multimodal reasoning.

Ten open- and closed-source large models (e.g., Gemma-12, GPT-4o, InternVL-78, Qwen2.5-VL-7, Llama-70) were benchmarked. Three prominent findings:
- Tool augmentation consistently increased accuracy: e.g., Gemma-12 improved from 61.5% (baseline) to 70.5% (+14.6%), GPT-4o from 66.7% to 72.9% (+9.3%) on the multimodal track.
- Longitudinal reasoning also benefited: Qwen2-VL-32 rose from 67.0% to 74.2% (+10.7%) on longitudinal tasks.
- Maximum task-level improvements reached 9.0% (digital pathology) and 11.2% (progression forecasting).

### Comparative Performance (selected models)

| Model         | Multimodal Baseline | Multimodal +Tools | Longitudinal Baseline | Longitudinal +Tools |
|---------------|--------------------|-------------------|----------------------|---------------------|
| Gemma-12      | 61.5%              | 70.5%             | 58.0%                | 63.5%               |
| GPT-4o        | 66.7%              | 72.9%             | 64.2%                | 67.8%               |
| Qwen2-VL-32   | –                  | –                 | 67.0%                | 74.2%               |

## 5. Foundation Model-Based Tool Stack

MTBBench integrates domain-specific foundation model-based tools as agent-callable functions for in-context expert reasoning:

- **Digital Pathology (CONCH):** Dual-encoder model trained on 1.17 M H&E image–caption pairs for image-to-label matching via dot-product in a joint embedding space.
- **Immunohistochemistry Quantification (UniV2 + ABMIL):** 1,536-dimensional patch embeddings generated via UNI2; aggregated by an attention-based multiple instance learning (ABMIL) head with 5-layer FC regressor, trained for 70 epochs using Adam on NVIDIA A100 hardware.
- **Knowledge Bases:** PubMed tool (retrieves top abstracts using BAAI-bge reranker) and DrugBank tool (clinical drug metadata mapping).

Seamless agent-tool integration enables on-demand access to digital pathology quantification, biomarker detection, and biomedical evidence retrieval, mirroring the MDT's (multidisciplinary team) use of specialized resources during MTB meetings.

## 6. Core Findings and Model Insights

Analysis revealed several critical phenomena:

- **Hallucinations and Information-Seeking Failures:** Baseline agents frequently fabricated file names, failed to retrieve novel critical files, or reused stale context—most commonly in outcome/recurrence prediction where performance hovered near random (≈50% accuracy).
- **Temporal Reasoning Deficiencies:** Models detected coarse survival signals but lacked fine-grained discrimination for progression-vs-recurrence without accurate re-access of evolving event timelines.
- **Multimodal Integration:** The number of files accessed (not model size) most strongly correlated with performance ($r\approx0.65$), underscoring that successful models must combine retrieval and integration of diverse modalities to achieve clinical accuracy.

Tool-augmented models demonstrated greater robustness to these challenges, though limitations persisted regarding deep longitudinal reasoning.

## 7. Limitations and Future Directions

Despite its rigor, MTBBench has several current limitations:

- **Offline Simulation:** Evaluation remains decoupled from live clinical settings; real-time patient-physician-agent dialogues are not yet simulated.
- **Dataset Scale:** The number of cases (66 patients) restricts statistical power for rare cancer subtypes and event types.
- **Tool Coverage:** No specialized agentic foundation model exists for temporal reasoning; longitudinal improvements stemmed primarily from general-purpose literature/drug lookups.
- **Clarification and Ambiguity:** Tasks assume complete, non-ambiguous data; interactive clarification protocols are planned for future benchmarks.

Planned expansions include broader cancer/organ coverage, support for additional imaging (e.g., radiology), real-time clinical trial database integration, and piloting MTBBench in simulated clinician–AI collaboration studies to assess usability and trust.

---

MTBBench establishes a rigorous, reproducible framework for measuring, analyzing, and improving the clinical reasoning abilities of AI agents in precision oncology, moving beyond traditional static QA to emulate and advance dynamic, tool-enabled workflows characteristic of real molecular tumor boards [2511.20490].

Source: https://www.emergentmind.com/topics/mtbbench