BreastStage-Bench: Multimodal Breast Cancer Benchmark

• BreastStage-Bench is a patient-level test partition of BreastStage, providing a balanced, leakage-free split for evaluating multimodal, stage-aware clinical reasoning in breast cancer care.
• It integrates data from 5 imaging modalities and 17 sub-datasets, aligning tasks with clinical workflow stages from screening and diagnosis to treatment planning.
• The benchmark employs diverse tasks—including closed-ended and open-ended VQA, captioning, and report generation—with strict patient-level splits to prevent data leakage.

Searching arXiv for the primary source paper to ground the article. {"query":"(Liu et al., 3 Jun 2026) BreastGPT BreastStage-Bench", "max_results": 5} I found the primary paper on arXiv corresponding to the provided id and title, which will be used as the sole cited source because the article must remain strictly faithful to the supplied data. BreastStage-Bench is the held-out, patient-level test partition of BreastStage, a workflow-aligned, multimodal breast cancer instruction corpus designed to evaluate stage-aware clinical reasoning across screening, diagnosis, and treatment planning. It is intended to measure multimodal, multi-stage performance in closed-ended and open-ended formats, grounded captioning, and report generation, rather than only modality-specific perception tasks. In the associated study, BreastStage contains 1.86 million instruction-following pairs curated from 17 sub-datasets across 5 imaging modalities and 136 expert-designed task templates, while BreastStage-Bench isolates a balanced, leakage-free evaluation split for comprehensive benchmarking across the breast cancer care continuum (Liu et al., 3 Jun 2026).

1. Definition and relationship to BreastStage

BreastStage reorganizes heterogeneous breast oncology data around the clinical workflow of screening, diagnosis, and treatment planning. Its scale is defined in terms of approximately 662,055 unique 2D/3D images, 606,226 records with bounding-box or mask annotations, and 1,856,068 instruction pairs, rounded to 1.86M in the paper text. BreastStage-Bench is the corresponding held-out test partition, constructed at the patient level and intended to preserve class and task distributions while preventing train–test leakage.

The benchmark’s purpose is explicitly stage-conditioned, cross-modality clinical reasoning. This distinguishes it from evaluations centered on isolated modalities or narrow task families. The organizing principle is the clinical progression of breast oncology care: screening tasks emphasize detection, risk, and triage; diagnosis tasks emphasize lesion characterization and pathology-related discrimination; treatment tasks emphasize prognosis, biomarkers, subtype, and planning. This workflow alignment is central to the benchmark’s definition.

BreastStage-Bench contains 12,182 evaluation records distributed across closed-ended VQA, open-ended VQA, lesion-grounded captioning, histopathology captioning, and report generation. Because the split is patient-level, the same image may appear across distinct tasks, but all records from a given patient remain within one split. This design supports benchmarking of task diversity without permitting patient leakage.

2. Dataset composition across modalities and clinical stages

BreastStage-Bench inherits BreastStage’s five-modality design: mammography, breast ultrasound, multiparametric MRI, chest CT, and digital pathology whole-slide images. These modalities are mapped onto the clinical stages in a stage-aware manner rather than treated as interchangeable inputs.

Modality	Data scale in BreastStage	Stage association
Mammography (2D)	592,470 images; 454,590 instruction pairs	Screening and diagnosis
Breast Ultrasound (BUS, 2D)	10,405 images; 190,730 instruction pairs	Screening and diagnosis
MRI (3D, multiparametric)	36,124 volumes; 926,634 instruction pairs	Diagnosis and treatment
CT (3D chest, non-contrast)	20,546 volumes; 254,041 instruction pairs	Opportunistic screening- and diagnosis-relevant tasks
Digital pathology WSI	2,510 slides; 30,073 instruction pairs	Treatment-stage tasks

Mammography data were curated from 11 public sets via MammoVQA: BMCD, CBIS-DDSM, CDD-CESM, CSAW-M, DMID, EMBED, INbreast, KAU-BCMD, MIAS, RSNA, and VinDr-Mammo. BUS data were sourced from BUS-CoT, described as chain-of-thought reasoning ultrasound covering 99 WHO histopathology categories with expert masks. MRI data came from an institutional multiparametric cohort acquired under IRB, with expert lesion masks by 10 board-certified breast specialists. CT data were sourced from CT-RATE after filtering to female patients and breast-inclusive field of view via QC. WSI data were assembled from BCNB, TCGA-BRCA, and TCGA-HISTAI.

The stage allocation is also quantified in instruction-pair counts. Screening accounts for 1,075,092 pairs and includes tasks such as modality, view, and laterality recognition; ACR density, fibroglandular tissue, and background parenchymal enhancement assessment; lesion presence and morphology; and risk or triage decisions. Diagnosis accounts for 680,409 pairs and includes BI-RADS and pathology classification, lesion signal and kinetics, and associated findings such as lymph nodes and invasion. Treatment accounts for 100,567 pairs and includes prognosis, surgical planning, urgency, systemic therapy and biomarkers, molecular subtype, and histological grading.

Representative task templates illustrate the benchmark’s stage semantics. Screening examples include Density_Risk, abnormal_presence, mass/non_mass enhancement descriptors, and follow_up_recommended. Diagnosis examples include BI-RADS, pathology main_type, mass.morphology (margin/shape), internal_enhancement pattern, and lymph_nodes involvement. Treatment examples include surgical_plan, treatment.urgency, biomarkers/ER/PR/HER2/Ki67, molecular_subtype, and histological grading.

3. Construction, preprocessing, and held-out split design

BreastStage-Bench is constructed by stratified sampling over a composite key consisting of modality, task type, and pathology label. The stated purpose of this procedure is to preserve class and task distributions while preventing leakage. The split is strict at the patient level.

Several modality-specific preprocessing pipelines are part of the benchmark’s provenance. For CT, 25,692 volumes were initially available from CT-RATE, then filtered to female patients and breast-inclusive field of view by QC, yielding 20,546 retained volumes; automated lesion segmentation via DRT-M3D was used to identify suspicious structures for screening tasks. For BUS, 11,439 images with 10,019 lesions across 4,838 patients and 18 devices were available in BUS-CoT, and 10,405 images were retained after QC. For mammography, patient-level stratification from the MammoVQA-derived union of 11 datasets was preserved while adding stage-aware task templates beyond MammoVQA’s 9 core templates. For MRI, de-identification was performed at the DICOM header level by removing PHI, images were reconstructed in breast-coil field of view, and structured reporting ensured no PHI in released instructions. For WSI, tiles were extracted at $20\times$ with $512\times512$ pixels, and patch features were produced via frozen CONCH v1.5.

Annotation sources also vary by modality. BUS masks come from BUS-CoT, mammography bounding boxes from EMBED were reformatted, CT tumor masks were produced by DRT-M3D, MRI masks were manually delineated by experts on T1 and T1dyn, and WSI tasks rely on subtype labels including ER, PR, HER2, Ki-67, and Nottingham grade.

Quality control combines automated and expert review. Modality-specific visual QC is performed by Qwen2.5-VL-72B agents that emit {validity, reason}; low-quality samples are reviewed by a breast specialist rather than being dropped. Heuristic filters remove hallucinated terms, malformed boxes, instruction-answer conflicts, and near-duplicates defined by MinHash similarity $> 0.85$. Three breast specialists independently audit the held-out test partition using 5 random samples per task per specialist, and non-trivial flag rates trigger prompt revision and regeneration. Reported post-revision approval rates exceed 95% across dimensions, with Fleiss’s $\kappa$ in $[0.74, 0.86]$; example values include Task validity 98.2% $(\kappa = 0.78)$, Answer correctness 97.4% $(\kappa = 0.79)$, and Clinical consistency 99.8% $(\kappa = 0.86)$. Twenty-two templates were re-generated after the audit loop (Liu et al., 3 Jun 2026).

4. Task families, formats, and evaluation metrics

The benchmark contains five evaluation families with explicitly enumerated record counts.

Task family	Count	Notes
Closed-ended VQA	5,369	BUS 987, CT 825, Mammography 1,828, MRI 908, Histopathology 821
Open-ended VQA	2,833	BUS 987, CT 825, MRI 908, Histopathology 113; mammography has no open-ended set
Ground caption	2,910	BUS 1,000; CT 510; Mammography 1,000; MRI 400
Histopathology captioning	70	Separate captioning set
Report generation	1,000	Multiparametric MRI reports

Closed-ended tasks use multiple-choice or enumerated outputs drawn strictly from schemas designed by breast specialists, including BI-RADS, density categories, lesion morphology enums, and biomarker status. Open-ended tasks are free-text outputs rewritten into fluent clinical language without adding facts beyond the schema. Grounded captioning requires lesion-level localization with aligned narrative description; 2D coordinates are used for BUS and mammography, while 3D coordinates are used for CT and MRI. Report generation follows ACR-aligned sections—Findings, Impression, Final Assessment, and Management—and forbids fabricated measurements while requiring probabilistic phrasing such as “findings suspicious for…”.

The evaluation metrics are explicitly defined. Closed-ended VQA uses accuracy:

$$\mathrm{Accuracy} = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}[y_i=\hat{y}_i].$$

Option extraction is robust to formats such as “A/B/C/D” and “Option A”, and invalid or non-committal responses are counted as incorrect.

Grounding uses IoU:

$$\mathrm{IoU} = \frac{\mathrm{Area}(\mathrm{pred}\cap \mathrm{gt})}{\mathrm{Area}(\mathrm{pred}\cup \mathrm{gt})},$$

averaged over items. For samples with no ground-truth box, the scoring rule assigns IoU $512\times512$0 when the model abstains and IoU $512\times512$1 when the model hallucinates a box.

Open-ended VQA and generation use raw metrics BERTScore F1, BLEU-4, and ROUGE-1, together with the weighted composite:

$512\times512$2

The normalized score per task cell is

$512\times512$3

where $512\times512$4 is the per-item weighted composite in $512\times512$5. Although the evaluator prioritizes clinical correctness and uses a few-shot rubric with examples at $512\times512$6, $512\times512$7, $512\times512$8, $512\times512$9, and $> 0.85$0, the final reported numbers are the automatic metrics above. Macro-F1 is not reported; the paper gives the standard definition only as a possible auxiliary metric and states that it is not used (Liu et al., 3 Jun 2026).

5. Benchmark protocol and methodological context

Inference on BreastStage-Bench is stage-aware by construction. System prompts assign a role corresponding to the clinical stage, such as screening radiologist, diagnostic radiologist, or treatment oncologist/pathologist, and then combine that role with task-specific instructions for closed_vqa, open_vqa, caption, groundcaption, or report. This protocol makes the benchmark sensitive not only to visual understanding but also to clinical-role conditioning.

Baseline inference settings are standardized. Proprietary models are evaluated through official APIs with temperature = 0 and max_tokens = 512. Open-source models use greedy decoding via Transformers, images are preprocessed by each model’s processor, and multi-image inputs are supplied in the same order across models. All baselines are evaluated zero-shot on the same instruction format, with no external retrieval or tools.

The benchmark also imposes modality-specific visual handling constraints. Radiology images are processed with a standard ViT branch using IMAGE_MAX_TOKEN_NUM=1024. Histopathology WSIs are represented by $> 0.85$1 tiles of $> 0.85$2 pixels, with features from frozen CONCH v1.5 $> 0.85$3-D), global context via LongNet with dilated attention, and concept-preserving token selection under budget $> 0.85$4. CT and MRI volumes are capped at $> 0.85$5 during inference. Ground captioning enforces modality-dependent coordinate formats: models emitting 2D boxes for 3D cases are scored as invalid, yielding IoU $> 0.85$6 on ground-truth-positive volumes. Report generation forbids mentions of internal coordinates.

The study situates BreastStage-Bench within the evaluation of BreastGPT, a unified multimodal LLM with a dual-branch visual encoder and concept-preserving token compression. The standard branch serves radiology and is aligned to Qwen3-VL, while the gigapixel branch for WSI uses frozen CONCH v1.5 patch features, trainable LongNet, and a universal token selector. Token selection operates on visual tokens $> 0.85$7 and chooses exactly $> 0.85$8 tokens by maximizing a dual coverage objective that combines text–vision coverage and vision–vision coverage:

$> 0.85$9

The paper further defines calibrated similarity matrices

$\kappa$0

normalized by softmax with temperatures $\kappa$1 and $\kappa$2, and a dual coverage function

$\kappa$3

Greedy selection is stated to achieve a $\kappa$4 approximation for the monotone submodular objective and is applied identically during training and inference. The unified budget $\kappa$5 is reported to lie on the saturation plateau for VQA and generation, retain approximately 99% of grounded-caption IoU achievable at $\kappa$6, reduce prefill latency to approximately 191.4 ms with selector overhead of approximately 9.3 ms, and keep memory within approximately 17 GB; by contrast, directly feeding all 5,987 patch tokens requires approximately 6.5 s, about $\kappa$7 slower (Liu et al., 3 Jun 2026).

6. Reported results, limitations, and practical access

BreastStage-Bench is evaluated against proprietary models, open-source general VLMs, and medical-specific VLMs. The proprietary set includes GPT-5.4, Claude-opus-4-6, Claude-sonnet-4-6, Gemini-3.1-Flash, and Gemini-3.1-Pro. Open-source general VLMs include Qwen2.5-VL-Instruct (3B/7B), Qwen3-VL-Instruct (4B/8B), MiMo-VL-SFT (7B), and InternVL3.5 (8B). Medical-specific VLMs include Lingshu (7B) and HuatuoGPT-V (7B). Additional baselines reported in the supplement include Grok-4.1-Fast, Gemma-4 (8B), GLM-4.6V-Flash (9B), LLaVA-OneVision-1.5 (8B), HealthGPT (32B), Hulu-Med (7B), MedDr (40B), and RadFM (14B). BreastGPT itself is reported in two variants, “cluster” and “learn”, both built on a Qwen3-VL-8B backbone.

The key benchmark results are given for BreastGPT (cluster). On closed-ended VQA, it achieves 75.66% average accuracy, compared with 54.00% for GPT-5.4, a difference of $\kappa$8 points. Its per-modality closed-ended accuracy is 86.81 on BUS, 77.21 on CT, 75.00 on mammography, 82.86 on MRI, and 71.38 on histopathology. On open-ended VQA, it achieves a normalized score of 89.92%, compared with 53.58% for GPT-5.4, a difference of $\kappa$9 points. Raw open-ended metrics reported for BreastGPT (cluster) include, for screening: BUS 99.15/90.67/94.91, CT 99.07/88.56/94.45, MRI 99.27/87.53/95.87; for diagnosis: BUS 98.63/85.08/90.60, MRI 99.12/88.94/95.68; and for treatment: MRI 98.09/75.47/88.08, histopathology 92.69/18.88/50.96, listed in the order BERTScore/BLEU/ROUGE.

On generation tasks, BreastGPT (cluster) records a BUS caption weighted score of 79.32, compared with 51.48 for Gemini-3.1-Flash; a mammography caption weighted score of 77.64 with IoU 23.14; a CT caption weighted score of 73.16; a histopathology caption weighted score of 66.78; and an MRI report weighted score of 67.67, compared with 55.16 for GPT-5.4. For 3D grounding, only BreastGPT variants are stated to produce volumetric 6D boxes. BreastGPT (cluster) achieves IoU 5.12 on CT and 33.49 on MRI; the Qwen3-VL SFT baseline reaches CT 3.89 and MRI 11.98, while other baselines either fail or output 2D boxes that are dimensionally incompatible (Liu et al., 3 Jun 2026).

The stated limitations are important to the benchmark’s interpretation. BreastStage-Bench does not include PET/CT or emerging modalities such as photoacoustic imaging. Longitudinal continuity across the same patients is limited in training, although a subset of the benchmark includes linked workflow cases. Pathology and MRI tasks are more sensitive to token budget and to the availability of precise clinical measurements, and adjacent-class boundaries such as BI-RADS 3 versus 4A or Nottingham grade II versus III induce errors. The benchmark is presented for research comparison rather than direct clinical deployment; site-specific validation and regulatory review are required before clinical use. The paper also notes caution regarding over-generation of biomarker claims in treatment-stage texts and states that prompts were tightened to reduce unsupported statements. A plausible implication is that high aggregate benchmark performance does not eliminate clinically consequential failure modes in subtype- or treatment-related outputs.

Practical access is provided through the project website, a ModelScope dataset release for BreastStage, ModelScope weights for BreastGPT-8B, and linked code and evaluation tooling. The released materials include stage-aware prompt templates, generator prompts for open-VQA, grounded caption, and report generation, and persona assembly rules intended to replicate benchmarking. The recommended use procedure is to download the benchmark split and task definitions, load modality-specific inputs and task templates, run models with the provided stage-aware system prompts, and compute accuracy for closed-ended VQA, IoU for grounding, and weighted BERT/BLEU/ROUGE for open-ended VQA and generation under the same zero-shot, temperature 0, no-external-tools protocol (Liu et al., 3 Jun 2026).