BreastGPT: Multimodal Breast Oncology AI

Updated 6 July 2026

BreastGPT is a workflow-centric multimodal large language model for breast cancer care, integrating imaging modalities with clinical reasoning across screening, diagnosis, and treatment planning.
It employs a dual-branch visual encoder with stage-aware prompts and token compression to efficiently process mammography, ultrasound, MRI, CT, and pathology images, achieving strong benchmark accuracy.
Its design establishes a unified platform that orchestrates specialized breast imaging tasks, offering practical insights and a research foundation for advancing breast oncology AI.

BreastGPT is a domain-specialized multimodal LLM for breast cancer care that is organized around the clinical workflow of screening, diagnosis, and treatment planning rather than around a single modality or isolated task. In its formal 2026 definition, it is trained within the BreastStage ecosystem, a workflow-aligned breast imaging instruction corpus spanning mammography, breast ultrasound, MRI, CT, and histopathology whole-slide images, and it is evaluated on BreastStage-Bench, a held-out benchmark for multimodal reasoning across the breast cancer care continuum (Liu et al., 3 Jun 2026). In a broader and earlier informal sense, “BreastGPT” has also been used to denote a robust breast-imaging AI system that must cope with multi-institutional heterogeneity, noisy labels, and clinically meaningful outputs, especially in mammography and breast density assessment (Lee et al., 2019).

1. Definition and conceptual scope

BreastGPT denotes a workflow-centric breast oncology MLLM whose central claim is not merely multimodal input support, but clinically staged reasoning. The motivating observation is that breast cancer management is distributed across three distinct but linked phases. In screening, the dominant tasks are triage, lesion presence assessment, tissue characterization, and recall decisions, primarily from mammography, breast ultrasound, and opportunistically chest CT. In diagnosis, the model must perform fine-grained lesion characterization and stage-associated reasoning from diagnostic mammography, targeted ultrasound, and multiparametric MRI. In treatment planning, histopathology whole-slide images, biomarker status, and treatment-related MRI become central for subtype inference, prognosis, and management recommendations (Liu et al., 3 Jun 2026).

This workflow framing distinguishes BreastGPT from narrower breast-AI systems. Earlier breast models typically solved one task at one scale: mammographic density grading, ultrasound lesion segmentation, fibroglandular tissue segmentation on MRI, pathology representation learning, or report-oriented question answering. BreastGPT instead treats those modality-specific tasks as components of a unified clinical routine. A plausible implication is that it functions less as a single classifier than as a general-purpose breast oncology reasoning layer that must absorb heterogeneous visual evidence, preserve modality-specific semantics, and switch between screening-style, diagnostic-style, and treatment-style outputs without changing backbone architecture (Liu et al., 3 Jun 2026).

The term also has a wider conceptual history. In the context of breast density prediction, a “BreastGPT”-type model was explicitly described as a robust, multi-institutional breast imaging AI system requiring learned image-conditional normalization and label correction to handle vendor heterogeneity and inter-reader variability (Lee et al., 2019). That usage did not denote a LLM, but it anticipated the broader idea that a clinically meaningful breast AI system must combine perception, normalization, and reasoning under distribution shift.

2. BreastStage and BreastStage-Bench

BreastGPT is trained on BreastStage, a workflow-aligned instruction corpus built from approximately 662,000 unique images/volumes, 17 sub-datasets, 5 imaging modalities, and 136 task templates, yielding 1.86M instruction-following pairs. Its stage distribution is 57.9% screening, 36.7% diagnosis, and 5.4% treatment. The modality set comprises mammography, breast ultrasound, MRI, CT, and histopathology whole-slide images (Liu et al., 3 Jun 2026).

The corpus construction pipeline is explicitly schema-driven. Clinical reports are translated, converted into structured JSON records aligned with modality-specific expert schemas, and then mapped to closed-ended VQA, open-ended VQA, grounded captioning, and report-generation tasks. The defining design choice is that instructions and answers are derived from expert-designed structured reports rather than free-form synthetic paraphrase alone. Closed-ended questions use answer spaces inherited from schema enums, while open-ended answers are rewritten from the same underlying records into short natural-language statements. This reduces answer leakage and constrains supervision to clinically grounded facts (Liu et al., 3 Jun 2026).

Quality control is likewise modality-specific. “Visual specialist agents” based on Qwen2.5-VL-72B inspect each image with prompts tailored to mammography, BUS, MRI, CT, or pathology, and output {validity, reason} as JSON. Low-quality samples are reviewed by human breast specialists rather than silently discarded. Spatial annotation is heterogeneous but explicit: BUS uses expert masks, mammography uses source-dataset boxes, CT uses DRT-M3D-derived tumor segmentations converted to boxes, MRI uses manual masks from 10 breast radiologists, and WSIs are retained without pixel annotations for VQA and report tasks (Liu et al., 3 Jun 2026).

BreastStage-Bench is the held-out benchmark built from the patient-level test split. It contains approximately 12k records across closed-ended VQA, open-ended VQA, ground captions, WSI captioning, and MRI report generation. Closed-ended VQA is scored by accuracy. Open-ended answers and reports are scored with a weighted metric,

$\text{Score} = 100 \times \Big( 0.5\,\text{BERTF1} + 0.25\,\text{BLEU} + 0.25\,\text{ROUGE1} \Big),$

and grounding is scored by mean IoU, with true negatives counted as IoU $=1$ and hallucinated positive boxes on negative cases counted as IoU $=0$ (Liu et al., 3 Jun 2026).

3. Architecture and cross-scale visual modeling

BreastGPT is built on Qwen3-VL-8B-Instruct and adds three defining mechanisms: stage-aware system prompting, a dual-branch visual encoder, and concept-preserving token compression. The stage-aware prompts assign one of three clinical personas: an expert screening breast radiologist, an expert diagnostic breast radiologist, or an expert breast oncologist and pathologist. The selected persona is prepended to the task instruction, so the same backbone is behaviorally conditioned to produce screening-style, diagnostic-style, or treatment-style responses (Liu et al., 3 Jun 2026).

The dual-branch encoder addresses the scale disparity between routine radiology and gigapixel pathology. The standard branch uses the Qwen3-VL ViT encoder for mammography, BUS, MRI, and CT. The GigaPixel branch processes whole-slide images by first tiling at 20×, 512×512, and non-overlapping resolution, then passing each tile through CONCH v1.5 to obtain 512-dimensional embeddings,

$\mathbf{v}_i = \text{CONCH}(\mathbf{x}_i) \in \mathbb{R}^{512}.$

Those tile embeddings are then contextualized by a 2-layer LongNet encoder,

$\mathbf{h}_i = \text{LongNet}(\{\mathbf{v}_1,\ldots,\mathbf{v}_N\})_i,$

before projection into the LLM token space. Routing is resolution-aware: radiology images are sent to the standard branch and WSIs to the GigaPixel branch (Liu et al., 3 Jun 2026).

The token compression layer is the technical core of BreastGPT. Let $\{v_j\}_{j=1}^N$ be projected visual tokens, $\{v'_j\}_{j=1}^N$ their pre-projection counterparts, and $\{t_i\}_{i=1}^m$ the text tokens. BreastGPT constructs text–vision and vision–vision similarity matrices,

$M^{tv}_{i,j} = t_i^\top v_j, \qquad M^{vv}_{i,j} = {v'}_i^{\!\top} v'_j,$

followed by temperature-scaled softmax normalization,

$\widetilde{M}^{tv}_{i,j} = \frac{\exp(M^{tv}_{i,j}/\tau_t)} {\sum_{j'} \exp(M^{tv}_{i,j'}/\tau_t)}, \qquad \widetilde{M}^{vv}_{i,j} = \frac{\exp(M^{vv}_{i,j}/\tau_v)} {\sum_{j'} \exp(M^{vv}_{i,j'}/\tau_v)}.$

For a selected subset $=1$ 0 of size $=1$ 1, it maximizes the submodular coverage objective

$=1$ 2

where the first term enforces text–vision coverage and the second term enforces global visual coverage. A greedy selector yields a $=1$ 3-approximation in $=1$ 4. BreastGPT uses $=1$ 5 visual tokens, which empirically saturates performance while controlling compute (Liu et al., 3 Jun 2026).

Training proceeds in two stages. In Stage 1, the LLM is frozen and the visual front end is warmed up on closed-ended VQA plus histopathology captioning/VQA; CONCH remains frozen. In Stage 2, the LLM and visual branches are unfrozen and jointly instruction-tuned on all four task types. Training uses AdamW, $=1$ 6 learning rate, cosine schedule, 3% warm-up, 1 epoch for warm-up and 2 epochs for end-to-end tuning, on 32 H100 GPUs with bfloat16, DeepSpeed ZeRO-2, and FlashAttention, for approximately 2,530 GPU-hours (Liu et al., 3 Jun 2026).

4. Benchmark performance and empirical profile

On BreastStage-Bench, BreastGPT is reported in two variants: BreastGPT (cluster), which uses the greedy coverage selector, and BreastGPT (learn), which replaces it with a learnable cross-attention resampler. The cluster variant is the stronger system. It achieves 75.66% closed-ended VQA accuracy and 89.92% open-ended score, outperforming both general-purpose and medical-specific multimodal baselines across stages and task formats (Liu et al., 3 Jun 2026).

Before the table, two empirical patterns are central. First, BreastGPT’s gains are largest in modalities where domain-specific structure and scale are hardest for generic VLMs, notably MRI and histopathology. Second, the cluster selector outperforms the learn selector, which suggests that explicit coverage over text-conditioned and globally representative visual tokens is more effective than a purely learnable resampler under the same backbone and data. A plausible implication is that token budgeting, not only encoder capacity, becomes a primary bottleneck once gigapixel pathology and volumetric imaging are brought under one language interface (Liu et al., 3 Jun 2026).

Setting	BreastGPT	Comparator
Closed-ended VQA overall	75.66%	Qwen3-VL-8B SFT 68.21%
Open-ended score overall	89.92%	Qwen3-VL-8B SFT 88.24%
BUS closed VQA	86.81%	GPT-5.4 64.89%
Mammography closed VQA	75.00%	GPT-5.4 68.51%
MRI closed VQA	82.86%	GPT-5.4 41.43%
Histopathology closed VQA	71.38%	GPT-5.4 32.28%

The same paper reports CT closed VQA of 77.21% for BreastGPT versus 78.55% for GPT-5.4, indicating that CT is the least differentiated modality in the benchmark. On text generation tasks, BreastGPT reaches 79.32 on BUS captioning, 73.16 on CT captioning, approximately 77.6 on mammography captioning, 66.78 on histopathology captioning, and 67.67 on MRI report generation, with the MRI report score exceeding 55.16 for GPT-5.4 (Liu et al., 3 Jun 2026).

The grounding results are particularly diagnostic of architectural intent. Generic multimodal baselines largely fail on 3D grounding because they emit 2D boxes rather than volumetric coordinates. BreastGPT reports 5.12% IoU on CT and 33.49% IoU on MRI for 3D grounding, which is modest in absolute terms but nontrivial given the volumetric target format. In histopathology-specific ablations, replacing the GigaPixel branch with naive random sampling of 32 tiles reduces closed VQA from 71.38% to 60.41%, confirming that whole-slide representation, not only pathology-aware pretraining, is essential (Liu et al., 3 Jun 2026).

Efficiency ablations show why token compression is treated as a first-class design requirement. On a WSI with approximately 5,987 tiles, selection at $=1$ 7 costs about 9.3 ms, LLM prefill about 191 ms, and total latency about 200 ms, with peak memory around 17 GB. Without selection, feeding all 5,144 visual tokens into the LLM yields about 6.5 s total latency, roughly 33× slower, and about 18 GB memory. These numbers make clear that BreastGPT’s pathology capability is conditioned as much by token routing as by foundation-model scale (Liu et al., 3 Jun 2026).

BreastGPT emerges from a preexisting technical ecology in breast imaging and pathology rather than from language modeling alone. In mammography, robust multi-site density prediction has already been framed around two problems that any broader breast AI must solve: image-conditional photometric normalization and label correction under inter-reader variability. The photometric transformer network and label distillation framework for breast density prediction showed that adaptive intensity remapping and iterative pseudo-label adjustment materially improve cross-site density assessment, with PTN plus label distillation reaching 0.7941 test accuracy and 0.9663 dAUC on the in-house test set, compared with 0.7509 and 0.9518 for PTN alone (Lee et al., 2019).

A separate line of work treated ChatGPT 3.5 turbo as an Expert System Shell for breast cancer self-screening rules derived from the American Cancer Society, introducing “reinforcement explainability” through prompt-level enforcement of rule tracing. On 50 structured and 50 unstructured synthetic use cases, the system achieved 47/50 correct for structured and 41/50 for unstructured inputs, illustrating an early attempt to make breast-cancer reasoning both rule-grounded and user-facing (Khan et al., 2024). This suggests that part of the BreastGPT agenda is not only perception across modalities but also controlled explanation.

In breast pathology, the organ-specific foundation model BRIGHT provides a close analogue to BreastGPT’s specialization strategy. BRIGHT is trained on approximately 210 million histopathology tiles from 51,836 H&E WSIs, derived from 40,203 patients across 19 independent clinical centers, and evaluated on 24 downstream breast pathology tasks. It outperforms generalist pathology foundation models on most internal tasks and several external tasks, demonstrating that organ-specific adaptation on top of generalist pretraining can be clinically advantageous in breast oncology (Guo et al., 3 Mar 2026). BreastGPT’s dedicated GigaPixel branch can be read as an imaging-language parallel to that organ-specific pathology turn.

At the modality level, several specialized breast-vision modules define the kinds of subsystems a generalized BreastGPT could absorb. For breast MRI, the transformer-based FGT segmentor TraBS improved external Dice from 0.824 ± 0.144 with nnUNet to 0.864 ± 0.081, supporting more reliable MRI-based density and BPE quantification (Müller-Franzes et al., 2023). For ultrasound, the Global Guidance Network improved lesion segmentation by adding long-range spatial and channel dependencies plus explicit boundary supervision (Xue et al., 2021), and a later long-tailed ultrasound subtype framework combined class-controllable diffusion augmentation, sketch-grounded perception, and RL-driven adaptive sampling to reach 35.23 macro F1 and 31.25 few-shot accuracy on an 8-class in-house long-tailed dataset (Chen et al., 30 Jul 2025). For mammography quality control, a CoordAtt UNet for MLO positioning assessment reached 88.63% ± 2.84 accuracy and 90.25% ± 4.04 specificity while localizing the nipple and pectoralis landmarks needed to draw the posterior nipple line (Tanyel et al., 2024).

These antecedents support a specific interpretation of BreastGPT. It is not a replacement for high-performing specialty modules; rather, it is a unifying multimodal reasoning layer that can potentially orchestrate modules for normalization, segmentation, quality assurance, pathology representation, and structured explanation. A plausible implication is that future versions may become progressively more modular internally even if they remain unified at the interface level.

6. Limitations, safety, and future directions

BreastGPT is explicitly presented as a research foundation model, not a clinically validated diagnostic device. The corpus has several acknowledged limits: MRI data are drawn from two institutions, WSI data from specific public cohorts, true longitudinal screening-to-diagnosis-to-treatment patient pathways are largely absent from training, and the modality set does not include PET/CT, PET/MR, or richer non-imaging clinical streams such as genomics, laboratory data, and medication history (Liu et al., 3 Jun 2026). Label noise also remains possible despite specialist auditing.

The system is therefore best understood as a benchmarked research platform for organ-specific medical MLLMs. Its current strengths lie in workflow breadth, cross-scale visual routing, and schema-grounded supervision, but those do not by themselves establish safe deployment. Any clinical use would require site-specific validation, regulatory review, and drift monitoring. The paper’s own framing is cautious: institutional raw DICOM volumes are not publicly redistributed, and the released artifacts are intended to support reproducible research rather than direct clinical integration (Liu et al., 3 Jun 2026).

External evidence from mammogram VQA reinforces that caution. In a separate zero-shot study of GPT-family models on four public mammography datasets, GPT-5 was the best general-purpose model tested, but still lagged behind both human experts and domain-specific fine-tuned systems. On CBIS-DDSM malignancy classification, GPT-5 reached 58.2% accuracy, 63.5% sensitivity, and 52.3% specificity, compared with human expert estimates of 86.9% sensitivity and 88.9% specificity (Li et al., 15 Aug 2025). This does not measure BreastGPT directly, but it clarifies the broader problem setting: general multimodal scaling alone is insufficient for high-stakes breast imaging, and specialized architecture plus workflow-aligned data are necessary but not obviously sufficient.

The future directions stated for BreastGPT follow directly from these constraints. They include building true longitudinal, per-patient cross-stage datasets, extending the approach to other organ systems, integrating richer EHR data, and adding more robust uncertainty estimation, calibrated abstention, and safety layers (Liu et al., 3 Jun 2026). The broader breast-AI literature points to further likely extensions: organ-specific pathology specialization (Guo et al., 3 Mar 2026), explicit rule-grounded explanation (Khan et al., 2024), robust multi-site mammography preprocessing and noisy-label handling (Lee et al., 2019), and specialized modules for MRI, ultrasound, pathology, and acquisition quality control (Müller-Franzes et al., 2023). Taken together, these directions suggest that the long-term significance of BreastGPT lies less in being a single monolithic model than in establishing a workflow-aligned template for breast oncology AI across modalities, scales, and clinical stages.