BreastGPT: A Multimodal Large Language Model for the Full Spectrum of Breast Cancer Clinical Routine

Abstract: Breast cancer remains a leading cause of cancer-related mortality among women. Its clinical management requires multimodal reasoning across a clinical workflow that spans \textit{screening}, \textit{diagnosis} and \textit{treatment planning}, where each stage involves distinct imaging modalities, task objectives, and reasoning patterns. However, constrained by data scarcity and model versatility, existing medical MLLMs are typically evaluated on isolated modalities or narrow task families, limiting their ability to support workflow-level clinical reasoning. In this work, we first introduce \textbf{BreastStage}, a workflow-aligned breast imaging instruction corpus comprising 1.86M instruction-following pairs curated from 17 sub-datasets across 5 imaging modalities and 136 task templates. Its held-out split, \textbf{BreastStage-Bench}, provides a comprehensive benchmark for evaluating multimodal reasoning across the breast cancer care continuum. Building on this corpus, we propose \textbf{BreastGPT}, a unified MLLM equipped with a dual-branch visual encoder and concept-preserving token compression to bridge the scale gap between standard radiology and gigapixel pathology. On BreastStage-Bench, BreastGPT achieves 75.66\% closed-ended accuracy and 89.92\% open-ended score, outperforming both general-purpose and medical-specific MLLMs across clinical stages and task formats. These results suggest that workflow-aligned data and cross-scale visual modeling are critical for clinically grounded medical MLLMs. All data, code, and model checkpoints are released at https://yangyy-liu.github.io/BreastGPT.io.

• The paper presents BreastGPT, a unified multimodal model that bridges breast cancer screening, diagnosis, and pathology using cross-scale visual reasoning.
• It employs modality-conditioned prompting with dual-branch visual encoding and a resolution-aware gating module to effectively process diverse imaging modalities.
• Efficient token compression via greedy submodular optimization achieves near-optimal accuracy with significant reductions in inference time for clinical use.

Unified Multimodal Clinical Reasoning for Breast Cancer: The BreastGPT Framework

Clinical Workflow Alignment and Dataset Construction

The management of breast cancer inherently demands reasoning across heterogeneous phases: screening, diagnosis, and treatment planning, each characterized by distinctive imaging modalities and reasoning patterns. Conventional medical MLLMs and breast imaging models have largely focused on individual modalities or isolated stage-specific task families. This fragmentation has led to models that fail to generalize effectively across the clinical workflow, limiting their practical applicability.

The BreastStage dataset addresses this gap by curating 1.86M instruction-following pairs across 17 sub-datasets, covering five major modalities (mammography, BUS, MRI, WSI, CT) and 136 stage-aligned task templates. The data construction pipeline features multi-stage quality control, expert-driven annotation of imaging and clinical attributes, and rigorous stage-specific QA pair synthesis, ensuring task validity, answer correctness, and clinical consistency. The stratified train-test split preserves modality, task, and label distributions, while enforcing patient-level separation to prevent leakage.

Figure 1: Overview of BreastStage, aligning imaging and instruction data with the breast cancer screening-diagnosis-treatment workflow.

Figure 2: Pipeline for BreastStage: modality curation, annotation, LLM-driven instruction generation, and expert audit.

Model Architecture: Modality and Resolution-Aware Visual Language Modeling

BreastGPT operationalizes workflow-level clinical reasoning by integrating modality-conditioned role prompting with a dual-branch visual encoder. The architecture leverages a resolution-aware gating module to route standard radiology (BUS, mammography, MRI, CT) to a ViT-based branch, while gigapixel WSIs are processed by a dedicated GigaPixel branch. WSI patch features are extracted via a frozen CONCHv1.5 encoder, globally contextualized using LongNet’s dilated attention, and compressed via a concept-based token selector grounded in a submodular coverage maximization objective.

Figure 3: BreastGPT architecture with dual-branch visual encoding and universal concept-based token compression; performance comparison against SOTA medical and general-purpose MLLMs.

This unified interface bridges the scale gap between radiology and pathology, enabling BreastGPT to process multi-image, multi-resolution, and multi-task clinical inputs without the need for task-specific architectural heads or modality-specific pooling.

Efficient Visual Token Compression and Scaling

The universal concept-based token selector distills high-dimensional visual inputs—tens of thousands of WSI tokens or hundreds of ViT tokens—into a fixed token budget ($k=128$) by simultaneously maximizing prompt-relevant (text-vision) and globally diverse (vision-vision) coverage. Greedy submodular optimization guarantees near-optimal representativeness and query alignment. Empirical sweep on BreastStage-Bench demonstrates performance saturation as early as $k=64$–$128$ across closed/open VQA, captioning, grounding, and report generation tasks, with minimal gains beyond this threshold.

Figure 4: Visual token budget sweep: accuracy and IoU saturate at $k=128$ across modalities and tasks.

Latency profiling confirms a substantial computational advantage: inference on a WSI (5,987 tokens) with $k=128$ incurs only 200ms GPU time versus 6.5s without compression, with negligible accuracy tradeoff.

Workflow-Level Evaluation and Comparative Benchmarking

BreastGPT establishes state-of-the-art performance across the held-out BreastStage-Bench, which evaluates cross-stage clinical workflow reasoning. On closed-ended VQA, BreastGPT achieves 75.66% average accuracy, exceeding the strongest commercial baseline (GPT-5.4) by 21.66 points. Open-ended VQA scores reach 89.92%, outperforming all general-purpose and medical-specific MLLMs by at least 25 points. Captioning, grounding, and report generation tasks show similarly pronounced gains, with BreastGPT producing volumetric bounding boxes for CT/MRI and comprehensive multiparametric MRI/histopathology reports.

Qualitative case studies further illustrate robust handling of categorical classification (BI-RADS), quantitative reasoning, spatial grounding, and clinical report synthesis across all modalities and stages.

Figure 5: Distribution of instruction pairs across stage, modality, and task taxonomy; highlighting workflow coverage.

Figure 6: Case 1—closed-ended vascularity attribute extraction for BUS.

Figure 7: Case 2—open-ended Doppler description reasoning for BUS.

Figure 8: Case 3—lesion-grounded captioning for BUS expert-cropped frames.

Figure 9: Case 4—multi-view BI-RADS classification for mammography.

Figure 10: Case 5—comprehensive multiparametric MRI reporting.

Figure 11: Case 6—pathology report synthesis for TCGA-BRCA gigapixel WSIs.

Ablations and Analysis

Controlled ablations confirm the necessity and impact of each architectural component:

• The GigaPixel branch materially enhances slide-level reasoning, outpacing ViT-based random patch sampling by 11 points for closed VQA and 4 points for captioning.
• Coverage-driven token selection outperforms learnable cross-attention retrievers for WSI compression, with the training-free greedy selector yielding higher accuracy and efficiency.
• Cross-stage system prompts enable role-consistent linguistic reasoning without task-specific heads, maintaining high clinical validity and consistency under expert audit.

Practical and Theoretical Implications

BreastGPT and BreastStage demonstrate that workflow-aligned data, cross-scale visual modeling, and principled token compression are critical prerequisites for clinical MLLMs capable of supporting real-world oncology workflows. This design enables practical deployment at single-device latency and memory envelope, while delivering highly reliable reasoning across imaging evidence, task modalities, and clinical objectives.

Theoretical implications extend to general multimodal AI: submodular coverage objectives for visual token selection, role-conditioned prompting, and dual-scale architectural integration offer robust strategies for any domain requiring heterogeneous, longitudinal, and high-resolution visual reasoning.

Conclusion

BreastGPT constitutes a unified MLLM for breast oncology, bridging screening, diagnosis, and pathology with cross-scale visual representation and workflow-oriented prompting. Strong numerical results and robust qualitative reasoning confirm the efficacy of this approach, substantially closing the performance gap in clinical multimodal reasoning relative to general and medical-specific MLLMs. Future work will extend toward longitudinal, per-patient multimodal records and temporally continuous workflow modeling, with the potential to set new paradigms in computational oncology AI.

(2606.04911)