MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models

Abstract: Vision and LLMs (VLMs) hold immense promise to transform biomedical imaging workflows, from detecting lesions in chest X-rays to profiling cellular features in microscopy. Realizing this potential, however, requires robust and fine-grained visual perception. Models need to correctly interpret subtle features in images, and they must do so across diverse biomedical modalities, scales, and contexts. Nevertheless, current benchmarks remain limited. To address these gaps, we introduce the Massive Multimodal Biomedical Understanding (MMBU) benchmark. It is the largest biomedical vision and language benchmark to date, covering 35 submodalities with rich structured metadata. It includes both open and closed versions of ungrounded classification, grounded classification, and object detection, enabling systematic evaluation of model performance across biological scales, clinical settings, and imaging modalities. Evaluating 15 open-weight and 2 frontier VLMs, we find that while medical adaptation provides measurable gains for some models, the high accuracy often reported on established benchmarks can mask deficiencies in visual perception and domain generalization.

• The paper introduces MMBU, a massive benchmark with 410 curated biomedical datasets spanning 11 imaging modalities to probe VLM performance.
• It employs a structured, metadata-driven annotation pipeline and eight evaluation configurations to reveal spatial and reasoning limitations in models.
• Findings show models underperform in open tasks, highlighting the need for stronger domain adaptation and robust pretraining data in biomedical applications.

MMBU: Expanding the Evaluation Frontier for Biomedical Vision-LLMs

Motivation and Benchmark Landscape

The rapid proliferation of vision-LLMs (VLMs) in the biomedical domain has introduced the need for evaluation resources that match the diversity and complexity of modern biomedical imaging workflows. Existing benchmarks, predominantly focused on limited modalities and task types with substantial overlap between training and evaluation data, are insufficient for fine-grained, reliable measurement of VLMs' perceptual and reasoning capacities.

Figure 1: MMBU addresses the inadequate diversity and data pollution inherent to current biomedical VLM benchmarks, vastly expanding the exploitation space for assessment.

Current benchmarks are limited both in the number of datasets and in their ability to stratify model performance, masking failure modes and domain-specific generalization bottlenecks. The expansion of VLM pretraining data necessitates broader, more granular evaluation. To this end, MMBU (Massive Multi-modal Biomedical Understanding) introduces the largest and most structurally annotated collection for systematic probing of biomedical VLMs.

Construction of MMBU: Design Principles and Data Taxonomy

MMBU encompasses 410 curated, quality-controlled biomedical datasets spanning 11 imaging modalities and 35 submodalities. The dataset selection and taxonomy are clinician-guided, emphasizing diversity across biological scales, technical preparation, acquisition parameters, and clinical contexts. MMBU samples images from clinical, laboratory, and biological imaging, rejecting datasets with data contamination risk or low annotation quality.

Figure 2: The MMBU curation and annotation pipeline combines structured metadata extraction with expert-driven, clinically grounded question generation over multiple modalities and tasks.

Annotation is structured through a metadata-driven template pipeline, incorporating specimen type, body part, modality, acquisition details, and context. This facilitates question construction across a unified set of task types—ungrounded classification, grounded classification (segmentation/bounding box), and object detection. Every sample is human-verified for quality and annotation accuracy.

Figure 3: The distribution of MMBU data spans the entire modality, submodality, domain, and specimen landscape, with coverage of pathologic, radiologic, microscopic, and organ-focused imaging.

Benchmark Organization and Evaluation Protocol

MMBU supports both open and closed question-answering formats for each of its eight core configurations, yielding 24 evaluation settings per sample. Closed-formats constrain answer space via candidate lists, while open-format tasks demand free-form responses. Each benchmark test is paired with structured metadata necessary for stratified performance analysis.

Seventeen representative VLMs, including both large open-source (e.g., InternVL3.5, Qwen3-VL, LLaVA-Med, MedGemma, OctoMed, Lingshu) and proprietary (GPT-5.4-mini, GPT-4.1-mini) systems, are evaluated, with careful separation between adaptation data and benchmark content. Baseline and medical-specialized models are directly compared to quantify the effects of domain adaptation.

Evaluation uses micro-averaged F1-score, with strict string normalization for closed tasks and a semantically robust LLM-based judge (Qwen3-VL-32B) for open tasks, enforcing biomedical synonym, negation, laterality, and specificity checks.

Empirical Findings: Model Performance and Generalization

Aggregate results indicate uniformly low F1 scores across all models on the full MMBU testbed.

Figure 4: Closed-format tasks outperform open-format variants, and no VLM consistently dominates on all tasks.

Most models exhibit closed-format F1 in the 0.3–0.5 range for classification, but open-ended performance collapses (F1 ≪ 0.2), with the best exception being Qwen2.5-VL-32B on segmentation-derived classification (F1 = 0.693). Object detection failure is nearly universal, with all models underperforming the random baseline in the open setting, revealing a pervasive lack of spatial localization ability.

Medically specialized models (e.g., OctoMed-7B, MedGemma-4B) show measurable, but not uniform, improvements over base versions. Adaptive advantages are inconsistent and highly domain/submodality-dependent.

Figure 5: Radial comparisons across domains reveal that domain adaptation effects are selective, not global—substantial headroom remains for robust cross-domain performance.

Medical adaptation delivers a win in under 20% of direct comparisons, but in certain models (e.g., OctoMed-7B yields +0.40 improvement), advantages are marked. Notably, the success of adaptation correlates more strongly with pretraining data volume than with model parameter count, reinforcing the critical role of data scale and coverage for medical domain transferability.

Legacy benchmark gains do not extrapolate to MMBU, with models highly tuned to previously widely-used tests exhibiting sharp degradation on the broader, more challenging MMBU image and question distributions.

Figure 6: Superior legacy benchmark performance does not generalize to MMBU, with several models plateauing or regressing on newly curated data.

Granular Error Analysis

Modalities and submodalities (Figures 9 and 10) manifest large variance in model performance, highlighting that overall means conceal significant failures in rare or fine-grained classes. Closed- vs open-format gaps (Figures 11–14) are widest in settings requiring unconstrained answer generation rather than template matching, indicating both poor robustness in semantic mapping and a failure to generalize beyond learned verbal cues.

Theoretical and Practical Implications

MMBU exposes that current biomedical VLMs' perceptual and spatial reasoning capacities are substantially overestimated when using prior evaluation standards. Medical domain adaptation yields inconsistent benefits and does not eliminate critical failure modes. The strong disparity between closed and open-format task performance further reveals that current models compensate via answer-choice priming rather than authentic biomedical understanding or localization.

The MMBU design underlines the necessity for benchmark diversity and metadata-rich evaluation: robust biomedical VLM assessment mandates cross-domain, cross-modality, and open-format task probing. Fine-tuning and adaptation must be guided by broad and clean data, with continued analysis at the modality and submodality levels.

Forward-Looking Trajectories in AI for Biomedical Imaging

The MMBU contribution sets a new requirement for evaluating future VLMs—comprehensive, high-fidelity benchmarks with systematic metadata stratification, supporting both spatial and semantic challenges. As model scale and multi-modality continue to rise, MMBU's approach identifies the urgent need for:

• Stronger spatial and grounding mechanisms in VLM architectures.
• Pretraining regimes emphasizing underrepresented modalities and sub-domains.
• Open-format, zero-shot, and free-form evaluation protocols to drive genuine reasoning capabilities.
• Transparent adaptation reporting to separate true domain transfer from benchmark overfitting.

Widespread adoption of MMBU can accelerate progress toward generalist biomedical VLMs with robust, domain-agnostic perception and reasoning faculties, as well as galvanize data collection efforts in imaging modalities where current models are consistently weak.

Conclusion

MMBU fundamentally alters the landscape of biomedical VLM assessment, challenging assumptions established by prior benchmarks and quantifying deficits in current systems with previously unattainable granularity. The findings highlight the persistence of perceptual, spatial, and generalization failures as limiting factors for biomedical VLM deployment. Progress requires high-quality, large-scale, metadata-rich benchmarks—such as MMBU—for reliable measurement and driving advances in multimodal AI for biomedicine (2606.06696).