MMBU: Massive Multimodal Biomedical Understanding

• MMBU is a research agenda that unifies diverse biomedical modalities, including images, text, molecular data, and signals, under a single multimodal framework.
• It overcomes the drawbacks of single-modal approaches by integrating heterogeneous data types to enhance clinical reasoning and localized evidence interpretation.
• Recent benchmark efforts and architectural innovations in MMBU demonstrate improved composite reasoning, grounded interpretation, and unified generation of biomedical insights.

Searching arXiv for the cited MMBU-related papers to ground the article and confirm metadata. Massive Multimodal Biomedical Understanding (MMBU) denotes a research program in biomedical AI that seeks models and benchmarks capable of operating across heterogeneous biomedical modalities, tasks, biological scales, and output formats, rather than remaining confined to single-image captioning, narrow visual question answering, or single-domain prediction. In recent work, the term is used both for a general agenda—large-scale multimodal biomedical learning across images, text, signals, molecular data, and grounded outputs—and for specific benchmark efforts designed to test whether biomedical vision-LLMs genuinely perceive, localize, and reason over biomedical evidence (Mo et al., 2024, Bansal et al., 2024, D'Cunha et al., 4 Jun 2026).

1. Conceptual scope and problem setting

MMBU emerges from the observation that biomedical data are inherently multimodal. MultiMed states this explicitly by framing medicine as a domain of images, text, molecular measurements, and signals, and by formalizing a modality set $X = \{X^{(1)}, X^{(2)}, \ldots, X^{(M)}\}$, outputs $Y = \{y_1, y_2, \ldots, y_N\}$, and tasks as functions $f : X \rightarrow Y$ (Mo et al., 2024). In that formulation, biomedical learning is not merely multimodal at the input level; it is also multitask, spanning disease classification, prognosis, protein structure prediction, gene expression prediction, and medical visual question answering.

A central motivation across the literature is that prior biomedical multimodal systems are fragmented. Some are trained only on one modality or a small subset of modalities; others support one task family, such as VQA or captioning; others generate fluent biomedical text without localizing the underlying lesion, organ, or tissue. UniBiomed frames this limitation as the disconnect between region-agnostic language generation and segmentation-only localization, arguing that grounded biomedical interpretation requires both text and pixel-level evidence in one system (Wu et al., 30 Apr 2025). M$^3$LLM makes an analogous argument at the level of image composition: most medical MLLMs are trained for single-image VQA or captioning, whereas clinical reasoning often requires comparison across time points, modalities, and panels (Chen et al., 27 Nov 2025).

This literature also narrows the meaning of “understanding.” In the MMBU benchmark paper, understanding is not equated with answering a familiar multiple-choice question; it includes ungrounded classification, grounded classification from detection, grounded classification from segmentation, and object detection, each in open and closed versions (D'Cunha et al., 4 Jun 2026). Taken together, these works suggest that MMBU is not a single architecture class but a broader attempt to align benchmark design, data curation, and model training with the multimodal and heterogeneous structure of biomedical inference.

2. Data substrates and corpus construction

Large-scale corpora are a recurring precondition for MMBU. BiomedCLIP established an early foundation with PMC-15M, a public corpus of 15,282,336 image-caption pairs from over 3 million distinct articles, extracted from the PubMed Central Open Access Subset, whose source pool contained 4.4 million publicly available full-text scientific articles as of June 15, 2022 (Zhang et al., 2023). Its importance lies not only in scale but in diversity: the corpus spans about 30 major biomedical image types, including radiology, digital pathology, microscopy, graphs, charts, tables, and forms.

Subsequent work has moved beyond caption-only supervision. PMC-InterCPT argues that biomedical multimodal continued pretraining requires interleaved, context-grounded evidence rather than isolated figure-caption pairs. It reconstructs samples by combining figures, captions, and figure-referencing body text; recovers missing captions from PMC XML; cleans markup, repetition, and incoherent paragraph aggregation; filters by LLM-supervised medical relevance and quality; and then resamples with a four-bucket evidence taxonomy. Its final mixture contains 10.11M samples and 9.63B tokens, improving over raw BIOMEDICA while using fewer CPT tokens (Zhu et al., 31 May 2026).

Instruction-tuning corpora have been expanded in parallel. MedMax presents 1.47 million instruction-tuning instances, about 1.7 billion multimodal discrete tokens, 725K unique images, and 947K unique words, covering biomedical VQA, image captioning and generation, visual chat, report understanding, and 88K instances of interleaved image-text outputs in MedMax-Instruct (Bansal et al., 2024). UniMedVL introduces UniMed-5M with over 5.6M samples, reformulating diverse unimodal datasets into multimodal pairs across nine primary imaging modalities and three task families: understanding, generation, and interleaved multimodal input-output tasks (Ning et al., 17 Oct 2025).

A distinct data-construction direction uses compound figures as proxies for real multi-image workflows. M$^3$LLM mines 237,137 compound figures from biomedical literature after license filtering, text-based pre-screening with fine-tuned PubMedBERT, compound-figure detection with a fine-tuned ViT, medical content screening with DenseNet-121, and textual quality thresholds on captions and sub-image captions (Chen et al., 27 Nov 2025). This suggests that MMBU data construction is increasingly oriented toward structured supervision that encodes relationships across panels, modalities, regions, or context paragraphs, rather than simple one-image/one-text pairs.

3. Benchmarks and evaluation regimes

Benchmarking in MMBU has progressed from narrow datasets toward broader, metadata-rich, and expert-validated evaluation. MultiMed provides a benchmark-scale framework with 2.56 million samples, 10 medical modalities, and 11 tasks, explicitly designed for multimodal learning, multitask learning, OOD generalization, few-shot adaptation, robustness studies, and modality-combination experiments (Mo et al., 2024). Its headline result is that multimodal multitask learning is best on every task reported in the main table, including disease classification, cell classification, expression prediction, protein structure prediction, and MedVQA.

The pathology domain has produced a more specialized but highly rigorous benchmark. PathMMU contains 33,428 multimodal multi-choice questions and 24,067 pathology images from heterogeneous sources, including PubMed scientific documents, pathology atlases, expert Twitter posts, classification datasets, and YouTube educational content (Sun et al., 2024). Its construction uses GPT-4V for caption enrichment and cascading question generation, filters out questions answerable without the image if at least three of four text-only models can solve them, and then applies manual review by seven professional pathologists. The benchmark exposes a substantial gap between current models and experts: GPT-4V reaches 49.8% zero-shot performance on the full test set, while human pathologists achieve 71.8% on the expert benchmark subset (Sun et al., 2024).

The benchmark explicitly titled MMBU broadens evaluation further. It is described as the largest biomedical vision-language benchmark to date, with 410 curated datasets, 78K total samples in the benchmark comparison table, 11 unique modalities, 35 submodalities, 27 medical and scientific domains, 20 distinct specimens, 95 regions of interest, 458 topics, and up to 13 structured attributes per sample (D'Cunha et al., 4 Jun 2026). Its four task categories—ungrounded classification, grounded classification from detection, grounded classification from segmentation, and object detection—each have open and closed variants. The results emphasize that high scores on established biomedical benchmarks can mask weak perception: open-format performance drops sharply relative to closed settings, and no model beats the random baseline on object detection in the closed setting (D'Cunha et al., 4 Jun 2026).

For composite image reasoning, PMC-MI-Bench offers a narrower but clinically targeted evaluation set. It contains 300 curated test cases spanning six task categories: multi-image VQA for holistic reasoning, single-panel reasoning within a compound figure, and spatial relationship reasoning; single-image VQA; text-only QA; and multi-choice VQA (Chen et al., 27 Nov 2025). Two board-certified physicians independently review the benchmark for medical correctness, answer accuracy and completeness, diagnostic appropriateness, and source consistency, with >85% inter-annotator agreement (Chen et al., 27 Nov 2025). This benchmark is significant because it tests a dimension of MMBU—composite multi-image understanding—that many earlier benchmarks do not cover.

4. Model architectures and learning objectives

Architecturally, MMBU research spans contrastive foundation models, instruction-tuned autoregressive assistants, grounded segmentation-language systems, and multimodal fusion baselines. BiomedCLIP adapts CLIP to biomedical data with PubMedBERT as text encoder, WordPiece tokenization, a context length 256, and a ViT-B/16 image backbone trained with the symmetric CLIP InfoNCE objective on PMC-15M (Zhang et al., 2023). The paper presents this as a generalist biomedical multimodal foundation model whose value lies in broad transfer rather than narrowly specialized supervision.

Grounded interpretation introduces a different training regime. UniBiomed couples an MLLM to SAM2-hiera-large, using language embeddings produced by the MLLM as textual prompts for SAM’s prompt encoder and mask decoder. Its joint loss is

$$L = L_{text} + L_{seg}, \qquad L_{seg} = 2*L_{BCE} + 0.5*L_{dice},$$

with $L = L_{text}$ for tasks without segmentation output (Wu et al., 30 Apr 2025). This formulation operationalizes a core MMBU claim: biomedical interpretation should bind generated findings to localized evidence.

Mixed-modal instruction tuning takes yet another form. MedMax fine-tunes Anole-7B, built on Chameleon-7B, with LoRA to preserve mixed-modal understanding and multimodal output capability. The model is trained under an autoregressive multimodal formulation in which image tokens and text tokens are part of a single token stream, and instruction tuning applies loss only on response tokens conditioned on the instruction or context (Bansal et al., 2024). This is notable because MMBU in that paper includes not only understanding but also text-to-image generation and interleaved text-image responses.

At the benchmark level, MultiMed evaluates early fusion, intermediate fusion, late fusion, and multimodal multitask settings, with modality-specific encoders such as Vision Transformers for imaging, DNABERT and scBERT for genomics and scRNA-seq, and LSTM-based RNNs for EEG (Mo et al., 2024). A plausible implication is that MMBU is less a single “best architecture” than a family of design patterns that attempt to preserve modality-specific inductive biases while learning shared representations across tasks.

5. Composite reasoning, grounding, and unified generation

Three capability clusters have become especially important within MMBU: composite multi-image reasoning, grounded region-aware interpretation, and unified understanding-generation.

For multi-image reasoning, M$^3$LLM proposes a five-stage, context-aware instruction generation paradigm under a divide-and-conquer strategy. The stages are inline text summarization, medical knowledge complementation, multi-modal medical visual perception enhancement, context-question-answer instruction generation, and leakage-prevented context refinement (Chen et al., 27 Nov 2025). The resulting model is trained to handle multi-image VQA, single-image VQA, text-only QA, and multi-choice VQA. On PMC-MI-Bench it reports BLEU@4: 15.0, ROUGE-L: 37.8, BERTScore: 70.1, and STS: 78.2 for multi-image VQA, and it generalizes to a longitudinal MIMIC chest X-ray benchmark of 1,326 pairs, achieving 73.9% diagnosis accuracy and 45.1% progression prediction accuracy (Chen et al., 27 Nov 2025). This directly targets a gap in prior MLLMs that could describe one panel but were not trained to compare, align, and synthesize across panels.

For grounded reasoning, UniBiomed is presented as a universal foundation model for grounded biomedical image interpretation. It is trained on 27 million triplets of images, annotations, and text descriptions spanning 10 biomedical imaging modalities, including 18 million images total, with 3D CT and MRI converted into 2D slices (Wu et al., 30 Apr 2025). Validation covers 84 internal and external datasets, and the paper reports that UniBiomed surpasses BiomedParse by an average of 10.25% in Dice scores across 60 internal and external segmentation datasets. It also reports improvements in grounded disease recognition, ROI classification, region-aware report generation, and end-to-end grounded report generation (Wu et al., 30 Apr 2025). In MMBU terms, this work shifts the field from ungrounded textual description to text explicitly tied to biomedical targets.

For unified understanding and generation, UniMedVL advances the Observation–Knowledge–Analysis (OKA) paradigm. At the observation level it uses UniMed-5M; at the knowledge level it applies Progressive Curriculum Learning in three stages; and at the analysis level it introduces a unified architecture with a ViT-based understanding encoder, a VAE-based generation encoder, and a mixture-of-transformer-experts (MoT) design (Ning et al., 17 Oct 2025). The model combines next-token prediction for understanding with flow matching in VAE latent space for generation under a unified loss. It reports strong performance on five medical understanding benchmarks, while generation quality across eight imaging modalities is summarized by Full UniMedVL: gFID 96.29, BiomedCLIP 0.706 (Ning et al., 17 Oct 2025). The paper’s central claim is bidirectional knowledge sharing: generation improves understanding, and understanding improves generation.

6. Limitations, controversies, and future directions

A recurrent controversy in MMBU concerns whether existing biomedical benchmarks genuinely measure visual perception. PathMMU shows that text-only models can still solve a substantial fraction of pathology questions, and that replacing images with Gaussian noise often reduces performance only slightly, suggesting shortcut exploitation rather than robust visual grounding (Sun et al., 2024). The MMBU benchmark extends this critique across biomedical domains by showing large open-versus-closed gaps and especially weak object detection, concluding that high scores on legacy benchmarks do not necessarily indicate strong visual perception (D'Cunha et al., 4 Jun 2026).

Data quality and modality balance are a second major concern. PMC-InterCPT argues that raw large-scale biomedical corpora contain missing captions, residual markup, repeated context, and incoherent multi-paragraph aggregation, and that the natural token distribution is heavily skewed toward QTE (Quantitative/Table Evidence) rather than BVE (Biomedical Visual Evidence) (Zhu et al., 31 May 2026). Its ablations suggest that quality filtering and modality-aware resampling are complementary rather than interchangeable. This implies that MMBU scaling is constrained not only by dataset size but also by the structure of evidence that pretraining exposes to the model.

Coverage limitations remain explicit. M$^3$LLM notes dependence on training-data diversity and weaker performance on underrepresented modalities such as ultrasound and fundus photography, while also observing that standard metrics like BLEU, ROUGE, and accuracy do not fully capture clinical reasoning quality (Chen et al., 27 Nov 2025). MedMax states that it does not address multiple-image reasoning or generation, citing both the lack of large, open, high-quality multi-image biomedical datasets and the limited context length of the base Chameleon model, where each image consumes 1024 tokens (Bansal et al., 2024). UniMedVL, for its part, states that current work focuses on 2D medical imaging and does not yet cover full unified 3D volumetric analysis, temporal reasoning, richer longitudinal patient modeling, or broader clinical multimodal inputs beyond imaging and text (Ning et al., 17 Oct 2025).

Future directions proposed across the literature are broadly convergent. They include more diverse and rare biomedical modalities, richer patient context beyond images and text, more efficient image representations, better spatial modeling and localization, stronger resistance to contamination and shortcut learning, broader evaluation across biological scales, and improved expert-aligned benchmarks (Bansal et al., 2024, Chen et al., 27 Nov 2025, D'Cunha et al., 4 Jun 2026). Taken together, these proposals suggest that the next phase of MMBU will depend less on a single architectural breakthrough than on coordinated advances in data curation, grounded evaluation, multimodal supervision, and clinically meaningful validation.