MedMASLab: Unified Medical MAS Benchmark

• MedMASLab is a unified orchestration and benchmarking framework for multimodal medical multi-agent systems, addressing fragmentation, non-uniform data ingestion, and evaluation inconsistencies.
• It integrates 11 heterogeneous MAS architectures across 24 medical modalities, enabling standardized comparisons and cost-performance analysis across diverse clinical tasks.
• Its zero-shot semantic evaluation engine assesses clinical meaning and visual grounding rather than relying on brittle lexical matching, ensuring robust medical reasoning.

MedMASLab is a unified orchestration and benchmarking framework for multimodal medical multi-agent systems (MAS). It was introduced to address architectural fragmentation, non-uniform data ingestion pipelines, inconsistent visual-reasoning evaluation, and the lack of cross-specialty benchmarking in medical MAS research. Its core contribution is infrastructural rather than algorithmic: a standardized inference environment for heterogeneous medical and general-purpose agentic methods, coupled to a zero-shot semantic evaluation engine that judges clinical meaning and visual grounding rather than relying on brittle lexical matching (Qian et al., 10 Mar 2026).

1. Origins and problem setting

MedMASLab emerged in a research context in which medical MAS systems were often built for one task, one modality, or one specialty, with bespoke prompting, custom preprocessing, incompatible agent communication schemes, and evaluation rules that made comparison unreliable. In this setting, architectural fragmentation obscured whether performance differences came from the collaboration design or from hidden implementation details such as data formatting, image handling, stopping rules, or parsing logic. The framework therefore targets four concrete bottlenecks: architectural fragmentation, inconsistent multimodal ingestion, weak evaluation, and poor cross-specialty comparability (Qian et al., 10 Mar 2026).

A broader infrastructure trajectory had already appeared in the general-domain MAS literature. "MASLab: A Unified and Comprehensive Codebase for LLM-based Multi-Agent Systems" (Ye et al., 22 May 2025) had argued that a medical specialization was plausible because MASLab already included medicine as a supported domain, integrated MedAgents, evaluated on MedMCQA, and emphasized extensibility to new domains, tools, and benchmarks. This suggests a clear lineage: MedMASLab concretizes, for multimodal medicine, the unification agenda that MASLab had formulated for LLM-based MAS more generally (Ye et al., 22 May 2025).

Within medical AI, the need for such a framework is sharper than in ordinary NLP benchmarking. Medical tasks span text, radiology images, pathology images, videos, and mixed evidence; non-uniform multimodal ingestion can therefore confound a comparison before reasoning begins. Likewise, many clinically correct outputs are semantically valid without being lexically identical to the reference answer. MedMASLab is designed around the claim that meaningful assessment of medical MAS requires standardized orchestration and semantic, multimodal-aware evaluation rather than repository-level comparison (Qian et al., 10 Mar 2026).

2. Orchestration model and system design

MedMASLab contains two main subsystems: a standardized inference-orchestration environment for heterogeneous medical MAS, and a multimodal semantic verification engine that evaluates answers based on clinical meaning and visual grounding. The framework is explicitly a decoupled orchestration layer between raw multimodal clinical inputs and agent reasoning modules, separating agent collaboration algorithms, multimodal data ingestion and normalization, serving infrastructure, and evaluation logic (Qian et al., 10 Mar 2026).

Its core interface requires every integrated method to expose a single inference function returning

$\mathcal{R} = (\mathbf{y}, \Gamma, \Theta)$

where $\mathbf{y}$ is the final medical response, $\Gamma$ is the token-usage metadata, and $\Theta$ is the agent-topology configuration used for that run. This interface does not force all methods into one internal architecture. Instead, each method preserves its native multi-turn reasoning and communication structure but enters and exits the framework through a shared contract (Qian et al., 10 Mar 2026).

The orchestration layer is organized around five stated principles. First, streamlined method abstraction wraps diverse paradigms under a common execution interface. Second, consistent multimodal inputs are emitted by a centralized dataset registry as standardized records containing prompt or question, standardized answer options, media paths, answer type, and evaluation flags. Third, shared inference resources route all methods through a shared serving layer based on dynamic vLLM and an OpenAI-compatible API. Fourth, unified configurations separate algorithmic hyperparameters from infrastructure parameters. Fifth, transparent cost accounting and resilient execution provide token and latency logging, persistent JSONL checkpointing, deduplication after restarts, and an auto-cleansing module that requeues corrupted entries (Qian et al., 10 Mar 2026).

Multimodal workflow standardization is a central design choice. For videos, MedMASLab uses an adaptive sampler that selects key frames under a configurable frame budget; in experiments it uses 4 to 8 frames per video for MedVidQA. The framework-level communication protocol is therefore less a low-level message schema than a standardized orchestration contract: common input abstraction, common execution interface, common output abstraction, common cost and state logging, and common evaluation pass. A structured per-sample ledger records VLM calls, correctness, latency, token usage, and configuration metadata (Qian et al., 10 Mar 2026).

The implementation also includes an interactive graphical user interface with five modules: API setup for decoupled base and judge model configuration, an interactive guide, single-sample quick test, large-scale batch evaluation with dynamic method-specific parameters, and a Custom MAS Builder with drag-and-drop roles and automatic code generation conforming to the MedMASLab interface. This GUI turns the framework into both a benchmarking platform and a prototyping environment (Qian et al., 10 Mar 2026).

3. Integrated architectures and benchmark construction

MedMASLab integrates 11 heterogeneous MAS architectures across 24 medical modalities. In the main comparison setting, the benchmarked methods are Single, Debate, MDAgents, MDTeamGPT, Discussion, Reconcile, Meta-Prompting, AutoGen, DyLAN, MedAgents, and ColaCare. The taxonomy also includes Vanilla VLM and CoT as baselines; Self-Consistency, Debate, Discussion, Reconcile, DyLAN, AutoGen, and Meta-Prompting as general-purpose MAS; and MedAgents, MDAgents, MDTeamGPT, and ColaCare as medical-specific MAS. Appendix taxonomy material also lists LINS, MedAgentAudit, MedLA, CXRAgent, MoMA, and MedOrch, indicating that the framework is broader than the core comparison table (Qian et al., 10 Mar 2026).

The benchmark is assembled from 11 existing medical or clinically oriented datasets and grouped into five task families.

Task family	Datasets	Test sizes
Medical Visual Understanding and Reasoning	Med-CMR, SLAKE-En, MedVidQA, MedXpertQA-MM	148 to 20,654
Diagnostic Decision-Making	DxBench	1,148
Medical Literature Reasoning	PubMedQA	500
Medical Question Answering	MedQA, MedBullets, MMLU, VQA-RAD	308 to 1,273
Evaluation of Medical Reasoning Chains	M3CoTBench	1,078

This suite spans 11 organ systems and 473 diseases. The medical spread comes from the union of standardized benchmarks: DxBench covers 461 disease types from 15 departments and 5,038 symptoms; MedXpertQA-MM spans 17 specialties and 11 body systems; Med-CMR spans 11 organ systems and 12 imaging modalities; and M3CoTBench spans 24 imaging modalities (Qian et al., 10 Mar 2026).

Benchmark construction involves substantive normalization rather than mere dataset aggregation. PubMedQA combines the original research question with background information to form a richer input. MedVidQA is augmented using Claude Sonnet 4.5 to generate multiple-choice options, one correct and three distractors, and its videos are reduced to 4–8 sampled frames for efficiency and token control. SLAKE-En uses only the English test subset, and MedQA primarily uses the US subset. These choices are important because MedMASLab’s stated goal is to eliminate benchmark-specific preprocessing idiosyncrasies that would otherwise confound architectural comparison (Qian et al., 10 Mar 2026).

4. Evaluation methodology

A defining feature of MedMASLab is its rejection of lexical matching as the primary evaluation paradigm for medical MAS. The framework compares five protocols: VLM-SJ (“Semantic Judge”), VLM-EC (“Extract-Compare”), Rule-MR (“Multi-Regex”), Rule-FL (“First-Letter”), and Rule-EM (“Exact Match”). Its default evaluator is VLM-SJ, a zero-shot semantic evaluation paradigm using Qwen2.5-VL-32B-Instruct as judge. The judge sees the same multimodal context as the agent, including the same radiographs, images, or sampled video frames, and assesses semantic equivalence, clinical faithfulness, and visual grounding (Qian et al., 10 Mar 2026).

The motivation is empirical rather than merely conceptual. Multi-agent systems often produce verbose or unconventionally formatted answers after extended deliberation, so lexical scoring can end up measuring formatting compliance rather than clinical reasoning quality. MedMASLab shows that rankings can invert under different protocols. On PubMedQA, MDTeamGPT ranks first under VLM-SJ with 79.40% but drops to last under Rule-MR with 0.40%. DyLAN drops from 71.60% to 0% under Rule-EM. On MedXpertQA, some methods drop near zero under VLM-EC but recover to greater than 20% under Rule-MR because extractor failures and regex heuristics behave differently. Nearly all methods collapse toward 0% under Rule-EM (Qian et al., 10 Mar 2026).

The evaluation setup for formal results is tightly controlled: zero-shot evaluation throughout, no external tools for MAS methods, default maximum token limit 1024, temperature 0.1, and VLM-SJ as the formal protocol. Accuracy under VLM-SJ is the main reported metric, but MedMASLab also logs latency, number of VLM calls, total token usage, and topology or configuration information. This enables cost-performance analysis rather than accuracy-only reporting (Qian et al., 10 Mar 2026).

A recurring misconception in agentic evaluation is that semantic judging simply replaces human adjudication. MedMASLab does not make that claim. Its contribution is that semantic, multimodal-aware verification is more faithful than exact match for this class of systems, not that model-based judging is equivalent to expert review. That limitation is explicit in the framework’s own discussion of autonomous clinical deployment (Qian et al., 10 Mar 2026).

5. Empirical findings and ablations

The main experimental conclusion is that no single MAS architecture dominates all 11 datasets. Under Qwen2.5VL-7B-Instruct, the highest average accuracy (Avg-V) is ColaCare at 57.3, followed by MDAgents at 56.8, MDTeamGPT at 56.2, Debate at 55.9, and MedAgents at 55.9. Under LLaVA-v1.6-mistral-7b-hf, ColaCare again leads with 49.9, followed by MDTeamGPT at 48.0 and MDAgents at 47.9. These shifts are not incidental; they support the framework’s central finding of a domain-specific performance gap, or “specialization penalty,” in which methods that perform very well in one domain or modality often do not transfer cleanly to another (Qian et al., 10 Mar 2026).

The benchmark does show gains from collaboration, but they are modest and inconsistent rather than universal. Under Qwen2.5VL-7B-Instruct, Single reaches Avg-V 54.1, Debate 55.9, MDAgents 56.8, and ColaCare 57.3. MedMASLab therefore rejects the stronger claim that multi-agent structure is intrinsically superior across medical reasoning tasks. Instead, collaborative gain depends heavily on the base model’s reasoning and instruction-following capacity (Qian et al., 10 Mar 2026).

Task-level results illustrate this conditionality. With Qwen2.5VL-7B-Instruct, MDTeamGPT is best on PubMedQA at 79.4 and on MMLU at 77.6; MedAgents is best on MedQA at 56.7, MedBullets at 41.9, Med-CMR at 72.9, and DxBench at 65.2; MDAgents is best on M3CoTBench at 36.8; Discussion is best on SLAKE-En at 65.3; and ColaCare is best on VQA-RAD at 59.5 and MedVidQA at 80.5. The winners change again under LLaVA-v1.6-mistral-7b-hf, reinforcing that architecture quality cannot be separated from backbone compatibility (Qian et al., 10 Mar 2026).

The ablations also undermine several common intuitions in medical MAS design. Increasing the number of agents does not monotonically increase accuracy: MDTeamGPT on MedQA performs best at 8 agents, and Debate’s best configuration differs by task, with Debate-A6-R2 on MedQA and Debate-A3-R2 on MedVidQA. Medical expert-playing is expensive and often not worth it: fixed and dynamic role-playing can average about 50,000 tokens per query while slightly improving M3CoTBench yet hurting MedVidQA. More communication is likewise not automatically better. On reasoning-heavy tasks such as DxBench, more tokens can help, but on MedXpertQA extra exchanges often inject semantic noise and degrade performance (Qian et al., 10 Mar 2026).

Model scaling produces similarly non-monotonic interactions. As the Qwen2.5VL family becomes larger, both single-agent and MAS accuracy generally improve, but the relative benefit of MAS can shrink or move across tasks. On MedQA, the largest gains from MAS appear with the 32B model; on MedXpertQA-MM, the greatest MAS gains appear at 7B. Supplementary results indicate that at 72B, on PubMedQA, nearly all methods except MedAgents underperform their 32B versions. The framework characterizes this as an amplification effect: MAS magnifies improvements in strong single agents and magnifies degradation when the single-agent backbone worsens (Qian et al., 10 Mar 2026).

6. Robustness, limitations, and significance

MedMASLab’s error analyses show that many failures are communication-protocol failures rather than purely medical-knowledge failures. Reconcile is a clear example: with Qwen2.5VL-3B, format errors reach 84.00% on PubMedQA and 75.33% on MedQA, while the same method on Qwen2.5VL-7B reduces those rates to 0.00% on PubMedQA and 0.55% on MedQA. This indicates that some MAS architectures are highly sensitive to the instruction-following reliability of the base model (Qian et al., 10 Mar 2026).

Backbone instability can also become operationally catastrophic. On MedQA, MDAgents with LLaVA-1.6-7B can inflate to roughly 150,000 tokens per query, nearly 100 times more than under other backbones, because agents fail to converge or terminate. MDTeamGPT with the same LLaVA backend shows failure samples in 58.2% of all MedQA cases; among failures, 41.9% are due to wrong model responses and 14.0% are due to round-limit failure. These results are directly relevant to clinical safety because they move the bottleneck from raw answer quality to protocol reliability and bounded execution (Qian et al., 10 Mar 2026).

Several limitations are explicit or strongly implied. Current MAS are fragile across specialties and modalities. Performance depends heavily on the base model. Communication overhead can be excessive. Role-playing and complex topologies can degrade efficiency without improving accuracy. Autonomous clinical deployment remains risky. Even the improved evaluator remains model-based rather than expert-adjudicated. A common misconception is that more agents, more rounds, or more elaborate medical role simulation will automatically yield better medical reasoning; the benchmark does not support that claim (Qian et al., 10 Mar 2026).

The enduring significance of MedMASLab is therefore infrastructural. It establishes a technical baseline for future medical MAS research: a standard input-output interface, a shared multimodal benchmark, a common inference environment, semantic multimodal-aware evaluation, and cost-aware analysis tools. By standardizing orchestration, multimodal ingestion, and semantic evaluation across 11 architectures, 11 medical benchmarks, 24 modalities, 11 organ systems, and 473 diseases, it makes heterogeneous multimodal medical MAS meaningfully comparable in a way that earlier repository-level comparisons could not. The source code and data are publicly available at the project repository, positioning MedMASLab as both a benchmark and a platform for subsequent medical multi-agent systems research (Qian et al., 10 Mar 2026).