MedXpert: Expert Medical AI Benchmark

• MedXpert is a polymorphic medical AI framework that includes the MedXpertQA benchmark and expert systems for radiology, triage, and follow-up support.
• It evaluates both text-only and multimodal clinical reasoning by separating understanding from multi-step diagnostic inference.
• The system leverages rigorous curation, expert filtering, and advanced evaluation metrics to drive improvements in clinical simulation and expert aggregation.

MedXpert is a polymorphic designation in medical artificial intelligence. In its most prominent contemporary use, it refers to MedXpertQA, a rigorously curated benchmark for expert-level medical reasoning and understanding in both text-only and multimodal settings, comprising 4,460 questions across 17 specialties and 11 body systems (Zuo et al., 30 Jan 2025). In a broader and partly derivative sense, the term also denotes a family of expert-oriented medical AI ideas: multi-expert radiology generation, expert-soft-guided follow-up summarization, on-device triage systems, physician-facing decision support, and adaptive aggregation of multiple medical experts (Wang et al., 2023, Wang et al., 2024, Li et al., 2020, 0810.1991, Bary et al., 4 Oct 2025).

1. Benchmark-centered meaning

MedXpertQA was introduced to address limitations of earlier medical QA datasets that had become insufficiently challenging or clinically narrow. Its design explicitly targets expert-level medical knowledge, advanced reasoning, and clinical realism. The benchmark covers 17 ABMS specialties and 11 body systems, and it is divided into a Text subset and an MM subset. Text is reported at 2,450 questions, while MM contains 2,005 questions, 2,839 images, and rich clinical context such as patient records, examination results, tables, and semistructured documents. Item metadata include specialty, body system, clinical task, fine-grained subtask, and a Reasoning versus Understanding label (Zuo et al., 30 Jan 2025).

The item format is multiple-choice with a single correct answer. Difficulty is increased by option augmentation: 10 options in Text and 5 in MM. The benchmark is evaluation-oriented rather than a training corpus, with each subset providing a few-shot development set of 5 questions and a held-out test set. The authors emphasize licensing constraints and leakage prevention: source materials are not released, and examples are not to be posted online (Zuo et al., 30 Jan 2025).

Curation is central to MedXpert’s identity. The benchmark begins from 37,543 initial questions drawn from authoritative exams and textbooks, including USMLE, COMLEX, 17 ABMS specialty boards, ACR DXIT/TXIT, EDiR, and the NEJM Image Challenge. It then applies AI expert filtering, human difficulty filtering using human response distributions and Brier score thresholds, similarity filtering with MedCPT embeddings and IQR-based outlier detection, distractor augmentation with gpt-4o-2024-11-20 and claude-3.5-sonnet-20241022, and multiple rounds of licensed-physician review (Zuo et al., 30 Jan 2025).

2. Cognitive taxonomy, measurement, and protocol

MedXpert is unusual in that it operationalizes both task modality and cognitive demand. The benchmark separates Text from MM, but it also separates Understanding from Reasoning. Understanding questions target visual recognition or factual retrieval, whereas Reasoning questions require multi-step inference, integration of clinical context, and diagnostic or therapeutic decision making. The MM subset further distinguishes diagnosis, treatment, and basic medicine, making the benchmark useful for fine-grained capability decomposition rather than only aggregate scoring (Zuo et al., 30 Jan 2025, Ding et al., 16 Jan 2026).

The primary evaluation metric is accuracy:

$$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\hat{y}_i = y_i].$$

This is the scoring rule used in MedXpertQA itself and in later MedXpert-based evaluations (Zuo et al., 30 Jan 2025).

A separate quantitative device appears during curation rather than scoring: the Brier score over human response distributions,

$B = \frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y_i})^2,$

with lower values interpreted as easier items. This score is used to filter questions by posterior human difficulty, not to evaluate model predictions (Zuo et al., 30 Jan 2025).

Evaluation prompting varies by study. MedXpertQA reports default zero-shot chain-of-thought prompting with greedy decoding. Later work standardizes a step-by-step instruction and requires the final answer to appear in boxed form, such as \boxed{C}, after which the boxed option is extracted and matched to the gold label. In AlphaMed, only the final boxed answer is scored; chain-of-thought supervision is not used in training, and no external tools or retrieval are used in evaluation (Liu et al., 23 May 2025, Liu et al., 15 Jul 2025).

3. MedXpert as a stress test for medical language-model reasoning

In “Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL,” MedXpert is positioned as a recent “hard+” benchmark explicitly targeting expert-level clinical knowledge and complex reasoning. Within that paper’s evaluation taxonomy, it lies above “normal” benchmarks such as MedQA, MedMCQA, and PubMedQA, and above “hard” benchmarks such as MMLU-ProM and GPQA-M. The benchmark is treated as out-of-domain for AlphaMed, since MedXpert is not used in training (Liu et al., 23 May 2025).

AlphaMed is trained purely by minimalist rule-based reinforcement learning on public multiple-choice QA datasets, without supervised fine-tuning on chain-of-thought data and without distilled preferences or verifier reward models. The main MedXpert results use Llama-3.1-8B-Instruct and Llama-3.1-70B-Instruct backbones with full-parameter tuning. The reward is binary:

$$r_i = \begin{cases} 1, & \text{if is\_answer\_correct}(\hat{y}_i, y_i) \ 0, & \text{otherwise.} \end{cases}$$

It is awarded only when both correctness and final boxed-answer formatting are satisfied. Optimization uses Group Relative Policy Optimization, with group-normalized advantage

$$\hat{A}_{i,t}=\frac{r_i-\mathrm{mean}(\{r_j\}_{j=1}^G)}{\mathrm{std}(\{r_j\}_{j=1}^G)}.$$

The reported training configuration is batch size 512, 64 QA pairs per batch, $G=8$ candidate answers per question, 300 training steps, and the verl framework (Liu et al., 23 May 2025).

The data-centric conclusion of that study is that MedXpert is especially sensitive to training difficulty and informativeness. The final AlphaMed training set contains all MedQA training samples (10,178) and 1,600 MedMCQA pairs from each difficulty level, for a total of 19,178 QA pairs; PubMedQA is excluded because of low informativeness and negative generalization effects. Hard samples at levels $L5$ and $L6$ improve MedXpert performance, and cumulative difficulty from $L1 \to L6$ raises MedXpert accuracy steadily, unlike several other benchmarks (Liu et al., 23 May 2025).

4. Multimodal benchmarking and state of the art

MedXpert and MedXpert-MM are now used as flagship evaluations for medical VLMs and multimodal reasoning systems. A 2025 benchmarking study evaluates general-purpose and medical-specific VLMs in zero-shot fashion and uses MedXpert’s explicit Understanding versus Reasoning labels to decompose performance. A 2026 reasoning MedVLM study, MMedExpert-R1, uses MedXpert-MM as a primary external benchmark and reports a finer-grained breakdown across Reasoning, Understanding, Treatment, Basic Science, and Diagnosis (Liu et al., 15 Jul 2025, Ding et al., 16 Jan 2026).

Study	Setting	Reported MedXpert result
AlphaMed	text MCQA, OOD	22.14% (8B), 32.56% (70B)
VLM benchmarking study	zero-shot MedXpert	Lingshu-32B: 0.3107 overall
MMedExpert-R1	MedXpert-MM	27.50 Total at 7B

In the benchmarking study, the best overall MedXpert score is Lingshu-32B at 0.3107, slightly above Qwen2.5-VL-72B at 0.2995. A distinctive observation is that, on MedXpert specifically, reasoning accuracy exceeds understanding accuracy for every listed model. For example, Qwen2.5-VL-72B scores 0.2953 on reasoning versus 0.2622 on understanding, and Lingshu-32B scores 0.2997 versus 0.2751. The authors note, however, that no model yet reaches the reliability threshold for clinical deployment (Liu et al., 15 Jul 2025).

MMedExpert-R1 reports MedXpert-MM performance under MedEvalKit. Its 7B model achieves Total 27.50, Reasoning 26.55, Understanding 29.96, Treatment 29.91, Basic Science 22.66, and Diagnosis 28.02. The method combines Domain-Specific Adaptation with specialty-specific LoRA modules, Guideline-Based Advantages for RL across Differential Diagnosis, Intuitive Reasoning, Analytical Reasoning, and Bayesian Reasoning perspectives, and Conflict-Aware Capability Integration via TIES-Merging. In the 2B ablation series, the full system reaches 23.35 on MedXpert-MM, above base, pooled-SFT, and GRPO variants (Ding et al., 16 Jan 2026).

These studies collectively position MedXpert as a benchmark where scale alone is not sufficient. AlphaMed uses it to show out-of-domain gains from minimalist RL, the VLM benchmarking study uses it to expose residual failure in medical multimodal reasoning, and MMedExpert-R1 uses it to validate domain-specific adaptation plus guideline-conditioned reinforcement (Liu et al., 23 May 2025, Liu et al., 15 Jul 2025, Ding et al., 16 Jan 2026).

5. MedXpert as a multi-expert and expert-guided design paradigm

A broader MedXpert paradigm in the literature emphasizes multiple expert perspectives rather than a single monolithic model. In radiology report generation, METransformer realizes a “multi-expert joint diagnosis” mechanism by appending $M$ learnable expert tokens to both encoder and decoder. These tokens attend to complementary image regions, are regularized by an orthogonality loss,

$$L_{OrL}(\hat{z}_L^e)=\frac{1}{M}\left\|\ell(\hat{z}_L^e)^\top \ell(\hat{z}_L^e)-I\right\|_F^2,$$

and produce $B = \frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y_i})^2,$0 candidate reports in parallel. Final selection uses metrics-based expert voting with

$B = \frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y_i})^2,$1

choosing $B = \frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y_i})^2,$2. On IU-Xray, METransformer reports BLEU-4 $B = \frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y_i})^2,$3, ROUGE-L $B = \frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y_i})^2,$4, METEOR $B = \frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y_i})^2,$5, and CIDEr $B = \frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y_i})^2,$6; on MIMIC-CXR, it reports BLEU-4 $B = \frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y_i})^2,$7, ROUGE-L $B = \frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y_i})^2,$8, METEOR $B = \frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y_i})^2,$9, and CIDEr $$r_i = \begin{cases} 1, & \text{if is\_answer\_correct}(\hat{y}_i, y_i) \ 0, & \text{otherwise.} \end{cases}$$0 (Wang et al., 2023).

A second expert-oriented formulation appears in follow-up chest X-ray summary generation. Here MedXpert denotes an expert insight–enhanced transformer framework built around expert soft guidance and masked entity modeling. A pretrained expert classifier produces a guidance vector $$r_i = \begin{cases} 1, & \text{if is\_answer\_correct}(\hat{y}_i, y_i) \ 0, & \text{otherwise.} \end{cases}$$1, injected into the generation model through an injection layer; in EIE-light, guidance is randomly dropped during training according to

$$r_i = \begin{cases} 1, & \text{if is\_answer\_correct}(\hat{y}_i, y_i) \ 0, & \text{otherwise.} \end{cases}$$2

with best $$r_i = \begin{cases} 1, & \text{if is\_answer\_correct}(\hat{y}_i, y_i) \ 0, & \text{otherwise.} \end{cases}$$3. Expanding guidance from 5 to 14 CheXpert observations improves performance, and the best reported variant, EIE-all-14, achieves BLEU-4 0.498, METEOR 0.391, ROUGE-L 0.639, and CIDEr 1.780 (Wang et al., 2024).

A third formulation is COVID-MobileXpert, a lightweight on-device chest-X-ray system for triage and follow-up. Its three-player knowledge transfer and distillation pipeline comprises an attending physician network, a resident fellow network, and a medical student network. The RF DenseNet-121 achieves 0.935 accuracy on 3-way triage. For the student models, MobileNetV2 reaches 0.880 under a representative KD+PC setting and AUROC 0.970 for COVID versus mixed pneumonia+normal; in longitudinal follow-up, MobileNetV2 reaches 0.800 accuracy and AUROC 0.883 for “Worse” versus “Improved.” The work frames expert hierarchy, distillation, and device-tier trade-offs as central design constraints (Li et al., 2020).

6. Physician-oriented systems, expert aggregation, and persistent limitations

An earlier systems-level use of MedXpert appears in a global physician-oriented medical information system. That design is free to use, Internet-based, open source, ad-free, and globally accessible over low-bandwidth connections. Physicians obtain patient consent, enter histories, physiological data, symptoms, diagnoses coded via ICD-10, and medications coded via ATC; the system then applies Bayesian-network reasoning for diagnostic assistance and uses accumulated outcome data to rank treatments and medications tailored to the patient. Researchers interact through a governed analytics interface that returns only aggregate outputs after proposal and code review (0810.1991).

Another expert-system lineage is rule-based and fuzzy rather than neural. For low back pain management, a lattice-based fuzzy medical expert system organizes knowledge over the power set of clinical attributes, uses Mamdani inference, triangular membership functions, and Centroid of Area defuzzification, and attaches reliability strengths to knowledge items. In a prototype using 20 patient records from ESI Hospital, Sealdah, the system’s inferences were acceptable to verifying medical experts in 18 of 20 cases (Santra et al., 2019).

A more recent operational perspective treats MedXpert as a multi-expert aggregation problem. In streaming annotation, expert trusts are modeled with Beta posteriors,

$$r_i = \begin{cases} 1, & \text{if is\_answer\_correct}(\hat{y}_i, y_i) \ 0, & \text{otherwise.} \end{cases}$$4

and labels are queried until posterior confidence exceeds a threshold $$r_i = \begin{cases} 1, & \text{if is\_answer\_correct}(\hat{y}_i, y_i) \ 0, & \text{otherwise.} \end{cases}$$5. The reported adaptive strategy reduces the number of expert queries by up to 50% while maintaining accuracy comparable to a non-adaptive baseline, thereby aligning MedXpert with real-time screening and workload optimization (Bary et al., 4 Oct 2025).

Across these strands, limitations are consistent. Benchmark scores on MedXpert do not imply safe clinical deployment; multiple-choice evaluation restricts the richness of clinical reasoning; divergent trends across benchmarks can misrepresent generalizable reasoning progress; and multimodal systems still fall far short of reliability thresholds for clinical use. The literature repeatedly calls for harder reasoning-oriented datasets, open-ended and human-in-the-loop benchmarks, stronger multimodal alignment, and rigorous clinical validation before deployment (Liu et al., 23 May 2025, Liu et al., 15 Jul 2025).

MedXpert therefore denotes more than a single artifact. It names a benchmark that has become a demanding yardstick for expert-level medical reasoning, and it also names a broader technical program: bringing expert structure, expert guidance, and expert aggregation into medical AI while preserving clinical relevance, interpretability, and evaluation rigor (Zuo et al., 30 Jan 2025, 0810.1991).