USMLE: Licensing, AI, and Clinical Benchmark

• USMLE is a comprehensive, multi-step examination assessing foundational science, clinical knowledge, and patient care competency in the U.S.
• The exam structure spans Step 1 to Step 3, utilizing multimodal content like images and case vignettes to rigorously evaluate diagnostic and therapeutic reasoning.
• USMLE has become a key benchmark for AI systems, driving innovations in chain-of-thought prompting, diagnostic simulations, and enhanced clinical decision support.

The United States Medical Licensing Examination (USMLE) is the principal multi-step standardized assessment for medical licensure in the U.S., evaluating candidates across basic science knowledge, clinical problem solving, and patient care decision-making. The exam is structured into three steps, each representing distinct domains of medical competency, and is frequently used as a high-fidelity benchmark in medical artificial intelligence research. Recent advancements in LLMs and domain-tuned medical models have transformed both performance and methodological approaches to USMLE-style question answering, explanation, and reasoning.

1. Structure and Purpose

The USMLE is a tripartite examination program designed to assess clinical competency and grant licensure for medical practice in the United States. The Steps are:

• Step 1: Focuses on foundational medical sciences including physiology, biochemistry, and pathology.
• Step 2 (CK and CS components): Emphasizes clinical knowledge (CK – multiple-choice questions on patient diagnosis and management) and, historically, clinical skills (CS – direct patient encounters, discontinued after 2020 but modeled in clinical skill research and simulation tools).
• Step 3: Centers on comprehensive patient management and ambulatory care in a standardized, case-based context.

Questions typically employ multiple-choice formats, with clinically realistic vignettes, and, increasingly, multimodal content (e.g., radiographic images, pathology slides, charts). Accuracy on the examination is conventionally evaluated as $$Accuracy = \frac{\text{Number of Correct Answers}}{\text{Total Questions}} \times 100\%$$, with a passing threshold of 60% or higher depending on the step and year.

2. USMLE as an AI Benchmark

USMLE materials have become the canonical evaluation suite for AI systems intended for medical reasoning (Nori et al., 2023, Singhal et al., 2023, Sharma et al., 2023, Chen et al., 2023, Schmidgall et al., 12 Feb 2024, Chen et al., 28 Feb 2024, Griot et al., 4 Jun 2024, Zuo et al., 30 Jan 2025, Wang et al., 11 Aug 2025). Key attributes include:

• Coverage: Broad spectrum of specialties and body systems.
• Complexity: Multi-hop, context-rich reasoning; tests not only rote recall, but integration and application of medical knowledge.
• Clinical Relevance: Vignettes often mirror authentic diagnostic dilemmas or therapeutic decision points.
• Multimodality: Integration of text, images, tables, and patient records.

Benchmarks such as MedQA, MedXpertQA, and clinical skill simulations draw directly from USMLE-style content or expand upon it to address more challenging, realistic scenarios.

3. LLM Performance

Recent studies report dramatic increases in performance on USMLE-style tasks using LLMs. GPT-4, evaluated on official self-assessment and sample exam materials, achieved average scores of approximately 86.65% (5-shot) and 83.76% (zero-shot), exceeding the passing threshold by more than 20 points and outperforming domain-specific models such as Med-PaLM (Nori et al., 2023). Med-PaLM 2, following substantial domain finetuning and the introduction of ensemble refinement prompting, scored up to 86.5%—a more than 19% improvement over its predecessor (Singhal et al., 2023). MEDITRON-70B, as an open-source alternative, reached 70.2% accuracy (Chen et al., 2023), while Meerkat-7B and Med42 provided competitive open-scale benchmarks (>71%) (Kim et al., 30 Mar 2024, Christophe et al., 23 Apr 2024). GPT-5, the latest reported generative model, achieved 95.22% average accuracy—surpassing both earlier models and human reference experts (Wang et al., 11 Aug 2025).

The performance gap between generalist LLMs and specialized medical models appears to be narrowing rapidly, with chain-of-thought, ensemble refinement, and retrieval-augmented generation contributing foundational improvements.

Model	USMLE Accuracy (%)	Calibration/Explanatory Features
GPT-4	86.7 (Sample Exam)	Superior certainty calibration, detailed explanations, personalized reasoning
Med-PaLM 2	86.5 (MedQA)	Ensemble refinement, human-preferred explanations, robust evaluation rubrics
MEDITRON-70B	~70.2 (MedQA-4opt)	Open-source, large-scale medical pretraining
Meerkat-7B	74.3 (MedQA)	Synthetic CoT training, interpretability
GPT-5	95.22 (Self-Assess)	State-of-the-art multimodal reasoning, stepwise explanation abilities

4. Calibration, Explanations, and Reasoning

Model evaluation increasingly emphasizes calibration—the correspondence of assigned probabilities with empirical correctness—a critical safety factor in medicine (Nori et al., 2023, Dhakal et al., 15 Feb 2024). For instance, GPT-4 outputs with 0.96 probability estimates align with a 93% true-correct frequency, whereas GPT-3.5 is much less reliable (55% at the same confidence).

Beyond accuracy, qualitative studies highlight models’ capacity to generate stepwise explanations, justify answer selection, and critique incorrect alternatives. Case studies demonstrate diagnostic reasoning, differential diagnosis list maintenance, and counterfactual scenario generation (Nori et al., 2023). These developments suggest new roles for LLMs as interactive teaching and assessment tools.

However, large-scale error annotation shows reasoning faults present even in high-scoring models: sticking to flawed diagnoses, vague conclusions, ignoring missing data, unsupported claims, and hallucinated detail are all documented as persistent error categories for GPT-4 (Roy et al., 20 Apr 2024). Annotation studies reveal that many explanations, though incorrect, are “reasonable” to human experts, complicating the validation of clinical reasoning.

5. Question Generation, Assessment, and Cognitive Bias

Efforts to automate question generation for USMLE-style assessments require domain-specific prompt engineering, multi-hop reasoning, and iterative critique-correction pipelines. The MCQG-SRefine framework employs self-refinement and LLM-as-Judge metrics to systematically improve the quality and difficulty of generated items; expert-driven prompts and scoring rubrics ensure alignment with NBME standards (Yao et al., 17 Oct 2024).

Research on cognitive bias in both humans and AI systems illustrates the vulnerability of models to recency, confirmation, “false consensus,” and others when decision cues are injected into question prompts. LLMs such as GPT-4 display resilience (average drop of 0.2% under confirmation bias) compared with smaller or less robust models—some of which experience 26% accuracy reductions (Schmidgall et al., 12 Feb 2024). Mitigation strategies include bias education, example demonstrations, and prompt calibration.

Notably, multiple-choice formats may reward pattern recognition and shallow test-taking heuristics rather than genuine clinical reasoning, as models obtained 64% accuracy on fully fictional benchmarks while physicians scored only 27% (Griot et al., 4 Jun 2024). These results challenge the validity of MCQ-only evaluations and motivate development of multimodal, consultation-based, and real-world scenario assessments.

6. Practical Applications and AI Integration in Medical Education

Recent implementations of AI tutors, virtual consultation simulators, and real-time diagnostic interfaces have proven effective on both USMLE-style tasks and simulated clinical encounters. Systems relying on retrieval-augmented generation (RAG), prompt engineering, and chain-of-thought reasoning architectures adapt expert knowledge for personalized learning, efficient paper planning, and spontaneous, context-aware question answering (Saxena, 31 Aug 2024).

Clinical trial data show that LLM-based diagnostic interfaces can match or exceed physicians in differential diagnosis accuracy, reduce time per encounter by 44.6%, cut costs by 98%, and maintain high patient satisfaction scores (AI: 3.9 vs. physician: ~4.2) (Park et al., 27 May 2025). These systems are poised to play supporting roles in primary care, tutoring, and self-evaluation for medical trainees.

Open-source benchmarks (MedXpertQA) now support comprehensive evaluation of reasoning and multimodal decision-making, spanning 17 specialties and including detailed images and patient records—offering publicly available code and data for research and educational integration (Zuo et al., 30 Jan 2025).

7. Limitations, Safety, and Future Directions

Despite substantial progress, limitations persist regarding hallucination risk, incomplete clinical context modeling, and error interpretability (Nori et al., 2023, Roy et al., 20 Apr 2024). Further, translation from MCQ performance to authentic patient-centered care remains an unresolved challenge. Calibration mechanisms, bias mitigation, chain-of-thought validation, and expert oversight remain indispensable before clinical deployment.

Future work in the field is poised to:

• Develop more robust, clinically meaningful assessment methods—incorporating realistic multi-modal input, consultation dialogue, longitudinal reasoning, and differential diagnosis simulation (Zuo et al., 30 Jan 2025, Liao et al., 2023).
• Advance the interpretability and error correction in LLM-generated explanations through large-scale annotation and semantic alignment tools (e.g., SemRep) (Roy et al., 20 Apr 2024).
• Secure AI-assisted medical decision-support pipelines against bias and error drift, and refine feedback mechanisms for model self-assessment (Dhakal et al., 15 Feb 2024).
• Expand open access to datasets, models, and evaluation code to promote reproducible, scalable progress in medical AI.

In summary, the USMLE remains both the gold standard for physician assessment and a focal benchmark for medical AI research, increasingly supported by high-performing generative models, advanced prompt strategies, multimodal datasets, and sophisticated question generation methodologies. These developments chart the pathway toward AI-augmented education and assessment at—or above—human expert level, while highlighting ongoing challenges in safety, reasoning, and deployment.