Capabilities of GPT-4 on Medical Challenge Problems (2303.13375v2)

Abstract: LLMs have demonstrated remarkable capabilities in natural language understanding and generation across various domains, including medicine. We present a comprehensive evaluation of GPT-4, a state-of-the-art LLM, on medical competency examinations and benchmark datasets. GPT-4 is a general-purpose model that is not specialized for medical problems through training or engineered to solve clinical tasks. Our analysis covers two sets of official practice materials for the USMLE, a three-step examination program used to assess clinical competency and grant licensure in the United States. We also evaluate performance on the MultiMedQA suite of benchmark datasets. Beyond measuring model performance, experiments were conducted to investigate the influence of test questions containing both text and images on model performance, probe for memorization of content during training, and study probability calibration, which is of critical importance in high-stakes applications like medicine. Our results show that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B). In addition, GPT-4 is significantly better calibrated than GPT-3.5, demonstrating a much-improved ability to predict the likelihood that its answers are correct. We also explore the behavior of the model qualitatively through a case study that shows the ability of GPT-4 to explain medical reasoning, personalize explanations to students, and interactively craft new counterfactual scenarios around a medical case. Implications of the findings are discussed for potential uses of GPT-4 in medical education, assessment, and clinical practice, with appropriate attention to challenges of accuracy and safety.

Capabilities of GPT-4 on Medical Challenge Problems

The paper presents a rigorous evaluation of GPT-4, an advanced LLM, specifically its performance on medical examinations and benchmarks. Notably, GPT-4 is not optimized for medical tasks through specialized training or fine-tuning yet surpasses earlier iterations and models designed for medical problem-solving.

Key Findings and Methodology

GPT-4's capabilities were assessed against the United States Medical Licensing Examination (USMLE) and the MultiMedQA suite. The results indicate that GPT-4 significantly exceeds the passing score on all steps of the USMLE, notably improving by over 20 points compared to GPT-3.5 and specialized models like Med-PaLM. Furthermore, GPT-4 shows superior calibration, predicting the correctness of its answers more accurately than previous models.

The evaluation methodology focused on zero-shot and few-shot prompting strategies. By employing straightforward prompts, the paper establishes a baseline performance without relying on complex techniques like chain-of-thought reasoning or retrieval augmentation.

Performance on Medical Examinations

In testing against the USMLE, GPT-4 demonstrated high accuracy across Step 1, Step 2, and Step 3 of the examination. The model achieved an average score of 86.65% on the Self-Assessment and 86.7% on the Sample Exam. Despite the inclusion of media elements in some questions, which the model could not process, GPT-4 maintained strong performance, suggesting its logic and reasoning capabilities were robust enough to handle text-based information effectively.

Multilingual and Benchmark Performance

GPT-4 was also evaluated on datasets from the MultiMedQA benchmark, which includes questions from different regions and languages, such as MedQA, PubMedQA, and MedMCQA. The model continued to outperform GPT-3.5 and scored higher than Flan-PaLM 540B in most tests, indicating its adaptability across diverse medical reasoning tasks.

Model Calibration and Prediction Confidence

The model's ability to assign calibrated probabilities to its answers underscores its suitability for high-stakes applications like medicine, where confidence levels guide decision-making. This enhanced calibration differentiates GPT-4 from its predecessors, offering credible confidence in its responses.

Qualitative Insights and Potential Applications

Beyond statistical results, the paper explores GPT-4's qualitative capabilities through case studies. Examples illustrate the model's potential in explaining medical reasoning, interacting with students, and crafting alternative scenarios. This highlights its potential utility in medical education and practice, where it could assist practitioners by providing second opinions, educational support, and more.

Challenges and Future Directions

The paper addresses the limitations and potential risks of deploying LLMs like GPT-4 in real-world medical settings. Significant work is required to address issues such as reliability, accuracy, and the ongoing risk of biased outcomes. Moreover, while calibration has improved, ensuring safety and trust within a clinical context remains critical.

Integration into healthcare systems will necessitate careful development of best practices for verification and a robust understanding of model biases. Future research should explore methods to mitigate errors, enhance the user interface for practitioners, and examine the broader societal impacts of AI on the medical profession.

Conclusion

GPT-4's evaluation conveys its high-level performance on medical proficiency examinations and suggests its broader applicability in healthcare. As LLMs become integral to medical practice, careful attention to validation, calibration, and bias detection will be essential for their successful deployment in clinical environments. This research marks a significant stride toward integrating AI in medicine, indicating a future where these models could substantially support healthcare delivery and education.