MedConceptsQA: Open Source Medical Concepts QA Benchmark

Abstract: We present MedConceptsQA, a dedicated open source benchmark for medical concepts question answering. The benchmark comprises of questions of various medical concepts across different vocabularies: diagnoses, procedures, and drugs. The questions are categorized into three levels of difficulty: easy, medium, and hard. We conducted evaluations of the benchmark using various LLMs. Our findings show that pre-trained clinical LLMs achieved accuracy levels close to random guessing on this benchmark, despite being pre-trained on medical data. However, GPT-4 achieves an absolute average improvement of nearly 27%-37% (27% for zero-shot learning and 37% for few-shot learning) when compared to clinical LLMs. Our benchmark serves as a valuable resource for evaluating the understanding and reasoning of medical concepts by LLMs. Our benchmark is available at https://huggingface.co/datasets/ofir408/MedConceptsQA