AfriMedQA: Pan-African Medical QA Benchmark
- AfriMedQA is a Pan-African English medical QA benchmark comprising 15,275 questions from over 60 African medical schools across 16 countries and 32 specialties.
- It evaluates large language model accuracy on a diverse set of clinical questions that capture African healthcare realities, showing notable performance differences from US-centric benchmarks.
- The benchmark’s broad contributor base, varied question formats, and regional focus emphasize both its cultural relevance and current limitations in guideline alignment.
AfriMedQA, published as AfriMed-QA, is a Pan-African English medical question-answering benchmark designed to evaluate LLMs on clinically relevant material drawn from African educational and healthcare contexts rather than from Global North examination syllabi alone. It comprises 15,275 questions sourced from over 60 medical schools across 16 countries and spans 32 medical specialties, with both open-ended and closed-ended formats. In subsequent comparative profiling of medical QA benchmarks, AfriMedQA was identified as one of the most contextually representative resources for African deployment, ranking second for African disease-burden and neglected tropical disease representation after the guideline-anchored Alama Health QA, while also being characterized as lacking formal linkage to national or WHO clinical guidelines (Olatunji et al., 2024, Mutisya et al., 22 Jul 2025).
1. Origin, rationale, and benchmark identity
AfriMed-QA was introduced against a background in which most widely used medical QA benchmarks—such as MedQA/USMLE, PubMedQA, MedMCQA, and MMLU medical subsets—largely reflect Global North examinations, research discourse, and disease distributions. The benchmark was motivated by the claim that these resources do not adequately transfer to African contexts, where endemic diseases, local guidelines, resource constraints, and intra-English linguistic variation shape medical reasoning differently. The dataset is explicitly presented as the first large-scale Pan-African English multi-specialty medical QA benchmark and was created to test whether LLMs that perform strongly on standard medical benchmarks generalize to African clinical questions, clinical vignettes, and consumer-health concerns (Olatunji et al., 2024).
Its position within the benchmarking landscape is defined by breadth rather than by guideline anchoring. AfriMed-QA aggregates material from many countries and specialties, and it includes consumer queries in addition to clinician-authored questions. This breadth distinguishes it from narrower, regulator-grounded datasets. A plausible implication is that AfriMed-QA is best understood as a continental coverage benchmark: it probes regional relevance, cross-specialty robustness, and African contextual knowledge, but it does not by itself encode a single national standard of care.
2. Corpus composition, sourcing, and data schema
The dataset contains 15,275 questions distributed across expert and crowdsourced contributions. The primary paper reports 3,910 expert MCQs, 359 expert SAQs, 129 crowdsourced MCQs, 877 crowdsourced SAQs, and 10,000 consumer queries. It is English-only, text-only, and available as afrimedqa_v2 on Hugging Face. The benchmark covers 32 specialties: Obstetrics and Gynecology, Pediatrics, General Surgery, Pathology, Neurology, Infectious Disease, Psychiatry, Cardiology, Endocrinology, Gastroenterology, Allergy and Immunology, Ophthalmology, Pulmonary Medicine, Hematology, Rheumatology, Nephrology, Internal Medicine, Otolaryngology, Orthopedic Surgery, Oncology, Other, Family Medicine, Neurosurgery, Radiology, Physical Medicine and Rehabilitation, Dermatology, Urology, Emergency Medicine, Plastic Surgery, Anesthesiology, Geriatrics, and Medical Genetics (Olatunji et al., 2024).
| Segment | Count | Notes |
|---|---|---|
| Expert MCQs | 3,910 | Answers only for expert MCQs |
| Expert SAQs | 359 | Reference long-form answers |
| Crowdsourced MCQs | 129 | Text-only |
| Crowdsourced SAQs | 877 | Text-only |
| Consumer queries | 10,000 | Free-form community-facing questions |
| Total | 15,275 | English-only benchmark |
The sourcing pipeline combined contributions from 621 contributors from over 60 medical schools across 16 countries, with contribution caps of 300 questions per contributor and compensation ranging from $5–$100/hour depending on task difficulty. The all-question country table reports contributions from Nigeria (7,577), Kenya (2,476), South Africa (2,444), Ghana (1,572), Malawi (465), Philippines (320), Uganda (175), Mozambique (77), Tanzania (36), Lesotho (33), United States (30), Zimbabwe (28), Australia (19), Botswana (17), Eswatini (4), and Zambia (2). Expert MCQs were concentrated in Ghana (1,495), Nigeria (1,452), Kenya (562), Malawi (347), and South Africa (54). The paper later notes a geographic skew, with over 60% of expert MCQs from West Africa (Olatunji et al., 2024).
AfriMed-QA’s schema includes question text, specialty, country, question type, contributor tier, and—depending on the item type—answer options, correct option(s), and rationales. MCQs contain 2–5 options; SAQs require 1–3 paragraph reference answers; consumer queries were seeded from 472 common conditions/symptoms relevant to African communities. A later profiling study described AfriMedQA as including multiple-choice questions (MCQs), short-answer questions (SAQs), yes/no items, factoids, and consumer queries, and as being drawn “exclusively from medical case content collected across 60 institutions in 16 African countries,” explicitly designed to reflect African clinical realities (Mutisya et al., 22 Jul 2025).
3. Evaluation design and principal empirical findings
AfriMed-QA was used to evaluate 30 LLMs spanning proprietary and open models, general-purpose and biomedical variants, and parameter scales from roughly 2B to 405B. The benchmark employs accuracy for expert MCQs, BERTScore for SAQs and CQs, and blinded human evaluation on axes including correctness, omission, hallucination or irrelevance, potential harm, and localization or African expertise. Human assessment involved 379 raters—58 clinicians and 321 non-clinicians—who provided 37,435 blinded ratings (Olatunji et al., 2024).
The main empirical result is that performance on AfriMed-QA expert MCQs is materially below performance on U.S.-centric medical benchmarks for many models. Expert MCQ accuracy ranged from 0.1728 (Gemma-2B) to 0.7928 (GPT-4o). The paper reports explicit MedQA-to-AfriMed-QA drops for several strong models: GPT-4o 0.8814 → 0.7928 (−8.86), GPT-4 0.7989 → 0.7568 (−4.21), Claude-3.5 Sonnet 0.8327 → 0.777 (−5.57), and Gemini Ultra 0.7879 → 0.739 (−4.89). A few models showed small gains, but the dominant pattern was degradation under African distributional shift (Olatunji et al., 2024).
The benchmark also exposes heterogeneity across specialties and countries. Averaged across models, expert MCQ accuracy was highest for Kenya (0.71) and Malawi (0.70), followed by Ghana (0.68), South Africa (0.57), and Nigeria (0.48). By specialty, internal medicine subspecialties such as Rheumatology, Nephrology, Gastroenterology, Endocrinology, and Pulmonary Medicine tended to be easier than Pediatrics, Infectious Disease, Surgery, Pathology, and Obstetrics & Gynecology. Human evaluation further reported high omission or hallucination rates in several specialties, including Ophthalmology (11.36% omission), Nephrology (10.13%), Pulmonary Medicine (10.64%), Neurosurgery (10.42%), Radiology (8.70% hallucination), and General Surgery (5.56% harm; 8.33% hallucination) (Olatunji et al., 2024).
A notable finding is that general LLMs outperform biomedical models of similar sizes on AfriMed-QA. For example, Meta Llama3-70B (0.7379) exceeded Llama3-OpenBioLLM-70B (0.6661), and several smaller biomedical models trailed general 7–9B models. The paper hypothesizes that domain fine-tuning may overfit to non-African distributions. Another important finding is that smaller, edge-friendly models struggle to achieve a passing score, which has direct implications for on-device or low-infrastructure deployment strategies in LMIC settings (Olatunji et al., 2024).
Human evaluation introduced an additional nuance: on consumer queries, respondents consistently preferred model-generated answers to clinician answers for relevance and informativeness, while clinician answers were often briefer and more omission-prone. This preference does not establish higher factual reliability; rather, it indicates that answer style, completeness, and presentation influence perceived utility.
4. Representativeness for African disease burdens and linguistic-cultural fit
A later benchmark-comparison study profiled AfriMedQA against Alama Health QA, MMLU Medical, PubMedQA, MedMCQA, and MedQA/USMLE. In that comparison, AfriMedQA ranked second for African disease-burden coverage and neglected tropical disease representation, behind Alama Health QA. The paper reports the following within-dataset disease proportions for AfriMedQA: malaria 1.51%, TB 1.07%, sickle cell disease 1.06%, and Ebola/viral hemorrhagic fevers 0.26%. HIV was acknowledged as important but not explicitly quantified for AfriMedQA in the paper text. The same study emphasized that global benchmarks often show minimal representation of African-burden diseases—for example, malaria absent in MMLU Medical, malaria 0.19% and HIV 0.42% in MedMCQA, and TB 0.25% in PubMedQA (Mutisya et al., 22 Jul 2025).
Semantic profiling placed AfriMedQA at 15,275 rows, with brief (15 tokens) sentence length, syntactic depth 4.7, FKGL ~11, TTR 0.95, and MTLD 34. In comparative terms, the study described it as more accessible than most global benchmarks, except MedMCQA at FKGL 10, while also having moderate lexical diversity. In blinded expert review on a 5-point Likert scale, AfriMedQA received 3.9/5 in the combined Clinical Relevance and Guideline Alignment discussion and 4.5/5 for language/cultural fit. Reviewers explicitly appreciated its reflection of African traditions and beliefs, while also criticizing some items for overgeneralized or ambiguous phrasing (Mutisya et al., 22 Jul 2025).
Two examples illustrate these strengths and weaknesses. One reviewer-approved item asks: “According to DSM 5, for a diagnosis of Bipolar 1, at least one episode of which the following is required?”, with the reviewer comment “Good case with mention of DSM 5”. Another, culturally framed item asks: “How do certain African societies manage symptoms of earwax buildup?”, with a reviewer observing that it was “a bit too wide of a net” despite its grounding in regional practices. These examples suggest that AfriMedQA captures cultural framing and non-U.S. clinical discourse, but that cultural contextualization can become overly generalized if not tightly specified.
5. Limitations, caveats, and recurrent misconceptions
AfriMed-QA has several documented limitations. It is English-only, text-only, and lacks predefined train/dev/test splits. Expert MCQs often provide answers without explanations, and the paper reports that free-form model rationales created regex extraction failures during MCQ evaluation, artificially lowering measured accuracy for some systems. The dataset is also described as ongoing, with potential annotation noise despite quality control, and later profiling work characterizes it as a one-off release in 2024 with no formal, ongoing update cadence (Olatunji et al., 2024, Mutisya et al., 22 Jul 2025).
A second limitation concerns alignment with policy and evidence standards. AfriMedQA is not linked to formal national or WHO guidelines; the later profiling study states that it “lacks formal alignment with national or WHO guidelines, limiting its utility for evidence-based model evaluation” and reports “No explicit mapping to national or WHO clinical guidelines described.” This matters because many African health systems are explicitly guideline-driven. A plausible implication is that strong benchmark scores on AfriMedQA do not, by themselves, establish conformance to country-specific standards of care (Mutisya et al., 22 Jul 2025).
A third limitation is imbalance. Topic distribution across the 32 specialties is uneven, with maternal health larger and rheumatology smaller in the later study’s summary. Country contributions are also uneven, and the primary paper notes a strong West African concentration among expert MCQs. This raises the possibility that aggregate benchmark scores can be disproportionately influenced by dominant specialties or regions rather than by uniformly distributed continental coverage (Olatunji et al., 2024, Mutisya et al., 22 Jul 2025).
Several misconceptions are directly challenged by the empirical results. One is that biomedical LLMs should outperform general LLMs on medical QA; AfriMed-QA shows the opposite pattern for many size-matched comparisons. Another is that good performance on USMLE-style datasets implies readiness for African deployment; the documented MedQA-to-AfriMed-QA accuracy drops refute that assumption. A further misconception is that consumer preference implies clinical superiority: the benchmark shows that users often prefer LLM answers for helpfulness and completeness, but safety, omission, hallucination, and localization remain separate evaluation axes.
6. Place within the emerging African medical QA ecosystem
AfriMed-QA now functions as one component of a broader African medical QA ecosystem rather than as a standalone solution to localization. The Kenyan methodology paper on Alama Health QA explicitly cites AfriMed-QA as a pan-African, multi-specialty medical QA benchmark built from African medical school and licensing exams and argues that Alama complements its breadth with country-level, guideline-grounded depth. That paper states: “Together, these efforts constitute an emerging AfriMedQA ecosystem: pan-African breadth (AfriMed-QA) plus country-level, guideline-grounded depth (Alama Health QA).” In this framing, AfriMed-QA is the broad continental benchmark, whereas Alama Health QA is optimized for Kenya MoH Level 2–3 alignment, multilingual grounding, and safety-centric metrics such as Decision Points, Needle-in-the-Haystack, CAS, and CBST (Mutisya et al., 19 Jul 2025).
A related systems paper, “Open-Source Retrieval Augmented Generation Framework for Retrieving Accurate Medication Insights from Formularies for African Healthcare Workers”, is not itself an AfriMedQA benchmark, but it is described as aligning with AfriMedQA task categories. Its Drug Insights RAG pipeline directly supports categories such as dosage, indications, contraindications, side effects, and special populations, and the paper presents it as a pragmatic baseline for future African medical QA infrastructure. The reported stack uses AzureOpenAIEmbeddings (1536-dimensional vectors), Pinecone, cosine similarity, threshold = 0.9, top-k = 3, and GPT-4o, with prompt guardrails specifying no speculation, no unverified sources, clear disclaimer, and context-limited responses. This suggests a path by which broad evaluation datasets such as AfriMed-QA may be complemented by retrieval-grounded clinical systems tied to formularies or guidelines rather than by parametric knowledge alone (AI et al., 28 Jan 2025).
In practical terms, AfriMed-QA is best used for assessing cultural fit, language appropriateness, baseline coverage of African-burden conditions, and broad cross-specialty robustness. The later profiling paper advises that it should not be relied upon alone for regulatory or patient-safety decisions and recommends pairing it with guideline-anchored datasets such as Alama Health QA. The same discussion prioritizes future steps that would strengthen AfriMedQA-like resources: formal linkage to national or WHO guidelines, regular update cycles, multilingual expansion, improved clarity and specificity, publication of dataset validation metrics such as inter-rater agreement, and disease-specific modules for HIV, malaria, sickle cell disease, and viral hemorrhagic fevers (Mutisya et al., 22 Jul 2025).
AfriMed-QA therefore occupies a foundational but delimited role. It established a continent-scale empirical basis for showing that LLM performance does not transfer cleanly from U.S.-centric medical exams to African medical questions. At the same time, later work has clarified that continental breadth, while necessary, is insufficient for regulator-aligned deployment. The benchmark’s enduring significance lies in making that gap measurable.