---
title: 'AfriMedQA: Pan-African Medical QA Benchmark'
url: https://www.emergentmind.com/topics/afrimedqa
type: topic
---

# AfriMedQA: Pan-African Medical QA Benchmark

AfriMedQA, published as **AfriMed-QA**, is a Pan-African English medical question-answering benchmark designed to evaluate large language models on clinically relevant material drawn from African educational and healthcare contexts rather than from Global North examination syllabi alone. It comprises **15,275 questions** sourced from **over 60 medical schools across 16 countries** and spans **32 medical specialties**, with both open-ended and closed-ended formats. In subsequent comparative profiling of medical QA benchmarks, AfriMedQA was identified as one of the most contextually representative resources for African deployment, ranking **second** for African disease-burden and neglected tropical disease representation after the guideline-anchored Alama Health QA, while also being characterized as lacking formal linkage to national or WHO clinical guidelines [2411.15640] [2507.16322].

## 1. Origin, rationale, and benchmark identity

AfriMed-QA was introduced against a background in which most widely used medical QA benchmarks—such as MedQA/USMLE, PubMedQA, MedMCQA, and MMLU medical subsets—largely reflect Global North examinations, research discourse, and disease distributions. The benchmark was motivated by the claim that these resources do not adequately transfer to African contexts, where endemic diseases, local guidelines, resource constraints, and intra-English linguistic variation shape medical reasoning differently. The dataset is explicitly presented as the **first large-scale Pan-African English multi-specialty medical QA benchmark** and was created to test whether LLMs that perform strongly on standard medical benchmarks generalize to African clinical questions, clinical vignettes, and consumer-health concerns [2411.15640].

Its position within the benchmarking landscape is defined by breadth rather than by guideline anchoring. AfriMed-QA aggregates material from many countries and specialties, and it includes consumer queries in addition to clinician-authored questions. This breadth distinguishes it from narrower, regulator-grounded datasets. A plausible implication is that AfriMed-QA is best understood as a continental coverage benchmark: it probes regional relevance, cross-specialty robustness, and African contextual knowledge, but it does not by itself encode a single national standard of care.

## 2. Corpus composition, sourcing, and data schema

The dataset contains **15,275 questions** distributed across expert and crowdsourced contributions. The primary paper reports **3,910 expert MCQs**, **359 expert SAQs**, **129 crowdsourced MCQs**, **877 crowdsourced SAQs**, and **10,000 consumer queries**. It is **English-only**, text-only, and available as **`afrimedqa_v2`** on Hugging Face. The benchmark covers **32 specialties**: Obstetrics and Gynecology, Pediatrics, General Surgery, Pathology, Neurology, Infectious Disease, Psychiatry, Cardiology, Endocrinology, Gastroenterology, Allergy and Immunology, Ophthalmology, Pulmonary Medicine, Hematology, Rheumatology, Nephrology, Internal Medicine, Otolaryngology, Orthopedic Surgery, Oncology, Other, Family Medicine, Neurosurgery, Radiology, Physical Medicine and Rehabilitation, Dermatology, Urology, Emergency Medicine, Plastic Surgery, Anesthesiology, Geriatrics, and Medical Genetics [2411.15640].

| Segment | Count | Notes |
|---|---:|---|
| Expert MCQs | 3,910 | Answers only for expert MCQs |
| Expert SAQs | 359 | Reference long-form answers |
| Crowdsourced MCQs | 129 | Text-only |
| Crowdsourced SAQs | 877 | Text-only |
| Consumer queries | 10,000 | Free-form community-facing questions |
| Total | 15,275 | English-only benchmark |

The sourcing pipeline combined contributions from **621 contributors** from **over 60 medical schools across 16 countries**, with contribution caps of **300 questions per contributor** and compensation ranging from **$5–$100/hour depending on task difficulty**. The all-question country table reports contributions from **Nigeria (7,577)**, **Kenya (2,476)**, **South Africa (2,444)**, **Ghana (1,572)**, **Malawi (465)**, **Philippines (320)**, **Uganda (175)**, **Mozambique (77)**, **Tanzania (36)**, **Lesotho (33)**, **United States (30)**, **Zimbabwe (28)**, **Australia (19)**, **Botswana (17)**, **Eswatini (4)**, and **Zambia (2)**. Expert MCQs were concentrated in **Ghana (1,495)**, **Nigeria (1,452)**, **Kenya (562)**, **Malawi (347)**, and **South Africa (54)**. The paper later notes a geographic skew, with **over 60% of expert MCQs from West Africa** [2411.15640].

AfriMed-QA’s schema includes question text, specialty, country, question type, contributor tier, and—depending on the item type—answer options, correct option(s), and rationales. MCQs contain **2–5 options**; SAQs require **1–3 paragraph reference answers**; consumer queries were seeded from **472 common conditions/symptoms relevant to African communities**. A later profiling study described AfriMedQA as including **multiple-choice questions (MCQs), short-answer questions (SAQs), yes/no items, factoids, and consumer queries**, and as being drawn “**exclusively from medical case content collected across 60 institutions in 16 African countries**,” explicitly designed to reflect African clinical realities [2507.16322].

## 3. Evaluation design and principal empirical findings

AfriMed-QA was used to evaluate **30 LLMs** spanning proprietary and open models, general-purpose and biomedical variants, and parameter scales from roughly **2B** to **405B**. The benchmark employs **accuracy** for expert MCQs, **BERTScore** for SAQs and CQs, and blinded human evaluation on axes including correctness, omission, hallucination or irrelevance, potential harm, and localization or African expertise. Human assessment involved **379 raters**—**58 clinicians** and **321 non-clinicians**—who provided **37,435 blinded ratings** [2411.15640].

The main empirical result is that performance on AfriMed-QA expert MCQs is materially below performance on U.S.-centric medical benchmarks for many models. Expert MCQ accuracy ranged from **0.1728 (Gemma-2B)** to **0.7928 (GPT-4o)**. The paper reports explicit MedQA-to-AfriMed-QA drops for several strong models: **GPT-4o 0.8814 → 0.7928 (−8.86)**, **GPT-4 0.7989 → 0.7568 (−4.21)**, **Claude-3.5 Sonnet 0.8327 → 0.777 (−5.57)**, and **Gemini Ultra 0.7879 → 0.739 (−4.89)**. A few models showed small gains, but the dominant pattern was degradation under African distributional shift [2411.15640].

The benchmark also exposes heterogeneity across specialties and countries. Averaged across models, expert MCQ accuracy was highest for **Kenya (0.71)** and **Malawi (0.70)**, followed by **Ghana (0.68)**, **South Africa (0.57)**, and **Nigeria (0.48)**. By specialty, internal medicine subspecialties such as Rheumatology, Nephrology, Gastroenterology, Endocrinology, and Pulmonary Medicine tended to be easier than **Pediatrics**, **Infectious Disease**, **Surgery**, **Pathology**, and **Obstetrics & Gynecology**. Human evaluation further reported high omission or hallucination rates in several specialties, including **Ophthalmology (11.36% omission)**, **Nephrology (10.13%)**, **Pulmonary Medicine (10.64%)**, **Neurosurgery (10.42%)**, **Radiology (8.70% hallucination)**, and **General Surgery (5.56% harm; 8.33% hallucination)** [2411.15640].

A notable finding is that **general LLMs outperform biomedical models of similar sizes** on AfriMed-QA. For example, **Meta Llama3-70B (0.7379)** exceeded **Llama3-OpenBioLLM-70B (0.6661)**, and several smaller biomedical models trailed general 7–9B models. The paper hypothesizes that domain fine-tuning may overfit to non-African distributions. Another important finding is that **smaller, edge-friendly models struggle to achieve a passing score**, which has direct implications for on-device or low-infrastructure deployment strategies in LMIC settings [2411.15640].

Human evaluation introduced an additional nuance: on consumer queries, respondents consistently preferred model-generated answers to clinician answers for relevance and informativeness, while clinician answers were often briefer and more omission-prone. This preference does not establish higher factual reliability; rather, it indicates that answer style, completeness, and presentation influence perceived utility.

## 4. Representativeness for African disease burdens and linguistic-cultural fit

A later benchmark-comparison study profiled AfriMedQA against Alama Health QA, MMLU Medical, PubMedQA, MedMCQA, and MedQA/USMLE. In that comparison, AfriMedQA ranked **second** for African disease-burden coverage and neglected tropical disease representation, behind Alama Health QA. The paper reports the following within-dataset disease proportions for AfriMedQA: **malaria 1.51%**, **TB 1.07%**, **sickle cell disease 1.06%**, and **Ebola/viral hemorrhagic fevers 0.26%**. HIV was acknowledged as important but **not explicitly quantified for AfriMedQA in the paper text**. The same study emphasized that global benchmarks often show minimal representation of African-burden diseases—for example, **malaria absent in MMLU Medical**, **malaria 0.19% and HIV 0.42% in MedMCQA**, and **TB 0.25% in PubMedQA** [2507.16322].

Semantic profiling placed AfriMedQA at **15,275 rows**, with **brief (15 tokens)** sentence length, **syntactic depth 4.7**, **FKGL ~11**, **TTR 0.95**, and **MTLD 34**. In comparative terms, the study described it as **more accessible than most global benchmarks, except MedMCQA at FKGL 10**, while also having moderate lexical diversity. In blinded expert review on a **5-point Likert scale**, AfriMedQA received **3.9/5** in the combined **Clinical Relevance and Guideline Alignment** discussion and **4.5/5** for **language/cultural fit**. Reviewers explicitly appreciated its reflection of **African traditions and beliefs**, while also criticizing some items for **overgeneralized or ambiguous phrasing** [2507.16322].

Two examples illustrate these strengths and weaknesses. One reviewer-approved item asks: **“According to DSM 5, for a diagnosis of Bipolar 1, at least one episode of which the following is required?”**, with the reviewer comment **“Good case with mention of DSM 5”**. Another, culturally framed item asks: **“How do certain African societies manage symptoms of earwax buildup?”**, with a reviewer observing that it was **“a bit too wide of a net”** despite its grounding in regional practices. These examples suggest that AfriMedQA captures cultural framing and non-U.S. clinical discourse, but that cultural contextualization can become overly generalized if not tightly specified.

## 5. Limitations, caveats, and recurrent misconceptions

AfriMed-QA has several documented limitations. It is **English-only**, **text-only**, and lacks **predefined train/dev/test splits**. Expert MCQs often provide **answers without explanations**, and the paper reports that free-form model rationales created **regex extraction failures** during MCQ evaluation, artificially lowering measured accuracy for some systems. The dataset is also described as **ongoing**, with potential annotation noise despite quality control, and later profiling work characterizes it as a **one-off release in 2024 with no formal, ongoing update cadence** [2411.15640] [2507.16322].

A second limitation concerns alignment with policy and evidence standards. AfriMedQA is **not linked to formal national or WHO guidelines**; the later profiling study states that it **“lacks formal alignment with national or WHO guidelines, limiting its utility for evidence-based model evaluation”** and reports **“No explicit mapping to national or WHO clinical guidelines described.”** This matters because many African health systems are explicitly guideline-driven. A plausible implication is that strong benchmark scores on AfriMedQA do not, by themselves, establish conformance to country-specific standards of care [2507.16322].

A third limitation is imbalance. Topic distribution across the **32 specialties** is uneven, with **maternal health larger** and **rheumatology smaller** in the later study’s summary. Country contributions are also uneven, and the primary paper notes a strong West African concentration among expert MCQs. This raises the possibility that aggregate benchmark scores can be disproportionately influenced by dominant specialties or regions rather than by uniformly distributed continental coverage [2411.15640] [2507.16322].

Several misconceptions are directly challenged by the empirical results. One is that biomedical LLMs should outperform general LLMs on medical QA; AfriMed-QA shows the opposite pattern for many size-matched comparisons. Another is that good performance on USMLE-style datasets implies readiness for African deployment; the documented MedQA-to-AfriMed-QA accuracy drops refute that assumption. A further misconception is that consumer preference implies clinical superiority: the benchmark shows that users often prefer LLM answers for helpfulness and completeness, but safety, omission, hallucination, and localization remain separate evaluation axes.

## 6. Place within the emerging African medical QA ecosystem

AfriMed-QA now functions as one component of a broader African medical QA ecosystem rather than as a standalone solution to localization. The Kenyan methodology paper on **Alama Health QA** explicitly cites AfriMed-QA as a **pan-African, multi-specialty medical QA benchmark built from African medical school and licensing exams** and argues that Alama complements its breadth with **country-level, guideline-grounded depth**. That paper states: **“Together, these efforts constitute an emerging AfriMedQA ecosystem: pan-African breadth (AfriMed-QA) plus country-level, guideline-grounded depth (Alama Health QA).”** In this framing, AfriMed-QA is the broad continental benchmark, whereas Alama Health QA is optimized for **Kenya MoH Level 2–3** alignment, multilingual grounding, and safety-centric metrics such as **Decision Points**, **Needle-in-the-Haystack**, **CAS**, and **CBST** [2507.14615].

A related systems paper, **“Open-Source Retrieval Augmented Generation Framework for Retrieving Accurate Medication Insights from Formularies for African Healthcare Workers”**, is not itself an AfriMedQA benchmark, but it is described as aligning with AfriMedQA task categories. Its **Drug Insights** RAG pipeline directly supports categories such as **dosage**, **indications**, **contraindications**, **side effects**, and **special populations**, and the paper presents it as a pragmatic baseline for future African medical QA infrastructure. The reported stack uses **AzureOpenAIEmbeddings (1536-dimensional vectors)**, **Pinecone**, **cosine similarity**, **threshold = 0.9**, **top-k = 3**, and **GPT-4o**, with prompt guardrails specifying **no speculation**, **no unverified sources**, **clear disclaimer**, and **context-limited responses**. This suggests a path by which broad evaluation datasets such as AfriMed-QA may be complemented by retrieval-grounded clinical systems tied to formularies or guidelines rather than by parametric knowledge alone [2502.15722].

In practical terms, AfriMed-QA is best used for assessing **cultural fit**, **language appropriateness**, **baseline coverage of African-burden conditions**, and broad cross-specialty robustness. The later profiling paper advises that it should **not** be relied upon alone for **regulatory or patient-safety decisions** and recommends pairing it with **guideline-anchored datasets** such as Alama Health QA. The same discussion prioritizes future steps that would strengthen AfriMedQA-like resources: formal linkage to national or WHO guidelines, regular update cycles, multilingual expansion, improved clarity and specificity, publication of dataset validation metrics such as inter-rater agreement, and disease-specific modules for **HIV**, **malaria**, **sickle cell disease**, and **viral hemorrhagic fevers** [2507.16322].

AfriMed-QA therefore occupies a foundational but delimited role. It established a continent-scale empirical basis for showing that LLM performance does not transfer cleanly from U.S.-centric medical exams to African medical questions. At the same time, later work has clarified that continental breadth, while necessary, is insufficient for regulator-aligned deployment. The benchmark’s enduring significance lies in making that gap measurable.

Source: https://www.emergentmind.com/topics/afrimedqa