Alama Health QA Dataset
- Alama Health QA is a guideline-grounded medical QA benchmark focused on Kenyan primary care, built on national clinical guidelines.
- It uses a retrieval-augmented generation pipeline, combining BM25 and vector similarity indices to generate over 40,000 guideline-linked multiple-choice questions.
- The benchmark addresses a validity gap by emphasizing high-burden African conditions like malaria, HIV, and TB, ensuring cultural and regulatory relevance.
Searching arXiv for the cited papers to ground the article. arxiv_search: query="(Mutisya et al., 22 Jul 2025) OR (Mutisya et al., 19 Jul 2025) OR (Bardhan et al., 2023)", max_results=10 I’m going to look up the arXiv records for the specified papers. Alama Health QA is a guideline-grounded medical question answering benchmark for evaluating LLMs on Kenyan primary care reasoning, with explicit alignment to the Kenyan Ministry of Health’s Level 2 and Level 3 clinical guidance rather than to high-income examination syllabi or generic biomedical corpora. It was introduced as a response to a representativeness problem in medical LLM evaluation: widely used quantitative benchmarks underrepresent malaria, HIV, TB, sickle cell disease, and neglected tropical diseases, and they do not enforce national guideline compliance in settings where Ministry of Health protocols govern care. The benchmark therefore grounds every item in Kenyan Clinical Practice Guidelines, targets frontline contexts such as dispensaries, health centres, community health volunteers, and primary care workflows, and stores guideline-linked rationales and metadata to support traceability, verifiability, and updates (Mutisya et al., 22 Jul 2025, Mutisya et al., 19 Jul 2025).
1. Purpose, scope, and intended validity target
Alama Health QA was designed to close what its authors describe as a validity gap in medical LLM evaluation for African health systems. The core concern is construct validity: a benchmark that primarily reflects high-income disease profiles or examination conventions may reward model behavior that is clinically unsafe or regulator-inconsistent in Kenya and related settings. In this framing, benchmark quality is not reducible to scale alone; it depends on whether questions test decisions that are actually indicated in the local system and can be checked against authoritative text (Mutisya et al., 22 Jul 2025).
The dataset is explicitly oriented to Kenyan Level 2 and Level 3 primary care, that is, dispensaries and health centres. The source guideline covers outpatient diagnostics and treatments together with referral criteria consistent with Kenya’s tiered care system. The methodological description states that the content spans adult and pediatric care, obstetrics and gynecology, surgery and trauma, oxygen therapy, transfusion, forensic medicine, and a COVID-19 annex. In the reported table-level distribution, Part I addresses Internal Medicine and related content, Part II Paediatrics, Part III Surgery and related, Part IV Obstetrics and Gynaecology, and Parts V–IX oxygen therapy, blood products, referral framework, forensic medicine, and COVID-19 (Mutisya et al., 19 Jul 2025).
This positioning distinguishes Alama Health QA from broader “African medical QA” resources that may improve cultural relevance without enforcing current national recommendations, and from global exam-style resources whose latent assumptions may reflect tertiary-care infrastructure or non-African disease prevalence. A plausible implication is that Alama Health QA is intended less as a general medical knowledge test than as an operational test of regulator-aligned clinical reasoning in a specific care environment.
2. Guideline source and retrieval-augmented construction pipeline
The benchmark is built from the Kenyan Ministry of Health document “Clinical Guidelines for Management and Referral of Common Conditions at Level 2 and 3 (Dispensaries and Health Centres)” in one description, and “Clinical Guidelines for Management and Referral of Common Conditions at Level 2 and 3 (Primary Care)” in the methodology paper. These guidelines are treated as regulator-endorsed standards for primary care decision making, and every benchmark item is linked back to this source through citations and rationale text (Mutisya et al., 22 Jul 2025, Mutisya et al., 19 Jul 2025).
The construction workflow is retrieval-augmented generation. Guideline documents were digitized, paragraph-chunked, and indexed for efficient retrieval. One paper states only that chunks were indexed and used in a RAG loop, without specifying the retrieval algorithm; the methodology paper adds that the 636-page guideline, comprising approximately 416,000 words, was segmented into 1,115 semantically coherent chunks tagged with metadata such as source, section title, and page. It also reports that a hybrid retrieval setup combining a BM25 lexical index and a vector similarity index was prepared, although for the proof-of-concept dataset generation the authors used all chunks rather than selective retrieval (Mutisya et al., 22 Jul 2025, Mutisya et al., 19 Jul 2025).
For item generation, guideline chunks were passed to an LLM with prompts that requested realistic Kenyan clinical scenarios, a single multiple-choice question, four answer options, a correct answer, and an explanation tied to the source passage. Three model families were trialed—GPT-4o mini, Gemini Flash 2.0 Lite, and Llama 3.1 or LLaMA-3.1 (8B)—with Gemini Flash 2.0 Lite selected for the production pipeline because it produced the best balance of guideline adherence and creativity during QA generation (Mutisya et al., 22 Jul 2025, Mutisya et al., 19 Jul 2025).
The methodological paper reports that co-creation and review feedback led to several prompt and schema constraints: contextual metadata such as rural dispensary setting, age, gender, and postpartum status; explicit guideline citations; language and cultural sensitivity; and respect for local resource constraints. It also notes multilingual support, with approximately 10% of items translated into Kiswahili, especially for patient advice scenarios, and refined by bilingual clinicians (Mutisya et al., 19 Jul 2025).
3. Dataset structure, schema, and content organization
In the representativeness study, Alama Health QA is reported to contain 40,607 rows. The format is multiple-choice question answering with four options, typically embedded in case-based vignettes, and each item includes an explanation referencing guideline text together with guideline metadata such as section or heading references. The evaluation in that study used the English version for cross-benchmark comparability, although a Swahili variant had already been created (Mutisya et al., 22 Jul 2025).
The reported item structure is stable across descriptions: a clinical vignette framed for Kenyan primary care, four MCQ options, a correct answer, a rationale anchored to the guideline source, and guideline metadata. Additional metadata may include location and persona cues such as “Kibera” or “community health volunteer,” topic codes, and fields intended to support targeted updates as guidelines evolve. The paper presenting the benchmark as an evaluation resource does not report train, validation, or test splits (Mutisya et al., 22 Jul 2025).
The methodology paper provides a more explicit implied schema, though it labels this schema as implied rather than formally specified. The fields listed are: id, language, clinical_condition, level_of_care, question_text, options (A–D), correct_option, rationale, guideline_citation, and metadata, where metadata can include contextual tags such as rural or urban setting, age, gender, resource constraints, generation model, and guideline version tag. It also states that file formats, version numbers, and official splits are not specified in that paper (Mutisya et al., 19 Jul 2025).
| Aspect | Reported characteristic |
|---|---|
| Source corpus | Kenyan Ministry of Health Level 2/3 clinical guidelines |
| Size | 40,607 rows |
| Core format | Case-based MCQs with four options |
| Explanations | Guideline-backed rationale with citation |
| Languages | English; Swahili variant created |
| Splits | Not reported |
| Maintenance | Tied to guideline revisions, approximately every 2–3 years |
| Access details | Repository link and license not specified |
Representative examples clarify the design logic. One frequently cited item asks about a severely malnourished child with visible edema encountered during a home visit in Kibera; the correct response is referral to the nearest health facility for assessment and management, directly aligned with national protocols for severe acute malnutrition. Another example asks for the adult first-line ART regimen for a person who injects drugs, with the correct answer reported as TDF + 3TC + DTG, consistent with Kenya’s national ART guidance (Mutisya et al., 22 Jul 2025). The methodology paper additionally gives example items on very severe pediatric pneumonia, immunization at 10 weeks in Kisumu, and Kiswahili diarrhoeal-disease management prompts (Mutisya et al., 19 Jul 2025).
4. Disease coverage and representativeness profile
Alama Health QA intentionally emphasizes high-burden African conditions and primary care decision points. In the six-corpus comparison reported in “Mind the Gap,” Alama captured more than 40% of all neglected tropical disease mentions across the combined corpora. Within its own item set, it achieved the highest reported frequencies for malaria, HIV, and TB among the compared datasets: malaria at 7.74%, HIV at 4.07%, and TB at 5.15% (Mutisya et al., 22 Jul 2025).
The representativeness analysis operationalizes this comparison through explicit corpus-level metrics. Let be the number of rows in a dataset. The NTD proportion is defined as
and for a disease , the disease-category distribution is
The paper also defines a recency proxy,
together with readability and lexical-diversity measures including Flesch Reading Ease, Flesch–Kincaid Grade Level, Type–Token Ratio, MTLD, and HDD (Mutisya et al., 22 Jul 2025).
The comparative findings are central to the dataset’s significance. AfriMed-QA ranked second on African disease coverage, with reported frequencies including malaria at 1.51%, sickle-cell disease at 1.06%, TB at 1.07%, and Ebola or viral hemorrhagic fevers at 0.26%, but it lacked formal guideline linkage. By contrast, global benchmarks such as MMLU-Medical, MedQA-USMLE, MedMCQA, and PubMedQA collectively contributed under 20% of total NTD mentions; malaria was absent from MMLU-Medical, dengue was absent from all four of those global sets, and sickle cell disease was absent in three global benchmarks despite high African prevalence (Mutisya et al., 22 Jul 2025).
This evidence is used to argue that benchmark scale and benchmark representativeness are distinct properties. The reported underrepresentation of malaria, dengue, sickle cell disease, and other locally important conditions suggests that models evaluated only on global medical QA datasets may appear stronger than they are for African frontline care. That is the specific benchmark-validity gap Alama Health QA was built to address.
5. Linguistic profile, expert review, and comparative benchmark performance
Alama Health QA was profiled quantitatively and then assessed through blinded expert review against AfriMed-QA, MMLU-Medical, PubMedQA, MedMCQA, and MedQA-USMLE. In the semantic profiling reported in Table 3, Alama Health QA has 40,607 rows, sentence length of approximately 43 tokens, syntactic depth of approximately 7.9, FKGL of 25, and MTLD of 75, which is the second-highest among the compared datasets. The paper interprets these values as indicating mid-long, guideline-style vignettes with high diversity appropriate for clinical reasoning at primary care to early specialist levels (Mutisya et al., 22 Jul 2025).
The blinded qualitative review used 25 randomly sampled items per dataset, organized into five reviewer packs and scored on a five-point Likert instrument across five dimensions: clinical relevance, guideline alignment, clarity and completeness, distractor plausibility, and language or cultural fit. Reviewers were multidisciplinary in one paper and licensed medical doctors in the methodology paper’s description of the review setup; precise reviewer numbers and inter-rater reliability statistics were not reported (Mutisya et al., 22 Jul 2025, Mutisya et al., 19 Jul 2025).
Alama Health QA received the highest reported score for clinical relevance, with mean approximately 4.7 out of 5, and was also described as strongest on guideline alignment. Reported mean scores were approximately 4.5 out of 5 for clarity and approximately 4.4 out of 5 for distractor plausibility, with reviewers noting culturally and linguistically appropriate scenarios. AfriMed-QA was reported at approximately 3.9 out of 5 for clinical relevance and approximately 4.5 out of 5 for cultural fit, but again lacked formal guideline mapping. PubMedQA was reported as lowest for clinical utility at approximately 2.2 out of 5 and for cultural fit at approximately 1.6 out of 5, with its research-abstract framing judged poorly suited to frontline decisions (Mutisya et al., 22 Jul 2025).
The methodology paper extends this benchmarking logic beyond final-answer correctness by proposing several evaluation tasks and composite metrics associated with Alama Health QA. These include Decision Points for stepwise logic, Needle-in-the-Haystack for rare decisive cues, Reverse QA for simulated patient persona coherence, Geographic-Contextual Response Variance with the composite CAS score for contextual adaptability, and a Cognitive-Bias Stress Test with a Bias Susceptibility Index for anchoring and premature closure. No numerical baseline results for those metrics are reported in the methodology paper, but the framework is explicitly presented as a move beyond accuracy toward testing safety, vigilance, adaptability, and bias resilience in localized clinical scenarios (Mutisya et al., 19 Jul 2025).
6. Position within medical QA research and distinction from EHR QA
Alama Health QA belongs to the medical QA benchmark landscape, but it occupies a different niche from both exam-derived medical MCQ sets and EHR question answering datasets. A 2023 scoping review of QA over electronic health records does not mention any dataset named “Alama Health QA,” “Alama,” or “Alama Health,” and its catalog instead centers on resources such as emrQA, emrKBQA, MIMICSQL, EHRSQL, RadQA, RxWhyQA, DiSCQ, ClinicalKBQA, DrugEHRQA, and MedAlign (Bardhan et al., 2023).
That absence is informative rather than anomalous. In the EHR QA literature summarized by the review, answers are obtained from patient-related medical records, including unstructured notes, structured relational tables, knowledge graphs, or multimodal combinations. By contrast, Alama Health QA is grounded in national clinical guidelines and designed for regulator-aligned evaluation of decision making in primary care workflows, not for retrieval from patient records. A common misconception is therefore to treat it as another EHR QA corpus; the available descriptions do not support that classification (Bardhan et al., 2023, Mutisya et al., 22 Jul 2025).
Within non-EHR medical QA, the methodological paper contrasts Alama with MedQA-USMLE, MedMCQA, MultiMedQA, and ARC-style reading-comprehension benchmarks, describing those resources as largely Western-curriculum-oriented and less sensitive to African resource tiers, endemic disease patterns, and policy constraints. It also contrasts Alama with AfriMed-QA: AfriMed-QA is broader and pan-African, whereas Alama narrows deliberately to Kenyan primary care and ties every item to a current national guideline recommendation (Mutisya et al., 19 Jul 2025).
This suggests a useful typological distinction. Alama Health QA is not primarily a scale-maximized academic benchmark, nor an EHR retrieval benchmark, nor a pan-African exam set. It is a national, regulator-aligned, primary-care benchmark whose defining feature is guideline anchoring.
7. Governance, access, limitations, and planned extensions
The dataset’s governance logic is built around traceability, safety, and updateability. Each item is linked to a Ministry of Health source passage, explanations cite the relevant guidance, and metadata are stored to support targeted refresh when recommendations change. The maintenance schedule is tied to guideline revisions, reported as approximately every 2–3 years, with versioning tags and scheduled reviews proposed in the methodology paper (Mutisya et al., 22 Jul 2025, Mutisya et al., 19 Jul 2025).
The papers also describe several safety and compliance features. Distractors are intended to be plausible yet incorrect under Ministry of Health guidance, rather than arbitrarily misleading. Review rubrics enforce language and cultural appropriateness, and the items are constrained to Level 2–3 resource realities such as local availability of tests, medications, and vaccine formulations. The methodology paper further suggests that regulators could use Alama Health QA as an approval “exam” before clinical deployment, so that LLMs demonstrate competence on local guidelines prior to use in practice (Mutisya et al., 19 Jul 2025).
Several limitations are explicitly acknowledged. The cross-benchmark representativeness study evaluated English items for comparability, although a Swahili iteration exists. Inter-rater statistics were not reported for the qualitative review. Train, validation, and test splits are not given. Public repository links, DOIs, and license terms are not specified in the available papers. The methodology paper adds that its contribution is methodological rather than a completed leaderboard: quantitative answering baselines are not yet reported, and some composite metric weights may be debated (Mutisya et al., 22 Jul 2025, Mutisya et al., 19 Jul 2025).
Planned extensions include disease-specific derivatives in HIV, malaria, sickle-cell disease, and viral hemorrhagic fevers; regional extensions across additional African countries and guidelines; continued English and Kiswahili coverage; and open-sourcing of evaluation scripts together with a benchmark subset. In that sense, Alama Health QA is presented not only as a fixed dataset but as a template for regulator-aligned, locally curated benchmark development across African health systems (Mutisya et al., 22 Jul 2025, Mutisya et al., 19 Jul 2025).