Alama Health QA: Kenyan Clinical LLM Benchmark
- Alama Health QA is a retrieval-augmented benchmark that evaluates clinical LLMs in Kenyan primary care by aligning with official national guidelines.
- It converts Kenya's 636-page clinical guideline into 1,115 structured recommendation chunks using OCR, parsing, and metadata tagging.
- It employs multi-module evaluation measuring clinical reasoning, decision points, context adaptation, and bias robustness under local resource constraints.
Alama Health QA is a retrieval-augmented, guideline-grounded benchmark for testing clinical LLMs in Kenyan primary care. It is both a dataset of regulator-aligned clinical question-answer items in English and Swahili and a framework of evaluation tasks and metrics focused on reasoning, safety, and context-sensitive decision-making in Kenya’s Level 2–3 outpatient settings. Its central design principle is explicit linkage to Kenya’s official primary-care guidance, so benchmark performance can be interpreted against the Kenya Ministry of Health’s own standards rather than against decontextualized global exam sets (Mutisya et al., 19 Jul 2025, Mutisya et al., 22 Jul 2025).
1. Definition, scope, and institutional grounding
Alama Health QA is centered on Kenya’s official “Clinical Guidelines for Management and Referral of Common Conditions at Levels 2 and 3 (Primary Care)”, a 2024 Ministry of Health document of about 636 pages and about 416,000 words. The benchmark targets outpatient and front-line primary care in dispensaries and health centres, with explicit attention to Kenyan epidemiology, rural and urban resource constraints, and tiered referral pathways (Mutisya et al., 19 Jul 2025).
The source guideline is organized into nine major parts, and Alama Health QA inherits that breadth of coverage. The benchmark therefore spans internal medicine, paediatrics, surgery and trauma, obstetrics and gynaecology, oxygen therapy, blood products, referral, forensic medicine, and a COVID-19 annex.
| Guideline part | Approximate share |
|---|---|
| Internal Medicine & related | 26% |
| Paediatrics | 31% |
| Surgery & related | 12% |
| Obstetrics & Gynaecology | 10% |
| Principles of Oxygen Therapy | <1% |
| Management of Blood & Blood Products | 2% |
| Referral Framework | 1% |
| Forensic Medicine | 8% |
| COVID-19 annex | 3% |
The intended use-cases are evaluating LLMs as clinical decision support tools for Kenyan primary care, checking adherence to national standards of care, and stress-testing safety, clinical reasoning, and contextual adaptability rather than factual recall alone. This positioning distinguishes Alama Health QA from physician-centric, English-language, high-income benchmarks whose disease mix and regulatory assumptions may not match African deployment settings (Mutisya et al., 19 Jul 2025, Mutisya et al., 22 Jul 2025).
2. Guideline-to-benchmark pipeline
Because WHO SMART Guideline L1 machine-readable formats were not yet available for Kenya, the benchmark was constructed by converting the PDF guideline into structured text using OCR and parsing, chunking the text into semantically coherent units roughly corresponding to discrete recommendations or small sections, and attaching metadata such as source document and version, part or section title, and page number. The resulting knowledge base indexed 1,115 guideline chunks in a hybrid retrieval system with a BM25 lexical index and a vector similarity index (Mutisya et al., 19 Jul 2025).
The benchmark generation process is explicitly retrieval-augmented. A guideline topic is selected, relevant chunk(s) are retrieved, and a generator model is prompted to produce Kenyan healthcare training vignettes, multiple-choice questions, and rationale-based answers grounded in the retrieved text. Although the first proof-of-concept dataset generation effectively used the whole guideline document rather than selective retrieval, the retrieval layer is a core design element for later question-generation pipelines and for contextual model testing in a RAG setting (Mutisya et al., 19 Jul 2025).
Three generator models were tested—GPT-4O mini, Gemini Flash 2.0 Lite, and LLaMA-3.1 (8B)—and Gemini Flash 2.0 Lite was selected because it gave the best balance between faithfulness to guidelines and clinical plausibility. The benchmark was not produced as a purely synthetic corpus: Kenyan physicians co-created and refined the design through interviews and roundtables, and a blinded expert review process was used to assess clinical relevance, guideline alignment, clarity and completeness, distractor plausibility, and language and cultural appropriateness (Mutisya et al., 19 Jul 2025).
This pipeline is also presented as reusable methodology. A later comparative analysis describes Alama Health QA not only as a dataset but as a benchmark pipeline for creating medical LLM evaluations in African primary care, with metadata-linked questions that can be regenerated when guideline sections change (Mutisya et al., 22 Jul 2025).
3. Item structure, bilingual design, and annotation logic
Each Alama Health QA item typically contains a short clinical vignette of two or three sentences, a single four-option multiple-choice question, a marked correct option, and a model rationale explaining why that option follows from the guideline. Vignettes usually specify patient age and sex, presenting symptoms, and setting details such as a rural dispensary or a health centre in Kisumu; some include temporal context such as postpartum day. Distractors are clinically plausible but incorrect, often representing wrong regimens, wrong referral decisions, or contextually inappropriate interventions (Mutisya et al., 19 Jul 2025).
The benchmark is primarily in English for compatibility and ease of review, but a subset of around 10% was generated in Kiswahili, particularly for patient-counselling or advice-oriented items. These Swahili items were reviewed and refined by bilingual Kenyan clinicians to ensure correct medical meaning, natural locally appropriate phrasing, and cultural sensitivity. The methodology paper also notes a subsequent HIV benchmark in English and Swahili using the same pipeline, indicating active expansion of bilingual coverage (Mutisya et al., 19 Jul 2025).
Physician involvement extends beyond post hoc review. Kenyan clinicians asked that questions specify age, gender, location, and facility level to force context-aware reasoning; that each correct answer reference its guideline source; that the corpus avoid over-representing any single domain; and that multilingual support be included where appropriate. These requests were translated into the benchmark’s generation constraints. Review was conducted in a web-based interface, with items mixed blindly alongside other factual benchmarks so that licensed Kenyan doctors, as well as pharmacists and nurses, could score them without dataset-name bias (Mutisya et al., 19 Jul 2025).
A common misconception is to treat Alama Health QA as merely another MCQ set. Its item design is closer to a structured, regulator-aligned clinical scenario corpus: questions are intended to encode setting, facility constraints, and local policy, not simply abstract textbook knowledge.
4. Evaluation modules and metrics
Alama Health QA includes a core MCQ accuracy task, but its main technical distinctiveness lies in additional evaluation modules aimed at reasoning, vigilance, adaptation, and bias robustness. The base MCQ score is standard accuracy, computed as the fraction of correctly answered questions (Mutisya et al., 19 Jul 2025).
Beyond that base task, the benchmark defines several challenge modules:
| Module | What it tests | Core signal |
|---|---|---|
| Accuracy on QA items | Guideline-anchored factual correctness | MCQ accuracy |
| Decision Points | Stepwise guideline-consistent information gathering | Precision/recall over critical decision nodes, rescaled as DPS |
| Needle-in-the-Haystack | Detection of rare but decisive clues | Mean Needle Score |
| Reverse QA | Medically consistent patient or caregiver role-play | Persona score from consistency, completeness, style, and linguistic appropriateness |
| Geographic-Contextual Adaptation | Adaptation to locale and facility constraints | Context Adaptation Score |
| Cognitive-Bias Stress Test | Robustness to anchoring, premature closure, and confirmation bias | Composite CBST score and Bias Susceptibility Index |
The Decision Points module evaluates whether a model asks for the right additional history, examination findings, or tests before deciding, rather than jumping directly to diagnosis or treatment. Critical decision nodes are derived from guideline algorithms, and the Decision-Point Score is defined as an -score over asked versus critical nodes, rescaled from 0 to 10. The Needle-in-the-Haystack module measures whether a model can identify a rare but decisive clue embedded among distractors and then elevate the correct diagnosis or management. The Reverse QA module reverses the usual direction of evaluation by asking the model to act as a patient or caregiver consistent with a fact sheet, scoring it for consistency, completeness, style realism, and linguistic appropriateness (Mutisya et al., 19 Jul 2025).
The Geographic-Contextual Adaptation module tests whether recommendations change appropriately across settings such as Kisumu versus Johannesburg or Level 2 versus higher-resourced facilities. The Cognitive-Bias Stress Test uses two-stage vignettes to examine whether the model revises an initial diagnosis when later red-flag information appears, with separate components for anchor flexibility, contradiction recognition, breadth of differential, and action appropriateness (Mutisya et al., 19 Jul 2025).
This multi-module design is central to the benchmark’s identity. It operationalizes the claim that safe primary-care evaluation requires more than top-line accuracy, especially in resource-constrained and policy-sensitive settings.
5. Representativeness and comparative position
A later benchmarking study situates Alama Health QA within a broader audit of quantitative medical LLM benchmarks for African disease burdens. In that analysis, six benchmark families underwent harmonized semantic profiling and blinded expert rating, and Alama Health QA emerged as the most guideline-aligned and one of the most epidemiologically representative corpora in the comparison set (Mutisya et al., 22 Jul 2025).
In the profiled version, Alama Health QA contains 40,607 questions. It alone captured more than 40% of all neglected tropical disease term mentions across the six analyzed corpora, and it had the highest within-set frequencies for several major African disease burdens: malaria 7.74%, HIV 4.07%, and TB 5.15%. By contrast, the same comparative analysis reports near-zero or absent coverage of several African-priority conditions in widely used global benchmarks, including complete absence of sickle-cell disease in three mainstream datasets and complete absence of dengue in all four global benchmarks examined (Mutisya et al., 22 Jul 2025).
The same study reports that Alama Health QA has a sentence length of about 43 tokens, syntactic depth 7.9, Flesch–Kincaid Grade Level 25, and MTLD 75, described as the second-highest lexical-diversity value among the compared corpora. In blinded expert review, Alama received 4.7/5 for clinical relevance, 4.5/5 for clarity and completeness, and 4.4/5 for distractor plausibility, while also scoring highest for relevance and guideline alignment overall. Reviewers specifically praised frontline scenarios in areas such as child malnutrition, HIV, and community health (Mutisya et al., 22 Jul 2025).
This comparative evidence clarifies what Alama Health QA is intended to correct. It is not simply an Africanized analogue of USMLE-style benchmarking; it is a benchmark explicitly constructed to evaluate alignment with African disease burdens and with national regulatory guidance.
6. Uses, limitations, and extensions
The methodology paper presents Alama Health QA as a regulatory tool, a comparative benchmark, a blueprint for other countries and domains, and an educational resource. Ministries of Health and regulators can treat it as an exam that LLM-based health assistants must pass before deployment; researchers can compare frontier and smaller models with or without RAG; other countries can replicate the pipeline with their own national guidelines; and trainees can use the question set, explanations, and citations as a study resource for Kenyan primary-care guidance (Mutisya et al., 19 Jul 2025).
The benchmark’s limitations are also explicit. The initial version covers only Kenyan Level 2–3 care and only one national guideline volume. Non-English Kenyan languages beyond Kiswahili are not yet included. Some evaluation modules rely on composite scores whose weights are acknowledged as somewhat ad hoc. Multiple-choice format does not fully capture uncertain scenarios, shared decision-making, or more complex clinical interaction, and guideline-grounding inherits any incompleteness or obsolescence present in the source guideline itself (Mutisya et al., 19 Jul 2025).
The later comparative paper adds a broader caution: using non-African, exam-centric benchmarks to validate models for African deployment may overestimate safety and utility, ignore regulatory requirements, and underrepresent the conditions clinicians most often see. It therefore recommends Alama-type benchmarks as complements to, or substitutes for, Western exam sets when evaluating models for African health systems (Mutisya et al., 22 Jul 2025).
Planned extensions include disease-specific derivatives for HIV, malaria, sickle-cell disease, and viral hemorrhagic fevers, country-specific variants grounded in other Ministries of Health, broader multilingual expansion, and eventual multimodal additions. A Swahili variant already exists, and the benchmark is presented as a dynamic, updateable resource tied to guideline revision cycles rather than as a static corpus (Mutisya et al., 19 Jul 2025, Mutisya et al., 22 Jul 2025).