AfriMMLU: African Multilingual QA Benchmark

• AfriMMLU is a human-translated MMLU-style benchmark designed to assess multilingual multiple-choice reasoning in African languages.
• It features 608 question-answer pairs across 5 subjects, ensuring feasibility and cultural relevance through carefully selected content.
• Evaluations reveal significant performance gaps between African and high-resource languages, highlighting the need for targeted language model specialization.

Searching arXiv for papers on AfriMMLU and closely related benchmarks. Tool call: arxiv_search({"query":"AfriMMLU IrokoBench African languages benchmark", "max_results": 10, "sort_by": "submittedDate"}) AfriMMLU is the multi-choice knowledge-based question answering component of IrokoBench, a human-translated benchmark introduced to evaluate LLMs on African languages beyond the basic classification tasks that have historically dominated the area. It is an African-language adaptation of MMLU-style evaluation, using four-option multiple-choice questions to probe broad factual and academic knowledge under multilingual prompting conditions. In the original IrokoBench presentation, AfriMMLU is not a translation of all MMLU, but a curated subset designed to remain feasible for human translation and less distorted by cultural or educational mismatch; later papers reuse it both as a direct model-comparison benchmark and as a substrate for studying multilingual transfer, prompting, continued pre-training, and tokenization effects (Adelani et al., 2024).

1. Origin within IrokoBench

AfriMMLU was introduced as one of the three tasks in IrokoBench, alongside AfriXNLI and AfriMGSM. Within that suite, it is the benchmark’s test of multilingual knowledge-based multiple-choice reasoning. The source task is MMLU, but the benchmark designers explicitly rejected a naïve full translation of MMLU into African languages. They state that many original MMLU subjects are highly technical, culturally specific, or taught in African contexts primarily through English or French, making faithful translation difficult and potentially unnatural. For that reason, they selected five subjects “that we believe present less tendency to be culturally unbiased and that are not too difficult to translate” (Adelani et al., 2024).

This design choice is central to the benchmark’s identity. AfriMMLU is not intended to reproduce the entire MMLU subject inventory; it is intended to preserve the spirit of MMLU-style knowledge evaluation while constraining the benchmark to subjects that can be rendered more reliably across low-resource African languages. In the original IrokoBench interpretation, AfriMMLU occupies a middle position in task difficulty: AfriMGSM is described as the hardest task, followed by AfriMMLU, then AfriXNLI. That placement makes AfriMMLU the suite’s primary benchmark for broad knowledge-based reasoning that is more demanding than NLI but less specialized than mathematical word-problem solving (Adelani et al., 2024).

2. Benchmark design, subjects, and language coverage

In IrokoBench, AfriMMLU preserves the core MMLU-style format: each instance is a multiple-choice question with four options and a single correct answer. The benchmark consists of 608 translated question-answer pairs in total, with 500 instances in the test split and 100 questions per subject. The five subjects are elementary mathematics, high-school geography, international law, global facts, and high-school microeconomics. The benchmark maps these into broader categories as follows: high-school geography and high-school microeconomics under social sciences, elementary mathematics under STEM, international law under humanities, and global facts under other (Adelani et al., 2024).

The original paper reports AfriMMLU results for English, French, and 16 African languages: Amharic, Ewe, Hausa, Igbo, Kinyarwanda, Lingala, Luganda, Oromo, chiShona, Sesotho, Swahili, Twi, Wolof, isiXhosa, Yorùbá, and isiZulu. The benchmark was created through human translation. Language coordinators were recruited for each of the 16 African languages and for French, and those coordinators recruited professional translators. For Ewe, Lingala, and Wolof, translation proceeded from French rather than directly from English because those languages are in Francophone regions; the authors also note that those translators often understood both English and French and could cross-check when needed. Human review by language coordinators was part of the quality-control process, and the benchmark release emphasizes sentence-level comparability across languages even while acknowledging that the data remain translations rather than native-origin compositions (Adelani et al., 2024).

Later papers operationalize the language inventory somewhat differently. The AI Language Proficiency Monitor includes AfriMMLU as a human-translated MMLU-style dataset covering 17 languages and cites IrokoBench as the source of a benchmark for “17 typologically-diverse low-resource African languages.” In that system, AfriMMLU is one of the datasets used for the “Q&A: Multitask Language Understanding” task family (Pomerenke et al., 11 Jul 2025). By contrast, “The Token Tax” reports AfriMMLU as 9,000 MCQA items over 5 subjects and 16 African languages in its main analysis, while also noting that a broader figure aggregates 17 combined African languages and excludes Amharic from the mixed-effects analysis because it “shares less similarity with the remaining languages.” This suggests that later reuse of AfriMMLU often reflects benchmark subsets, row restrictions, or analysis-specific exclusions rather than a single immutable reporting convention (Lundin et al., 5 Sep 2025).

3. Evaluation protocols and metrics

AfriMMLU does not have a single invariant evaluation protocol across the literature. In the original IrokoBench paper, the benchmark is evaluated in zero-shot, few-shot, and translate-test settings. The core metric is option prediction accuracy. Open models are scored with the EleutherAI LM Evaluation Harness using log-likelihood over answer options, while closed models are evaluated through a manually created verbalizer because API access does not expose token log probabilities. The benchmark prompt is a subject-conditioned multiple-choice template, and the few-shot configuration for AfriMMLU uses 5-shot evaluation. Translate-test is implemented by automatically translating the test instances into English with NLLB-200 (3.3B) before evaluation (Adelani et al., 2024).

Subsequent work varies this setup substantially. InkubaLM uses the EleutherAI LM Evaluation Harness, evaluates AfriMMLU in zero-shot, uses prompts in the target African language, follows the IrokoBench prompt template, and reports F1 scores rather than accuracy; its AfriMMLU table covers Swahili, Hausa, Yoruba, isiXhosa, and isiZulu (Tonja et al., 2024). Lugha-Llama also evaluates with the EleutherAI LM Evaluation Harness, but uses 5-shot prompting and English prompt templates for AfriMMLU because IrokoBench does not provide native-language prompt templates for that task in the setup they follow; its reported numbers are interpreted as multiple-choice accuracy in percent, with an average over the 16 low-resource African languages that excludes English and French, denoted as $\mathrm{Avg}^{\dagger}$ (Buzaaba et al., 9 Apr 2025).

The AI Language Proficiency Monitor further standardizes the task. It states that all leaderboard evaluations are conducted in a few-shot manner, avoids translated long-form instructions, uses minimal language-agnostic prompts together with few-shot prompting, converts all classification and question-answering tasks into a multiple-choice format with one correct answer, evaluates them using accuracy, and then incorporates them into a min-max-normalized aggregate Language Proficiency Score (Pomerenke et al., 11 Jul 2025). AfriqueLLM, in turn, reports 5-shot accuracy through the AfroBench lm-eval task definitions, with deterministic decoding and either Hugging Face or vLLM backends depending on the experiment (Yu et al., 10 Jan 2026). “The Token Tax” uses an explicitly constrained answer format—Final Answer: X, where $X \in \{A,B,C,D\}$—and standard accuracy on MCQA (Lundin et al., 5 Sep 2025).

A recurring technical point is that several papers do not provide a formal AfriMMLU-specific scoring equation, macro/micro-averaging convention, or benchmark-specific mathematical loss. The original benchmark paper gives accuracy and option prediction accuracy; InkubaLM reports F1; the AI Language Proficiency Monitor contributes AfriMMLU to a broader normalized aggregate score. As a result, reported AfriMMLU numbers are comparable only within a paper’s stated protocol, prompt regime, language subset, and metric.

4. Reported model results and performance trends

The original IrokoBench results establish AfriMMLU as a difficult multilingual benchmark with a large gap between African languages and high-resource reference languages. On the African-language average in in-language evaluation, GPT-4o is reported at 52.3, GPT-4-Turbo at 47.0, Claude Opus at 43.0, LLaMa 3 70B at 34.1, Aya-101 at 30.3, and LLaMa 3 8B at 27.5. Translate-test leaves GPT-4o unchanged at 52.3, leaves GPT-4-Turbo at 47.0, raises Claude Opus to 47.6, raises LLaMa 3 70B to 47.4, raises LLaMa 3 8B to 34.4, and raises Aya-101 to 31.1. The paper also reports that in-language prompting is better on 10 of the 16 African languages evaluated, even though English-centric models such as LLaMa 3 70B benefit strongly from translate-test on average (Adelani et al., 2024).

Later model-adaptation papers use AfriMMLU to show that targeted African-language specialization can materially improve performance. InkubaLM-0.4B reports AfriMMLU F1 scores of 25.0 for Swahili, 29.4 for Hausa, 24.8 for Yoruba, 27.4 for isiXhosa, and 24.2 for isiZulu, with an average of 26.16. In that table, the model is below Gemma-7B at 30.28 and LLaMa 3-8B at 28.12, but above BLOOMZ-7B at 24.2, lola_v1-7.4B at 23.0, MobiLlama-1B at 22.28, and SmolLM-1.7B at 21.88. The paper highlights the Hausa score of 29.4 as essentially tied with the strongest systems and the isiXhosa score of 27.4 as the best score in the table (Tonja et al., 2024).

Lugha-Llama uses AfriMMLU as one of its main knowledge-intensive evaluations and reports an African-language $\mathrm{Avg}^{\dagger}$ of 30.0 for Llama-3.1-8B, 33.2 for Lugha-Llama, 34.3 for Lugha-Llama-edu, and 33.0 for Lugha-Llama-math. The strongest model, Lugha-Llama-edu, also exceeds aya-101 13B at 31.6, LLaMAX3 8B at 29.7, AfroLlama-V1 8B at 25.3, AfriInstruct 7B at 26.7, and InkubaLM 0.4B at 24.8. The paper attributes especially large gains to languages such as Igbo, Hausa, Shona, Sesotho, Xhosa, and Zulu, while noting that gains are smaller or absent in some languages not covered by WURA (Buzaaba et al., 9 Apr 2025).

AfriqueLLM places AfriMMLU among its central reasoning and knowledge evaluations. In its main task-level comparison, AfriMMLU scores are 32.27 for Llama 3.1 8B, 33.89 for Gemma 3 4B, 48.76 for Gemma 3 12B, 36.56 for Qwen 3 8B, and 39.66 for Qwen 3 14B; after continued pre-training, the corresponding AfriqueLLM models reach 36.57, 36.73, 49.47, 46.91, and 52.22. The table also includes Gemma 3 27B at 55.47. These results are used to support the paper’s claim that data composition is the primary driver of continued-pretraining gains and that architecture can dominate size when comparing across families (Yu et al., 10 Jan 2026).

Study	Setup	Selected reported result
IrokoBench	In-language / translate-test accuracy	GPT-4o 52.3; best open in-language LLaMa 3 70B 34.1; translate-test LLaMa 3 70B 47.4
InkubaLM	Zero-shot native-prompt F1 on 5 languages	InkubaLM-0.4B AVG 26.16; isiXhosa 27.4
Lugha-Llama	5-shot $\mathrm{Avg}^{\dagger}$ over 16 African languages	Lugha-Llama-edu 34.3; base Llama-3.1-8B 30.0
AfriqueLLM	5-shot accuracy after CPT	AfriqueQwen-14B 52.22; Gemma 3 27B 55.47

Taken together, these results show two stable empirical regularities. First, AfriMMLU exposes a persistent performance gap between English/French and low-resource African languages. Second, careful specialization or continued pre-training can reduce that gap substantially, although later open models still do not uniformly erase the advantage of the strongest proprietary or larger systems.

5. AfriMMLU as a diagnostic substrate

AfriMMLU is not used only for leaderboard-style ranking. The AI Language Proficiency Monitor integrates it as one of six MMLU-family datasets inside its multilingual benchmark system. In that framework, AfriMMLU is preferred where available because the system explicitly prioritizes human-translated datasets over machine-translated alternatives. A key harmonization decision is that the Masakhane datasets, including AfriMMLU, cover only a subset of the original English datasets, so the monitor restricts all corresponding datasets to exactly the same rows as the Masakhane subsets to ensure that the combined dataset is parallel. AfriMMLU therefore functions there as a human-translated African-language anchor for the broader MMLU-style task family rather than as an isolated benchmark (Pomerenke et al., 11 Jul 2025).

“The Token Tax” uses AfriMMLU differently: as the core evaluation substrate for studying tokenization inefficiency. In that paper, AfriMMLU is treated as a multilingual MCQA benchmark with 9,000 items, 5 subjects, and 16 African languages in the main analysis. The study defines fertility as

$F = \frac{T}{W},$

where $T$ is token count and $W$ is word count, and then regresses AfriMMLU accuracy on fertility by model and subject. Every reported slope is negative, the main text summarizes the slope range as approximately $-0.08$ to $-0.18$, and fertility is said to explain 20–50% of the variance in accuracy. The same paper reports that African languages trail English by about 25 accuracy points on average, with French generally intermediate, and that reasoning-oriented models such as DeepSeek and o1 reduce but do not eliminate that gap (Lundin et al., 5 Sep 2025).

A narrower derivative use appears in “Improving Multilingual Math Reasoning for African Languages,” which does not evaluate full AfriMMLU but reports an AfriMMLU Math Subset. There, Llama 3.1 8B Instruct scores 45.0 for Yoruba, 38.0 for Igbo, 44.0 for Hausa, 62.0 for Swahili, and 37.0 for Zulu, while a 30k synthetic SFT variant reaches 46.0, 51.0, 52.0, 53.0, and 51.0 respectively. The same paper treats these numbers cautiously and emphasizes that its main contribution concerns multilingual math reasoning rather than full MMLU-style subject coverage (Ogundepo et al., 26 May 2025).

These later uses show that AfriMMLU now serves at least three roles: as a benchmark for direct multilingual model comparison, as a component in broader multilingual monitoring systems, and as an empirical substrate for studying structural causes of performance variation such as tokenization inefficiency and training-data composition.

6. Limitations, controversies, and interpretive caveats

Several limitations recur across the literature. The most basic is that AfriMMLU is a translated benchmark, not a benchmark originally authored in the target languages. The IrokoBench paper explicitly notes that “translationese effects” may remain and that it would have been preferable for examples to be originally generated in native African languages. The benchmark also uses only five subjects rather than the full MMLU inventory, precisely because many original MMLU subjects are difficult to translate naturally and may be culturally or educationally misaligned in African-language settings (Adelani et al., 2024).

A related issue is cultural and linguistic bias. Lugha-Llama explicitly notes that MMLU may contain Western- and US-centric knowledge, that translated MMLU can suffer from cultural bias, and that some questions may preserve mathematical symbols and entity names across languages, potentially allowing partial answerability from surface cues. The AI Language Proficiency Monitor cites GlobalMMLU as prior evidence that translated MMLU can suffer from cultural bias and translation artefacts, but it does not claim to correct those biases directly; instead, it prefers human-translated resources such as AfriMMLU when possible (Buzaaba et al., 9 Apr 2025, Pomerenke et al., 11 Jul 2025).

Cross-paper comparability is also limited. InkubaLM reports AfriMMLU with zero-shot native prompts and F1 over five languages, and does not report standard deviations, confidence intervals, or significance tests; the paper explicitly warns against over-interpreting small differences such as 27.4 versus 27.0 on isiXhosa or 25.0 versus 26.0 on Swahili (Tonja et al., 2024). Other papers use 5-shot accuracy, English prompt templates, minimal language-agnostic prompting, or strict answer-format constraints. The AI Language Proficiency Monitor adds another source of instability by sampling only 10 instances per combination of model, task, and language because of compute limits, which it identifies as a major limitation for stable per-language estimation (Pomerenke et al., 11 Jul 2025).

Contamination and overlap remain underexplored. The original IrokoBench paper does not present a benchmark-contamination audit for AfriMMLU. InkubaLM likewise does not claim a decontamination pipeline or assess whether AfriMMLU overlaps with its own or baseline training data. The AI Language Proficiency Monitor includes an operational safeguard through OpenRouter endpoints that do not allow providers to train on user data, but this is a benchmarking-pipeline safeguard rather than a dataset-level memorization audit (Adelani et al., 2024, Tonja et al., 2024, Pomerenke et al., 11 Jul 2025).

For these reasons, AfriMMLU is best understood as a high-value but bounded benchmark: a human-translated, multiple-choice, knowledge-based African-language adaptation of MMLU that has become central to evaluating multilingual LLMs for low-resource African languages, while still carrying the constraints of translation, subject selection, protocol heterogeneity, and incomplete contamination analysis.