---
title: 'AfriMMLU: African MCQA Benchmark'
url: https://www.emergentmind.com/topics/afrimmlu-benchmark
type: topic
---

# AfriMMLU: African MCQA Benchmark

AfriMMLU is a multilingual multiple-choice question answering (MCQA) benchmark specifically designed to evaluate the language understanding and reasoning abilities of large language models (LLMs) across African languages. Serving as a subset of the IrokoBench suite, AfriMMLU encompasses a broad typological spectrum of languages and domains, systematically quantifying model performance disparities between high-resource and low-resource environments. It is characterized by rigorous human translation protocols, closed-form MCQA tasks, and diverse evaluation methodologies, addressing both linguistic and computational inequities in current NLP systems [2509.05486][2406.03368].

## 1. Dataset Composition and Coverage

AfriMMLU comprises human-translated MCQA items reflecting significant phylogenetic and regional diversity. The benchmark covers 17 languages—English, French, and 15 African languages classified under two major families: Afro-Asiatic (Amharic, Hausa, Oromo) and Niger-Congo, which further splits into Volta-Niger (Igbo, Yorùbá), Kwa (Twi, Ewe, Wolof), and Bantu (Swahili, Kinyarwanda, Luganda, chiShona, isiXhosa, isiZulu, Sesotho, Lingala).

Each language supports five culturally neutral domains: elementary mathematics, global facts, high-school geography, high-school microeconomics, and international law. The MCQA dataset consists of 608 questions per language, separated into 500-item test sets (100 per subject), 25-item train pools, and 83-item development sets for prompt calibration and few-shot evaluation [2406.03368].

Translations undergo a rigorous protocol: professional translators are supervised and remunerated conditional on two-stage review and quality control, including COMET QE and AfriCOMET metrics. For 13 languages with relevant QE coverage, sentence-level scores range between 0.7–1.0; deficiency in metric coverage for certain languages is resolved by manual checks rather than surrogate inference.

## 2. Task Formalism and Evaluation Protocols

The AfriMMLU task constructs a closed MCQA challenge. Each instance is comprised of a subject label, a question prompt, and four answer choices (A–D), formatted for unambiguous option selection. The evaluation criterion is prediction accuracy, defined as
\[
\text{Accuracy} = \frac{\#\text{(correctly chosen options)}}{\#\text{(total questions)}} \times 100\%.
\]

Three principal evaluation paradigms are adopted:
- **Zero-shot in-language:** direct prompting in the target language without any in-context examples.
- **Zero-shot translate-test:** MCQA prompts are first translated to English (via NLLB-200), and models respond in English.
- **Few-shot in-language (k=5):** exemplars are prepended in-context within the target language (only for “best-in-class” models).

This design prohibits fine-tuning or supervised adaptation on the evaluated languages, enforcing strict zero-shot and “few-shot” conditions to test generalization and transfer.

## 3. Model Coverage and Fertility Analysis

AfriMMLU supports comprehensive LLM evaluation, encompassing both open and proprietary models. Recent assessments feature ten major LLMs in [2509.05486], including:

- **Reasoning-capable models:** DeepSeek R1, DeepSeek V3, o1 (proprietary, tokenizer details undisclosed)
- **Non-reasoning models:** Sonnet-3.5, Aya23 35B, Gemini 1.5 Pro, Llama 3.1 405B, Phi4, GPT-4o, Pixtral 12B, Qwen 2.5 32B

A key structural variable is **fertility** (\(f\)), defined as the average number of tokens per word when text is tokenized by model \(m\):
\[
f_{m,\ell} = \frac{T_{m,\ell}}{W_\ell}
\]
where \(T_{m,\ell}\) is the total token count for model \(m\) on language \(\ell\) and \(W_\ell\) the word count. High fertility indicates excessive token fragmentation—more subword segments per word—common in morphologically complex, low-resource African languages.

The statistical relationship between fertility and model accuracy (\(a_{m,\ell}\)) is quantified using the Pearson correlation coefficient (\(r_m\)) and linear regressions of the form:
\[
a_{m,\ell} = \alpha_m + \beta_m f_{m,\ell} + \varepsilon_{m,\ell}
\]
In all cases, \(\beta_m < 0\): increased tokenization inefficiency (higher fertility) predicts lower MCQA accuracy. Across all subjects and models, regression slopes \(\beta\) fall in the range \(-0.08\) to \(-0.18\)—for example, Llama 3.1 405B on Macroeconomics reports \(\beta = -0.185\), and Qwen 2.5 32B on Geography yields \(\beta = -0.155\). With \(R^2\) as high as 0.48, fertility accounts for up to 48% of accuracy variance per model [2509.05486].

## 4. Accuracy and Cross-Linguistic Disparities

Extensive evaluation highlights a persistent performance gradient between high-resource (English, French) and low-resource (African) languages. 

Mean MCQA accuracy by model class and language resource tier is summarized as:

| Language Class        | Non-reasoning LLMs | Reasoning LLMs |
|----------------------|--------------------|---------------|
| English (high-resource) | ≈75%           | ≈75–80%       |
| French (mid-resource)   | ≈60%           | ≈60%          |
| African (low-resource)  | ≈50%           | ≈60%          |

Subject-specific analysis (e.g., on Global Facts) shows non-reasoning models achieving ≈55% on African languages versus 80% on English (25 percentage point gap); reasoning models improve African-language accuracy to ≈68% (narrowing the gap to 12 points).

Proprietary models consistently outperform open-weight counterparts. For instance, GPT-4o achieves 52.3% average accuracy on African languages in in-language settings, while the top-performing open model (Gemma-7B) reaches only 38.3%—approximately 58% of GPT-4o’s benchmark [2406.03368].

Translation to English raises the performance of large, English-centric models (e.g., LLaMa-3-70B) by as much as +13.3 pp. For Bantu staples such as Swahili and isiZulu, in-language accuracies cluster around 40–50%, while ultra-low-resource languages like Ewe, Lingala, Twi, and Wolof register 20–30%.

## 5. Economic Implications of Tokenization Inefficiency

Tokenization inefficiency imposes substantial computational and financial penalties—a phenomenon referred to as the “token tax.” Training cost for transformers scales as \(O(n^2)\), where \(n\) is sequence length. For languages with fertility twice that of English (\(f_\ell \approx 2 f_{\mathrm{EN}}\)), training costs quadruple:
\[
\mathrm{Cost}_\ell \propto (2n)^2 = 4n^2 \propto 4\ \mathrm{Cost}_{\mathrm{EN}}
\]
Empirical reporting indicates training Llama 3.1 405B on English costs \$105M, while training on a 2x-fertility language raises this to \$420M. Generation cost also doubles: producing 1M English-equivalent tokens with GPT-4o costs \$5–\$20, but \$10–\$40 for high-fertility languages, with inference latency increasing commensurately. This systemic cost amplification restricts access and advances for low-resource languages [2509.05486].

## 6. Recommendations and Future Directions

Benchmarking with AfriMMLU reveals structural deficits in both model architecture and ecosystem infrastructure. Several recommendations follow from these findings:

- **Morphologically aware tokenization:** Develop and adopt algorithms that better accommodate the subword complexity of African languages.
- **Efficient transformer variants:** Advance attention mechanisms with subquadratic or adaptive computational cost profiles.
- **Equitable pricing:** Revise commercial API pricing to reflect the inflated tokenization and associated costs for high-fertility languages.
- **Data augmentation:** Enrich pretraining corpora with diverse African textual domains (news, educational, synthetic QA).
- **Domain and cultural expansion:** Extend benchmarks like AfriMMLU to include locally salient domains (e.g., regional history, law).
- **Instruction tuning and prompt engineering:** Prioritize language-specific prompts and few-shot exemplars to mitigate native prompting penalties.
- **Avoidance of forced translation:** While translate-test protocols narrow performance gaps, sustainable progress depends on closing the in-language accuracy disparity, rather than reinforcing English-centric pipelines [2406.03368][2509.05486].

## 7. Significance and Broader Impact

AfriMMLU quantifies and elucidates the interdependence of representational, computational, and economic inequalities facing low-resource, morphologically complex languages under prevailing LLM paradigms. Its cross-linguistic, cross-model assessment surfaces both systematic “token tax” effects and the model scaling plateau, providing a reliable metric for future advances in equitable NLP system design. The benchmark establishes a high-precision, data-rich foundation for evaluating architectural, tokenization, and training innovations targeted at underrepresented linguistic communities [2509.05486][2406.03368].

Source: https://www.emergentmind.com/topics/afrimmlu-benchmark