AfriMMLU: African MCQA Benchmark

• The paper introduces AfriMMLU to evaluate LLM language reasoning across 17 African languages using closed-form MCQA tasks.
• It employs rigorous human translation protocols and diverse evaluation methods including zero-shot, translate-test, and few-shot settings.
• Findings show that increased tokenization fertility in low-resource languages triggers computational penalties and lowers LLM accuracy.

AfriMMLU is a multilingual multiple-choice question answering (MCQA) benchmark specifically designed to evaluate the language understanding and reasoning abilities of LLMs across African languages. Serving as a subset of the IrokoBench suite, AfriMMLU encompasses a broad typological spectrum of languages and domains, systematically quantifying model performance disparities between high-resource and low-resource environments. It is characterized by rigorous human translation protocols, closed-form MCQA tasks, and diverse evaluation methodologies, addressing both linguistic and computational inequities in current NLP systems (Lundin et al., 5 Sep 2025, Adelani et al., 2024).

1. Dataset Composition and Coverage

AfriMMLU comprises human-translated MCQA items reflecting significant phylogenetic and regional diversity. The benchmark covers 17 languages—English, French, and 15 African languages classified under two major families: Afro-Asiatic (Amharic, Hausa, Oromo) and Niger-Congo, which further splits into Volta-Niger (Igbo, Yorùbá), Kwa (Twi, Ewe, Wolof), and Bantu (Swahili, Kinyarwanda, Luganda, chiShona, isiXhosa, isiZulu, Sesotho, Lingala).

Each language supports five culturally neutral domains: elementary mathematics, global facts, high-school geography, high-school microeconomics, and international law. The MCQA dataset consists of 608 questions per language, separated into 500-item test sets (100 per subject), 25-item train pools, and 83-item development sets for prompt calibration and few-shot evaluation (Adelani et al., 2024).

Translations undergo a rigorous protocol: professional translators are supervised and remunerated conditional on two-stage review and quality control, including COMET QE and AfriCOMET metrics. For 13 languages with relevant QE coverage, sentence-level scores range between 0.7–1.0; deficiency in metric coverage for certain languages is resolved by manual checks rather than surrogate inference.

2. Task Formalism and Evaluation Protocols

The AfriMMLU task constructs a closed MCQA challenge. Each instance is comprised of a subject label, a question prompt, and four answer choices (A–D), formatted for unambiguous option selection. The evaluation criterion is prediction accuracy, defined as

$$\text{Accuracy} = \frac{\#\text{(correctly chosen options)}}{\#\text{(total questions)}} \times 100\%.$$

Three principal evaluation paradigms are adopted:

• Zero-shot in-language: direct prompting in the target language without any in-context examples.
• Zero-shot translate-test: MCQA prompts are first translated to English (via NLLB-200), and models respond in English.
• Few-shot in-language (k=5): exemplars are prepended in-context within the target language (only for “best-in-class” models).

This design prohibits fine-tuning or supervised adaptation on the evaluated languages, enforcing strict zero-shot and “few-shot” conditions to test generalization and transfer.

3. Model Coverage and Fertility Analysis

AfriMMLU supports comprehensive LLM evaluation, encompassing both open and proprietary models. Recent assessments feature ten major LLMs in (Lundin et al., 5 Sep 2025), including:

• Reasoning-capable models: DeepSeek R1, DeepSeek V3, o1 (proprietary, tokenizer details undisclosed)
• Non-reasoning models: Sonnet-3.5, Aya23 35B, Gemini 1.5 Pro, Llama 3.1 405B, Phi4, GPT-4o, Pixtral 12B, Qwen 2.5 32B

A key structural variable is fertility ($f$), defined as the average number of tokens per word when text is tokenized by model $m$: $f_{m,\ell} = \frac{T_{m,\ell}}{W_\ell}$ where $T_{m,\ell}$ is the total token count for model $m$ on language $\ell$ and $W_\ell$ the word count. High fertility indicates excessive token fragmentation—more subword segments per word—common in morphologically complex, low-resource African languages.

The statistical relationship between fertility and model accuracy ($a_{m,\ell}$) is quantified using the Pearson correlation coefficient ($r_m$) and linear regressions of the form: $f$0 In all cases, $f$1: increased tokenization inefficiency (higher fertility) predicts lower MCQA accuracy. Across all subjects and models, regression slopes $f$2 fall in the range $f$3 to $f$4—for example, Llama 3.1 405B on Macroeconomics reports $f$5, and Qwen 2.5 32B on Geography yields $f$6. With $f$7 as high as 0.48, fertility accounts for up to 48% of accuracy variance per model (Lundin et al., 5 Sep 2025).

4. Accuracy and Cross-Linguistic Disparities

Extensive evaluation highlights a persistent performance gradient between high-resource (English, French) and low-resource (African) languages.

Mean MCQA accuracy by model class and language resource tier is summarized as:

Language Class	Non-reasoning LLMs	Reasoning LLMs
English (high-resource)	≈75%	≈75–80%
French (mid-resource)	≈60%	≈60%
African (low-resource)	≈50%	≈60%

Subject-specific analysis (e.g., on Global Facts) shows non-reasoning models achieving ≈55% on African languages versus 80% on English (25 percentage point gap); reasoning models improve African-language accuracy to ≈68% (narrowing the gap to 12 points).

Proprietary models consistently outperform open-weight counterparts. For instance, GPT-4o achieves 52.3% average accuracy on African languages in in-language settings, while the top-performing open model (Gemma-7B) reaches only 38.3%—approximately 58% of GPT-4o’s benchmark (Adelani et al., 2024).

Translation to English raises the performance of large, English-centric models (e.g., LLaMa-3-70B) by as much as +13.3 pp. For Bantu staples such as Swahili and isiZulu, in-language accuracies cluster around 40–50%, while ultra-low-resource languages like Ewe, Lingala, Twi, and Wolof register 20–30%.

5. Economic Implications of Tokenization Inefficiency

Tokenization inefficiency imposes substantial computational and financial penalties—a phenomenon referred to as the “token tax.” Training cost for transformers scales as $f$8, where $f$9 is sequence length. For languages with fertility twice that of English ($m$0), training costs quadruple: $m$1 Empirical reporting indicates training Llama 3.1 405B on English costs \$m$2420M. Generation cost also doubles: producing 1M English-equivalent tokens with GPT-4o costs \$m$320, but \$m$440 for high-fertility languages, with inference latency increasing commensurately. This systemic cost amplification restricts access and advances for low-resource languages (Lundin et al., 5 Sep 2025).

6. Recommendations and Future Directions

Benchmarking with AfriMMLU reveals structural deficits in both model architecture and ecosystem infrastructure. Several recommendations follow from these findings:

• Morphologically aware tokenization: Develop and adopt algorithms that better accommodate the subword complexity of African languages.
• Efficient transformer variants: Advance attention mechanisms with subquadratic or adaptive computational cost profiles.
• Equitable pricing: Revise commercial API pricing to reflect the inflated tokenization and associated costs for high-fertility languages.
• Data augmentation: Enrich pretraining corpora with diverse African textual domains (news, educational, synthetic QA).
• Domain and cultural expansion: Extend benchmarks like AfriMMLU to include locally salient domains (e.g., regional history, law).
• Instruction tuning and prompt engineering: Prioritize language-specific prompts and few-shot exemplars to mitigate native prompting penalties.
• Avoidance of forced translation: While translate-test protocols narrow performance gaps, sustainable progress depends on closing the in-language accuracy disparity, rather than reinforcing English-centric pipelines (Adelani et al., 2024, Lundin et al., 5 Sep 2025).

7. Significance and Broader Impact

AfriMMLU quantifies and elucidates the interdependence of representational, computational, and economic inequalities facing low-resource, morphologically complex languages under prevailing LLM paradigms. Its cross-linguistic, cross-model assessment surfaces both systematic “token tax” effects and the model scaling plateau, providing a reliable metric for future advances in equitable NLP system design. The benchmark establishes a high-precision, data-rich foundation for evaluating architectural, tokenization, and training innovations targeted at underrepresented linguistic communities (Lundin et al., 5 Sep 2025, Adelani et al., 2024).