---
title: 'AfroLM: Tailored LMs for African Languages'
url: https://www.emergentmind.com/topics/afrolm
type: topic
---

# AfroLM: Tailored LMs for African Languages

AfroLM refers to a family of pre-trained and adapted large language models (LLMs) and Transformer-based masked language models explicitly tailored for African languages. These models are designed to address the chronic underrepresentation of African languages in mainstream multilingual corpora and to provide SOTA (state-of-the-art) performance on a range of Natural Language Processing (NLP) tasks, including Language Identification (LID), Named Entity Recognition (NER), text classification, sentiment analysis, and cross-lingual knowledge-intensive reasoning. Several architectures and adaptation methods fall under the AfroLM designation, including models trained from scratch on African data, as well as variants adapted from strong multilingual baselines via self-active learning or multilingual adaptive fine-tuning.

## 1. Model Architectures and Variants

Several distinct model architectures are referred to as AfroLM in the literature, ranging from moderate-size masked language models to large LLMs:

- **AfroLM (Self-Active Learning Variant)**: Adopts a Transformer encoder architecture similar to XLM-RoBERTa. The “Large” variant uses 10 layers, hidden size of 768, 6 self-attention heads, 264M parameters, and a max sequence length of 256 tokens [2211.03263].
- **AfroLM for LID (XLM-R–Style Base)**: 12 Transformer encoder layers, 12 heads per layer, hidden size 768, 3,072-unit feed-forward, dropout 0.1, and ≈278M parameters. Vocabulary is ≈250K subwords via SentencePiece [2410.08728].
- **AfroLM (LLM Variant, "Lugha-Llama" Paradigm)**: Built on Llama-3.1-8B (32-layer causal Transformer, ~8B parameters), inheriting the standard decoder stack and existing multilingual embeddings. No architectural modifications or new adapters are introduced; subword vocabulary may be supplemented with African scripts for expanded coverage [2504.06536].

All models apply the masked language modeling (MLM) or standard causal language modeling objective during pre-training/fine-tuning.

## 2. Data Collection, Composition, and Pre-training Strategies

Coverage of African languages and careful corpus preparation are central to the AfroLM approach.

- **Multilingual Data Scope**: Training data covers up to 23 African languages, including but not limited to Hausa, Yoruba, Igbo, Somali, Malagasy, Kinyarwanda, isiXhosa, Sesotho, and others. Variants may also include continentally relevant languages such as English, French, Arabic, and Portuguese [2204.06487, 2211.03263, 2504.06536, 2410.08728].
- **Data Sourcing**: Major corpora include WURA (10B tokens), MasakhaNER, KinNews, mT5-mC4, news sources (VOA, BBC, Isolezwe), and web-scraped resources subjected to extensive cleaning.
- **Sampling and Balancing**: UniMax sampling with up-sampling (up to 4×) for rare languages ensures more equitable exposure [2504.06536].
- **Pre-training Procedures**:
  - **Self-Active Learning Loop**: In this protocol, the model alternates between MLM training, selection of unlabelled sentences, and pseudo-label generation, expanding training data by generating new variants, improving robustness to domain shift [2211.03263, 2410.08728].
  - **Multilingual Adaptive Fine-Tuning (MAFT)**: Continues pre-training a base PLM (e.g., XLM-R, AfriBERTa) on African monolingual corpora. Non-African-script subwords are pruned (especially effective on Latin/Ge’ez script languages) to create smaller, more focused vocabularies [2204.06487].
  - **Inclusion of High-Quality Educational Content**: Mixing web-classified educational English (FineWeb-Edu) or machine-translated educational texts (e.g., English to Swahili) with African-language core data substantially increases knowledge-intensive task performance [2504.06536].
  - **Tokenization**: SentencePiece models with shared 250K vocabulary; optional pruning to ~70K tokens for African scripts [2204.06487].

## 3. Training Regimens and Optimization

Distinct training regimens are employed for self-supervised pre-training, adaptation, and task-specific fine-tuning.

- **Optimization Hyperparameters**:
  - Self-active models: Adam optimizer (β₁=0.9, β₂=0.999), peak LR=1e–4, warmup/decay scheduling, 500,000 steps per round, effective batch size of 256 [2211.03263].
  - LID models: Fine-tuning on Vuk’zenzele + NCHLT splits, batch size 16, LR=2e–5, up to 20 epochs [2410.08728].
  - “Lugha-Llama” LLM: 10B tokens, 2,400 gradient steps, batch of 512×8,192 tokens, 1e–5 initial LR with linear warmup and cosine decay [2504.06536].

- **Supervised Fine-Tuning**: Conducted for NER (MasakhaNER), topic and sentiment classification (AG News, KINNEWS, YOSM, NaijaSenti), and LID.

- **Loss Functions**:
  - MLM: $\mathcal{L}_{\text{MLM}} = -\sum_{i \in M} \log P_\theta(x_i|X_{\setminus M})$
  - Causal LM: $\mathcal{L}(\theta) = -\sum_{t=1}^T \log p_\theta(w_t|w_{<t})$
  - Cross-entropy fine-tuning for classification: $L_\text{CE} = -\sum_{k=1}^K y_k \log \hat{y}_k$

## 4. Evaluation Benchmarks and Empirical Results

Systematic evaluation against strong baselines demonstrates AfroLM's efficacy across tasks.

| Task                  | AfroLM w/ self-active learning | AfriBERTa | mBERT   | XLM-R   | AfroXLMR | Reference      |
|-----------------------|---------------------|-----------|---------|---------|----------|----------------|
| NER (MasakhaNER v1)   | 80.13%              | 79.10%    | 71.55%  | 79.16%  | 81.90%   | [2211.03263]   |
| NER (MasakhaNER v2)   | 83.26%              | 81.31%    | 80.68%  | 83.09%  | 84.55%   | [2211.03263]   |
| LID (11 S. African)   | 97.4% F1            | 97.6%     | 96.7%   | 97.1%   | 97.7–98% | [2410.08728]   |
| News Classification   | (Hausa/Yoruba) 91.0/82.9% | 90.86/83.22% | — | —   | —        | [2211.03263]   |
| Sentiment (In/Cross)  | 85.4% / 68.7%       | 82.7% / 65.9% | — | —   | —        | [2211.03263]   |
| AfriMMLU (14 langs)   | 34%                  | —         | —       | —      | —        | [2504.06536]   |
| AfriQA (QA, Llama-3.1-8B base) | 34.2% (AfroLM) | —  | — | 20.1% (base) | — | [2504.06536]   |

- **AfroLM (LID task):** Macro F1 of 97.4% over 11 South African languages, outperforming mBERT, XLM-R base/large, RemBERT, and matching closely related Afri-centric PLMs such as AfriBERTa, Afro-XLM-R, and Serengeti. Off-the-shelf LID tools (CLD v3, AfroLID, OpenLID) lag substantially (35.7–75% F1) [2410.08728].
- **NER and Classification**: Consistently competitive or superior to AfriBERTa and XLM-R on MasakhaNER, KINNEWS, and YOSM, with a robust performance in cross-domain settings due to self-active data augmentation [2211.03263].
- **Lugha-Llama LLM AfroLM**:
  - On IrokoBench (AfriMMLU): +4 percentage point gain over Llama-3.1-8B base (from ~30% to ~34%).
  - On AfriQA: 34.2% (vs 20.1% for Llama-3.1-8B base, and 19–22% for other open-weight 8B models).
  - Gains on reasoning/mathematical tasks with math-data augmentation reach +6 points over the base [2504.06536].

## 5. Innovations: Active Learning, Adaptation, and Vocabulary Pruning

- **Self-Active Learning**: AfroLM is the first documented multilingual LM to employ self-active learning at pre-training scale for African languages, automatically generating pseudo-labeled sentence augmentations to diversify and densify low-resource corpora [2211.03263, 2410.08728].
- **MAFT vs. LAFT**: AfroLM’s Multilingual Adaptive Fine-Tuning approach retains cross-lingual transfer within one compact model, outperforming vanilla PLMs on NER, news, and sentiment by ≈+2–4 F1 points, and only trailing language-adaptive fine-tuned (LAFT) models by <0.5 F1 [2204.06487].
- **Non-African Script Pruning**: Reducing vocabulary size (from 250K to ~70K tokens) by pruning non-Latin/non-Ge’ez pieces yields smaller, more efficient models with minimal accuracy loss, though a small penalty is observed for non-Latin languages (e.g., Amharic, Arabic) [2204.06487].
- **Educational Content Translation**: Experiments demonstrate that machine-translated educational corpora (e.g., GPT-4o-translated English-to-Swahili) improve downstream performance over English-only data. The key driver is domain-relevant content quality in the target language rather than the source language [2504.06536].

## 6. Impact, Current Limitations, and Recommendations

AfroLM establishes a new benchmark for African language processing, particularly for low-resource and closely related language clusters.

- **Generalization**: AfroLM’s self-active learning yields heightened cross-domain robustness for tasks such as sentiment analysis (Twitter → Movies, e.g., 68.7% F1 vs 41.3% for no-AL model) [2211.03263].
- **LID Cluster Discrimination**: AfroLM offers strong discrimination among Nguni and Sotho–Tswana clusters, regions where previous PLMs struggled [2410.08728].
- **Parameter and Disk Efficiency**: MAFT enables a “single model for all African languages,” reducing the disk/storage requirement typically associated with LAFT, at minimal loss in downstream accuracy [2204.06487].
- **Recommendations**:
  1. Curate ≥1GB monolingual clean corpus per target language.
  2. Apply vocabulary pruning for script relevance if feasible.
  3. Utilize self-active learning or MAFT over all languages for maximal cross-lingual performance.
  4. Mix educational/encyclopedic (and if possible, translated) content for knowledge-centric tasks.
  5. Use balanced sampling (e.g., UniMax, up to 4× for rare tongues) to prevent dominance by high-resource languages.
  6. Retain some English data in pre-training to avoid catastrophic forgetting, but maximize in-language educational content [2211.03263, 2504.06536, 2204.06487].

Persistent limitations include moderate cross-domain performance drops (4–5 F1), especially on out-of-domain text; a lack of full per-language fine-grained reporting in some studies; and challenges with vocabulary coverage for scripts other than Latin or Ge’ez [2410.08728, 2204.06487].

## 7. Availability and Extensions

- **Model and Code Release**: Checkpoints, tokenizers, training pipelines, and processed corpora for all major AfroLM models are publicly released (e.g., https://github.com/bonaventuredossou/MLM_AL; HuggingFace Hub deployments for AfroXLMR variants) [2211.03263, 2204.06487].
- **Extensibility**: The underlying protocols (active learning, MAFT, curated translation) generalize to additional African language families and other low-resource regimes.
- **Future Directions**: Unexplored avenues include deeper model architectures, extension to sub-100M parameter “mini” LMs, improved tokenization strategies for non-Latin scripts, and systematic domain- and task-specific data augmentation [2410.08728, 2204.06487].

AfroLM sets a robust baseline for efficient, high-utility NLP in African languages, bridging the capacity gap between under-resourced LID or task models and massive, general-purpose PLMs.

Source: https://www.emergentmind.com/topics/afrolm