AfroLM: Tailored LMs for African Languages

• AfroLM is a family of pre-trained and adapted language models for African languages, leveraging self-active learning and multilingual fine-tuning.
• It incorporates diverse architectures—from masked language models to causal LLMs—with optimized data sampling, vocabulary pruning, and adaptive pre-training strategies.
• Empirical benchmarks show AfroLM consistently outperforms standard models on tasks like NER, LID, and sentiment analysis, setting a new baseline for African NLP.

AfroLM refers to a family of pre-trained and adapted LLMs and Transformer-based masked LLMs explicitly tailored for African languages. These models are designed to address the chronic underrepresentation of African languages in mainstream multilingual corpora and to provide SOTA (state-of-the-art) performance on a range of NLP tasks, including Language Identification (LID), Named Entity Recognition (NER), text classification, sentiment analysis, and cross-lingual knowledge-intensive reasoning. Several architectures and adaptation methods fall under the AfroLM designation, including models trained from scratch on African data, as well as variants adapted from strong multilingual baselines via self-active learning or multilingual adaptive fine-tuning.

1. Model Architectures and Variants

Several distinct model architectures are referred to as AfroLM in the literature, ranging from moderate-size masked LLMs to large LLMs:

• AfroLM (Self-Active Learning Variant): Adopts a Transformer encoder architecture similar to XLM-RoBERTa. The “Large” variant uses 10 layers, hidden size of 768, 6 self-attention heads, 264M parameters, and a max sequence length of 256 tokens (Dossou et al., 2022).
• AfroLM for LID (XLM-R–Style Base): 12 Transformer encoder layers, 12 heads per layer, hidden size 768, 3,072-unit feed-forward, dropout 0.1, and ≈278M parameters. Vocabulary is ≈250K subwords via SentencePiece (Sindane et al., 2024).
• AfroLM (LLM Variant, "Lugha-Llama" Paradigm): Built on Llama-3.1-8B (32-layer causal Transformer, ~8B parameters), inheriting the standard decoder stack and existing multilingual embeddings. No architectural modifications or new adapters are introduced; subword vocabulary may be supplemented with African scripts for expanded coverage (Buzaaba et al., 9 Apr 2025).

All models apply the masked language modeling (MLM) or standard causal language modeling objective during pre-training/fine-tuning.

2. Data Collection, Composition, and Pre-training Strategies

Coverage of African languages and careful corpus preparation are central to the AfroLM approach.

• Multilingual Data Scope: Training data covers up to 23 African languages, including but not limited to Hausa, Yoruba, Igbo, Somali, Malagasy, Kinyarwanda, isiXhosa, Sesotho, and others. Variants may also include continentally relevant languages such as English, French, Arabic, and Portuguese (Alabi et al., 2022, Dossou et al., 2022, Buzaaba et al., 9 Apr 2025, Sindane et al., 2024).
• Data Sourcing: Major corpora include WURA (10B tokens), MasakhaNER, KinNews, mT5-mC4, news sources (VOA, BBC, Isolezwe), and web-scraped resources subjected to extensive cleaning.
• Sampling and Balancing: UniMax sampling with up-sampling (up to 4×) for rare languages ensures more equitable exposure (Buzaaba et al., 9 Apr 2025).
• Pre-training Procedures:
  • Self-Active Learning Loop: In this protocol, the model alternates between MLM training, selection of unlabelled sentences, and pseudo-label generation, expanding training data by generating new variants, improving robustness to domain shift (Dossou et al., 2022, Sindane et al., 2024).
  • Multilingual Adaptive Fine-Tuning (MAFT): Continues pre-training a base PLM (e.g., XLM-R, AfriBERTa) on African monolingual corpora. Non-African-script subwords are pruned (especially effective on Latin/Ge’ez script languages) to create smaller, more focused vocabularies (Alabi et al., 2022).
  • Inclusion of High-Quality Educational Content: Mixing web-classified educational English (FineWeb-Edu) or machine-translated educational texts (e.g., English to Swahili) with African-language core data substantially increases knowledge-intensive task performance (Buzaaba et al., 9 Apr 2025).
  • Tokenization: SentencePiece models with shared 250K vocabulary; optional pruning to ~70K tokens for African scripts (Alabi et al., 2022).

3. Training Regimens and Optimization

Distinct training regimens are employed for self-supervised pre-training, adaptation, and task-specific fine-tuning.

• Optimization Hyperparameters:
  • Self-active models: Adam optimizer (β₁=0.9, β₂=0.999), peak LR=1e–4, warmup/decay scheduling, 500,000 steps per round, effective batch size of 256 (Dossou et al., 2022).
  • LID models: Fine-tuning on Vuk’zenzele + NCHLT splits, batch size 16, LR=2e–5, up to 20 epochs (Sindane et al., 2024).
  • “Lugha-Llama” LLM: 10B tokens, 2,400 gradient steps, batch of 512×8,192 tokens, 1e–5 initial LR with linear warmup and cosine decay (Buzaaba et al., 9 Apr 2025).
• Supervised Fine-Tuning: Conducted for NER (MasakhaNER), topic and sentiment classification (AG News, KINNEWS, YOSM, NaijaSenti), and LID.
• Loss Functions:
  • MLM: $$\mathcal{L}_{\text{MLM}} = -\sum_{i \in M} \log P_\theta(x_i|X_{\setminus M})$$
  • Causal LM: $$\mathcal{L}(\theta) = -\sum_{t=1}^T \log p_\theta(w_t|w_{<t})$$
  • Cross-entropy fine-tuning for classification: $L_\text{CE} = -\sum_{k=1}^K y_k \log \hat{y}_k$

4. Evaluation Benchmarks and Empirical Results

Systematic evaluation against strong baselines demonstrates AfroLM's efficacy across tasks.

Task	AfroLM w/ self-active learning	AfriBERTa	mBERT	XLM-R	AfroXLMR	Reference
NER (MasakhaNER v1)	80.13%	79.10%	71.55%	79.16%	81.90%	(Dossou et al., 2022)
NER (MasakhaNER v2)	83.26%	81.31%	80.68%	83.09%	84.55%	(Dossou et al., 2022)
LID (11 S. African)	97.4% F1	97.6%	96.7%	97.1%	97.7–98%	(Sindane et al., 2024)
News Classification	(Hausa/Yoruba) 91.0/82.9%	90.86/83.22%	—	—	—	(Dossou et al., 2022)
Sentiment (In/Cross)	85.4% / 68.7%	82.7% / 65.9%	—	—	—	(Dossou et al., 2022)
AfriMMLU (14 langs)	34%	—	—	—	—	(Buzaaba et al., 9 Apr 2025)
AfriQA (QA, Llama-3.1-8B base)	34.2% (AfroLM)	—	—	20.1% (base)	—	(Buzaaba et al., 9 Apr 2025)

• AfroLM (LID task): Macro F1 of 97.4% over 11 South African languages, outperforming mBERT, XLM-R base/large, RemBERT, and matching closely related Afri-centric PLMs such as AfriBERTa, Afro-XLM-R, and Serengeti. Off-the-shelf LID tools (CLD v3, AfroLID, OpenLID) lag substantially (35.7–75% F1) (Sindane et al., 2024).
• NER and Classification: Consistently competitive or superior to AfriBERTa and XLM-R on MasakhaNER, KINNEWS, and YOSM, with a robust performance in cross-domain settings due to self-active data augmentation (Dossou et al., 2022).
• Lugha-Llama LLM AfroLM:
  • On IrokoBench (AfriMMLU): +4 percentage point gain over Llama-3.1-8B base (from ~30% to ~34%).
  • On AfriQA: 34.2% (vs 20.1% for Llama-3.1-8B base, and 19–22% for other open-weight 8B models).
  • Gains on reasoning/mathematical tasks with math-data augmentation reach +6 points over the base (Buzaaba et al., 9 Apr 2025).

5. Innovations: Active Learning, Adaptation, and Vocabulary Pruning

• Self-Active Learning: AfroLM is the first documented multilingual LM to employ self-active learning at pre-training scale for African languages, automatically generating pseudo-labeled sentence augmentations to diversify and densify low-resource corpora (Dossou et al., 2022, Sindane et al., 2024).
• MAFT vs. LAFT: AfroLM’s Multilingual Adaptive Fine-Tuning approach retains cross-lingual transfer within one compact model, outperforming vanilla PLMs on NER, news, and sentiment by ≈+2–4 F1 points, and only trailing language-adaptive fine-tuned (LAFT) models by <0.5 F1 (Alabi et al., 2022).
• Non-African Script Pruning: Reducing vocabulary size (from 250K to ~70K tokens) by pruning non-Latin/non-Ge’ez pieces yields smaller, more efficient models with minimal accuracy loss, though a small penalty is observed for non-Latin languages (e.g., Amharic, Arabic) (Alabi et al., 2022).
• Educational Content Translation: Experiments demonstrate that machine-translated educational corpora (e.g., GPT-4o-translated English-to-Swahili) improve downstream performance over English-only data. The key driver is domain-relevant content quality in the target language rather than the source language (Buzaaba et al., 9 Apr 2025).

6. Impact, Current Limitations, and Recommendations

AfroLM establishes a new benchmark for African language processing, particularly for low-resource and closely related language clusters.

• Generalization: AfroLM’s self-active learning yields heightened cross-domain robustness for tasks such as sentiment analysis (Twitter → Movies, e.g., 68.7% F1 vs 41.3% for no-AL model) (Dossou et al., 2022).
• LID Cluster Discrimination: AfroLM offers strong discrimination among Nguni and Sotho–Tswana clusters, regions where previous PLMs struggled (Sindane et al., 2024).
• Parameter and Disk Efficiency: MAFT enables a “single model for all African languages,” reducing the disk/storage requirement typically associated with LAFT, at minimal loss in downstream accuracy (Alabi et al., 2022).
• Recommendations:

1. Curate ≥1GB monolingual clean corpus per target language.
2. Apply vocabulary pruning for script relevance if feasible.
3. Utilize self-active learning or MAFT over all languages for maximal cross-lingual performance.
4. Mix educational/encyclopedic (and if possible, translated) content for knowledge-centric tasks.
5. Use balanced sampling (e.g., UniMax, up to 4× for rare tongues) to prevent dominance by high-resource languages.
6. Retain some English data in pre-training to avoid catastrophic forgetting, but maximize in-language educational content (Dossou et al., 2022, Buzaaba et al., 9 Apr 2025, Alabi et al., 2022).

Persistent limitations include moderate cross-domain performance drops (4–5 F1), especially on out-of-domain text; a lack of full per-language fine-grained reporting in some studies; and challenges with vocabulary coverage for scripts other than Latin or Ge’ez (Sindane et al., 2024, Alabi et al., 2022).

7. Availability and Extensions

• Model and Code Release: Checkpoints, tokenizers, training pipelines, and processed corpora for all major AfroLM models are publicly released (e.g., https://github.com/bonaventuredossou/MLM_AL; HuggingFace Hub deployments for AfroXLMR variants) (Dossou et al., 2022, Alabi et al., 2022).
• Extensibility: The underlying protocols (active learning, MAFT, curated translation) generalize to additional African language families and other low-resource regimes.
• Future Directions: Unexplored avenues include deeper model architectures, extension to sub-100M parameter “mini” LMs, improved tokenization strategies for non-Latin scripts, and systematic domain- and task-specific data augmentation (Sindane et al., 2024, Alabi et al., 2022).

AfroLM sets a robust baseline for efficient, high-utility NLP in African languages, bridging the capacity gap between under-resourced LID or task models and massive, general-purpose PLMs.