MzansiLM: SA Multilingual Language Model

• MzansiLM is a multilingual decoder-only Transformer model designed to support 11 South African languages, focusing on low-resource settings.
• It employs a LLaMA-style architecture with 125M parameters and leverages a rigorously filtered MzansiText corpus for training.
• The model is openly released to provide a reproducible baseline for multilingual NLP research and downstream language adaptation.

MzansiLM is a 125-million-parameter decoder-only Transformer LLM, trained from scratch on the MzansiText corpus to explicitly support all eleven official written South African languages, nine of which are low-resource. It is designed to offer an open, reproducible baseline for multilingual language modeling and downstream adaptation in African low-resource settings, with systematic evaluation on natural language generation, understanding, and reasoning tasks. Both the MzansiText corpus and the MzansiLM models, along with full code and checkpoints, are released openly for the research community (Lombard et al., 21 Mar 2026).

1. MzansiText: Multilingual Corpus Construction

MzansiText is a curated multilingual corpus that aggregates eight public sources to comprehensively cover all eleven official South African languages: Afrikaans, English, isiZulu, isiXhosa, Sesotho, Setswana, Sepedi (Northern Sotho), Xitsonga, siSwati, Tshivenda, and isiNdebele. Data is sourced from mC4, CulturaX, WURA, Glot500-c, NCHLT Text, CC100, ParaCrawl, and Inkuba-Mono, spanning web crawl, news, and monolingual data.

The corpus undergoes a fully deterministic seven-stage filtering pipeline implemented via HuggingFace's DataTrove. The pipeline includes language ID (line and document), normalization, structural and heuristic quality filtering, deduplication (including MinHash-based near-duplicate removal at Jaccard > 0.8), privacy filtering (PII), and packing into ~1,000-token records.

Token counts per language (training split):

Language	Train Tokens	%Train
Afrikaans	2.48 B	64.96
English	0.74 B	19.44
isiZulu	0.32 B	8.40
isiXhosa	0.15 B	3.99
Sesotho	0.10 B	2.56
Setswana	10.08 M	0.26
Sepedi	6.70 M	0.18
Xitsonga	3.01 M	0.08
siSwati	1.93 M	0.05
Tshivenda	1.85 M	0.05
isiNdebele	0.82 M	0.02
Total	3.81 B	100.0

Each language has separate held-out validation and test splits capped at 2 million tokens to ensure partitioned evaluation.

2. MzansiLM: Model Architecture and Pretraining

MzansiLM is a decoder-only Transformer, MobileLLM variant, following a LLaMA-style architecture with:

• 6 Transformer blocks,
• Hidden dimension: 512,
• FFN inner dimension: 2,048,
• Self-attention heads: 8,
• Maximum context window: 2,048 tokens,
• Vocabulary: 65,536 BPE tokens,
• Total parameters: ≈125M.

The pretraining objective is standard causal language modeling:

$$\mathcal{L}(\theta) = -\sum_{t=1}^T \log P_\theta(x_t \mid x_{<t}),$$

where sequences per training step are up to 2,048 tokens long.

Training utilized 3.81B tokens, across five epochs, with AdamW (β₁=0.9, β₂=0.95, ε=1e-8), linear warmup of 2,000 steps to peak LR (1e-4) followed by cosine decay, batch size of 512 sequences (~1,024 tokens/seq), on 4× NVIDIA A100 80GB GPUs over ~27 hours.

3. Adaptation and Finetuning Regimes

MzansiLM was adapted using three finetuning regimes:

3.1. Monolingual Task-Specific Finetuning

A LoRA-based adapter (rank 128, α=256, dropout=0.1) is trained per task-language pair, using only in-language examples and task-specific prompts for all tasks (classification/generation). Example configurations:

• T2X (isiXhosa data-to-text): 3,859 examples, 346,518 tokens,
• AfriHG (isiXhosa/isiZulu headline-generation): up to 14,209 examples, 7.49M tokens,
• MasakhaNEWS (topic classification): English 3,309/isiXhosa 1,032 examples.

3.2. Multilingual Task-Specific Finetuning

Single LoRA adapters are trained on pooled multi-language data for a given task (e.g., MasakhaNEWS jointly on English + isiXhosa), with a unified label space, leveraging cross-lingual transfer, particularly among related Bantu languages.

3.3. General Multi-Task Instruction Finetuning

All task-language datasets are combined with balanced sampling. Prompts follow the EleutherAI LM Evaluation Harness zero/one/three-shot templates, and instruction formats are cycled during training to maximize exposure to task variety.

4. Evaluation Results and Benchmarks

MzansiLM was evaluated on NLG (generation), NLU (classification/NER/POS), and reasoning (reading comprehension, QA), with comparisons to encoder-only (AfroXLMR, AfriBERTa), encoder–decoder (mT5, Aya-101), and larger decoder-only (AfroLlama, Llama-3.1) baselines.

4.1. Generation Metrics

Data-to-text (T2X) – isiXhosa:

Model	Params	BLEU	ROUGE-L
MzansiLM (mono-ft)	0.125B	20.65	41.19
mT5-base	0.58B	16.8	38.7
Aya-101	13B	8.9	33.9

Monolingual finetuning of MzansiLM (isiXhosa) outperforms much larger encoder–decoder baselines on BLEU and ROUGE-L.

Headline Generation (AfriHG) – isiXhosa BLEU/chrF:

• Mono-ft: BLEU 0.63 / chrF 4.47 (underline, second best)
• Multi-ft: BLEU 0.95 / chrF 5.67
• mT5-base, Aya-101: higher chrF but lower BLEU for corresponding sizes.

4.2. NLU Tasks

• Topic classification (MasakhaNEWS, macro-F1):
  • isiXhosa (multi-ft): 78.5 (AfroXLMR-base: 94.7, InkubaLM-0.4B: 7.4)
  • English (mono-ft): 63.5
• NER (MasakhaNER 2.0, macro-F1, isiXhosa mono-ft): 48.4 (vs. InkubaLM-0.1, AfroLlama: 5.6)
• POS tagging (MasakhaPOS, token accuracy, isiXhosa mono-ft): 30.5 (Llama-8B: 58.1, AfroXLMR-base: 88.5)
• SIB-200 topic classification (isiZulu multi-ft): 47.2 (AfroXLMR-large: 85.8)

4.3. Few-shot Reasoning and Reading Comprehension

• Belebele (isiXhosa normalized accuracy): all decoder-only <30%, MzansiLM (general-ft): 21.9%
• IrokoBench (AfriXNLI, AfriMMLU, AfriMGSM): MzansiLM ~32% (chance), Llama-70B Instruct: 40–76%

At 125M scale, MzansiLM achieves competitive results on supervised generation in low-resource languages, but remains at chance-level on few-shot reasoning and comprehension tasks, a limitation common even for much larger LMs in these languages.

5. Recommendations, Limitations, and Reproducibility

Key findings:

• Monolingual task-specific finetuning is optimal for structured generation in agglutinative low-resource languages, even at 125M scale.
• Multilingual finetuning yields substantial gains via cross-lingual transfer in related Bantu language classification.
• General multi-task instruction-tuning broadens task coverage but dilutes signal at this scale and does not enhance few-shot reasoning or structured classification.
• For high-accuracy structured prediction, encoder-only (AfroXLMR) and encoder–decoder (mT5) architectures are consistently superior.
• Few-shot reasoning tasks require orders-of-magnitude larger decoder-only models (≥70B parameters).

MzansiText and MzansiLM are openly released to support reproducible work in African low-resource NLP, establishing the first decoder-only baseline for all official South African languages. Code for the corpus pipeline, model pretraining, adaptation, and evaluation is available at https://github.com/Anri-Lombard/sallm, datasets and models at https://huggingface.co/anrilombard/mzansi-text and https://huggingface.co/anrilombard/mzansilm-125m (Lombard et al., 21 Mar 2026).

A plausible implication is that the MzansiLM methodology scales to other multilingual low-resource settings, given sufficient corpus coverage and careful language-specific adaptation.