MzansiText: SA Multilingual NLP Corpus

• MzansiText is a curated multilingual corpus and LID system for eleven South African languages, combining high- and low-resource data sources.
• It employs deterministic filtering, deduplication, and lexicon-based classification to ensure reproducible benchmarks and high accuracy (up to 99.34%).
• The open-source resource underpins model pretraining and downstream evaluation, powering tools like MzansiLM and AfriE5 for African language NLP.

MzansiText is a curated, large-scale multilingual corpus and accompanying suite of computational resources designed to advance NLP for the eleven official South African languages. It is specifically architected for both model pretraining and robust empirical benchmarking, with a strong emphasis on under-resourced Bantu languages. Through deterministic filtering, deduplication, and quality control pipelines, MzansiText provides stable, reproducible, and linguistically representative data for downstream decoder-only language modeling and embedding research (Lombard et al., 21 Mar 2026). In complementary usage, “MzansiText” also appears as the short-text language identification (LID) system that combines hierarchical naïve Bayes and lexicon-based disambiguation optimized for the same language spectrum (Duvenhage, 2019).

1. Linguistic Coverage and Corpus Composition

MzansiText encompasses the two high-resource written languages of South Africa—Afrikaans (afr) and English (eng)—alongside nine low-resource Bantu languages: isiZulu (zul), isiXhosa (xho), Sepedi/Northern Sotho (nso), Sesotho (sot), Setswana (tsn), siSwati (ssw), Tshivenda (ven), Xitsonga (tso), and isiNdebele (nbl). This coverage reflects the primary linguistic demographics of South Africa, with only 8.7% of the population being home English speakers, underlining the importance of high-quality Bantu-language resources (Lombard et al., 21 Mar 2026).

Raw input to the corpus is drawn from diverse, widely-used public language resources, including:

• mC4: filtered Common Crawl multilingual web text.
• CulturaX: multilanguage web crawl with enhanced cleaning.
• WURA: mC4 filtered for African languages and annotated with news-scraping.
• Glot500-c: wide-coverage multilingual web aggregation.
• NCHLT Text: curated monolingual corpora from the South African SADiLaR project.
• CC100: curated Common Crawl subsets.
• ParaCrawl v7.1: monolingual extractions from parallel data.
• Inkuba-Mono: newly released African language corpora, especially for isiXhosa and isiZulu.

The dataset achieves a post-filtered total of approximately 3.81 billion train tokens, with significant language imbalance (e.g., Afrikaans at 64.96%, isiZulu at 8.40%), reflecting both the underlying web landscape and source data availability (Lombard et al., 21 Mar 2026).

2. Data Filtering, Preprocessing, and Deduplication

All preprocessing steps are formalized in the DataTrove pipeline, ensuring determinism and reproducibility with explicit parameterization (Lombard et al., 21 Mar 2026). The pipeline includes the following stages:

1. Language Identification: Document and line-level filtering retains only segments where $P(L|\text{segment}) \geq 0.5$ for the designated South African language.
2. Normalization: HTML stripping, Unicode NFC normalization, and whitespace and control character removal.
3. Structural Filtering: Documents are filtered for length ($50 \leq |\text{chars}| \leq 15\,000$), script, and digit ratio ($\leq 0.3$).
4. Deduplication: Documents with MinHash Jaccard similarity $> 0.9$ are removed, yielding a global 4.0% reduction in word count.
5. Safety & Quality Filtering: Explicit patterns matching personal information (e.g., phone, email) are filtered, as well as low-information strings and boilerplate.
6. Packing: Qualifying text is concatenated into $\sim$1,000-token records.

Retention rates vary by source, with CulturaX preserving 96.3% of its original content post-filtering, while ParaCrawl retains only 3.5% (Lombard et al., 21 Mar 2026).

3. Benchmark Tasks, Evaluation, and LID Methods

Within short-text LID, “MzansiText” refers to a classification pipeline for discriminating among South Africa's 11 written languages at the snippet level (15–20 characters) (Duvenhage, 2019). The hierarchical two-stage system is as follows:

• Stage 1: Multinomial Naive Bayes (NB) over six “families” (Nguni, Sotho, and four singletons), each parameterized as

$$\log P(f \mid T) = \log P(f) + \sum_{g\in G(T)} \operatorname{count}(g, T) \log P(g \mid f)$$

where $G(T)$ denotes features (character n-grams ($n=2,4,6$), word 1- and 2-grams), and Laplace smoothing is used with $\alpha=0.01$.

• Stage 2: For the Nguni or Sotho sub-families, a word-form lexicon-based classifier computes

$$\mathrm{score}_{\rm lex}(L|T) = \sum_{i=1}^m \mathbf{1}[w_i \in \mathcal{X}_L]$$

Confidence in the lexicon's top prediction is governed by the threshold $50 \leq |\text{chars}| \leq 15\,000$0. For singletons, NB is reapplied.

Overall accuracy is reported as the principal metric. On the NCHLT test split, the best “stacked” NB+Lex system achieves 96.12% accuracy, and as high as 99.34% on the DSL’15 shared task dataset, exceeding both classical SVM and deep recurrent neural baselines. A pure lexicon-based model can achieve $50 \leq |\text{chars}| \leq 15\,000$197–99%—but only with unrealistic access to test vocabulary—dropping to $50 \leq |\text{chars}| \leq 15\,000$275% when restricted to train-only forms, highlighting the importance of robust lexicon management (Duvenhage, 2019).

4. Embedding Models, Downstream Tasks, and Adaptation

MzansiText supports embedding-centric and generative downstream evaluation. It forms the pretraining backbone for MzansiLM, a 125M parameter, decoder-only, BPE tokenized (65,536 vocab) LLM (Lombard et al., 21 Mar 2026). Fine-tuning regimes include:

• Monolingual and multilingual task-specific adaptation.
• Multi-task instruction tuning.

For isiXhosa data-to-text (T2X), MzansiLM achieves 20.65 BLEU (mono-task finetuning), outperforming much larger encoder-decoder baselines. In news topic classification (MasakhaNEWS), multilingual adaptation yields 78.5% macro-F1 for isiXhosa, with similar trends in NER (MasakhaNER 2.0) and SIB-200 classification.

For text embeddings, the AfriMTEB benchmark provides task suites for bitext mining, news topic classification, hate-speech, STS, and clustering—all with substantial South African language inclusion (Uemura et al., 27 Oct 2025). The AfriE5 model, trained via cross-lingual contrastive distillation, matches or outperforms multilingual baselines for these tasks, with 63–80% classification accuracy and $50 \leq |\text{chars}| \leq 15\,000$375% retrieval recall@1 for Xhosa/Zulu.

Recommended adaptation strategies for MzansiText to state-of-the-art embedding models include mean pooling + L2 normalization, lightweight classification layers for labeled tasks, or paired-contrastive objectives for unlabeled/semantically paired data (Uemura et al., 27 Oct 2025).

5. Reproducibility and Release Protocols

MzansiText is fully open-sourced, with all data-preparation scripts, preprocessing pipelines, and model training routines available publicly on GitHub and Hugging Face datasets (see https://github.com/Anri-Lombard/sallm and anrilombard/mzansi-text). DataTrove pipeline versioning (v0.4.x), fixed deduplication seeds, and explicit preprocessing parity enable exact re-derivation of splits and language proportions (Lombard et al., 21 Mar 2026).

Model training adheres closely to transparent recipes:

• Tokenizer: BPE on the training set, vocab size $50 \leq |\text{chars}| \leq 15\,000$4.
• Up to 2M validation/test tokens per language.
• Training hardware and environment specifics (e.g., 4 $50 \leq |\text{chars}| \leq 15\,000$5 NVIDIA A100, PyTorch 2.0).

MzansiText’s LID components are similarly reproducible, with code and lexicons provided by the original authors (Duvenhage, 2019).

6. Open Challenges and Research Directions

Several persisting challenges are identified in MzansiText-related work:

• Dataset Standardization: There is no unified short-text LID benchmark for all eleven South African languages, particularly for realistic, out-of-vocabulary and code-switching scenarios.
• Diagnostics and Metrics: Reporting aggregate accuracy obscures systematic errors (e.g., among Nguni and Sotho languages); per-language precision/recall/F₁ and full confusion matrices remain essential unreported desiderata (Duvenhage, 2019).
• Lexicon Maintenance: Lexicon-based LID methods demand continuous updates to handle slang, borrowings, and typographic variants. Research in self-supervised and active learning for dynamic lexicon growth is ongoing.
• Few-Shot and Reasoning Tasks: With decoder-only models at sub-billion scale (e.g., 125M), few-shot reasoning on tasks such as NLI or MMLU remains near chance, indicating a parameter-size threshold for emergent capabilities (Lombard et al., 21 Mar 2026).
• Transfer and Adaptation: Multilingual task-specific finetuning provides consistent performance gains for classification, especially in related language clusters, but monolingual generation tasks benefit most from explicit single-language adaptation.
• Support for Multilingual Code-Switching: Extending the pipeline to segment and label language-mixed text is an unresolved task, crucial for representative modeling of South African social streams.

7. Significance and Adoption

MzansiText constitutes a foundational multilingual corpus, LID method, and benchmarking resource for South African language technology, enabling strong baselines for both conventional “shallow” classification (Nb+Lex) and modern transfer learning architectures (transformer-based LMs and embedding models). It has informed the training of models such as MzansiLM and AfriE5, supporting macro-F1 improvements in NLU and generation for both high- and low-resource languages. The open reproduction recipes, documented impact in both benchmarking and real-world language technology, and articulated research priorities position MzansiText as the primary substrate for computational language research in the South African context, directly addressing both linguistic inclusion and methodological rigor (Lombard et al., 21 Mar 2026, Duvenhage, 2019, Uemura et al., 27 Oct 2025).