AFRIDOC-MT: African MT Corpus

• AFRIDOC-MT Corpus is a multi-parallel, document-level machine translation dataset covering English and five African languages in the health and technology domains.
• It comprises 605 human-translated news documents with detailed train, dev, and test splits, enabling robust benchmarking for NMT and LLM models.
• The dataset supports pseudo-document creation and employs metrics like BLEU, chrF, and COMET, highlighting its role in low-resource multilingual MT research.

The AFRIDOC-MT Corpus is a document-level, multi-parallel machine translation dataset spanning English and five African languages—Amharic, Hausa, Swahili, Yoruba, and Zulu. It consists of 605 human-translated news documents from the health and information technology domains. AFRIDOC-MT enables benchmarking and analysis of both neural machine translation (NMT) models and LLMs in low-resource, document-level multilingual settings, with extensive evaluation at both sentence and pseudo-document scales (Alabi et al., 10 Jan 2025).

1. Dataset Composition and Statistics

AFRIDOC-MT features five languages (Amharic, Hausa, Swahili, Yoruba, Zulu). The corpus covers 334 health and 271 technology news documents, totaling 20,000 English sentences—each sentence was human-translated into all five African languages.

Train/dev/test splits and document counts per domain are as follows:

Domain	Split	#Documents	#Sentences
Health	Train	240	7,041
Health	Dev	33	977
Health	Test	61	1,982
Tech	Train	187	7,048
Tech	Dev	25	970
Tech	Test	59	1,982

Average English tokens per sentence are 21.6 (health) and 17.8 (tech); for African languages, values range from 13.4–28.1. At the document level, English documents average ≈650 tokens, while African language documents span ≈500 (Zulu) to ≈840 (Hausa/Yoruba).

Document curation involved scraping English articles from the World Health Organization (health) and Techpoint Africa (tech). Sentences were segmented using NLTK and verified by a linguist. Each of four expert translators per language worked “in context” (seeing entire documents), supported by terminology workshops and guidelines. Quality control involved Google language-ID, AfriCOMET QE, and manual review for QE < 0.65.

To fit context window constraints of various models, documents were also split into fixed-size “pseudo-documents” of $k$ sentences ($k\in \{1, 5, 10, 25\}$), with $k=1$ representing the sentence-level baseline and $k=10$ serving as default for document-level experiments.

2. Document-Level Alignment and Pseudo-Document Construction

After translation, output segments indicated both the document ID and sentence index. Full documents were reconstructed by concatenating outputs in original sentence order, regardless of whether the translation proceeded sentence-wise ($k=1$) or in multi-sentence pseudo-documents ($k>1$).

Heuristic analysis of model tokenizers identified context limits: 4096 tokens for LLaMA-based LLMs and 1024–2048 tokens for T5-based models. $k=10$ balanced context length with hardware and model constraints, ensuring over 99% of pseudo-documents fit within context windows at the 95th percentile.

3. Benchmarking: Models and Experimental Protocols

AFRIDOC-MT benchmarks both encoder–decoder NMT architectures and decoder-only LLMs:

• Encoder–Decoder NMT Systems: M2M-100 (0.4B/1.2B), NLLB-200 (0.6B/1.3B/3.3B), Toucan-1.2B, MADLAD-400 (3B/7.2B), Aya-101 (13B, mT5-based instruction-tuned).
• Decoder-Only LLMs: LLaMA 3.1 (8B) & LLaMA 3.1-IT, Gemma-2 (9B) & Gemma-2-IT, LLaMAX3 (8B) & LLaMAX3-Alpaca, OpenAI GPT-3.5 Turbo, GPT-4o.

Fine-tuning (SFT) was applied:

• NLLB-200 (1.3B): Jointly on all 30 directions (EN$\leftrightarrow${amh, hau, swa, yor, zul}), both domains, for 50K steps, learning rate $5\times10^{-5}$, batch size 2048 tokens, gradient accumulation=2, with early stopping at minimum dev loss.
• LLaMA 3.1, LLaMAX3: SFT with instruction-augmented prompts on sentence pairs or $k=10$ pseudo-docs, $k\in \{1, 5, 10, 25\}$0, batch=64, for one epoch.

Pseudo-document (multi-sentence chunk) setup directly targeted document-level learning and evaluation, as models trained solely at the sentence level showed poor document generalization.

4. Evaluation Metrics

Corpus-level translation quality was assessed using:

• BLEU:

$k\in \{1, 5, 10, 25\}$1

with $k\in \{1, 5, 10, 25\}$2 and brevity penalty $k\in \{1, 5, 10, 25\}$3 if $k\in \{1, 5, 10, 25\}$4, $k\in \{1, 5, 10, 25\}$5 otherwise ($k\in \{1, 5, 10, 25\}$6 = candidate length, $k\in \{1, 5, 10, 25\}$7 = reference).

• chrF:

Character-$k\in \{1, 5, 10, 25\}$8-gram $k\in \{1, 5, 10, 25\}$9-score with weighting $k=1$0.

• COMET:

Pre-trained neural regression model producing $k=1$1 quality scores for sentences.

Each metric was computed at both the sentence (s-) and document (d-) levels using SacreBLEU v2.3.1 with 1,000 bootstrap samples for 95% confidence intervals.

Sentence-level metrics assess individual sentences; document-level metrics concatenate all translations and compare to fully concatenated references.

5. Quantitative Results

Sentence-level outputs realigned as documents (averaged d-CHRF across African languages):

Model	Health	Tech
M2M-100 (1.2B)	51.2	50.7
NLLB-200 (1.3B)	60.8	60.1
NLLB-200 (3.3B)	61.9	60.4
MADLAD-400 (3B)	48.3	47.3
Aya-101 (13B)	45.5	45.8
+NLLB-200-SFT (1.3B)	68.0	66.2
LLaMAX3-Alpaca	46.0	46.9
LLama3.1-IT	35.5	36.4
Gemma2-IT	42.1	43.6
GPT-3.5	42.8	44.2
GPT-4o	55.8	55.6
+LLaMAX3-SFT	61.5	58.8
+LLM-SFT (k=10)	61.5	60.2

The NLLB-200-SFT model provides the highest average d-CHRF (68.0/66.2). Among LLMs, GPT-4o performs best (55.8/55.6), but SFT substantially improves LLaMAX3 results, nearly equating GPT-4o performance.

For pseudo-document ($k=1$2) evaluation:

Model	Health	Tech
MADLAD-400	34.6	31.8
Aya-101	37.1	42.8
Gemma2-IT	23.0	24.3
LLama3.1-IT	16.0	16.2
LLaMAX3-Alpaca	22.7	33.7
GPT-3.5	30.0	32.6
GPT-4o	54.5	54.0
+LLaMAX3-SFT₁₀	54.2	55.0
+LLama3.1-SFT₁₀	47.8	44.5

Pseudo-document SFT narrows the performance gap for LLMs; LLaMAX3-SFT₁₀ closely matches GPT-4o.

6. Qualitative Analysis and Error Patterns

GPT-4o was leveraged as a proxy human judge for qualitative analysis. Error patterns include:

• Under-generation (<70% reference length):
  • Aya-101: >3% in eng→afr.
  • LLaMAX3-Alpaca: ~10% into English.
  • GPT-3.5/GPT-4o: <5% overall.
• Off-target translations (language-ID mismatch):
  • Rare when translating into English (<1%).
  • Up to 5–10% for open LLMs into African languages.
• Repetition:
  • GPT-3.5 and GPT-4o occasionally repeat words/phrases, especially for long pseudo-docs (>25 sentences).
  • LLaMA-based LLMs show fewer repetitions, attributed to instruction-tuned SFT.
• Language-specific challenges:
  • Yoruba exhibits the lowest sentence-level d-CHRF ($k=1$343), largely due to diacritic-based segmentation/tokenization errors.
  • Amharic’s script causes segmentation/tokenization mismatches, with scores 5–7 points lower than Swahili.

A plausible implication is that document-level translation for African languages will require tailored tokenization and evaluation to mitigate script and orthographic complications.

7. Conclusions and Future Directions

AFRIDOC-MT constitutes the first multi-parallel, document-level MT corpus for five African languages within the health and technology domains (Alabi et al., 10 Jan 2025). The combination of sentence-level NMT and SFT establishes state-of-the-art performance (d-CHRF ≈ 68/66). GPT-4o delivers the best results among LLMs, but domain-/task-specific SFT is crucial to bridge the quality gap, especially for general-purpose LLMs and pseudo-document translation.

There are divergences between classical document-level metrics (BLEU/chrF) and GPT-4o-based human proxy assessments, highlighting the need for new metrics attuned to low-resource settings and discourse phenomena.

Key future directions include expanding the coverage to additional African languages and domains, developing embedding-based and discourse-aware document MT metrics, professional human evaluation to corroborate proxy findings, and designing domain-adaptive pretraining or contrastive alignment methods tailored to low-resource document translation.