English-Tigrinya NLP Evaluation Dataset

• The English-Tigrinya evaluation dataset is a human-aligned parallel corpus from diverse domains like religious texts, news, health, and education.
• It uses rigorous manual alignment, script normalization, and quality filtering to ensure reliable performance metrics such as BLEU and chrF.
• The dataset supports transfer learning and custom tokenization approaches to improve machine translation quality for Tigrinya's complex morphology.

An English-Tigrinya evaluation dataset is a human-aligned, parallel resource used for the rigorous assessment of NLP models—chiefly machine translation (MT) and language understanding systems—spanning both high-resource languages (English) and the severely underrepresented Tigrinya language. These datasets address core challenges posed by Tigrinya's complex morphology, non-Latin Ge’ez script, and severe resource scarcity, enabling reproducible benchmarking, error analysis, transfer learning validation, and the development of linguistically robust NLP systems.

1. Dataset Construction Principles and Domain Composition

A high-quality English-Tigrinya evaluation dataset is typically generated through the manual alignment of parallel sentences across diverse domains, including Religious, News, Health, and Education. Domain-specific sources such as JW.org and Bible.com for religious texts, BBC and GlobalVoices for news, and curated materials for health/education are collected, emphasizing accurate sentence-level correspondence and eliminating noisy alignments and script normalization inconsistencies (Teklehaymanot et al., 24 Sep 2025).

Rigorous human review ensures that each sentence pair is both semantically and syntactically aligned, with orthographic normalization (especially for the Ge’ez script) critical for downstream MT and evaluation. Sentence pairs undergo verification and quality-filtering procedures, producing a gold-standard evaluation corpus for benchmarking MT systems and other cross-lingual NLP tasks.

Domain	Example Source(s)	Alignment Comments
Religious	JW.org, Bible.com	Ge’ez script normalization
News	BBC, GlobalVoices	Mixed formal and informal styles
Health	Medical educational	Domain-specific terminology
Education	Textbooks, lessons	Structured, curriculum-aligned

This multi-domain approach is aimed at mitigating domain bias and supporting the assessment of generalization beyond in-domain translation.

2. Evaluation Metrics and Statistical Rigor

Two principal metrics dominate evaluation: BLEU (Bilingual Evaluation Understudy) and chrF (character F-score). BLEU measures n-gram overlap to gauge semantic and syntactic adequacy at the word level, while chrF quantifies character-level matching and is specifically sensitive to the intricate subword variation inherent in morphologically rich languages such as Tigrinya (Teklehaymanot et al., 24 Sep 2025). Together, these metrics capture both broad fluency and detailed morphological accuracy.

Statistical robustness is achieved through the Bonferroni correction, a multiple-testing adjustment method that ensures the family-wise error rate is properly controlled when comparing several experimental configurations. Given $m$ independent tests and target significance $\alpha$, the per-test threshold is $\alpha_{\text{adjusted}} = \alpha/m$:

$p_i < \frac{\alpha}{m}$

where $p_i$ is the individual test's p-value. This adjustment underpins the statistical significance claims for performance gains introduced by domain adaptation, custom tokenization, or transfer learning.

3. Transfer Learning, Tokenization, and Model Adaptation

Efforts to maximize translation quality in the low-resource English-Tigrinya setting leverage pretrained multilingual architectures (e.g., MarianMT). Transfer learning is operationalized by fine-tuning these models on Tigrinya data after initializing with high-resource language weights. A critical refinement is the deployment of a custom SentencePiece tokenizer, engineered for Tigrinya’s Ge’ez script, which:

• Incorporates script normalization,
• Segments subwords respecting Tigrinya’s non-concatenative morphology,
• Minimizes out-of-vocabulary rates and cross-lingual interference, especially vis-à-vis Amharic.

Embedding initialization strategies further adapt pretrained model representations to Tigrinya’s script and structure, substantially improving output quality over zero-shot baselines and generic tokenization (Teklehaymanot et al., 24 Sep 2025). Quantitative results demonstrate marked BLEU improvements and error reduction upon applying this linguistically informed fine-tuning pipeline.

4. Challenges Revealed by Error Analysis

Even with advanced transfer learning and tokenization, systematic error analysis exposes persistent limitations:

• Morphological confusions: Shared Ge’ez script elements between Tigrinya and Amharic often lead to erroneous token interchanges.
• Domain adaptation gaps: Out-of-domain data (e.g., health, education) introduces misalignments not present in news or religious data.
• Incomplete handling of dialectal and orthographic variation: Current systems may underperform on less-standardized text forms.

Fine-tuned models close much of the gap versus zero-shot performance, but a notable residual discrepancy remains compared to human translation references. This motivates future research in dialect-sensitive segmentation, improved normalization strategies, and possibly advanced subword modeling or morphology-aware neural architectures.

5. Accessibility, Reproducibility, and Community Impact

The resulting evaluation datasets, tokenizers, and fine-tuned models are openly distributed via platforms such as GitHub and Hugging Face (Teklehaymanot et al., 24 Sep 2025). This ensures that fellow researchers can download, benchmark, and extend resources, thus facilitating reproducible research pipelines and iterative model improvement. By providing these resources alongside clearly articulated evaluation standards and error analysis tools, the English-Tigrinya benchmark ecosystem supports fair comparison and robustness in empirical NLP research for under-resourced languages.

Resource	Access Point	Usage
Dataset (parallel, multi-domain)	https://github.com/hailaykidu/MachineT_TigEng	Development/evaluation
Custom SentencePiece tokenizer	https://github.com/hailaykidu/MachineT_TigEng	Model training/preprocessing
Fine-tuned translation models	https://huggingface.co/Hailay/MachineT_TigEng	Inference, further fine-tuning

6. Broader Implications for Low-Resource Language NLP

The introduction and expansion of high-quality English-Tigrinya evaluation datasets address foundational bottlenecks in comparative evaluation and reproducible research. Such benchmarks:

• Enable rigorous ablation studies and fair assessment of transfer learning strategies,
• Catalyze development of more effective morphology-aware and linguistically tailored NMT architectures,
• Support broader multi-domain and multi-task benchmarking, promoting cross-lingual transfer beyond common domains.

The iterative cycle of resource creation, model innovation, and shared benchmarking, exemplified by these datasets, constitutes a strategic path toward closing the performance gap for underrepresented languages in NLP and ensuring methodological parity with high-resource language settings.

7. Outlook: Future Directions

Planned enhancements include:

• Expanded domain coverage and enlargement of reference sets supporting rare dialects and genres,
• Improved script normalization and encoding, robust against orthographic variation,
• Integration of qualitative human evaluation with more granular error typology,
• Expansion of dialectal and societal bias auditing, especially as Tigrinya NLP becomes more widely adopted.

Such efforts are necessary to ensure that future English-Tigrinya evaluation datasets remain both comprehensive and sensitive to the linguistic complexity underpinning real-world Tigrinya language use, thereby reinforcing the foundation for robust, generalizable, and equitable NLP system development.