ARGEN: Arabic Natural Language Generation Benchmark

• The ARGEN benchmark provides a unified suite evaluating 7 Arabic NLG tasks with 19 datasets and 59 test splits.
• It employs standardized metrics like BLEU and ROUGE to assess performance on translation, summarization, and other text-to-text applications.
• Evaluations reveal that Arabic-centric models, such as AraT5, outperform multilingual systems, especially on dialectal and code-switched data.

The Arabic Natural Language Generation Benchmark (ARGEN) is a comprehensive evaluation suite designed to facilitate rigorous, multifaceted assessment of Arabic NLG models across Modern Standard Arabic (MSA), dialectal Arabic varieties, and romanized ("Arabizi") or code-mixed forms. Initiated by Nagoudi et al., ARGEN aggregates a diverse set of text-to-text generation tasks and datasets, providing a critical resource for advancing and benchmarking the capabilities of Arabic language generation systems (Nagoudi et al., 2021).

1. Scope, Design, and Objectives

ARGEN was constructed to address the lack of unified, reproducible, and diverse benchmarks in Arabic natural language generation. It spans seven major text-to-text NLG task clusters: machine translation (MT), code-switched translation (CST), text summarization (TS), news title generation (NTG), question generation (QG), paraphrasing (PPH), and transliteration (TR). The benchmark encompasses 19 datasets and 59 distinct test splits, balancing MSA with dialectal and mixed-language data.

The underlying objectives are to:

• Enable direct, fair comparison of Arabic and multilingual models on a broad spectrum of generative tasks.
• Encourage the development of Arabic-specific architectures and pre-training regimes.
• Expose gaps in performance, particularly on under-resourced varieties such as dialectal Arabic and Arabizi.

2. Task Clusters and Data Sources

ARGEN’s seven tasks target key NLG benchmarks, each admitted with careful curation to ensure task diversity and linguistic representativity.

Task	Core Data Sources	Main Objective
Machine Translation (ARGEN-MT)	UN Parallel, IWSLT, AraBench, OPUS	Ar↔En, X→Ar (X∈{EN, FR, DE, RU}); city/dialect splits
Code-Switched Translation (ARGEN-CST)	ALG-CST, JOR-CST (real), synthetic UN splits (back-translation)	Code-switched Ar–Fr/En↔monolingual translations
Text Summarization (ARGEN-TS)	WikiLingua-AR, EASC	Abstractive article summarization
News Title Generation (ARGEN-NTG)	AraNews	Headline generation from news articles
Question Generation (ARGEN-QG)	ARCD, XTREME, MLQA, XQuAD, TyDi QA	Passage+answer→question
Paraphrasing (ARGEN-PPH)	AraPara, ASEP, APB	Meaning-preserving re-expression
Transliteration (ARGEN-TR)	BOLT Egyptian (SMS/Chat/Transliteration)	Arabizi/atypical orthographies→Arabic script

MT covers both standard and dialectal splits. CST incorporates both real-world (Algerian-French, Jordanian-English) and synthetic code-mixed data. Summarization includes both contemporary (WikiLingua) and legacy (EASC) domains. Paraphrasing derives large-scale pairs via MT and semantic similarity filtering, complemented by human-annotated sets.

3. Evaluation Methodology and Metrics

ARGEN standardizes evaluation across tasks using widely adopted automatic metrics:

• BLEU-N (Papineni et al. 2002): n-gram precision, with brevity penalty:

$$\mathrm{BLEU} = \exp\Bigl(\min\{0,\,1 - \tfrac{L_{\mathrm{ref}}}{L_{\mathrm{hyp}}}\}\Bigr) \times \exp\Bigl(\sum_{n=1}^N w_n \log p_n\Bigr)$$

Used for MT, CST, QG, and TR.

• ROUGE-N/L (Lin 2004): recall of n-grams and longest common subsequence:

$$\mathrm{ROUGE}\!-\!N = \frac{\sum_{\text{n-gram} \in \text{Ref}} \min\{\#_{\text{match}},\,\#_{\text{ref}}\}} {\sum_{\text{n-gram} \in \text{Ref}} \#_{\text{ref}}}$$

$$\mathrm{ROUGE}\!-\!L = \frac{\mathrm{LCS}_{\text{len}}}{\mathrm{len}(\text{Ref})}$$

Applied to summarization and, where applicable, paraphrasing.

• Exact Match (EM) and F₁ Overlap: for sequence-to-sequence QA, measuring exact prediction equality and token-level harmonic mean.
• Task-Specific Formats: All inputs/outputs are tokenized using a fixed-size (typically 64–110k) byte-level or subword vocabulary shared with the pre-trained model.

4. Experimental Protocol

The ARGEN evaluation pipeline enforces strict reproducibility by:

• Uniform task preprocessing: Data are left untokenized (UTF-8), with normalization settings consistent with leading Arabic NLU/NLG models.
• Split control: For datasets like WikiLingua, deterministic shuffling and an 80/10/10 train–dev–test split (seed 42) are used.
• Training/fine-tuning settings: Grid search over optimizers (e.g., Adafactor), learning rates (1e-3 to 1e-5), batch sizes (4–64), and dropout (0.1, 0.2) to select optimal configurations.
• Decoding: T5 default settings—greedy decoding, maximum 114 decoder tokens per example.

A summary of the model types evaluated:

Model	Architecture	Pre-training Data
AraT5 variants	Arabic T5 (Base/Small)	MSA, Twitter, or combined (7.1–29B tokens)
mT5	Multilingual T5	mC4 corpus, 101 languages, 26TB data
Baseline S2S	Fairseq-trained from scratch	MSA/En pairs (2M/10M)

5. Key Results and Analysis

Experimental evidence demonstrates that Arabic-centric models (e.g., AraT5 variants) systematically outperform mT5 and baseline encoder-decoder systems. Across 59 test sets, AraT5 models exhibited state-of-the-art results on 52, with the strongest gains on dialectal and code-switched varieties (Nagoudi et al., 2021).

Select results:

• Machine Translation (MT, Ar→En): Average BLEU improvement of +3–5 over mT5.
• Question Generation (QG): BLEU increase of +1.70 on test sets.
• Summarization (WikiLingua): ROUGE-1 score increase of +3 (AraT5Tw over mT5).
• Transliteration: Substantial BLEU gain (+5.1), strong performance on Romanized and non-standard forms.

Analysis further revealed strengths in code-switch handling and dialect translation, attributed to pre-training over large, diverse Twitter corpora (~21.9B tokens; 28% dialect) (Nagoudi et al., 2021). However, performance was less robust on legacy/web corpora likely overrepresented in mT5’s pre-training.

6. Strengths, Limitations, and Impact

ARGEN’s strengths include:

• Multi-faceted coverage (19 datasets, 59 splits), MSA and dialectal representation, and incorporation of Arabizi/code-mixed resources.
• Public availability of splits and scripts for reproducibility ((Nagoudi et al., 2021); (Ghaddar et al., 2022)).
• Rigorous task formulation and evaluation parity with international NLG benchmarks.

Limitations identified:

• Some dataset splits and licenses are not fully public, restricting a portion of ARGEN to non-reproducible experiments (Ghaddar et al., 2022).
• Absence of a public leaderboard or a private, held-out test set for many ARGEN tasks impedes standardized progress tracking compared to newer benchmarks like Dolphin (Nagoudi et al., 2023).
• Most generative datasets are derived from automatic translation of English data, introducing domain mismatch and noise, particularly for dialogue and summarization (Ghaddar et al., 2022).

Despite these, ARGEN catalyzed systematic evaluation of Arabic NLG models and spurred efforts to develop larger, more diverse benchmarks.

7. Relation to Broader Landscape and Successors

ARGEN established early conventions for Arabic NLG benchmarking but has since been surpassed in breadth and openness by newer initiatives. The Dolphin benchmark, for example, is three times larger by dataset coverage (40 datasets, 13 task clusters, 50 test splits), spans a wider range of varieties (Classical, Arabizi, mixed), includes a modular public leaderboard, and adds tasks such as diacritization, grammatical error correction, and data-to-text (Nagoudi et al., 2023).

Comparison summary:

Benchmark	Datasets	Tasks	Varieties	Public Leaderboard
ARGEN	13	7	MSA, dialect, Arabizi/code-mixed	No
Dolphin	40	13	MSA, DA, CA, Arabizi, code-switched	Yes

A plausible implication is that ARGEN influenced subsequent benchmark design, motivating the shift towards fully open, linguistically richer evaluation suites and live community scoreboards. Its detailed task protocols and variant results remain a reference point for state-of-the-art comparisons, especially in the context of model pre-training and domain/dialect adaptation.