---
title: 'ARGEN: Arabic Natural Language Generation Benchmark'
url: https://www.emergentmind.com/topics/arabic-natural-language-generation-benchmark-argen
type: topic
---

# ARGEN: Arabic Natural Language Generation Benchmark

The Arabic Natural Language Generation Benchmark (ARGEN) is a comprehensive evaluation suite designed to facilitate rigorous, multifaceted assessment of Arabic NLG models across Modern Standard Arabic (MSA), dialectal Arabic varieties, and romanized ("Arabizi") or code-mixed forms. Initiated by Nagoudi et al., ARGEN aggregates a diverse set of text-to-text generation tasks and datasets, providing a critical resource for advancing and benchmarking the capabilities of Arabic language generation systems [2109.12068].

## 1. Scope, Design, and Objectives

ARGEN was constructed to address the lack of unified, reproducible, and diverse benchmarks in Arabic natural language generation. It spans seven major text-to-text NLG task clusters: machine translation (MT), code-switched translation (CST), text summarization (TS), news title generation (NTG), question generation (QG), paraphrasing (PPH), and transliteration (TR). The benchmark encompasses 19 datasets and 59 distinct test splits, balancing MSA with dialectal and mixed-language data.

The underlying objectives are to:

- Enable direct, fair comparison of Arabic and multilingual models on a broad spectrum of generative tasks.
- Encourage the development of Arabic-specific architectures and pre-training regimes.
- Expose gaps in performance, particularly on under-resourced varieties such as dialectal Arabic and Arabizi.

## 2. Task Clusters and Data Sources

ARGEN’s seven tasks target key NLG benchmarks, each admitted with careful curation to ensure task diversity and linguistic representativity.

| Task                          | Core Data Sources                                                                            | Main Objective                                        |
|-------------------------------|---------------------------------------------------------------------------------------------|-------------------------------------------------------|
| Machine Translation (ARGEN-MT)          | UN Parallel, IWSLT, AraBench, OPUS                                                       | Ar↔En, X→Ar (X∈{EN, FR, DE, RU}); city/dialect splits |
| Code-Switched Translation (ARGEN-CST)   | ALG-CST, JOR-CST (real), synthetic UN splits (back-translation)                          | Code-switched Ar–Fr/En↔monolingual translations       |
| Text Summarization (ARGEN-TS)           | WikiLingua-AR, EASC                                                                      | Abstractive article summarization                     |
| News Title Generation (ARGEN-NTG)       | AraNews                                                                                  | Headline generation from news articles                |
| Question Generation (ARGEN-QG)          | ARCD, XTREME, MLQA, XQuAD, TyDi QA                                                       | Passage+answer→question                              |
| Paraphrasing (ARGEN-PPH)                | AraPara, ASEP, APB                                                                       | Meaning-preserving re-expression                      |
| Transliteration (ARGEN-TR)               | BOLT Egyptian (SMS/Chat/Transliteration)                                                 | Arabizi/atypical orthographies→Arabic script          |

MT covers both standard and dialectal splits. CST incorporates both real-world (Algerian-French, Jordanian-English) and synthetic code-mixed data. Summarization includes both contemporary (WikiLingua) and legacy (EASC) domains. Paraphrasing derives large-scale pairs via MT and semantic similarity filtering, complemented by human-annotated sets.

## 3. Evaluation Methodology and Metrics

ARGEN standardizes evaluation across tasks using widely adopted automatic metrics:

- **BLEU-N** (Papineni et al. 2002): n-gram precision, with brevity penalty:
  $$
  \mathrm{BLEU}
      = \exp\Bigl(\min\{0,\,1 - \tfrac{L_{\mathrm{ref}}}{L_{\mathrm{hyp}}}\}\Bigr)
      \times \exp\Bigl(\sum_{n=1}^N w_n \log p_n\Bigr)
  $$
  Used for MT, CST, QG, and TR.

- **ROUGE-N/L** (Lin 2004): recall of n-grams and longest common subsequence:
  $$
  \mathrm{ROUGE}\!-\!N = \frac{\sum_{\text{n-gram} \in \text{Ref}} \min\{\#_{\text{match}},\,\#_{\text{ref}}\}} {\sum_{\text{n-gram} \in \text{Ref}} \#_{\text{ref}}}
  $$
  $$
  \mathrm{ROUGE}\!-\!L = \frac{\mathrm{LCS}_{\text{len}}}{\mathrm{len}(\text{Ref})}
  $$
  Applied to summarization and, where applicable, paraphrasing.

- **Exact Match (EM)** and **F₁ Overlap**: for sequence-to-sequence QA, measuring exact prediction equality and token-level harmonic mean.

- **Task-Specific Formats**: All inputs/outputs are tokenized using a fixed-size (typically 64–110k) byte-level or subword vocabulary shared with the pre-trained model.

## 4. Experimental Protocol

The ARGEN evaluation pipeline enforces strict reproducibility by:

- Uniform task preprocessing: Data are left untokenized (UTF-8), with normalization settings consistent with leading Arabic NLU/NLG models.
- Split control: For datasets like WikiLingua, deterministic shuffling and an 80/10/10 train–dev–test split (seed 42) are used.
- Training/fine-tuning settings: Grid search over optimizers (e.g., Adafactor), learning rates (1e-3 to 1e-5), batch sizes (4–64), and dropout (0.1, 0.2) to select optimal configurations.
- Decoding: T5 default settings—greedy decoding, maximum 114 decoder tokens per example.

A summary of the model types evaluated:

| Model           | Architecture          | Pre-training Data                        |
|-----------------|----------------------|------------------------------------------|
| AraT5 variants  | Arabic T5 (Base/Small) | MSA, Twitter, or combined (7.1–29B tokens) |
| mT5             | Multilingual T5       | mC4 corpus, 101 languages, 26TB data      |
| Baseline S2S    | Fairseq-trained from scratch      | MSA/En pairs (2M/10M)                        |

## 5. Key Results and Analysis

Experimental evidence demonstrates that Arabic-centric models (e.g., AraT5 variants) systematically outperform mT5 and baseline encoder-decoder systems. Across 59 test sets, AraT5 models exhibited state-of-the-art results on 52, with the strongest gains on dialectal and code-switched varieties [2109.12068].

Select results:

- **Machine Translation (MT, Ar→En):** Average BLEU improvement of +3–5 over mT5.
- **Question Generation (QG):** BLEU increase of +1.70 on test sets.
- **Summarization (WikiLingua):** ROUGE-1 score increase of +3 (AraT5Tw over mT5).
- **Transliteration:** Substantial BLEU gain (+5.1), strong performance on Romanized and non-standard forms.

Analysis further revealed strengths in code-switch handling and dialect translation, attributed to pre-training over large, diverse Twitter corpora (~21.9B tokens; 28% dialect) [2109.12068]. However, performance was less robust on legacy/web corpora likely overrepresented in mT5’s pre-training.

## 6. Strengths, Limitations, and Impact

ARGEN’s strengths include:

- Multi-faceted coverage (19 datasets, 59 splits), MSA and dialectal representation, and incorporation of Arabizi/code-mixed resources.
- Public availability of splits and scripts for reproducibility ([2109.12068]; [2205.10687]).
- Rigorous task formulation and evaluation parity with international NLG benchmarks.

Limitations identified:

- Some dataset splits and licenses are not fully public, restricting a portion of ARGEN to non-reproducible experiments [2205.10687].
- Absence of a public leaderboard or a private, held-out test set for many ARGEN tasks impedes standardized progress tracking compared to newer benchmarks like Dolphin [2305.14989].
- Most generative datasets are derived from automatic translation of English data, introducing domain mismatch and noise, particularly for dialogue and summarization [2205.10687].

Despite these, ARGEN catalyzed systematic evaluation of Arabic NLG models and spurred efforts to develop larger, more diverse benchmarks.

## 7. Relation to Broader Landscape and Successors

ARGEN established early conventions for Arabic NLG benchmarking but has since been surpassed in breadth and openness by newer initiatives. The Dolphin benchmark, for example, is three times larger by dataset coverage (40 datasets, 13 task clusters, 50 test splits), spans a wider range of varieties (Classical, Arabizi, mixed), includes a modular public leaderboard, and adds tasks such as diacritization, grammatical error correction, and data-to-text [2305.14989].

Comparison summary:

| Benchmark | Datasets | Tasks | Varieties           | Public Leaderboard |
|-----------|----------|-------|---------------------|-------------------|
| ARGEN     | 13       | 7     | MSA, dialect, Arabizi/code-mixed | No                |
| Dolphin   | 40       | 13    | MSA, DA, CA, Arabizi, code-switched | Yes               |

A plausible implication is that ARGEN influenced subsequent benchmark design, motivating the shift towards fully open, linguistically richer evaluation suites and live community scoreboards. Its detailed task protocols and variant results remain a reference point for state-of-the-art comparisons, especially in the context of model pre-training and domain/dialect adaptation.

Source: https://www.emergentmind.com/topics/arabic-natural-language-generation-benchmark-argen