ALHD: Arabic LLM Text Benchmark
- ALHD is a multigenre, multidialectal benchmark dataset designed to evaluate the detection of human versus LLM-generated Arabic texts.
- It covers news, social media, and reviews with balanced samples from Modern Standard Arabic and various dialects, supporting controlled experiments.
- Empirical findings show that fine-tuned BERT-based models outperform traditional ML and prompt-only LLM approaches, highlighting generalization challenges.
ALHD is a large-scale benchmark for detecting Arabic LLM-generated text, introduced as the first comprehensive Arabic dataset explicitly designed to distinguish between human- and LLM-generated texts across genres, registers, and dialectal varieties (Khairallah et al., 3 Oct 2025). It spans news, social media, and reviews; covers both Modern Standard Arabic (MSA) and dialectal Arabic; and contains over 400K balanced samples generated from multiple human sources and three leading LLMs. The benchmark is designed not only to measure in-domain discrimination between human and machine text, but also to expose a harder problem: generalization across genres, sources, dialectal variation, and stylistic regimes in Arabic.
1. Definition and research objective
ALHD is described as a multigenre, multidialectal benchmark for human-versus-LLM text detection in Arabic, with an emphasis on reproducibility, generalizability, and systematic evaluation (Khairallah et al., 3 Oct 2025). Its motivating premise is that Arabic has lacked a resource of comparable scale and breadth for this task, despite growing practical concerns around misinformation, academic dishonesty, and cyber threats enabled by generative models.
The dataset is positioned as a benchmark rather than merely a corpus dump. This distinction matters because ALHD includes rigorous preprocessing, rich annotations, and standardized balanced splits. The benchmarking component evaluates traditional classifiers, BERT-based models, and prompt-only LLMs under both standard and cross-genre conditions. This makes ALHD relevant to detector construction, robustness analysis, and failure-mode characterization, rather than only to supervised training.
A central result is that strong performance on random or source-mixed splits does not guarantee out-of-domain robustness. This suggests that ALHD is as much a stress test for generalization as it is a dataset for binary classification.
2. Dataset composition and linguistic coverage
ALHD contains 405,456 samples, of which 101,364 are human and 304,092 are LLM-generated; the generated portion corresponds to three machine generations for each human sample (Khairallah et al., 3 Oct 2025). The corpus covers three genres: news, social media, and reviews. The human texts are drawn from six Arabic corpora: SANAD and ANAD for news, MDAT and MASC for social media, and HARD and BRAD for reviews.
The linguistic coverage includes both MSA and dialectal Arabic. The dialectal inventory explicitly includes Algerian, Egyptian, Lebanese, Moroccan, and Tunisian dialects, with coverage varying by source. Social media and review sources are noted as particularly skewed toward dialectal content, whereas the news sources are MSA-oriented.
Each sample includes structured metadata fields that support controlled experiments. These fields are text, label, generator, source, category, subcategory, token_count, and document_id. The generator field takes values from {GPT-3.5-turbo, Gemini-2.5-flash, Command-R, Human}, while document_id links the three machine generations to the corresponding human text. This paired structure enables source-controlled comparison between human and generated writing on the same topic and in the same dialectal regime.
| Aspect | Value |
|---|---|
| Total samples | 405,456 |
| Human samples | 101,364 |
| LLM-generated samples | 304,092 |
| Genres | News, social media, reviews |
| Generators | GPT-3.5-turbo, Gemini-2.5-flash, Command-R |
| Language varieties | MSA and dialectal Arabic |
The corpus is balanced at the construction stage. The paper states that each human document has exactly three corresponding LLM-generated versions, and that each of the six sources contributes equally with 16,894 documents per generator. This design reduces trivial priors tied to label frequency, source imbalance, or generator imbalance.
3. Construction methodology, preprocessing, and annotation
The human side of ALHD is assembled from publicly available Arabic corpora after preprocessing steps that remove duplicates, nulls, and extremely short texts, with a minimum length of 4 tokens (Khairallah et al., 3 Oct 2025). The LLM side is generated by prompting three models to produce a new, original Arabic text on the same topic and in the same dialect as the reference, while matching its length and avoiding simple paraphrase or summarization.
The three generators used are GPT-3.5-turbo, Gemini-2.5-flash, and Command-R. The summary reports the associated model characteristics as follows: GPT-3.5-turbo with 16,385-token context and 4,096-token maximum output; Gemini-2.5-flash with up to 1M context tokens and 65K output; and Command-R with 128K context and approximately 4K output, optimized for Arabic and RAG-enabled. Default API parameters are used, with system prompts for GPT-3.5-turbo and Command-R only. Post-processing includes sanity checks for empty, null, or duplicate texts.
Balancing is formalized through the minimum number of eligible document IDs across sources: where is the set of eligible document IDs for source . This ensures uniform contribution from each source under the benchmark construction procedure.
The annotation process is primarily automatic. Labels are assigned from the generation pipeline itself, distinguishing human texts from those produced by a specific generator. Genre and, where possible, dialect are derived from the source datasets. The paper also emphasizes strict cleaning, deduplication, and inspection to reduce contamination or leakage.
The dataset is publicly released through Zenodo, and the benchmarking code is released separately. A plausible implication is that the authors intended ALHD to function as a reproducible community benchmark rather than as a one-off experimental resource.
4. Benchmark design and evaluation protocol
ALHD defines a standardized benchmarking setup with train/validation/test splits of 70/15/15 under random splitting, as well as harder cross-source and cross-genre settings such as leave-one-source-out and leave-entire-genre-out evaluation (Khairallah et al., 3 Oct 2025). The benchmark also includes 10% and 100% size variants to support data-scaling studies and resource-constrained experimentation.
The reported metrics are accuracy, macro F1, and ROC-AUC, with precision, recall, and confusion matrices used for error analysis. The paper explicitly gives the standard accuracy formula,
and defines macro F1 and ROC-AUC in their standard classification sense.
Three model families are evaluated. Traditional machine learning models use TF-IDF features with max_features=10k and 1–2 grams, followed by Logistic Regression, LinearSVC, Complement Naive Bayes, LightGBM, or Random Forest, with 3-fold grid search for essential hyperparameters. BERT-based transformers include Arabic-specific and multilingual encoders such as AraBERTv2, AraElectra, ARBERTv2, MARBERTv2, Asafaya-BERT, mBERT, and XLM-RoBERTa, all fine-tuned on ALHD. Prompt-only LLM classifiers include Qwen2.5-7B-Instruct, JAIS-13B-Chat, GPT-OSS-20B, ALLaM-7B-Instruct-preview, C4AI-Command-R7B-Arabic, and Gemma-3-12B-IT in zero-shot and few-shot settings, with outputs constrained to 0 or 1.
This benchmark design is technically important because it separates three distinct questions: how well detectors fit the ALHD distribution, how well they transfer to unseen sources or genres, and whether large generative models can act as competent detectors without fine-tuning.
5. Empirical findings and generalization failures
The main empirical finding is that fine-tuned BERT-based models outperform both traditional baselines and prompt-only LLMs on ALHD (Khairallah et al., 3 Oct 2025). The mean accuracy for BERT-based models is reported as 0.902, with the best models reaching up to 0.994 accuracy in certain splits and ROC-AUC above 0.98. Traditional machine learning methods provide strong baselines, with mean accuracy 0.804 and best all-source results around 0.91–0.93 for Linear SVC and Logistic Regression. Prompt-only LLMs perform worst, with mean accuracy 0.497, כלומר effectively near chance, and JAIS-13B-Chat as the best zero-shot model at 0.62.
| Model family | Summary performance |
|---|---|
| BERT-based models | Mean accuracy 0.902; best up to 0.994 |
| Traditional ML | Mean accuracy 0.804; best around 0.91–0.93 |
| Prompt-only LLMs | Mean accuracy 0.497; best zero-shot 0.62 |
A recurring misconception is that general-purpose LLMs can reliably detect machine-generated text by prompting alone. ALHD directly contradicts that view: prompt-only LLM classification is reported as consistently ineffective for robust detection without fine-tuning. By contrast, encoder-based classifiers fine-tuned on the task are much stronger.
The benchmark also shows a clear data-scaling effect. Moving from 10% to full dataset size improves the mean accuracy of BERTs and classical models; the summary gives an example in which BERT mean accuracy rises from 0.892 to 0.911. This indicates that ALHD is large enough to support nontrivial scaling analyses.
The most consequential result, however, concerns cross-genre generalization. Performance degrades sharply when models are tested on unseen genres or sources. News is the most difficult regime: the paper reports that both BERTs and traditional methods can drop to approximately 0.62 or lower in news-related cross-genre settings, with some traditional models collapsing to about 0.55 accuracy. The authors attribute this difficulty to the stylistic similarity between human and LLM-generated news articles, making news a particularly hard domain for Arabic LLM-text detection.
The dataset analysis also identifies variation in dialect prevalence and sequence length as important complicating factors. LLMs and humans exhibit different lexical fingerprints, with LLMs reportedly overusing connectors and modal particles, while human texts contain more formal news terms and numerals. This suggests that some detectors may exploit superficial cues, which could partly explain degraded transfer when those cues shift out of domain.
6. Research significance, applications, limitations, and future directions
ALHD is presented as a foundational benchmark for Arabic NLP safety and for the broader study of human-versus-LLM detection in morphologically rich and stylistically diverse settings (Khairallah et al., 3 Oct 2025). The paper explicitly identifies applications in misinformation mitigation, academic integrity, cybersecurity, and digital forensics. In cybersecurity, the dataset is framed as relevant to detecting phishing, smishing, and social-engineering content generated in Arabic. In academic settings, it supports work on detecting student essays or papers generated by LLMs.
The benchmark’s broader scientific significance lies in the gap it fills. According to the paper, no prior Arabic dataset offered comparable scale, genre breadth, and dialectal coverage for this task. ALHD therefore establishes a common reference point for detector development, cross-genre robustness studies, and reproducibility.
The paper also states several limitations. The covered genres are limited to news, social media, and reviews; dialectal coverage is extensive but not exhaustive; only three LLMs are used as generators; and the LLM baselines are prompt-only rather than fine-tuned because of resource limits. These limitations are significant because they define the present scope of the benchmark without overstating its universality.
The proposed future directions follow directly from the observed failure modes. The authors point to the need for improved domain adaptation, robustness to subtle genre and style shifts, and potentially more adversarial or more human-like generations in future datasets. This suggests that ALHD should be understood not as a solved benchmark, but as a deliberately challenging starting point for robust Arabic LLM-generated text detection.
