AraBERT: Arabic NLP Transformer

• AraBERT is a deep transformer-based language model family designed for Arabic NLP that employs specialized tokenization and morphological segmentation.
• It leverages large-scale Arabic corpora and state-of-the-art fine-tuning strategies to excel in sentiment analysis, NER, QA, and dialect identification.
• Empirical studies show that pre-segmentation and ensemble approaches significantly improve AraBERT's performance in handling Arabic linguistic complexities.

AraBERT is a family of deep transformer-based LLMs purpose-built for Arabic NLP. Designed to address the linguistic, orthographic, and morphological complexities inherent in Arabic, AraBERT employs tailored tokenization strategies, large-scale Arabic corpora, and architectures derived from BERT to establish state-of-the-art performance for a variety of downstream NLP tasks, including sentiment analysis, named entity recognition, question answering, dialect identification, and domain-specific classification (Antoun et al., 2020, Wadhawan, 2021, Hamdi et al., 2 Sep 2025).

1. Model Architecture, Tokenization, and Pretraining

AraBERT adopts the standard BERT transformer architecture with configurations matching BERT-Base (12 layers, 768 hidden size, 12 attention heads; ≈110M–136M parameters; max sequence length 512) and BERT-Large (24 layers, 1024 hidden size, 16 attention heads; ≈371M parameters) for selected releases (Antoun et al., 2020, Wadhawan, 2021). Crucially, Arabic-specific adjustments are implemented at the tokenization level to address high morphological dispersion and surface-form sparsity.

Morphological Segmentation and Subword Modeling

AraBERTv1 and later variants employ Farasa for morphological segmentation, splitting each token into prefix clitics, stem, and suffix clitics (e.g., “اللغة” becomes “ال + لغ + ة”). The segmented text is then processed using a subword unigram model (SentencePiece), resulting in a 64k-token vocabulary—significantly mitigating token redundancy caused by Arabic's concatenative morphology (Antoun et al., 2020). AraBERT releases trained exclusively on raw text (v0.1, v0.2) and those incorporating pre-segmentation (v1, v2) enable empirical comparisons regarding the impact of segmentation.

Pretraining Data and Objectives

Pretraining leverages 24–77 GB of deduplicated, predominantly news and Wikipedia Arabic corpora (encompassing the 1.5B-word Arabic Corpus, OSIAN, Arabic Wikipedia, OSCAR, Assafir news), equivalent to ~70–200 million sentences and up to 8.66 billion words (Antoun et al., 2020, Wadhawan, 2021). The masked language modeling (MLM) objective with whole-word masking is used alongside next sentence prediction (NSP), consistent with standard BERT training protocols:

• MLM loss:

$$L_\mathrm{mlm} = - \mathbb{E}_{x\sim D} \sum_{i \in M} \log P(x_i | \tilde{x})$$

• NSP loss:

$$L_\mathrm{nsp} = - \mathbb{E}_{(A,B,y)\sim D} [y \log P(\mathrm{IsNext}|A,B) + (1-y)\log P(\mathrm{NotNext}|A,B) ]$$

• Total loss: $L = L_\mathrm{mlm} + L_\mathrm{nsp}$

Training is performed on TPU-v2/8 with 1,250,000 steps, employing batch sizes of 512 (seq-len=128) and 128 (seq-len=512) (Antoun et al., 2020).

2. Preprocessing Pipelines: Farasa Segmentation and LLM-based Methods

For all tasks, input texts are preprocessed to ensure maximal compatibility between pretraining and fine-tuning data distributions. This includes, for pre-segmented models:

• Applying Farasa segmentation: token decomposition into morphemes (clitics and stems)
• Normalizing noisy artifacts: replacing URLs with [رابط], emails with [بريد], and mentions with [مستخدم]; stripping HTML, emojis, emoticons, and collapsing repeated characters, as outlined in specific pipelines (Wadhawan, 2021)
• Ensuring orthographic normalization and explicit whitespace for numeric and symbolic tokens

Morphological segmentation is succinctly defined as follows: Given $w$ (a word), Farasa decomposes

$w = c_1 \ldots c_k \cdot s \cdot e_1 \ldots e_m$

with $c_i$ prefix clitics, $s$ the stem, and $e_j$ suffix clitics.

LLM-based preprocessing, as explored in disease classification tasks (Hamdi et al., 2 Sep 2025), introduces three multi-layered strategies before AraBERT fine-tuning:

1. Refinement: Cleaning, grammatical correction, and condensation of user-generated text while retaining context.
2. Summarization: Abstractively generating concise representations preserving primary symptoms and background.
3. NER Extraction: Listing explicit medical entities (symptoms, durations, conditions) as flat entity sets.

Each variant is paired with the raw input for model fine-tuning.

3. Downstream Fine-Tuning Strategies

General Principles

Fine-tuning is performed end-to-end with classification heads—generally a single linear layer with output size matching the number of classes—added atop the [CLS] vector (Antoun et al., 2020, Wadhawan, 2021, Hamdi et al., 2 Sep 2025). Model parameters are fully unfrozen, and no architectural modifications are made apart from dropout (rate = 0.05) and new task-specific heads.

Representative Tasks and Hyperparameters

• Dialect Identification (NADI 2021):
  • Datasets: 21,000 train, 5,000 validation, 5,000 test tweets
  • Maximum sequence length: 256
  • Batch size: 40 (base), 4 (large)
  • Optimizer: Adam ($\epsilon=1e{-8}$)
  • Learning rate: $1e{-5}$
  • No early stopping or LR scheduling; fixed 5 epochs
  • Cross-entropy loss:

  $$L_\mathrm{nsp} = - \mathbb{E}_{(A,B,y)\sim D} [y \log P(\mathrm{IsNext}|A,B) + (1-y)\log P(\mathrm{NotNext}|A,B) ]$$0

• Medical Text Classification:

  • Four variants: Raw, Refined, Summarized, NER
  • Seven-class output (disease specialties)
  • Batch size: 4, learning rate: $$L_\mathrm{nsp} = - \mathbb{E}_{(A,B,y)\sim D} [y \log P(\mathrm{IsNext}|A,B) + (1-y)\log P(\mathrm{NotNext}|A,B) ]$$1, epochs: 25, weight decay: 0.01, AdamW (Hamdi et al., 2 Sep 2025)

4. Task Performance and Benchmarking

AraBERT has been evaluated on a comprehensive suite of standard Arabic NLP tasks:

• Sentiment Analysis (SA): On datasets such as HARD, ASTD, LABR, AJGT, and ArSenTD-Lev, AraBERTv1 achieves state-of-the-art accuracy, e.g., 96.1% (HARD), 92.6% (ASTD), 93.8% (AJGT), outperforming both mBERT and older SOTA models (Antoun et al., 2020).
• Named Entity Recognition (NER): On ANERcorp, AraBERT outperforms mBERT and BiLSTM-CRF baselines with macro-F1 up to 84.2%. However, segmentation can occasionally interfere with IOB boundaries.
• Question Answering (QA): On ARCD and machine-translated SQuAD, AraBERTv1 achieves F1 of 62.7% and Exact Match of 30.6%. Error analysis reveals most failures occur for missing function words or prepositions.

Dialect identification (NADI): Macro-F1 scores from 0.216–0.235 for country-level, 0.043–0.054 for province-level tasks (MSA/dialect tweets), with best development set results from AraBERTv2-large (Wadhawan, 2021).

Domain-level Text Classification (Arabic medical telehealth): Stand-alone AraBERT achieved 71.79–72.41% accuracy post-refinement, slightly lower than CAMeLBERT and AsafayaBERT. Majority-voting ensemble raised overall accuracy to 80.56% (Hamdi et al., 2 Sep 2025).

Task (Dataset)	Key Metric	mBERT	AraBERTv0.1	AraBERTv1
SA (ASTD)	Accuracy	80.1%	92.2%	92.6%
NER (ANERcorp)	Macro-F1	78.4%	84.2%	81.9%
QA (ARCD, F1/EM/SM)	62.7% / 30.6% / 92.0%
Medical classif. (Refined)	Accuracy	–	71.79%	72.41%

5. Analytical Insights and Model Variants

Pre-segmentation vs. Raw Tokenization

Empirical ablation studies confirm significant gains from Farasa pre-segmentation: SAs and QA scores improve by 0.4–2.1 points, attributed to lower effective vocabulary size and improved morpheme coverage (Antoun et al., 2020). For NER, non-segmented models sometimes outperform due to label misalignment caused by boundary fragmentation.

Layer Capacity and Variant Selection

AraBERTv2-large is optimal for coarse-grained dialect/country-level tasks (NADI), while base variants suffice for high-resolution province or fine-grained tasks, balancing parameter count and overfitting risk (Wadhawan, 2021).

Multi-layered Ensemble Effects

Aggregating predictions from multiple architectures and input views (original, refined, summarized, NER) via majority voting increases robustness and nets higher accuracy (up to +10 percentage points over any single variant) in domain-classification setups (Hamdi et al., 2 Sep 2025).

6. Limitations and Ongoing Challenges

Persistent issues include subtle semantic errors (dropping function words in QA), NER boundary confusion under morphological splitting, code-switching and dialectal divergence across tasks, and degradation on noisy or domain-shifted data (Antoun et al., 2020, Wadhawan, 2021, Hamdi et al., 2 Sep 2025). Systematic per-class error analysis is typically lacking, limiting interpretability regarding error localization.

A plausible implication is that best-practice application of AraBERT in real-world scenarios requires careful alignment of input preprocessing pipeline with the pretraining configuration, as even small deviations can significantly lower downstream accuracy.

7. Summary and Impact

AraBERT, through Arabic-aware tokenization, large-scale monolingual pretraining, and robust fine-tuning, has established new state-of-the-art performance on a broad spectrum of Arabic NLP benchmarks (Antoun et al., 2020). Its design exemplifies the importance of morphology-informed processing pipelines. Model variants allow for effective adaptation to task granularity (country vs. province dialect identification), and ensemble strategies further augment overall accuracy in complex domain-specific scenarios.

The AraBERT series is foundational for ongoing Arabic NLP research, with open pretrained models made available for reproducibility and further development (Antoun et al., 2020, Wadhawan, 2021, Hamdi et al., 2 Sep 2025).