AS-RoBERTa: Arabic-Script RoBERTa Model

• The paper introduces AS-RoBERTa, which leverages script-aware pretraining to balance shared orthographic cues with language-specific features in Arabic-script languages.
• It employs a RoBERTa-base transformer with dual tokenizers and an Orthographic Consistency Adapter, yielding measurable improvements over multilingual baselines.
• The model consistently enhances news classification accuracy and reduces log loss across Kurdish, Arabic, Persian, and Urdu datasets.

Arabic Script RoBERTa (AS-RoBERTa) is a family of script-focused, language-adapted RoBERTa models for four languages that share the Arabic/Perso-Arabic script: Kurdish Sorani, Arabic, Persian, and Urdu. It is proposed on the premise that multilingual models such as mBERT and XLM-RoBERTa can exploit broad cross-lingual sharing, but often underperform when the target languages share script yet differ in orthography, morphology, lexicon, and domain conventions. The paper positions AS-RoBERTa as a middle ground between fully multilingual models and monolingual models, with the central claim that script-centric pretraining can recover some of the benefits of multilingual transfer while preserving language-specific discrimination (Abdullah et al., 24 Jul 2025).

1. Conceptual scope and motivation

The motivating problem is not generic multilingual transfer in the abstract, but the specific case of Arabic-script languages whose writing systems overlap while their linguistic behavior diverges. The paper argues that shared character shapes and subword patterns can help, but that generic multilingual pretraining can also blur language-specific cues. AS-RoBERTa is therefore designed to learn orthographic regularities and language-specific representations more directly by pretraining on a balanced Arabic-script corpus and then fine-tuning on language-specific classification tasks (Abdullah et al., 24 Jul 2025).

The paper characterizes AS-RoBERTa in two closely related ways. In the abstract, it introduces “the Arabic Script RoBERTa (AS-RoBERTa) family: four RoBERTa-based models, each pre-trained on a large corpus tailored to its specific language.” In the detailed description, it is presented as a RoBERTa-base-style transformer trained from scratch on a unified multilingual Arabic-script corpus spanning Arabic, Persian, Urdu, and Kurdish Sorani. This suggests that the paper’s central emphasis is less on a single monolithic multilingual encoder than on script-aware specialization over a restricted Arabic-script language set (Abdullah et al., 24 Jul 2025).

A common point of confusion is the relation between script sharing and language identity. The paper explicitly treats these as distinct: shared script advantage and language-specific divergence are simultaneous facts. The proposed model is therefore not based on the assumption that Arabic-script languages are interchangeable; rather, it attempts to exploit orthographic commonality without collapsing morphology, lexicon, syntax, and domain conventions into a single undifferentiated representation space (Abdullah et al., 24 Jul 2025).

2. Model formulation and architectural components

AS-RoBERTa is described as a RoBERTa-base-style transformer built around a 12-layer encoder with hidden size $d = 768$. For an input sequence $x = [x_1, \ldots, x_n]$, the model uses two tokenizers, Byte-Pair Encoding (BPE) and WordPiece (WP), and fuses their embeddings by averaging. The fused representations are then passed through $L = 12$ Transformer encoder layers, matching a RoBERTa-base configuration (Abdullah et al., 24 Jul 2025).

A distinctive internal component is the Orthographic Consistency Adapter (OCA), which is inserted inside each transformer layer to encourage alignment between orthographically related characters. The forward pass is described as embeddings $\rightarrow$ transformer layers $\rightarrow$ OCA $\rightarrow$ final contextual states. The final contextual representation is denoted $H^{(L)}$, and the sequence representation used for classification is the first token, $h_{\text{CLS}} = H[1]$ (Abdullah et al., 24 Jul 2025).

Before classification, the CLS embedding is passed through a domain-adaptive projection: $$h_{\text{DA}} = \phi(W_{\text{DA}} h_{\text{CLS}} + b_{\text{DA}}),$$ where $\phi$ is a nonlinearity such as GeLU. The classifier then predicts

$x = [x_1, \ldots, x_n]$0

The paper emphasizes that this decouples representation learning from the final decision boundary (Abdullah et al., 24 Jul 2025).

The pretraining objective combines masked language modeling with an orthographic masking term. The MLM loss is written as

$x = [x_1, \ldots, x_n]$1

the orthographic masking loss as

$x = [x_1, \ldots, x_n]$2

and the total pretraining objective as

$x = [x_1, \ldots, x_n]$3

with $x = [x_1, \ldots, x_n]$4. Fine-tuning uses cross-entropy together with cross-linguistic KL regularization based on a transliterated version $x = [x_1, \ldots, x_n]$5: $x = [x_1, \ldots, x_n]$6 with $x = [x_1, \ldots, x_n]$7 (Abdullah et al., 24 Jul 2025).

3. Pretraining data, tokenization, and orthographic consistency

The pretraining corpus is explicitly balanced at 4.0 billion tokens total, with 1.0 billion tokens per language. The source corpora are language-specific: Arabic Wikipedia, OSCAR web crawl, and assorted news articles for Arabic; Persian Wikipedia, newswire feeds, and social media texts for Persian; Urdu Wikipedia, news websites, and online forums for Urdu; and Kurdish Wikipedia, news articles, AsoSoft Open Corpus (Veisi et al., 2019), and additional web sources for Kurdish Sorani (Abdullah et al., 24 Jul 2025).

All languages undergo HTML/script removal, punctuation normalization, whitespace normalization, and lowercasing where appropriate. For Kurdish Sorani, the preprocessing additionally includes conversion of Arabic-specific characters to Kurdish equivalents and diacritic removal. The paper also states that a single BPE tokenizer with 100k vocabulary is learned on the combined corpus, although this is slightly in tension with the earlier architectural description that uses dual tokenizers. The text nevertheless makes clear that a shared subword strategy is used to support consistent segmentation across languages (Abdullah et al., 24 Jul 2025).

Orthographic consistency is defined as the model’s ability to treat script-level variants that correspond to the same linguistic unit as equivalent. The examples given are different Unicode forms of the same character, Arabic/Kurdish character substitutions, and script variants that preserve meaning but differ in orthography. In operational terms, orthographic consistency is enforced through the orthographic masking loss $x = [x_1, \ldots, x_n]$8 and the OCA modules within the transformer stack (Abdullah et al., 24 Jul 2025).

The broader methodological claim is that script-specific pretraining means restricting the corpus to the Arabic script family languages of interest rather than relying on a broad multilingual corpus. The paper argues that such restriction is beneficial because shared script helps transfer orthographic and subword patterns, whereas broad multilingual training can dilute attention to language-specific statistics. A plausible implication is that the method is intended to preserve script-mediated sharing while avoiding the vocabulary and capacity fragmentation typical of massively multilingual encoders (Abdullah et al., 24 Jul 2025).

4. Downstream evaluation protocol and reported performance

The downstream tasks are language-specific news/topic classification benchmarks. For Kurdish Sorani, the dataset is Kurdish News Dataset Headlines (KNDH), with 50,000 headlines and five categories: Politics, Society, Sports, Technology, and Culture; the split is 45,000 train and 5,000 test. For Arabic, the dataset is from OSAC Open Arabic Corpora, supplemented with articles from Assabah and Al Jazeera, comprising 100,000 sentences or short paragraphs with five classes: Politics, Business, Sports, Tech, and Entertainment; the split is 90% train and 10% test. For Persian, the benchmark is based on the Hamshahri news corpus and BBC Persian, with 60,000 documents and five classes similar to Arabic; the split is 90% train and 10% test. For Urdu, the dataset contains 50,000 news headlines from Urdu news websites with a four-category scheme: Business, Technology, Entertainment, and Sports; the split is 90% train and 10% test (Abdullah et al., 24 Jul 2025).

The evaluation protocol uses Accuracy, Precision, Recall, F1 score, and Log loss. A validation protocol holds out 10% of each training set for validation, applies early stopping based on validation loss, fine-tunes each model three times with different random seeds, and reports averages. Statistical significance is tested using paired t-tests on per-document accuracy, and all reported improvements are said to be significant at $x = [x_1, \ldots, x_n]$9. The datasets are described as balanced, which the paper uses to justify accuracy as the primary metric, while macro-averaged F1 is also reported to ensure class-balanced performance (Abdullah et al., 24 Jul 2025).

Language	Baseline comparison	AS-RoBERTa result
Kurdish	mBERT 75.8%, XLM-RoBERTa 77.8%	80.5% accuracy, F1 80.1, log loss 0.35
Arabic	mBERT 88.3%, XLM-RoBERTa 90.3%	92.4% accuracy, F1 92.1, log loss 0.28
Persian	mBERT 86.1%, XLM-RoBERTa 88.1%	90.0% accuracy, F1 89.7, log loss 0.30
Urdu	mBERT 83.5%, XLM-RoBERTa 85.5%	88.2% accuracy, F1 87.9, log loss 0.33

The reported gains are consistent across languages. For Kurdish, AS-RoBERTa improves over mBERT by 4.7 points and over XLM-RoBERTa by 2.7 points. For Arabic, the improvements are 4.1 and 2.1 points respectively. For Persian, they are 3.9 and 1.9 points. For Urdu, they are 4.7 and 2.7 points. The paper summarizes the pattern as approximately 2–5 percentage points over multilingual baselines, and it also notes that AS-RoBERTa achieves the lowest log loss in every language (Abdullah et al., 24 Jul 2025).

5. Ablation evidence and error-analysis findings

The ablation study compares models trained from random initialization, multilingual pretrained models such as mBERT and XLM-R, and AS-RoBERTa. The main conclusion is that models trained from scratch are the weakest, performing about 15–20 percentage points below pretrained variants, that multilingual pretraining helps substantially, and that language-adapted pretraining helps further. The paper provides a concrete Kurdish example in which a scratch model reaches 60.4% while AS-RoBERTa reaches 80.5% (Abdullah et al., 24 Jul 2025).

The same section reports that AS-RoBERTa improves over XLM-RoBERTa by about 2.5 points on average and reduces log loss by 0.07–0.10 across languages. The interpretation offered is that unsupervised pretraining is essential, but that script-focused pretraining yields an additional measurable gain beyond generic multilingual pretraining. In the paper’s framing, the improvement is therefore attributed not merely to using a transformer, but to pretraining on the right data distribution, namely Arabic-script corpora (Abdullah et al., 24 Jul 2025).

The confusion-matrix analysis shows strong diagonal dominance across all languages, with most misclassifications occurring between semantically adjacent categories. The reported examples are confusion between Politics and Business in Arabic, between Sports and Entertainment in Persian together with some Politics/World-News-like overlap, between Technology and Business in Urdu, and more overall confusion in Kurdish, especially Culture versus Politics. The authors interpret these errors as arising from shared script traits, domain overlap, and language-specific borrowing and terminology (Abdullah et al., 24 Jul 2025).

The error analysis is significant because it constrains the interpretation of orthographic consistency. The paper argues that shared script traits can help, but can also obscure fine distinctions when classes share vocabulary. It also notes that many errors are intuitive and even plausible to human annotators, especially for articles on government economic policy, technology/business overlap, and cultural discussions involving religion or language policy. This suggests that orthographic regularization can improve topical discrimination without eliminating ambiguity where class boundaries are themselves fuzzy (Abdullah et al., 24 Jul 2025).

6. Position within Arabic-script encoder research and model boundaries

AS-RoBERTa belongs to the broader Arabic-script encoder landscape, but it is distinct from other prominent model families. ARBERT and MARBERT are Arabic-specific deep bidirectional transformer-based models rather than models literally named AS-RoBERTa. ARBERT targets Modern Standard Arabic and related formal text, while MARBERT is aimed at dialectal Arabic and social media, especially Twitter. MARBERT is the closest conceptual match to an Arabic-script RoBERTa variant in the earlier literature because it uses BERT-base architecture, is trained from scratch on massive Arabic tweet data, and drops next sentence prediction, explicitly motivated by the RoBERTa finding that NSP is not crucial (Abdul-Mageed et al., 2020).

AraModernBERT occupies a different architectural point. It is an adaptation of the ModernBERT encoder architecture to Arabic with 22 transformer layers, 768 hidden size, 12 attention heads, alternating local/global attention, native long-context support up to 8,192 tokens, rotary positional embeddings, and a training objective based on MLM. The paper explicitly states that it is best understood as an Arabic ModernBERT adaptation or an Arabic-script long-context encoder, and only loosely as a RoBERTa-style Arabic model because it remains an encoder-only masked LM (Elshehy et al., 10 Feb 2026).

Another boundary concerns task modality. The paper “Cross-Language Learning within Arabic Script for Low-Resource HTR” studies low-resource handwritten text recognition with cross-script joint training across Arabic, Urdu, and Persian. Its architecture is CRNN with 3 stacked bidirectional LSTM layers and CTC loss, with an auxiliary CTC shortcut branch used only during training. It is explicitly not AS-RoBERTa because it addresses handwritten line images rather than text-only language modeling, uses CRNN + BiLSTM + CTC rather than a RoBERTa transformer, and focuses on visual/script overlap and character-level transfer rather than masked-language-model pretraining (Al-Azzawi et al., 3 May 2026).

These comparisons clarify that AS-RoBERTa is specifically a script-aware RoBERTa-based text classification framework. It is neither a generic label for Arabic encoders nor a synonym for Arabic-script transfer methods more broadly. Its defining ingredients are RoBERTa-base-style encoding, script-restricted pretraining over four Arabic-script languages, orthographic consistency modeling, and downstream evaluation on balanced news classification benchmarks (Abdullah et al., 24 Jul 2025).

7. Limitations and stated future directions

The paper explicitly notes that the experiments are limited to balanced news-domain datasets only. As stated, the results may not generalize to social media, technical text, or other genres. The study also uses a single model size, namely RoBERTa-base, and does not explore larger or smaller architectures (Abdullah et al., 24 Jul 2025).

A second limitation is the absence of cross-lingual label transfer evaluation. The paper avoids zero-shot or few-shot transfer comparisons, which simplifies experimental design but limits insight into transfer settings. Language coverage is also restricted to four Arabic-script languages, and the study does not test other Arabic-script varieties or unrelated scripts (Abdullah et al., 24 Jul 2025).

The final caveat is conceptual: orthography versus semantics. The authors state that while orthographic consistency helps, it is not sufficient to solve all ambiguity, especially where classes are semantically close. The future directions they suggest are larger architectures, more diverse genres such as social media and technical content, hybrid pretraining strategies combining multilingual and monolingual signals, zero-shot and few-shot cross-lingual transfer, and broader validation of script-based strategies across other linguistic settings. Within those limits, the paper’s main message is that for Arabic-script NLP, script-aware pretraining matters, and that orthographic consistency plus language-specific distributional learning can yield consistent and statistically significant gains over strong multilingual baselines (Abdullah et al., 24 Jul 2025).