CAMeLBERT: Arabic Pre-trained Models

• CAMeLBERT is a suite of controlled Arabic pre-trained language models designed to isolate the effects of language variants and data size on downstream NLP tasks.
• It systematically compares variant-specific models for MSA, dialectal Arabic, and classical Arabic, revealing that task performance depends more on variant proximity than on large corpus size.
• The suite serves as both a robust benchmark and a complementary model in ensemble systems, enhancing Arabic NLP applications in areas like sentiment analysis, retrieval, and disease prediction.

CAMeLBERT is a suite of pre-trained LLMs for Arabic designed to study, under controlled conditions, how language variant, data size, and task type affect downstream NLP performance. The model family comprises variant-specific and mixed-data models for Modern Standard Arabic (MSA), dialectal Arabic (DA), classical Arabic (CA), and a combined setting, and its central empirical result is that variant proximity of pre-training data to fine-tuning data is more important than pre-training data size for Arabic NLP (Inoue et al., 2021).

1. Controlled formulation of the model family

CAMeLBERT is presented as a deliberately controlled Arabic pre-trained LLM suite rather than a single monolithic checkpoint. The original study varies three principal factors: the Arabic variant used in pre-training, the amount of pre-training data, and the downstream task type. This design is significant because earlier Arabic models often differed along multiple dimensions simultaneously, which made causal interpretation difficult. In the CAMeLBERT formulation, the intent is to isolate which variables materially drive downstream behavior (Inoue et al., 2021).

The same study situates CAMeLBERT against eight publicly available competitors: mBERT, AraBERT v0.1/v0.2, ArabicBERT, Multi-dialect-Arabic-BERT, GigaBERT, MARBERT, and ARBERT. That comparison places CAMeLBERT within the broader Arabic pre-trained LLM landscape while preserving the main analytical emphasis on controlled model construction rather than unconstrained model scaling (Inoue et al., 2021).

A recurrent theme in later work is that CAMeLBERT is valuable not only as an Arabic-specific baseline but also as a probe for understanding when Arabic variant matching, dialectal coverage, or ensemble complementarity matter most. This suggests that the importance of CAMeLBERT is methodological as much as architectural.

2. Variant structure and system selection

The original CAMeLBERT study reports four principal pre-trained models, plus reduced-size MSA models for size analysis (Inoue et al., 2021).

Variant	Pre-training data
CAMeLBERT-MSA	Only MSA, 107GB
CAMeLBERT-DA	Only dialectal Arabic, 54GB
CAMeLBERT-CA	Only classical Arabic, 6GB
CAMeLBERT-Mix	Mix of all three, 167GB

This organization makes variant selection a first-class modeling decision. The paper further defines CAMeLBERT-Star, a system-selection configuration in which the model is chosen according to task variant, such as using the MSA model for MSA data and DA/CA/Mix for DA/CA. In aggregate evaluation, this selection strategy becomes one of the strongest CAMeLBERT configurations (Inoue et al., 2021).

The model family also includes additional reduced-size MSA models. Their purpose is not to produce a separate deployment line, but to test how much downstream performance changes as pre-training data scale is reduced. That experiment underpins one of the main conclusions of the paper: data size matters less than expected once variant mismatch enters the picture.

3. Benchmark evidence on task type, variant proximity, and OOV effects

CAMeLBERT was evaluated on five core NLP task types spanning 12 datasets: NER on ANERcorp and PATB; POS tagging on PATB, ARZTB, and Gumar; sentiment analysis on ASTD, ArSAS, and SemEval; dialect identification on MADAR-26, MADAR-6, MADAR-Twitter-5, and NADI; and poetry meter classification on APCD (Inoue et al., 2021).

Several benchmark findings are central to CAMeLBERT’s interpretation. First, AraBERTv02 has the highest reported macro-average at 79.5, while CAMeLBERT-Star is a close second at 79.4, followed by CAMeLBERT-MSA at 79.2 and CAMeLBERT-Mix at 78.7. Second, CAMeLBERT-Star is best or ties for best in 4/12 subtasks, while AraBERTv02 is best or ties for best in 6/12 subtasks; taken together, CAMeLBERT-Star + AraBERTv02 win/tie for best in 9/12 tasks. This complementarity is one of the most important empirical outcomes of the paper (Inoue et al., 2021).

The variant-specific patterns are explicit. CAMeLBERT-MSA is reported as best overall for MSA tasks. CAMeLBERT-DA is best for highly dialectal tasks such as MADAR-6. CAMeLBERT-CA is best for CA tasks, specifically poetry classification, despite using much less pre-training data. By contrast, CAMeLBERT-Mix tends to help on dialect-rich or mixed tasks, but less so on pure MSA/CA. These findings support the paper’s central claim that downstream alignment between task variant and pre-training variant dominates raw corpus scale (Inoue et al., 2021).

The quantitative size analysis reinforces that claim. Scaling MSA pre-training data from 6GB to 107GB has a very minor effect on downstream performance, with a macro-average difference only $\approx 0.6\%$, whereas variant proximity produces much larger changes, including up to 14.5% performance swing in NER. An additional OOV analysis shows that the CAMeLBERT model with the lowest OOV rate for a dataset is the one that performs best, with a strong negative correlation between OOV rate and task performance of $-0.82$ (Inoue et al., 2021).

A common misconception is that Arabic PLM quality is primarily a function of pre-training scale. CAMeLBERT’s benchmark results directly contradict that simplification.

4. Task-specific fine-tuning beyond the original benchmark

Later studies use CAMeLBERT in a variety of downstream settings that stress different aspects of Arabic modeling. In SemEval-2023 Task 12, Bert-base-arabic-camelbert-da-sentiment is used for Algerian Arabic (dz) and Moroccan Arabic/Darija (ma) sentiment classification, where it achieves 65.10 weighted F1 on Algerian Arabic and 58.50 on Moroccan Darija. For these two dialects, the model is reported as best or tied-best among the evaluated systems, which include Afro-xlmr-large, AfriBERTa-Large, mBERT, and BERT (Salahudeen et al., 2023).

In Quran passage retrieval, CAMeLBERT-Base is fine-tuned on an expanded dataset that grows from 251 questions to 629 questions and then to 1895 questions categorized into single-answer, multi-answer, and zero-answer types. On the expanded setting, CAMeLBERT reaches MAP@10 = 0.34 and MRR = 0.47, and improves No Answer Recall and No Answer Precision from 0.25 to 0.50. The same study reports that CAMeLBERT outperforms RoBERTa, BERT, and AraBERT-large, while remaining below AraBERT-base on MAP@10 and MRR (Basem et al., 2024).

In Arabic AI-generated text detection, CAMeLBERT is fine-tuned with the Hugging Face transformers library and, notably, does not apply any specific Arabic text normalization; instead, it relies entirely on its pre-trained tokenizer. On the AraGenEval test set, it achieves F1-score 0.7290, accuracy 0.710, precision 0.6842, recall 0.780, specificity 0.640, and balanced accuracy 0.710. It substantially outperformed AraELECTRA across all metrics, but XLM-RoBERTa achieved the highest performance with F1-score 0.7701 (Zain et al., 23 Oct 2025).

Taken together, these applications show that CAMeLBERT is competitive across dialectal sentiment analysis, retrieval-style religious QA, and authorship-origin classification, but not uniformly dominant. A plausible implication is that its strengths are most pronounced when either Arabic variant specialization or model complementarity is central to the task.

5. CAMeLBERT in ensemble-centric systems

One of the most detailed recent uses of CAMeLBERT appears in Arabic disease prediction from social telehealth data. In that framework, Arabic medical posts are preprocessed with LLAMA3 using Text Refinement, Summarization, and Named Entity Recognition (NER). For each post, the original text is paired with one preprocessed version, yielding (original + refined), (original + summarized), and (original + NER) variants. The CAMeLBERT model used is CAMeL-Lab/bert-base-arabic-camelbert-mix, and each fine-tuned model adds a dropout layer and linear classification head for 7-class (disease types) prediction. The reported hyperparameters are dropout 0.05, learning rate $1e{-4}$, batch size 4, epochs 25, weight decay 0.01, and Cross-Entropy loss (Hamdi et al., 2 Sep 2025).

The individual CAMeLBERT accuracies in that study are 70.53% for CAMeLBERT_Post, 75.55% for CAMeLBERT_Refined, 74.61% for CAMeLBERT_NER, and 64.89% for CAMeLBERT_Summarized. The reported interpretation is that Refined and NER variants significantly outperform the base Post variant, while Summarization hurts CAMeLBERT’s performance, likely due to loss of clinical details. In the larger majority voting ensemble across 12 models, the best overall accuracy is 80.56%, and CAMeLBERT_Refined plus CAMeLBERT_NER are described as critical sources of diversity and uplift (Hamdi et al., 2 Sep 2025).

CAMeLBERT also appears as a core component in a winning Arabic readability assessment system. There it is part of a confidence-weighted ensemble with AraBERTv2, AraELECTRA, and MARBERT, and is fine-tuned under three different objectives: Cross-Entropy Loss (CE), Regression Loss (REG/MSE), and Conditional Ordinal Regression Loss (COR) via the CORAL framework. The training configuration includes batch size 16, learning rate $2 \times 10^{-5}$, epochs 5, AdamW, EarlyStopping, D3TOK (CAMeL Tools) for morphological segmentation, and mixed precision via torch.cuda.amp (Basem et al., 12 Sep 2025).

Its reported single-model readability results are 83.1 QWK for CE, 84.5 QWK for REG, and 83.0 QWK for COR. The ALL ensemble, which includes CAMeLBERT, reaches 87.5% QWK at the sentence level and 87.4% at the document level. The same paper attributes part of the final performance to model and loss diversity, with a 6.3 percent QWK gain from targeted post-processing (Basem et al., 12 Sep 2025).

These ensemble-focused studies show CAMeLBERT functioning less as an isolated winner and more as a high-value component whose error profile, variant coverage, and tokenization behavior enrich multi-model systems.

6. Domain adaptation, comparative limitations, and broader significance

A further perspective emerges from Arabic mental health disorders detection, where AraBERT, CAMeLBERT, and MARBERT each undergo Domain-Adaptive and Task-Adaptive Pretraining (DAPT and TAPT) on a corpus of ~250,000 unique tweets, totaling about 20 million tokens, followed by supervised evaluation on 50,670 tweets across six categories. For CAMeLBERT, DAPT+TAPT uses Masked Language Modeling (MLM) with a 15% dynamic masking rate, monitored by validation loss and perplexity, with learning rate $4.93\times 10^{-5}$, batch size 32, epochs 5, and AdamW. Supervised fine-tuning then uses learning rate $2\times 10^{-5}$, batch size 32, and epochs 3 within a hierarchical two-stage classification setup (Almalki et al., 10 Jun 2026).

The comparative effect of adaptation is narrow but instructive. For CAMeLBERT, accuracy moves from 0.8668 to 0.8669, macro precision from 0.8459 to 0.8446, macro recall from 0.8622 to 0.8774, and macro-F1 from 0.8530 to 0.8525. Only the recall improvement, +0.0152 with $p=0.0028$, is reported as significant. By contrast, MentalMARBERT shows statistically significant improvements in both accuracy and macro-F1, and is selected for the subsequent phase of the study (Almalki et al., 10 Jun 2026).

This result is important because it tempers overly broad claims about CAMeLBERT. The evidence across the literature does not support the notion that CAMeLBERT is universally the best Arabic model. The original benchmark already showed that CAMeLBERT-Mix is not always superior, that variant-matched selection is often preferable, and that AraBERTv02 slightly leads on macro-average (Inoue et al., 2021). Later work further shows that a multilingual model, XLM-RoBERTa, can outperform CAMeLBERT on Arabic AI-generated text detection (Zain et al., 23 Oct 2025).

The broader significance of CAMeLBERT lies in three durable contributions. First, it provides a publicly released collection of Arabic models organized around explicit linguistic variation. Second, it demonstrates that variant matching is a central design principle for Arabic NLP, often more consequential than brute-force corpus growth. Third, it remains a strong complementary model in contemporary Arabic systems, especially where dialect diversity, morphological segmentation, retrieval robustness, or ensemble diversity are operationally important. This suggests that CAMeLBERT’s enduring value is as both an empirical benchmark for Arabic representation learning and a practical component in high-performing Arabic NLP pipelines.