chDzDT: Character-Level PLM for Algerian Dialects
- chDzDT is a character-level pre-trained language model tailored for Algerian dialects with complex morphology, code-switching, and multiple scripts.
- It trains on isolated words to capture morphological patterns without relying on subword tokenization or standardized orthography.
- Extensive evaluations show strong morphological encoding and robustness to orthographic variability, aiding various downstream tasks.
chDzDT is a character-level pre-trained LLM tailored for Algerian morphology and designed for Algerian social media text, where complex morphology, frequent code-switching, multiple scripts, and highly variable, nonstandard spelling complicate tokenization and reduce the effectiveness of conventional word- or subword-level approaches. Unlike conventional PLMs trained on sentence-level token sequences, chDzDT is trained on isolated words and encodes morphological patterns without depending on token boundaries or standardized orthography. Its contributions are threefold: a detailed morphological analysis of Algerian dialect using YouTube comments, the construction of a multilingual Algerian lexicon dataset, and the development and extensive evaluation of a character-level PLM as a morphology-focused encoder for downstream tasks (Aries, 1 Sep 2025).
1. Problem setting and linguistic motivation
The Algerian dialect remains under-represented in pretrained language modeling, despite the fact that Algerian social media text exhibits several properties that are particularly hostile to standard tokenization pipelines: complex morphology, frequent code-switching across Arabic, Berber, French, and English, multiple scripts including Arabic, Latin, Arabizi, and Tifinagh, and strong lexical influences from other languages (Aries, 1 Sep 2025). In this setting, sentences and even words often change languages or scripts mid-form, while loanwords and borrowings may be integrated into non-Arabic morphology.
The paper frames these phenomena as a direct challenge to conventional word- and subword-based PLMs. Small or fixed vocabularies are ill-suited to a high-variance environment, tokenization methods are poorly matched to noisy and code-mixed data, and conventional models provide no direct morphological modeling. The same word may appear in multiple orthographic realizations, as in 3lech, 3lach, and علاش, which makes token fragmentation and out-of-vocabulary behavior especially problematic (Aries, 1 Sep 2025).
The resulting design objective is explicitly morphology-focused rather than primarily sentence-semantic. chDzDT therefore targets word-level structure, with the claim that robust character-level representations can better absorb orthographic variation, script mixing, and code-switching than subword segmentation in this domain. This suggests a shift from tokenization-centric modeling toward morphology-centric encoding for low-resource, multiscript dialectal text.
2. Model design and formalization
chDzDT is presented as a lightweight BERT-style transformer operating at the character level and on isolated words rather than full sentences (Aries, 1 Sep 2025). Its symbolic notation is given as , described in the paper as character-level, deep representations, and time-variant transformation.
The model pipeline is straightforward. A word is tokenized into a sequence of characters; each character is mapped to a vector and combined with positional embeddings; a stack of Transformer blocks produces contextualized character representations. The paper specifies special tokens as \sout{C} for classification, \sout{P} for padding, \sout{K} for masking in MLM, and \sout{S} for separation if needed (Aries, 1 Sep 2025). All special tokens use single characters for robustness.
The core encoder is formalized as
where denotes character embeddings, positional embeddings, and the number of Transformer blocks (Aries, 1 Sep 2025).
The training objective is dual-head. First, masked language modeling is applied at the character level: random characters are masked and predicted through a character softmax. Second, the embedding of the classification token is used for word-level multi-label classification, predicting via sigmoid which language or languages a word belongs to; the possible labels are Arabic, Berber, French, English, and Algerian Dialect (Aries, 1 Sep 2025). The overall loss jointly optimizes the MLM and multi-label objectives.
This architecture differs from conventional PLMs in three explicit ways. It operates on characters rather than subwords, it is trained on isolated words rather than sentences, and it is optimized to encode word-level morphological structure rather than primarily accumulating top-down semantic context. A plausible implication is that the model is structurally aligned with orthographic instability and affixal variation, while remaining less specialized for sentence-level semantics.
3. Morphology-aware representation of Algerian social media text
The paper situates character-level modeling as a direct response to Algerian word-level morphology. Algerian social media forms are described as inconsistent and variable, including phonetic spelling shifts, elongation, code-mixing, and creative writing using emojis, numbers, and symbols (Aries, 1 Sep 2025). Because the model operates directly over character sequences, it can capture phonetic and morphological variation without requiring a canonical orthography.
Its character vocabulary is intentionally multiscript. The model includes Arabic script for Standard Arabic and dialect, basic and extended Latin for Berber, French, English, and Arabizi, Tifinagh for Berber, and emojis, punctuation, IPA, and other symbol sets (Aries, 1 Sep 2025). This design allows chDzDT to process code-mixed words and irregular borrowings without first segmenting by script or language.
The paper also grounds the model in a detailed morphological analysis of Algerian YouTube comments. That analysis identifies variability induced by nonstandard dialect, regional variants, code-switching, script mixing, and hybridization with foreign and loan words (Aries, 1 Sep 2025). It further argues for script- and context-aware models that do not rely on standard tokenization or morphological rules.
A common misconception is that morphological variation in dialectal text can be resolved mainly through better subword tokenization. The evidence presented here points in a different direction: tokenization itself is one of the central failure points when spelling is unstable, scripts are mixed, and the same lexical item may circulate in several orthographic realizations. chDzDT addresses this by making characters, rather than subwords, the basic representational unit.
4. Training corpus and multilingual lexicon construction
The pretraining corpus is multilingual, multiscript, and Algerian-centric. Its principal source is YouTube comments: 15M+ comments from 347 Algerian channels and 45,842 videos, explicitly described as informal, code-mixed, script-mixed, real usage data (Aries, 1 Sep 2025). Additional sources include Wikipedia in Arabic with 2.2M articles, French with 0.5M, English with 0.38M, and Kabyle with 8,600+ articles, as well as all Tatoeba sentences labeled as Kabyle or Berber.
Preprocessing includes comprehensive normalization, specifically deduplication, emoji normalization, repeated letter truncation, script unification, emoji translation, and punctuation normalization, together with regular expressions to filter out regionally non-Algerian vocabulary (Aries, 1 Sep 2025). After preprocessing and deduplication, all unique words are collected into a master vocabulary and labeled for presence in Arabic, Berber, French, English, and Algerian Dialect sources.
This yields a multilingual vocabulary in which words are annotated with the set of languages in which they occur. The paper states that there are 31 label combinations total and that multi-label tags enable more nuanced modeling of language overlap and help with code-switching and cross-lingual awareness (Aries, 1 Sep 2025). It also reports that over 99% of words are under 20 characters, and the Transformer window is set accordingly.
| Label | Word Count |
|---|---|
| DZ-only | 3,808,202 |
| AR-only | 2,304,728 |
| EN-only | 1,104,555 |
| FR-only | 975,697 |
| BER-only | 147,471 |
| All five labels | 9,244 |
The lexicon is identified as a major contribution in its own right: a multilingual Algerian word-level lexicon with language provenance metadata. This suggests that the resource is not only auxiliary to model training but also a reusable substrate for future research on code-switching, script mixing, and dialectal lexical overlap.
5. Evaluation protocol and empirical profile
The evaluation spans both intrinsic and downstream tasks. Intrinsic evaluation tests isolated word-level embeddings for morphological consistency, robustness to orthographic noise and obfuscation, morphemic probing, compositional vector arithmetic, and semantic similarity, with the paper noting that semantic similarity is limited in character-level models (Aries, 1 Sep 2025). Downstream evaluation includes morphological tagging, PoS tagging using word embeddings plus a lightweight BiGRU encoder, and sentiment analysis using sentence encodings built from word embeddings.
Evaluation is conducted across Arabic, Algerian dialect including Arabizi, French, and English (Aries, 1 Sep 2025). The main findings are differentiated by task type. On morphological tasks, chDzDT outperforms or matches strong baselines such as DziriBERT and CANINE on derivational and inflectional clustering and morphemic probing, especially for Arabic and Arabizi. It also excels in robustness to spelling variation and code-switching, especially where subword-based models fail because of out-of-vocabulary behavior or tokenization errors.
For downstream tasks, the picture is more mixed. Morphological tagging attains high accuracy, indicating strong encoding of word structure. On PoS tagging, chDzDT nearly matches larger contextual PLMs, which the paper interprets as evidence that word morphology alone is highly predictive for syntactic categories in many settings. On sentiment analysis, however, chDzDT lags behind full-contextual PLMs, though it still achieves nontrivial accuracy (Aries, 1 Sep 2025). With fine-tuning, downstream performance further improves on most tasks.
The ablation study identifies embedding size as having the most impact on downstream task quality, while increasing depth and attention improves morphology but less so for syntactic or semantic tasks (Aries, 1 Sep 2025). A common oversimplification would be to view character-level modeling as uniformly superior. The reported results instead indicate a more specific profile: strong advantages for morphology and orthographic robustness, competitiveness for PoS, and weaker performance when broader contextual semantics dominate the task.
6. Significance, scope, and limitations
The paper presents chDzDT as a tokenization-free, morphology-centric approach for dialects with high out-of-vocabulary rates, code-mixing, orthographic non-standardness, and complex morphology (Aries, 1 Sep 2025). Because it operates at the character level, no vocabulary pruning is required, and the model is described as robust to spelling creativity and script mixing. The work also emphasizes low computational footprints through several small, efficient model variants, making the approach portable to resource-constrained environments.
Its broader contribution lies in combining model design with resource creation. The annotated multilingual lexicon, the preprocessing and labeling pipeline, and the released models, code, and evaluation data provide a concrete foundation for further work on similarly under-resourced varieties (Aries, 1 Sep 2025). The paper explicitly situates this as advancing language modeling for morphologically rich, low-resource dialects.
At the same time, the empirical record in the paper delimits the model’s scope. chDzDT is optimized for word-level morphological structure rather than sentence-level contextual semantics, and its weaker sentiment-analysis performance relative to full-contextual PLMs is consistent with that design (Aries, 1 Sep 2025). This suggests that chDzDT is best understood not as a universal replacement for sentence-level PLMs, but as a morphology-focused encoder whose strengths are most evident in noisy, multiscript, code-switched lexical modeling.
The concluding perspective is that character-level PLMs can fill critical gaps for dialects that are highly variable, code-mixed, multi-script, and underserved by existing language technologies. The paper further argues that the results open pathways for hybrid models combining morphology-centric word representations with higher-level context and semantic models. That implication remains prospective, but it follows directly from the observed contrast between strong word-structure encoding and more limited sentence-level semantic performance (Aries, 1 Sep 2025).
