Arabic Generality Score (AGS)

• Arabic Generality Score (AGS) is a scalar measure that quantifies a word’s usage breadth across Arabic dialects independent of its divergence from Modern Standard Arabic.
• It employs a pipeline that integrates cross-dialect word alignment, etymology-aware edit distance with logistic smoothing, and contextual regression.
• AGS aids cross-dialect NLP by distinguishing widely shared forms from localized variants, enhancing applications like lexicon design, translation, and domain adaptation.

Arabic Generality Score (AGS) is a scalar measure for Arabic lexical variation that quantifies how widely a word is used across dialects, irrespective of its distance from Modern Standard Arabic (MSA). It was introduced to complement single-axis measures of dialectness, especially the Arabic Level of Dialectness (ALDi), on the premise that Arabic dialects form a continuum rather than cleanly separated classes and that speakers fluidly mix MSA with dialectal varieties in everyday text. AGS is defined at the word level, can be aggregated to the sentence level, and is operationalized through a pipeline that combines cross-dialect word alignment, etymology-aware edit distance, logistic smoothing, and contextual regression (Shaban et al., 24 Aug 2025).

1. Conceptual motivation and relation to Arabic dialectness

AGS addresses a limitation of discrete dialect labeling and of one-dimensional dialectness scores. The motivating observation is that ALDi captures how far a text diverges from MSA, but can conflate distinct phenomena: a highly dialectal word may be widely shared across many dialects, or may be highly localized. AGS introduces a complementary axis that captures lexical breadth across dialects at the word level, rather than degree of divergence from MSA alone (Shaban et al., 24 Aug 2025).

Within this framework, AGS ranges from 0 to 1. A value near 1 indicates maximal generality, meaning that a form is widely used across dialects or MSA; a value near 0 indicates maximal specificity, meaning localized usage. The intended analytical effect is a two-dimensional space in which ALDi and AGS together distinguish widely shared dialectal forms from localized dialectal forms, as well as standard but pervasive forms. A word such as مافي or شوي can therefore be far from MSA while still receiving high AGS, whereas some MSA forms that are common in colloquial usage can have high AGS but low ALDi (Shaban et al., 24 Aug 2025).

This distinction is directly relevant to cross-dialect NLP. Lexical generality determines whether a form is broadly usable or likely to fail outside its home dialect. In that sense, AGS is not primarily a measure of standardness, prestige, or correctness; it is a measure of cross-dialect spread. The examples given for the meaning ‘a little (bit)’—شُوي, شوية, قليل, حبة, نتفة—illustrate the target phenomenon: multiple realizations may be dialectal, but differ in how broadly they are shared.

2. Mathematical definition

AGS is computed from aligned parallel data. The construction begins with a set of parallel sentences across dialects,

$s = \{s(d_1), \ldots, s(d_k)\},$

where each

$s(d_i) = (w_1^{(d_i)}, \ldots, w_{n_i}^{(d_i)})$

belongs to dialect $d_i \in D$. Cross-dialect word alignments are extracted sentence by sentence and aggregated across the corpus to form $A(w,d)$, the multiset of cross-dialect equivalents for a word $w$ in dialect $d$ (Shaban et al., 24 Aug 2025).

For each dialect $d'$, the method computes $\delta_{d'}(w)$, the minimum augmented edit distance between $w$ and any aligned form in $d'$. These distances are then transformed through a logistic smoothing kernel,

$s(d_i) = (w_1^{(d_i)}, \ldots, w_{n_i}^{(d_i)})$0

which softly treats distances below a threshold $s(d_i) = (w_1^{(d_i)}, \ldots, w_{n_i}^{(d_i)})$1 as matches and distances above $s(d_i) = (w_1^{(d_i)}, \ldots, w_{n_i}^{(d_i)})$2 as non-matches. The word-level score is the across-dialect average of the smoothed match scores:

$s(d_i) = (w_1^{(d_i)}, \ldots, w_{n_i}^{(d_i)})$3

The paper evaluates thresholds $s(d_i) = (w_1^{(d_i)}, \ldots, w_{n_i}^{(d_i)})$4 with fixed steepness $s(d_i) = (w_1^{(d_i)}, \ldots, w_{n_i}^{(d_i)})$5, and selects $s(d_i) = (w_1^{(d_i)}, \ldots, w_{n_i}^{(d_i)})$6 on development performance; the best setting uses $s(d_i) = (w_1^{(d_i)}, \ldots, w_{n_i}^{(d_i)})$7. By construction, $s(d_i) = (w_1^{(d_i)}, \ldots, w_{n_i}^{(d_i)})$8 (Shaban et al., 24 Aug 2025).

The score is defined over tokens but can be aggregated. Two sentence-level formulations are used. The proposed aggregation emphasizes the least-general items through the harmonic mean of the $s(d_i) = (w_1^{(d_i)}, \ldots, w_{n_i}^{(d_i)})$9 lowest word-level AGS scores:

$d_i \in D$0

where $d_i \in D$1 are the $d_i \in D$2 smallest word AGS values in the sentence. For comparison against MDID sentence labels, the paper also uses simple averaging over all tokens.

3. Annotation pipeline

The AGS annotation pipeline has four components: word alignment across dialects, an etymology-aware edit distance, smoothing, and aggregation into word-level labels on a parallel corpus (Shaban et al., 24 Aug 2025).

3.1 Word alignment across dialects

Words are aligned across the MADAR parallel corpus using AWESOME-Align, described as a neural aligner based on multilingual contextual embeddings. Given token sequences $d_i \in D$3 and $d_i \in D$4, AWESOME computes contextual embeddings $d_i \in D$5 and $d_i \in D$6 from a shared encoder, forms a similarity matrix

$d_i \in D$7

and extracts alignments by symmetric agreement after row-wise softmax, subject to a confidence threshold $d_i \in D$8.

The encoder is CAMeL-BERT, fine-tuned with all AWESOME objectives over approximately 100k MSA–DA sentence pairs from MADAR-26. The objectives are MLM, TLM, SO, PSI, and CO. To scale from pairwise to multiway parallel sentences, each dialectal sentence is aligned to an MSA anchor, and all dialectal words aligned to the same MSA token are treated as mutually aligned. With $d_i \in D$9, sentence-level alignment cliques are represented as

$A(w,d)$0

Aggregating these cliques across the corpus yields candidate equivalents for downstream distance computation.

3.2 Etymology-aware edit distance

A central technical claim is that plain Levenshtein distance over orthography over-penalizes systematic alternations induced by phonology and orthographic convention. The paper uses the example قلب qlb versus ألب Alb versus گلب glb for ‘heart’. To address this, it augments character substitution cost by the probability that two observed graphemes share the same etymological character:

$A(w,d)$1

This probability is estimated from three components derived from the MADAR Lexicon and the CAPHI table:

1. $A(w,d)$2, learned by aligning CODA characters to CAPHI phonemes.
2. $A(w,d)$3, learned similarly from orthography.
3. $A(w,d)$4, estimated through an etymology heuristic that tests whether a CODA grapheme maps to different default versus non-default CAPHI phonemes across dialects for the same lexical item.

The posterior over etymological characters is obtained by marginalizing over phonemic realizations:

$A(w,d)$5

with

$A(w,d)$6

The shared-etymology probability is then

$A(w,d)$7

This substitution model is inserted into a Levenshtein-style dynamic program, while insertions and deletions keep standard unit cost. The resulting augmented edit distance is intended to downweight systematic phonology- and etymology-induced variation rather than treating it as arbitrary orthographic divergence.

3.3 Normalization, smoothing, and label generation

The pipeline adopts CODA normalization to reduce surface variability and CAPHI phonology to model dialect-specific pronunciations of CODA-normalized words. It also extends with unnormalized forms from MADAR-CODA where available in order to better estimate $A(w,d)$8 for non-standard spellings.

Once minimum distances $A(w,d)$9 are computed, they are softly binarized by the logistic function above. The paper characterizes this as avoiding brittle hard thresholds: values well below $w$0 map close to 1, values well above $w$1 map close to 0, and $w$2 controls transition sharpness. For each word $w$3 in dialect $w$4, the method collects aligned candidates in each other dialect, computes augmented edit distances, takes the minimum per dialect, smooths the results, and averages across dialects to derive $w$5 (Shaban et al., 24 Aug 2025).

4. Contextual prediction of AGS

After generating gold word-level AGS labels on parallel data, the paper trains a contextual regressor to estimate AGS for any word in sentence context. The model is CAMeL-BERT fine-tuned as a regression model, specifically bert-base-arabic-camelbert-mix. Inputs are CODA-normalized sentences in which the target word is enclosed by special tokens [[TGT](https://www.emergentmind.com/topics/task-graph-transformer-tgt)], for example: … أنا [TGT] متوقف [TGT] عن العمل …. No explicit dialect metadata is added; the stated design assumption is that the model can learn contextual cues and cross-dialect regularities directly from the training examples (Shaban et al., 24 Aug 2025).

Training minimizes mean squared error between predicted and gold AGS:

$w$6

The reported hyperparameters are batch size 32, AdamW, learning rate $w$7 with linear decay and zero warmup, early stopping, fixed seed 42, and a cap of 10,000 training steps. Implementation uses PyTorch and transformers, and all experiments are reported as running on an NVIDIA A100 in Google Colab.

Gold labels are generated from the MADAR-6 and MADAR-26 parallel corpora, and separate regressors are trained on each in order to test whether fewer, geographically diverse dialects suffice. Threshold $w$8 is selected on MDID-DEV for best sentence-level RMSE. The paper states that no explicit calibration beyond clipping predictions to $w$9 is required.

At inference time, the algorithm CODA-normalizes the sentence, inserts [TGT] markers around the target token, tokenizes the marked sentence, passes it through the CAMeL-BERT regressor, and returns the scalar prediction as $d$0 (Shaban et al., 24 Aug 2025).

5. Evaluation design and empirical results

The training and annotation resources are MADAR-26, described as MSA plus 25 city dialects over 2k sentences, and MADAR-6, described as MSA plus five dialects—CAI, BEI, DOH, TUN, and RAB—extended with 10k sentences per dialect. Additional resources are the MADAR Lexicon with 47,466 dialectal entries and the CAPHI table. Evaluation uses MDID-DEV with 120 tweets and MDID-TEST with 1000 tweets from NADI 2024, annotated with multi-label “dialect validity” for 11 country-level dialects (Shaban et al., 24 Aug 2025).

For MDID, sentence-level gold AGS is derived by converting multi-label validity into a proportion:

$d$1

where $d$2. The evaluation metric is RMSE:

$d$3

Three baselines are compared against the AGS regressor. X1 MADAR Lookup assigns each token its lexicon-based AGS if present and defaults to 0.5 for out-of-vocabulary items, then averages to the sentence level. X2 B2BERT, from the NADI 2024 leaderboard, uses per-dialect binary classifiers and is reported with macro-F1 0.5963. X3 NADI2024-baseline is the official top-p inference baseline with macro-F1 0.4697. For X2 and X3, predicted dialect sets are converted into sentence AGS via a ratio and aggregated as specified in the paper.

System	Training source	MDID-TEST RMSE
CAMeL-BERT regressor	MADAR-26	0.2698
CAMeL-BERT regressor	MADAR-6	0.2704
X1 MADAR Lookup	Lexicon lookup	0.2901
X2 B2BERT	NADI 2024 leaderboard system	0.3003
X3 NADI2024-baseline	Official baseline	0.2985

The reported best threshold for logistic smoothing is $d$4 with fixed $d$5, selected on MDID-DEV. Performance from MADAR-26 and MADAR-6 is described as very similar, which the paper interprets as evidence that a geographically diverse subset already provides a strong generality signal. Formal significance testing is not reported, though the differences are said to be consistent across development and test sets (Shaban et al., 24 Aug 2025).

The paper also gives token-level examples from MDID. In one sentence with gold $d$6 and predicted $d$7, high-AGS items include من mn (0.941) and مين myn (0.986), while low-AGS items include مخنوق mxnwq (0.229) and طايق ṭāyiq (0.139). In another sentence with gold 0.273 and predicted $d$8, high-AGS items include ما في mā fī (0.821), من mn (0.987), and نص nuṣṣ (0.943), while lower-AGS items include وآخرتنا w-ākhritnā (0.251) and مفر mafarr (0.312). These examples are used to illustrate that the least-general words drive sentence-level specificity.

6. Interpretation, uses, and limitations

The interpretive scheme is explicit. AGS near 1 corresponds to vocabulary broadly shared across dialects or MSA, often including function words, common verbs, and frequent colloquialisms spanning regions. AGS near 0 corresponds to highly specific forms, including localized lexemes, culturally specific idioms, and rare spellings or phonological realizations. This makes AGS a measure of breadth of usage rather than of formality or proximity to standard Arabic (Shaban et al., 24 Aug 2025).

The paper reports several linguistic observations. Distributional analysis shows that DOH and BEI skew toward higher AGS, possibly acting as hubs of cross-dialect overlap. It also reports that MSA sentences are longer, more formal, and more semantically over-specified, which reduces surface overlap with dialects and lowers AGS. This suggests that high standardness does not automatically imply high lexical generality.

The practical applications described in the paper span several NLP settings. AGS can support data normalization and cleaning by prioritizing broadly shared forms, guiding lexicon consolidation, and flagging highly specific variants. It can be used in cross-dialect NLP and domain adaptation to select training examples and vocabulary likely to generalize across dialects, or to control style transfer in MT, ASR, and NLU. It can assist lexicon design and search by ranking candidate forms according to coverage across dialects, and in information retrieval by expanding queries with high-AGS synonyms. It can also be integrated as a feature in DID, MT, and tagging systems to bias toward generalizable tokens or detect when specialized vocabulary requires special handling.

Several limitations are stated. MADAR is domain-specific, covering travel expressions, so learned generality may not transfer directly to news or code-switched speech. AWESOME alignment is not backed by gold alignments and may introduce noise; MSA anchoring mitigates but does not eliminate misalignments and idiomatic mismatches. The etymology heuristic depends on CAPHI defaults and lexicon-derived patterns, so code-switching and loanwords can confound it. The regressor may overfit frequent patterns and underperform on rare varieties, and the paper notes that explicit metadata could help. A further inconsistency arises at the sentence level: the proposed sentence AGS uses a harmonic mean over the least-general words, but MDID evaluation uses simple averaging, which under-weights the impact of specific tokens (Shaban et al., 24 Aug 2025).

The paper argues that AGS generalizes conceptually to unseen dialects and genres provided that aligned data or a trained regressor is available, but also states that robust transfer depends on alignment quality and reliable etymology-aware distance estimation. It identifies expansion to multiword expressions, incorporation of domain and socio-geographic metadata, and direct integration into downstream tasks such as translation and retrieval as future directions.