Absher: Saudi Dialects LLM Benchmark

• Absher is a dialectal NLP benchmark evaluating LLMs on Saudi regional dialects by testing meaning, cultural interpretation, and geographic inference.
• It comprises over 18,000 multiple-choice questions derived from authentic words, phrases, and proverbs across diverse Saudi regions.
• The benchmark employs six evaluation tasks with controlled design and human validation to ensure reliable measurement of dialectal competence.

to=arxiv_search 天天中json code: {"^^^^^^^^3query3 A Benchmark for Evaluating LLMs Understanding of Saudi Dialects\"^^^^^^^^3 OR abs:\3^^^^^^^^"Absher: A Benchmark for Evaluating LLMs Understanding of Saudi Dialects\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"} to=arxiv_search 天天爱彩票json code: {"^^^^^^^^3query3 Saudi dialects benchmark","max_results":^^^^3ti:\3query3^^^^,"sort_by":"relevance","sort_order":"descending"} Absher is a dialectal NLP benchmark for Arabic that evaluates LLMs’ understanding of Saudi regional dialects and culturally embedded expressions, rather than the Saudi e-government “Absher” portal. It is introduced as a comprehensive benchmark specifically designed to assess LLMs performance across major Saudi dialects, using over ^^^^3ti:\3^^^^8,^^^^3query3query3query3^^^^ multiple-choice questions derived from authentic dialectal words, phrases, and proverbs sourced from various regions of Saudi Arabia. The benchmark is intended to surface how well models interpret meaning, context, culture, and geography in a linguistically diverse setting where daily communication depends heavily on underrepresented dialectal tones, idioms, culturally grounded practices, and context-rich expressions (&&&^^^^3query3^^^^&&&).

^^^^3ti:\3^^^^. Motivation and conceptual scope

Absher is motivated by a gap in Arabic NLP evaluation: most evaluation focuses on Modern Standard Arabic (MSA), with limited coverage of dialects and culture. The benchmark therefore targets major Saudi dialects—Central, Western, Southern, Eastern, and Northern—as well as general Saudi terms. Its stated purpose is to provide a culturally grounded, fine-grained benchmark for assessing how LLMs handle dialectal items such as words, phrases, and proverbs, especially when correct interpretation requires more than lexical matching (&&&^^^^3query3^^^^&&&).

The benchmark assesses six capabilities: Meaning, True/False, Fill-in-the-Blank, Contextual Usage, Cultural Interpretation, and Location Recognition. These task types collectively operationalize different aspects of dialect competence. Meaning emphasizes direct semantic mapping; Contextual Usage tests whether an item can be placed appropriately in natural usage; Cultural Interpretation targets figurative and socially embedded understanding; and Location Recognition links linguistic forms to regional provenance. This suggests that Absher is designed not merely as a lexical resource but as an evaluation instrument for culturally aligned language understanding.

A central real-world rationale is inclusivity. The benchmark is presented as supporting dialect-aware capabilities that reflect regional identities, social norms, and cultural heritage, while mitigating marginalization of underrepresented communities. The paper further connects this need to practical Arabic applications, including understanding local idioms and contextually sensitive scenarios. In that framing, Absher functions as both a diagnostic benchmark and a critique of evaluation regimes that over-index on MSA while under-measuring regional and sociocultural competence.

^^^^3 OR abs:\3^^^^. Dataset composition and regional coverage

Absher comprises ^^^^3ti:\3^^^^8,564 multiple-choice questions built from 3,^^^^3query3^^^^94 unique dialectal contents, filtered from an initial raw pool of 5,483 items. The curated contents consist of ^^^^3 OR abs:\3^^^^,533 words, 478 phrases, and 83 proverbs. Question generation yields ^^^^3ti:\3^^^^5,^^^^3ti:\3 questions for words, ^^^^3 OR abs:\3^^^^,868 for phrases, and 498 for proverbs (&&&^^^^3query3^^^^&&&).

The item inventory is explicitly heterogeneous. Words include examples such as “Rabshah,” phrases include “Allah Yta’anni,” and proverbs include “اللي ما يعرف الصقر يشويه” with the gloss “He who doesn’t recognize the falcon cooks it.” This breadth is important because words, phrases, and proverbs differ sharply in compositionality, figurativeness, and dependence on shared background knowledge.

All multiple-choice questions use four options, labeled A, B, C, and D, with randomized correct answer positions. True/False questions use two options—(A) True and (B) False—and are balanced 5^^^^3query3^^^^/5^^^^3query3^^^^ across the set. The benchmark is therefore controlled at the option-design level, which matters for interpreting chance performance and cross-category comparability.

The regional distribution is uneven, with Central and Western varieties most heavily represented and Eastern and Northern varieties relatively sparse:

Region	Number of questions
Central	5,65^^^^3 OR abs:\3^^^^
Western	5,^^^^3 OR abs:\3submittedDate3 OR abs:\3^^^^
Southern	^^^^3 OR abs:\3^^^^,68^^^^3 OR abs:\3^^^^
Northern	4^^^^3ti:\3^^^^4
Eastern	^^^^3 OR abs:\3ti:\3^^^^6
General Saudi Terms	4,338

This regional imbalance is explicitly acknowledged as a limitation. A plausible implication is that benchmark-level coverage mirrors the documentation asymmetries present in available dialect resources: some varieties are more visible and more easily collectible, while others remain underrepresented despite their linguistic distinctiveness.

3. Data sources, preprocessing, and quality control

The primary source is Mo3jam (ar.mo3jam.com), described as a digital repository aggregating dialectal vocabulary across Arabic regions. The authors scraped Saudi-labeled entries and organized them into words, phrases, and proverbs, annotating each with meaning, dialect label, contextual usage, and translation. Preprocessing involved deduplication, normalization, and filtering so that only Saudi dialect labels—Central, Western, Southern, Northern, Eastern—and general Saudi terms were retained. Spelling and colloquial variants were standardized to harmonize prompt generation and reduce ambiguity (&&&^^^^3query3^^^^&&&).

Question generation was programmatic. For each item, prompt construction paired the dialect label, meaning, and usage example; GPT-4o was then used to generate six question types per item. Controls were added to randomize answer positions and to balance True/False statements. The benchmark is thus not a simple scrape of existing dictionary entries; it is a derived evaluation layer that converts lexical-dialectal content into a structured MCQ suite.

Human validation was performed by four native Saudi Arabic speakers on a stratified random sample comprising ^^^^3ti:\3query3^^^^% of contents and 6^^^^3query3query3^^^^ questions. The annotation tasks were Question Quality and Answer Correctness. Question Quality covered clarity, fluency, logic, dialectal appropriateness, and cultural sensitivity, while Answer Correctness verified the labeled correct option. A pilot study with quality control questions was used to ensure annotator calibration.

Inter-annotator agreement is reported using Cohen’s Kappa. For Question Quality, the values are: Meaning ^^^^3query3^^^^.97, True/False ^^^^3query3^^^^.95, Fill-in-the-Blank ^^^^3query3^^^^.7^^^^3ti:\3 Contextual Usage ^^^^3query3^^^^.7^^^^3 OR abs:\3^^^^, Cultural Interpretation ^^^^3query3^^^^.83, and Location Recognition ^^^^3query3^^^^.85. For Answer Correctness, the values are: Meaning ^^^^3query3^^^^.97, True/False ^^^^3query3^^^^.87, Fill-in-the-Blank ^^^^3query3^^^^.9^^^^3ti:\3 Contextual Usage ^^^^3query3^^^^.88, Cultural Interpretation ^^^^3query3^^^^.94, and Location Recognition ^^^^3query3^^^^.9^^^^3 OR abs:\3^^^^. Overall alignment between model-generated answers and human judgments is 93.^^^^3query33^^^^%. These figures indicate that the benchmark construction pipeline achieved strong human-model agreement on labeled correctness, while also revealing that some task families—especially Fill-in-the-Blank and Contextual Usage—are intrinsically harder to standardize.

The ethical framing is explicit: the dataset uses only publicly available data, contains no sensitive or identifiable information, and emphasizes minimizing cultural bias, avoiding stereotypes, and promoting responsible use of LLM outputs in low-resource, culturally nuanced scenarios.

4. Evaluation protocol and metrics

Absher is presented as an evaluation benchmark rather than a trainable dataset with prescribed train/dev/test splits. The paper does not define train/dev/test splits. Its recommended protocol is zero-shot evaluation on the MCQ tasks with standard classification metrics (&&&^^^^3query3^^^^&&&).

The principal metric is accuracy:

PRESERVED_PLACEHOLDER_^^^^3query3^^^^

Chance-level accuracy is defined as

PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^

where PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^ is the number of answer choices. For four-option MCQs, $A_{\mathrm{chance}} = 1/4 = 0.25$; for True/False, $A_{\mathrm{chance}} = 1/2 = 0.5$.

Macro-average accuracy across categories is defined as

$$\mathrm{Acc}_{\mathrm{macro}}=\frac{1}{C}\sum_{c=1}^{C}\mathrm{Acc}_c.$$

The paper also specifies standard classification metrics:

$$P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN},\quad F1=\frac{2PR}{P+R}.$$

For confidence intervals on accuracy, the benchmark adopts the binomial proportion form

$\hat{p} \pm z \sqrt{\frac{\hat{p}(1-\hat{p})}{n}},$

with $z = 1.96$ for a 95% confidence interval. Statistical significance tests such as $t$-tests are not reported.

The evaluated LLMs are LLaMA-3 8B Instruct, Jais-^^^^3ti:\33^^^^B, ALLaM-7B, Mistral-7B, Qwen^^^^3 OR abs:\3^^^^.5-7B Instruct, and AceGPT-7B-chat. The multilingual models are LLaMA-3, Mistral, and Qwen^^^^3 OR abs:\3^^^^.5 Instruct; the Arabic-centric models are Jais, ALLaM, and AceGPT. All models are evaluated in zero-shot mode, with no task-specific fine-tuning, under a fixed prompt structure containing an instruction to “Choose only one answer…”. Because the prompt structure is fixed across models, the reported differences are attributable to model behavior under a controlled inference setting rather than to prompt engineering differences.

5. Empirical results and model behavior

The overall accuracy ranking, averaged across tasks, is led by Qwen^^^^3 OR abs:\3^^^^.5-7B Instruct at 5^^^^3query3^^^^.35%, followed by Mistral-7B at 45.93%, ALLaM-7B at 43.78%, LLaMA-3-8B Instruct at 43.34%, Jais-^^^^3ti:\33^^^^B at 4^^^^3 OR abs:\3^^^^.^^^^3ti:\3^^^^4%, and AceGPT-7B-chat at 38.88% (&&&^^^^3query3^^^^&&&).

Performance varies substantially by content type. For words, Qwen is strongest at 63.^^^^3query3query3^^^^% accuracy with F^^^^3ti:\3^^^^ 4^^^^3ti:\3^^^^.^^^^3ti:\3 while the others range at roughly 4^^^^3ti:\3^^^^–48% accuracy. For phrases, Mistral achieves the highest F^^^^3ti:\3^^^^ at ^^^^3 OR abs:\3^^^^6.^^^^3query3 OR abs:\3^^^^%, with accuracies around 4^^^^3query3^^^^–44%. For proverbs, ALLaM reaches the highest accuracy at 48.74% with F^^^^3ti:\3^^^^ 3^^^^3ti:\3^^^^.43%, while most models struggle and Jais records lower F^^^^3ti:\3^^^^. This pattern supports the paper’s interpretation that multilingual models often outperform Arabic-native models on general word- and phrase-level tasks, whereas Arabic-native models can exhibit targeted strengths on culturally laden content such as proverbs.

Detailed category results identify several model-specific strengths. At the word level, Qwen attains 64.67% on Contextual Usage, 63.74% on Cultural Interpretation, 75.^^^^3ti:\33^^^^% on True/False, 64.47% on Meaning, and 6^^^^3query3^^^^.5^^^^3ti:\3 on Fill-in-the-Blank. For proverbs, ALLaM attains 6^^^^3query3^^^^.^^^^3query3query3 on Cultural Interpretation, 55.56% on Fill-in-the-Blank, and 45.^^^^3query3query3^^^^% on Contextual Usage. Mistral performs especially strongly on True/False, with 58.4^^^^3query3^^^^% for words and 56.^^^^3query3query3^^^^% for phrases. Jais is notably strong on Location Recognition—67.57% for words, 65.7^^^^3 OR abs:\3^^^^% for phrases, and 54.55% for proverbs—and LLaMA shows similar strengths, with 66.3^^^^3 OR abs:\3^^^^%, 63.3^^^^3 OR abs:\3^^^^%, and 54.55% respectively. AceGPT’s relative strengths appear in Contextual Usage, where it reaches 54.53% for words and 49.^^^^3ti:\3 OR abs:\3^^^^% for phrases.

Across categories, Location Recognition is the strongest question type overall, with average performance above 58%, and Jais and LLaMA are particularly strong, reaching at least 63% in several content types. The interpretation offered is that models can often link dialectal cues to regions even when deeper semantic or cultural reasoning remains weak. By contrast, True/False is described as the most challenging when derived from short, low-context inputs such as words and phrases. Accuracy ranges from approximately ^^^^3 OR abs:\3 OR abs:\3^^^^% for Jais on proverb True/False to approximately 75% for Qwen on word True/False, showing substantial instability under binary reasoning when contextual grounding is sparse.

6. Error patterns, comparisons, and future directions

The paper’s case studies make the benchmark’s diagnostic function concrete. For the general proverb “إذا حجت البقرة على قرونها,” whose correct meaning is “something impossible,” five of the six models—ALLaM, LLaMA, Jais, Qwen, and AceGPT—answer correctly, while only Mistral errs. The stated explanation is that widespread proverbs are better represented in training data. By contrast, for the Southern fill-in-the-blank proverb “خطط في ماء واقبص في حيد,” all six models fail. The paper interprets this as reflecting low coverage of underdocumented local idioms. For the Central location-recognition item “أويلاوه,” only ALLaM and Jais correctly associate it with Al-Qassim, while the other models default to Riyadh or Jeddah, suggesting urban-centric corpus bias. In an Eastern example, only AceGPT correctly interprets the contextual usage of a time-related word, underscoring the difficulty of context-sensitive dialect items in less-represented regions (&&&^^^^3query3^^^^&&&).

These results are synthesized into two major difficulty classes. Cultural Interpretation is hard because proverbs encode metaphor and cultural knowledge that may not be explicitly documented or uniformly represented in web corpora. Contextual Usage is hard because appropriate sentence placement depends on both semantic fit and pragmatic appropriateness tied to local usage patterns. The benchmark therefore exposes limitations that would remain invisible in purely lexical or MSA-centric evaluations.

Relative to prior Arabic resources, Absher is distinguished from Dallah, Palm, and AL-QASIDA by its close focus on Saudi dialects with fine-grained, region-specific coverage and multiple question types spanning words, phrases, and proverbs. It is also described as complementing SaudiCulture by providing a structured MCQ format across six task types and by combining linguistic and cultural dimensions with human validation and detailed error analysis. This suggests that Absher occupies a more tightly controlled evaluation niche centered on Saudi dialectology and cultural grounding.

The recommendations are correspondingly targeted. The paper calls for dialect-aware training with geographically diverse and balanced dialectal corpora, especially for Southern, Eastern, and Northern regions and underdocumented idioms and proverbs. It also recommends culturally aligned evaluation that measures cultural inference and context handling rather than only lexical tasks, and proposes exploring instruction-tuning, few-shot adaptation, sociolinguistic features, and region-tagged data to reduce urban-center bias. The reported application domains include customer support, social media analysis, healthcare communication, and public services in Saudi settings.

Availability is stated in conditional form: the complete dataset will be made publicly available for research purposes upon acceptance. Code and license are not specified. Reproducibility is supported by documentation of the full pipeline—data collection, preprocessing, prompt design, GPT-4o generation, and human validation—and by inclusion of the question template and annotation guidelines in the appendices. The paper also notes several limitations: computational constraints restricted exploration of alternative configurations and repeated trials; a fixed prompt structure may have influenced model behavior; human evaluation covered only a stratified ^^^^3ti:\3query3^^^^% sample rather than the entire ^^^^3ti:\3^^^^8,564 questions; and regional coverage is imbalanced. Planned extensions include adding dialects, question formats, and model types, assessing few-shot and instruction-tuned settings, and integrating sociolinguistic features and region-specific datasets.

Taken together, Absher is presented as a comprehensive, culturally grounded benchmark for evaluating LLM understanding of Saudi dialects and cultural expressions. Its core empirical finding is that models exhibit clear strengths in some categories—especially Location Recognition and selected contextual tasks—while continuing to fail on cultural inference, underrepresented regional varieties, and low-context binary reasoning. In that sense, the benchmark formalizes dialect understanding as a multidimensional evaluation problem rather than a single lexical-accuracy task.