PalmX: Arabic & Islamic Cultural Evaluation

• PalmX is a benchmark that evaluates LLMs’ cultural competence in Arabic and Islamic domains using structured multiple-choice questions.
• The task consists of two subtasks focusing on general Arabic culture and Islamic practices, drawing from diverse data across 22 Arab countries.
• Parameter-efficient fine-tuning, such as LoRA, emerged as the most effective approach, significantly improving model accuracy over baseline.

Searching arXiv for the PalmX shared task and closely related cultural evaluation work. PalmX 2025 is the first shared task designed to benchmark the cultural competence of LLMs in Arabic and Islamic domains, using multiple-choice questions in Modern Standard Arabic and focusing on underrepresented cultural knowledge that is often obscured by Western- and Anglocentric data distributions (Alwajih et al., 2 Sep 2025). The task comprises two subtasks—General Arabic Culture and General Islamic Culture—and evaluates whether models can answer culturally grounded questions about traditions, food, history, religious practices, and language expressions from across 22 Arab countries (Alwajih et al., 2 Sep 2025). Within the broader literature on cultural evaluation, PalmX occupies a benchmark-centered position: it operationalizes cultural competence as performance on structured domain-grounded knowledge tasks, while related work on Arabic instruction data, extrinsic cultural competence, and intentionally cultural evaluation provides the surrounding methodological and conceptual context (Alwajih et al., 28 Feb 2025, Bhatt et al., 2024, Oh et al., 1 Sep 2025).

1. Origins and problem setting

PalmX 2025 was introduced in response to the observation that LLMs “inherently reflect the vast data distributions they encounter during their pre-training phase,” and that this data is “predominantly sourced from the web,” creating a “high chance” of skew toward “high-resourced languages and cultures, such as those of the West” (Alwajih et al., 2 Sep 2025). The reported consequence is a “diminished understanding” of Arabic and Islamic cultures, particularly for “increasingly under-represented topics” (Alwajih et al., 2 Sep 2025).

The shared task is situated within a broader movement in cultural evaluation that argues factual cultural knowledge should be benchmarked explicitly rather than inferred indirectly from generic multilingual performance. The Palm dataset, a year-long community-driven project covering all 22 Arab countries, had already established a culturally inclusive and linguistically diverse Arabic resource with 17,411 instruction–response pairs spanning 20 topical domains and both Modern Standard Arabic and dialectal Arabic (Alwajih et al., 28 Feb 2025). PalmX builds on that ecosystem but narrows the evaluation setting to a controlled shared-task format centered on multiple-choice assessment in Modern Standard Arabic (Alwajih et al., 2 Sep 2025).

A plausible implication is that PalmX was designed to prioritize comparability, leaderboard evaluation, and reproducibility over the broader generative diversity captured by Palm. This suggests a deliberate shift from open-ended cultural instruction following toward a bounded benchmarking regime suitable for system ranking.

2. Task design and benchmark composition

PalmX 2025 comprises two main subtasks, each operationalized as a multiple-choice question test in Modern Standard Arabic (Alwajih et al., 2 Sep 2025).

Subtask 1: General Arabic Culture assesses model knowledge of “traditions, etiquette, cuisine, history, famous figures, geography, dialects, arts, sports, politics, and more from 22 Arab countries” (Alwajih et al., 2 Sep 2025).

Subtask 2: General Islamic Culture focuses on “Islamic rituals and practices, Qur’anic studies, Hadith literature, history of Islam, religious holidays, jurisprudence, ethics, and related topics” (Alwajih et al., 2 Sep 2025).

Each question provides “four plausible options” (Alwajih et al., 2 Sep 2025). The dataset statistics reported for the shared task are as follows:

Component	Subtask 1	Subtask 2
Train	2,000	600
Dev	500	300
Test	2,000	1,000

For Subtask 1, the questions were “sourced from the Palm corpus and curated web sources, reviewed by professional linguists” (Alwajih et al., 2 Sep 2025). For Subtask 2, the data came “from university exam banks and the Mawdoo3 website, again with professional review” (Alwajih et al., 2 Sep 2025). The data was released publicly post-competition “for reproducibility” (Alwajih et al., 2 Sep 2025).

The benchmark was intended to go beyond superficial pattern matching. The description states that the multiple-choice questions are “diverse” and that each question provides “four plausible options to challenge genuine knowledge” (Alwajih et al., 2 Sep 2025). Examples include questions about Saudi National Day and Sudanese wedding rituals, as well as Islamic-history questions such as the identity of the first nurse in Islam (Alwajih et al., 2 Sep 2025).

This structure places PalmX within what later work calls a “trivia-centered paradigm,” in which culture is tested through multiple-choice or short-answer questions about static facts or proxies (Oh et al., 1 Sep 2025). That critique does not invalidate PalmX’s design, but it frames the benchmark as one specific operationalization of cultural competence rather than a complete account of culturally appropriate behavior.

3. Evaluation protocol and participation

PalmX enforced several constraints intended to isolate internalized model knowledge. The shared task did not permit “retrieval-augmented methods or live internet access”; only the models’ “internalized knowledge” was tested (Alwajih et al., 2 Sep 2025). Submissions were capped at “Max 13B parameters,” and participants had to provide “weights, not APIs,” with “reproducibility and transparency required” (Alwajih et al., 2 Sep 2025).

Evaluation used multiple-choice accuracy computed via the log-likelihood of the correct label, “following best practices (as in EleutherAI Evaluation Harness)” and with “No length bias” (Alwajih et al., 2 Sep 2025). The scoring rule reported in the task description is

$$\text{score}(C_i) = \sum_{j=1}^{k} \log p(c_{i,j} \mid P, c_{i,1}, \dots, c_{i,j-1})$$

where $C_i$ is a choice label and $P$ is the prompt (Alwajih et al., 2 Sep 2025).

Participation was substantial. The initiative drew “considerable interest,” with “26 teams registering for Subtask 1 and 19 for Subtask 2,” and it concluded with “nine and six valid submissions, respectively” (Alwajih et al., 2 Sep 2025). Five teams participated in both subtasks (Alwajih et al., 2 Sep 2025).

The combination of parameter caps, no-retrieval rules, and likelihood-based multiple-choice scoring made PalmX a tightly controlled evaluation. A plausible implication is that the benchmark was intended not only to measure cultural knowledge, but also to compare adaptation strategies under realistic resource constraints for Arabic-centric open models.

4. Methods adopted by participants

The most prominent methodological pattern in PalmX 2025 was parameter-efficient fine-tuning. The task summary states that “Parameter-efficient fine-tuning emerged as the predominant and most effective approach among participants” (Alwajih et al., 2 Sep 2025). LoRA was used by most teams because it enabled “affordable, effective adaptation to PalmX data” (Alwajih et al., 2 Sep 2025).

Several additional approaches were documented:

Data augmentation: “paraphrasing” and “sampling MCQs using LLMs” were used, but the impact was “only in Subtask 2 (Islamic)” and “offered little/no development gain” in Subtask 1 (Alwajih et al., 2 Sep 2025).

Full fine-tuning: The winning Subtask 1 system, ADAPT-MTU HAI, used “full FT on NileChat” (Alwajih et al., 2 Sep 2025).

Cross-task learning: Some teams combined data or methods across both subtasks, with “mixed success” (Alwajih et al., 2 Sep 2025).

Other innovations: Reported techniques included “Chain-of-Thought cues (‘Let’s think step-by-step’) and model ensembling” (Alwajih et al., 2 Sep 2025).

The CultranAI system description provides a more detailed example of PalmX participation. That system combined the PalmX dataset with the Palm dataset and a new “PalmX-ext” resource of “22,000+ high-quality, culturally-aligned MCQs,” then fine-tuned Fanar-1-9B-Instruct with LoRA (Bhatti et al., 24 Aug 2025). It reported that adding augmented data increased PalmX Dev accuracy from 76.6 to 80.6, and the final blind-test score was 70.5%, ranking fifth (Bhatti et al., 24 Aug 2025). Because the PalmX overview states that the utility of data augmentation was “domain-dependent” (Alwajih et al., 2 Sep 2025), CultranAI’s results illustrate that augmentation could be useful but was not uniformly beneficial across subtasks or teams.

This pattern suggests that PalmX primarily rewarded domain-aligned adaptation of Arabic-centric models rather than generic multilingual scale alone. That interpretation is supported by the explicit observation that “Arabic-centric models (NileChat, ALLaM, Fanar) outperform generalist/translated models” (Alwajih et al., 2 Sep 2025).

5. Results and empirical findings

The shared task reported both baseline and best-system accuracies for each subtask (Alwajih et al., 2 Sep 2025).

Subtask	Baseline	Best system	Accuracy
General Arabic Culture	Zero-shot, NileChat-3B	ADAPT-MTU HAI, FT (NileChat-3B)	72.15%
General Islamic Culture	Zero-shot, NileChat-3B	AYA, LoRA+aug (ALLaM-7B)	84.22%

For Subtask 1, the baseline was 67.55%, and the winning score was 72.15%, a gain of +4.6 (Alwajih et al., 2 Sep 2025). For Subtask 2, the baseline was 75.12%, and the winning score was 84.22%, a gain of +9.1 (Alwajih et al., 2 Sep 2025).

The top systems differed by subtask. In General Arabic Culture, ADAPT-MTU HAI ranked first with NileChat-3B and full fine-tuning; RGIPT ranked second using NileChat-3B with LoRA; and AYA ranked third using Fanar-9B with LoRA and paraphrase augmentation (Alwajih et al., 2 Sep 2025). In General Islamic Culture, AYA ranked first with ALLaM-7B and LoRA plus augmentation; Phoenix ranked second with ALLaM-7B and “FT + paraphrase aug”; and ADAPT-MTU HAI ranked third with ALLaM-7B, LoRA, and a Chain-of-Thought cue (Alwajih et al., 2 Sep 2025).

Several substantive findings were highlighted. First, “task-specific fine-tuning markedly improves over baseline” (Alwajih et al., 2 Sep 2025). Second, “Islamic questions are easier for current LLMs,” as shown by the higher best accuracy and stronger gains over baseline (Alwajih et al., 2 Sep 2025). The task analysis states that this “suggests canonical, widely-taught knowledge is easier to model than diverse country-specific culture” (Alwajih et al., 2 Sep 2025). Third, “Model size is not the sole driver”: “Smaller well-tuned Arabic models (NileChat-3B) outperformed larger ones (Fanar-9B) on culture when paired with strong tuning” (Alwajih et al., 2 Sep 2025).

These findings align with results from the Palm dataset evaluation, which showed that even frontier Arabic-capable models vary substantially by country and dialect, with some countries better represented than others and with strong reversion to Modern Standard Arabic even under dialectal prompts (Alwajih et al., 28 Feb 2025). PalmX restricts the linguistic surface to Modern Standard Arabic, but the broader Palm results indicate that Arabic cultural competence remains uneven even when strong general-purpose models are used (Alwajih et al., 28 Feb 2025).

6. Position within cultural evaluation research

PalmX belongs to a rapidly expanding literature on cultural evaluation, but it reflects one specific design choice: controlled multiple-choice benchmarking in a culturally concentrated domain. Other work has explored extrinsic, multilingual, open-ended, or participatory alternatives.

The paper “Extrinsic Evaluation of Cultural Competence in LLMs” distinguishes intrinsic competence from extrinsic competence and evaluates open-ended question answering and story generation by perturbing nationality cues across 195 nationalities (Bhatt et al., 2024). That study found that outputs vary by nationality and contain culturally relevant words, but also found “weak correlations” between output similarity and standard cultural value metrics such as Hofstede’s Cultural Dimensions and the World Values Survey (Bhatt et al., 2024). Relative to PalmX, this underscores a methodological contrast: PalmX measures culturally grounded factual accuracy, whereas extrinsic evaluation probes how culture manifests in downstream text generation.

MCEval expands multilingual cultural evaluation to 13 cultures and 13 languages, using dynamic construction, Counterfactual Rephrasing, and Confounder Rephrasing to assess both cultural awareness and cultural bias (Huang et al., 13 Jul 2025). It reports performance disparities across linguistic scenarios and fairness issues when English-only evaluation is used (Huang et al., 13 Jul 2025). PalmX, by contrast, is monolingual in Modern Standard Arabic and culture-specific to Arabic and Islamic domains.

The position paper “Culture is Everywhere: A Call for Intentionally Cultural Evaluation” criticizes the “trivia-centered paradigm” for reducing culture to “static facts or values” and for relying on “multiple-choice or short-answer questions” (Oh et al., 1 Sep 2025). PalmX fits that description in a literal sense. However, the benchmark’s focus on Arabic and Islamic domains, professional review, and country-level topical breadth means it still addresses a concrete gap left by Western-centered general benchmarks. A plausible implication is that PalmX should be understood not as a complete model of cultural alignment, but as a foundational benchmark for one measurable slice of that broader construct.

The Palm dataset further reinforces this interpretation. Palm is “the first fully human-created Arabic instruction dataset targeting both cultural and linguistic diversity covering all 22 Arab countries,” with human-created instructions, dialectal breadth, and LLM-as-judge plus human validation for evaluation (Alwajih et al., 28 Feb 2025). PalmX translates that resource context into a leaderboard-driven shared task.

7. Significance, limitations, and future directions

PalmX establishes a “critical and standardized public benchmark for Arabic and Islamic cultural alignment” and provides “the first quantitative framework for tracking advances in LLM cultural competence for Arabic and Islamic domains” (Alwajih et al., 2 Sep 2025). It also demonstrates that “instruction-tuned, Arabic-specific, and efficiently fine-tuned models yield rapid performance gains even at modest parameter budgets” (Alwajih et al., 2 Sep 2025). In a field where general multilingual performance often obscures underrepresentation, PalmX makes Arabic and Islamic culture a first-class evaluation target.

At the same time, the task description identifies several limitations. There are “country/topic imbalances” (Alwajih et al., 2 Sep 2025). The scope is limited to “MSA, MCQ, and only on Arabic + Islamic worlds,” and therefore “does not capture dialects or other Islamic cultural contexts worldwide” (Alwajih et al., 2 Sep 2025). The benchmark also carries “Potential for Benchmark Overfitting or Misuse,” and the task summary advises that benchmarks should not be treated as proxies for deployment readiness (Alwajih et al., 2 Sep 2025).

These constraints are consistent with broader critiques in the literature. Extrinsic evaluation work argues that output adaptation must also be judged for desirability, pragmatic correctness, and representational harms (Bhatt et al., 2024). Intentionally cultural evaluation argues that all aspects of evaluation—including what is evaluated and how—carry cultural assumptions (Oh et al., 1 Sep 2025). PalmX does not resolve those broader issues, but it does make one previously neglected domain measurable in a standardized way.

A plausible implication is that future PalmX-like evaluations may move in several directions already visible in adjacent work: multilingual or dialectal expansion, short-answer or open-ended formats, dynamic or causal evaluation procedures, and more participatory benchmark construction (Huang et al., 13 Jul 2025, Iranmanesh et al., 26 Feb 2026, Montalan et al., 2024). Within the current evidence base, however, PalmX 2025 stands as the first shared task specifically benchmarked on Arabic and Islamic culture, with clear evidence that domain-specific fine-tuning, Arabic-centric pretraining, and task-sensitive adaptation materially improve performance on culturally grounded questions (Alwajih et al., 2 Sep 2025).