---
title: PalmX Dataset Benchmark
url: https://www.emergentmind.com/topics/palmx-dataset
type: topic
---

# PalmX Dataset Benchmark

Searching arXiv for the PalmX shared task and related benchmark papers.
PalmX is a shared-task benchmark and accompanying dataset suite introduced for evaluating the cultural competence of Arabic-centric large language models through multiple-choice questions written in Modern Standard Arabic (MSA). In its 2025 edition, PalmX comprises two subtasks—General Arabic Culture and General Islamic Culture—and is designed to probe internalized knowledge rather than retrieval, using blind evaluation and a fixed multiple-choice protocol with four options and one correct answer per item [2509.02550].

## 1. Scope and benchmark identity

PalmX 2025 was introduced as the first shared task on benchmarking LLMs on Arabic and Islamic culture. Its stated objective is to provide a standardized evaluation of cultural competence in domains that are often underrepresented in web-scale pretraining corpora, particularly Arabic and Islamic knowledge. The benchmark is explicitly framed as a response to skew in pretraining data distributions toward high-resourced languages and cultures, and it targets two complementary domains: General Arabic Culture and General Islamic Culture [2509.02550].

The first subtask measures grounded knowledge of everyday and country-specific Arab culture across 22 Arab countries. The second subtask measures religious literacy and contextual sensitivity in Islamic topics. In both subtasks, all items are authored in MSA, even when questions concern local practices or dialectal expressions. The benchmark therefore emphasizes cross-country and cross-domain knowledge while holding the surface register constant.

A defining design choice is the use of multiple-choice question answering rather than free-form generation. This reduces ambiguity in scoring, supports blind evaluation, and enables consistent likelihood-based assessment across decoder-only models. The benchmark also restricts submissions to models with at most 13B parameters and prohibits external retrieval or live internet access during evaluation, thereby targeting the knowledge internalized in the model parameters rather than the behavior of retrieval-augmented systems.

## 2. Dataset structure and content distributions

PalmX 2025 is organized into two subtasks with fixed train, development, and test splits.

| Subtask | Domain | Splits |
|---|---|---|
| General Arabic Culture | Culture across 22 Arab countries | Train 2,000, Dev 500, Test 2,000 |
| General Islamic Culture | Islamic knowledge and practice | Train 600, Dev 300, Test 1,000 |

Every item has exactly four options labeled A–D and a single correct answer. Items with multiple correct answers were removed during curation. No rationales or explanations are part of the released dataset. Distractors are described as plausible yet incorrect, and answer options were shuffled to reduce positional bias [2509.02550].

For Subtask 1, topic coverage includes customs, etiquette, cuisine, history, arts, geography, famous figures, local language or dialectal expressions, and related cultural domains. The paper reports the test-set topic distribution at a descriptive level: History 35.2%, Geography/Environment 10.0%, Food 7.9%, Customs 7.6%, Arts 6.0%, Sports 5.1%, Literature 4.6%, Economics 4.6%, Religion 3.9%, Language 2.8%, Clothing 2.4%, Education 1.5%, Politics 1.3%, Tourism 1.3%, Law 1.3%, and the remaining 10 topics combined 5.0%.

For Subtask 2, the reported test-set topic distribution is History 25.5%, Worship 18.2%, Ethics 12.4%, Fiqh 12.3%, Quranic Sciences 10.3%, Aqidah 9.4%, Hadith Sciences 3.5%, Mu‘amalat 2.4%, Contemporary issues 2.1%, Sirah 2.0%, and Philosophy 2.0%. The benchmark therefore spans both factual recall and contextual religious knowledge, but the released format remains a uniform MSA MCQ schema.

The organizers state that the Subtask 1 test distribution was designed to approximate the training distribution in domain and country while introducing new entities and less frequent cultural items to test generalization. At the same time, the paper acknowledges country-level imbalance, including underrepresentation of Iraq and Algeria.

## 3. Construction, sourcing, and curation

The two subtasks were assembled through different sourcing pipelines. For Subtask 1, one pipeline converted 4,000 candidate items from the Palm training split into MCQs using Qwen3 30B. A second pipeline generated 1,000 candidate items from culturally focused websites—Cultural Crossing, Commisceo, Cultural Atlas, and Expatica—by segmenting pages and using GPT-4o-mini to generate Arabic MCQs. Although 5,000 MCQs were produced during collection, the released Subtask 1 splits total 4,500 items after curation [2509.02550].

For Subtask 2, one pipeline produced 900 candidate items based on public Islamic competitions and a university “Islamic Culture” question bank from Al-Balqa Applied University. A second pipeline generated 1,000 candidate items with GPT-4o-mini from 200 Islamic articles crawled from Mawdoo3. The released Subtask 2 splits total 1,900 items after curation.

In both subtasks, two professional linguists independently reviewed all items for correctness and formatting. The stated filters included removal of off-topic questions, questions with multiple correct answers, biased content, trivial items, and items with grammatical errors. Discrepancies were resolved in consolidation sessions, and answer options were shuffled to minimize positional bias.

The curation process combined LLM-based generation or reformulation with human validation. The organizers explicitly note that parts of the dataset were generated or reformulated using LLMs and that, despite multiple levels of review, small artifacts or stylistic biases may remain. This places PalmX in a hybrid construction regime: machine-assisted generation followed by professional linguistic adjudication.

## 4. Official evaluation protocol and shared-task outcomes

PalmX uses a likelihood-based MCQ evaluation protocol for decoder-only generative models. Given a question prompt $P$ and candidate labels $C_1,\dots,C_n$ with $n=4$, the score for a choice is the log-likelihood of the label tokens as the continuation of “Answer:”:
$$
\mathrm{Score}(C_i) = \log p(C_i \mid P) = \sum_{j=1}^{k} \log p(c_{i,j} \mid P, c_{i,1}, \ldots, c_{i,j-1}).
$$
The resulting scores are normalized with softmax, and the predicted answer is the argmax over the four labels. The official metric is accuracy:
$$
\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\hat{y}_i = y_i].
$$
Macro-F1 and other metrics are not part of the official leaderboard [2509.02550].

The competition drew 26 team registrations for Subtask 1 and 19 for Subtask 2, yielding nine and six valid submissions, respectively. The organizers’ zero-shot NileChat-3B baseline achieved 70.00% dev and 67.55% test on Subtask 1, and 64.00% dev and 75.12% test on Subtask 2. The top-performing systems reached 72.15% on General Arabic Culture and 84.22% on General Islamic Culture.

For Subtask 1, the winning system, ADAPT-MTU HAI, achieved 72.15% with full fine-tuning of NileChat-3B for three epochs under a CLM objective with full-prompt supervision. The next-ranked systems used LoRA on NileChat-3B and Fanar-1-9B-Instruct. For Subtask 2, the winning system, AYA, achieved 84.22% with LoRA fine-tuning of ALLaM-7B-Instruct and data augmentation. Across both subtasks, parameter-efficient fine-tuning, especially LoRA, was the predominant and most effective paradigm. The organizers also report that augmentation was domain-dependent: it benefited Islamic knowledge more consistently than general culture, whereas some teams reported no development-set gains from paraphrase augmentation in Subtask 1.

## 5. Augmentation pipelines and experimental use around PalmX

A detailed example of PalmX’s use as a core task dataset appears in CultranAI’s participation in Subtask 1. That system centered PalmX as the official benchmark and augmented it with two external resources: the Palm dataset and a newly curated PalmX-ext resource of 22,000 MCQs. PalmX itself, in this setup, remained the official train/dev/test benchmark with Train = 2,000, Dev = 500, and Blind Test = 2,000, all in MSA and all formatted as four-option MCQs with one correct answer [2508.17324].

The Palm augmentation used the public Palm test portion of 1,926 QA pairs, split into two halves using stratified sampling by country: 950 for fine-tuning and 950 for evaluation. Because Palm consists of instruction-style QA pairs rather than MCQs, the authors converted each item into a four-option MCQ using GPT-4.1, generating three plausible distractors per question. This augmentation was motivated by Palm’s inclusion of both MSA and dialects across approximately 20 cultural topics, thereby complementing PalmX’s MSA-only design.

The PalmX-ext pipeline was more elaborate. For each PalmX question, GPT-4.1 was used to infer an associated Arab country; this inferred country tag was not an official PalmX field. The inferred country then drove retrieval through the NativQA framework in two iterations to maximize topical diversity. NativQA’s Domain Reliability Check was applied to keep answers from verified domains, and GPT-4.1 was used again to filter and refine answers for cultural relevance, factual accuracy, and concision. The resulting QA entries were converted into MCQs with three distractors, again using GPT-4.1. Manual review of 50 samples yielded an average score of approximately 7.4/10 for accuracy and clarity.

In the reported ablations, training on PalmX Train only yielded 76.6% on PalmX Dev, whereas training on PalmX Train + Palm + PalmX-ext yielded 80.6% on PalmX Dev. The fine-tuning comparison with Fanar-7B-Instruct as base also showed 80.00% for QLoRA and 80.60% for LoRA, indicating a marginal advantage for full-precision adapters under the reported setup. The final submission used Fanar-1-9B-Instruct fine-tuned via LoRA for three epochs with learning rate $2\times10^{-4}$, LoRA rank 64, dropout 0.1, and scaling factor $\alpha = 16$, trained on PalmX Train and Dev, Palm, and PalmX-ext. It achieved 70.50% accuracy on the blind test and ranked fifth in the shared task. The same paper reports 84.1% on an additional held-out evaluation, but also notes an attribution inconsistency: the abstract assigns 84.1% to “PalmX development,” whereas the Results section assigns it to “Palm test set.” The clearest PalmX development-set improvement reported in that study is the increase from 76.6% to 80.6%.

## 6. Access, reproducibility, and practical use

PalmX resources are available through the shared task page and the public repository hosting the data and evaluation code. The test sets were held private during the competition and released publicly after the leaderboard announcement. The organizers state that all resources are publicly available, but the paper does not specify file formats or exact field names in the paper text. Based on the examples shown, typical fields include split, question, options A–D, answer label, and metadata such as country and topic, although the exact schema is deferred to the repository [2509.02550].

For practical evaluation, the prompt template consists of the question, four options, and an “Answer:” field, after which only the label tokens A/B/C/D are scored. The organizers explicitly recommend preserving the original A–D order and not reshuffling at evaluation time. Because the harness scores labels rather than full option strings, no option-length normalization is required under the official protocol.

The PalmX-related experimental literature also exposes a set of reproducibility details around fine-tuning. In CultranAI’s reported setup, the selected model used Fanar’s native tokenizer, a batch size of 4, gradient accumulation 4, and three epochs; tested learning rates ranged from $5\mathrm{e}{-5}$ to $2\mathrm{e}{-4}$; LoRA ranks were 32 or 64; $\alpha \in \{16,32\}$; and dropout $\in \{0.05,0.1,0.15\}$. The paper further notes a PalmX limit of 512 tokens during fine-tuning, which became relevant when Palm-derived items exceeded that length after conversion to MCQ format [2508.17324].

Licensing is an area where the record remains incomplete. The PalmX shared-task paper does not report the license, and the CultranAI paper states that licensing and public download links for PalmX are not given there either. As a result, access conditions are operationally clear through the task infrastructure and repository, but licensing terms must be checked in the released resources themselves.

## 7. Limitations, interpretation, and terminological ambiguity

Several limitations are explicit in the PalmX 2025 documentation. The benchmark is MSA-only, so dialectal nuances are not directly represented in the official items, even when the subject matter is region-specific. Country-level imbalance remains, with the organizers naming Iraq and Algeria as underrepresented examples. Because parts of the dataset were generated or reformulated with LLMs, subtle stylistic artifacts may persist despite human review. The organizers also state that benchmark performance should not be interpreted as a guarantee of culturally appropriate deployment, and they do not report a formal audit of foundation-model pretraining data for contamination of PalmX items [2509.02550].

The augmentation-based experimental literature adds further practical caveats. In the CultranAI pipeline, some Palm-derived QAs exceeded the 512-token limit after MCQ conversion, and distractors in Palm and PalmX-ext were often shorter than the correct answers, potentially reducing difficulty. That paper also notes a training protocol caveat for the final submission: PalmX train and development splits were both used for training, so PalmX Dev was excluded from final evaluation to avoid contamination [2508.17324].

A separate source of confusion is the name “PalmX” itself. In Arabic NLP, PalmX refers to the 2025 shared-task benchmark on Arabic and Islamic culture. However, the same string is used differently in other literatures. In palmprint recognition, the 2020 benchmark officially introduced as the “Xi’an Jiaotong University Unconstrained Palmprint (XJTU-UP) database” is explicitly not called PalmX in the paper, even though the name may appear elsewhere informally [2004.03303]. In cross-domain palmprint authentication, the official name is “X-Palm,” with the paper noting that some posts or informal references may abbreviate or reorder the name as PalmX [2606.08437]. In remote sensing, “PalmX” is used as shorthand for the “Open Land Use Reference Dataset for Palm Oil Landscapes in Indonesia (version 0.1)” [2509.08303]. This suggests that “PalmX Dataset” is not globally unique as a label; within Arabic NLP, the canonical referent is PalmX 2025.

Source: https://www.emergentmind.com/topics/palmx-dataset