ArabicNLP QIAS 2025 Challenge

• ArabicNLP QIAS 2025 Challenge is a shared task that benchmarks Arabic question answering systems for Islamic legal and inheritance reasoning.
• It features a two-subtask structure: one on detailed Islamic inheritance reasoning through narrative cases and one on general Islamic knowledge via retrieval-enhanced MCQs.
• Leading systems achieved up to 92.7% accuracy, highlighting critical trade-offs between inference strategies, numerical reasoning, and genuine jurisprudential analysis.

The ArabicNLP QIAS 2025 Challenge was a shared task on question answering over Islamic sources, organized under ArabicNLP 2025 in Suzhou, China, and centered on evaluating Arabic-language systems for Islamic inheritance reasoning and broader Islamic knowledge assessment. Published descriptions characterize it as a multiple-choice benchmark in Arabic with scholar-verified questions, while the later QIAS 2026 overview presents it as the first major answer-focused stage in a larger research program on Islamic legal and religious reasoning in Arabic (Ahmad et al., 28 Sep 2025, Bouchekif et al., 11 Jun 2026).

1. Historical Position and Scope

QIAS 2025 occupies a specific niche within Arabic NLP: it targets domains in which textual interpretation, formal legal rules, and numerical reasoning interact. In participant descriptions, the challenge is named “Question Answering on Islamic Sources,” and its scope is explicitly tied to Islamic knowledge understanding and reasoning in Arabic rather than to generic factoid QA or broad language understanding benchmarks (Ahmad et al., 28 Sep 2025).

The successor overview of QIAS 2026 frames QIAS 2025 as a shared task on Islamic inheritance reasoning and knowledge assessment, primarily MCQ-based, and treats it as the predecessor to a more explicit end-to-end reasoning benchmark. That retrospective positioning is important because it places QIAS 2025 at the transition point between answer-selection evaluation and structured jurisprudential reasoning evaluation (Bouchekif et al., 11 Jun 2026).

2. Task Structure and Reported Resources

Published task descriptions converge on a two-subtask design but diverge in their accounting of released data. One retrieval-augmented submission describes Subtask 1 as 9,450 training questions, 1,500 validation questions, 1,000 test questions, and an auxiliary corpus of 32,000 IslamWeb fatwas; the same paper describes Subtask 2 as using 800 validation questions, 1,000 test questions, and the full text of 25 classical Islamic books. An encoder-based inheritance system instead reports 9,446 training, 1,000 validation, and 1,000 test instances for Subtask 1, with six answer labels $\{A,B,C,D,E,F\}$. A separate LoRA+RAG submission reports 20,000 training, 1,000 development, and 1,000 test examples for the same inheritance subtask (Ahmad et al., 28 Sep 2025, Bekhouche et al., 30 Aug 2025, AL-Smadi, 20 Aug 2025). This suggests that participant papers were prepared against different task releases, packaging conventions, or internally expanded task variants.

Despite those discrepancies, the reported structure is stable. Subtask 1 is Islamic Inheritance Reasoning, formulated as Arabic multiple-choice reasoning over inheritance scenarios. Subtask 2 is Islamic General Knowledge or Islamic Assessment, formulated as Arabic multiple-choice question answering over classical Islamic source material. Both subtasks are evaluated by accuracy, and both are described as stratified by difficulty, typically beginner, intermediate, and advanced (Ahmad et al., 28 Sep 2025, Bekhouche et al., 30 Aug 2025).

3. Islamic Inheritance Reasoning as the Core Technical Problem

Subtask 1 concentrates on $\textit{ʿilm al-mawārīth}$, where the model must interpret a narrative description of a deceased person, surviving relatives, and sometimes estate value, then identify valid heirs, classify heir categories such as أصحاب الفروض and عصبات, apply fixed-share rules, handle $\textit{ʿawl}$ when shares exceed the estate and $\textit{radd}$ when shares fall short, compute exact fractions or monetary shares, and finally choose the correct MCQ answer (Bekhouche et al., 30 Aug 2025).

In NLP terms, at least one system description formalizes the subtask not as free-form chain-of-thought generation but as ranking six candidates $\{c_1,\dots,c_6\}$ for a question $q$, with prediction $\hat{y} = \arg\max_i r_i$, where $r_i$ is a learned relevance score between the scenario and each candidate answer (Bekhouche et al., 30 Aug 2025). This design choice is significant because it recasts legal reasoning into semantic relevance ranking over fully specified candidate outcomes.

The later QIAS 2026 overview makes explicit the jurisprudential pipeline that QIAS 2025 was implicitly probing through MCQs: case understanding, heir identification and blocking ($\textit{ḥajb}$), share assignment ($\textit{furūḍ}$), global adjustment through $\textit{ʿilm al-mawārīth}$0 or $\textit{ʿilm al-mawārīth}$1, and final numerical distribution. In that sense, QIAS 2025 tested whether models could select the right endpoint of that pipeline, even when intermediate reasoning was not itself an evaluated output (Bouchekif et al., 11 Jun 2026).

4. System Families and Reported Performance

Participant systems for Subtask 1 span lightweight encoder ranking models, prompt-based frontier LLMs, fine-tuned Arabic LLMs, and ensemble methods. The CVPD submission reports that its best local encoder, MARBERT combined with Attentive Relevance Scoring, reaches 69.87% test accuracy, while the same study reports Gemini-2.5-flash at 87.60% under single-question prompting. A separate legal-reasoning study reports 92.7% test accuracy for a majority-vote ensemble of Gemini Flash 2.5, Gemini Pro 2.5, and GPT o3, which that paper identifies as third place overall in Task 1. QU-NLP reports 85.8% overall accuracy for a two-phase LoRA fine-tuning and RAG system over Fanar-1-9B, with 74.0% on beginner items and 97.6% on advanced items (Bekhouche et al., 30 Aug 2025, AlDahoul et al., 13 Aug 2025, AL-Smadi, 20 Aug 2025).

Subtask 2 elicited a somewhat different methodological profile because retrieval against authoritative source corpora was directly useful. The “Transformer Tafsir” submission reports that a three-stage hybrid RAG pipeline—BM25 retrieval, dense retrieval, and cross-encoder reranking—raises Fanar from 55% to 80% accuracy and Mistral from 69% to 79% on the general Islamic knowledge subtask. On the inheritance subtask, the same pipeline yields only modest gains, from 44% to 45% for Fanar and from 35% to 39% for Mistral, indicating that retrieval helps fact- and concept-centered Islamic QA more than it helps inheritance arithmetic and rule interaction (Ahmad et al., 28 Sep 2025).

5. Methodological Tensions and Recurring Misreadings

A persistent issue in the QIAS 2025 literature is that answer accuracy and genuine jurisprudential competence do not coincide perfectly. One participant study shows that inference strategy alone can dominate headline performance: gemini-2.5-flash rises from 68.65% under 50-question batched prompting to 87.60% under single-question prompts, with batched prompts introducing cross-question interference. The same study contrasts cloud LLMs with fixed-size local encoders and emphasizes the trade-off between peak accuracy and properties such as offline deployment, predictable latency, and privacy for sensitive family and financial data; in that framing, a lower-accuracy local model is positioned as an assistive system under human oversight rather than as an autonomous legal decision-maker (Bekhouche et al., 30 Aug 2025).

The QIAS 2026 overview makes the same concern more explicit at the benchmark level: MCQ-style evaluation can confirm that a model selected a correct option, but it cannot by itself determine whether the internal reasoning was legally valid, whether heir blocking was understood correctly, or whether apparent success relied on heuristic answer matching. That retrospective critique is one of the main reasons the successor task moved from option selection to structured outputs with stage-wise evaluation (Bouchekif et al., 11 Jun 2026).

6. Legacy and Transition to Structured Reasoning

The direct successor, QIAS 2026, recasts the problem as end-to-end Islamic inheritance reasoning from natural-language cases and replaces answer-only evaluation with the MAWARITH benchmark and the MIR-E metric. MAWARITH contains 12,500 Arabic inheritance cases annotated with intermediate reasoning steps and final answers, and MIR-E scores systems over heir identification and blocking, share assignment, adjustment type, and final distribution. Sixteen teams participated in that successor task, with leading systems spanning hybrid RAG, neuro-symbolic pipelines, prompting-only approaches, and QLoRA fine-tuning; the top reported MIR-E scores are 0.935 for CVPD, 0.931 for Simplicity, 0.916 for KMS, and 0.907 for QU-NLP (Bouchekif et al., 11 Jun 2026).

Seen in that trajectory, ArabicNLP QIAS 2025 established the problem space, its MCQ formulation, and the empirical landscape of Arabic Islamic reasoning systems. It also exposed the main technical axes that now organize subsequent work: prompt sensitivity, retrieval grounding, Arabic-specific fine-tuning, privacy-preserving local deployment, and the unresolved gap between selecting a correct legal answer and producing a faithful, auditable legal derivation (Bouchekif et al., 11 Jun 2026).