--- title: ArabicNLP QIAS 2025 Challenge url: https://www.emergentmind.com/topics/arabicnlp-qias-2025-challenge type: topic --- # ArabicNLP QIAS 2025 Challenge The ArabicNLP QIAS 2025 Challenge was a shared task on question answering over Islamic sources, organized under ArabicNLP 2025 in Suzhou, China, and centered on evaluating Arabic-language systems for Islamic inheritance reasoning and broader Islamic knowledge assessment. Published descriptions characterize it as a multiple-choice benchmark in Arabic with scholar-verified questions, while the later QIAS 2026 overview presents it as the first major answer-focused stage in a larger research program on Islamic legal and religious reasoning in Arabic [2509.23793][2606.13756]. ## 1. Historical Position and Scope QIAS 2025 occupies a specific niche within Arabic NLP: it targets domains in which textual interpretation, formal legal rules, and numerical reasoning interact. In participant descriptions, the challenge is named “Question Answering on Islamic Sources,” and its scope is explicitly tied to Islamic knowledge understanding and reasoning in Arabic rather than to generic factoid QA or broad language understanding benchmarks [2509.23793]. The successor overview of QIAS 2026 frames QIAS 2025 as a shared task on Islamic inheritance reasoning and knowledge assessment, primarily MCQ-based, and treats it as the predecessor to a more explicit end-to-end reasoning benchmark. That retrospective positioning is important because it places QIAS 2025 at the transition point between answer-selection evaluation and structured jurisprudential reasoning evaluation [2606.13756]. ## 2. Task Structure and Reported Resources Published task descriptions converge on a two-subtask design but diverge in their accounting of released data. One retrieval-augmented submission describes Subtask 1 as 9,450 training questions, 1,500 validation questions, 1,000 test questions, and an auxiliary corpus of 32,000 IslamWeb fatwas; the same paper describes Subtask 2 as using 800 validation questions, 1,000 test questions, and the full text of 25 classical Islamic books. An encoder-based inheritance system instead reports 9,446 training, 1,000 validation, and 1,000 test instances for Subtask 1, with six answer labels \( \{A,B,C,D,E,F\} \). A separate LoRA+RAG submission reports 20,000 training, 1,000 development, and 1,000 test examples for the same inheritance subtask [2509.23793][2509.00457][2508.15854]. This suggests that participant papers were prepared against different task releases, packaging conventions, or internally expanded task variants. Despite those discrepancies, the reported structure is stable. Subtask 1 is Islamic Inheritance Reasoning, formulated as Arabic multiple-choice reasoning over inheritance scenarios. Subtask 2 is Islamic General Knowledge or Islamic Assessment, formulated as Arabic multiple-choice question answering over classical Islamic source material. Both subtasks are evaluated by accuracy, and both are described as stratified by difficulty, typically beginner, intermediate, and advanced [2509.23793][2509.00457]. ## 3. Islamic Inheritance Reasoning as the Core Technical Problem Subtask 1 concentrates on \( \textit{ʿilm al-mawārīth} \), where the model must interpret a narrative description of a deceased person, surviving relatives, and sometimes estate value, then identify valid heirs, classify heir categories such as أصحاب الفروض and عصبات, apply fixed-share rules, handle \( \textit{ʿawl} \) when shares exceed the estate and \( \textit{radd} \) when shares fall short, compute exact fractions or monetary shares, and finally choose the correct MCQ answer [2509.00457]. In NLP terms, at least one system description formalizes the subtask not as free-form chain-of-thought generation but as ranking six candidates \( \{c_1,\dots,c_6\} \) for a question \( q \), with prediction \( \hat{y} = \arg\max_i r_i \), where \( r_i \) is a learned relevance score between the scenario and each candidate answer [2509.00457]. This design choice is significant because it recasts legal reasoning into semantic relevance ranking over fully specified candidate outcomes. The later QIAS 2026 overview makes explicit the jurisprudential pipeline that QIAS 2025 was implicitly probing through MCQs: case understanding, heir identification and blocking (\( \textit{ḥajb} \)), share assignment (\( \textit{furūḍ} \)), global adjustment through \( \textit{ʿawl} \) or \( \textit{radd} \), and final numerical distribution. In that sense, QIAS 2025 tested whether models could select the right endpoint of that pipeline, even when intermediate reasoning was not itself an evaluated output [2606.13756]. ## 4. System Families and Reported Performance Participant systems for Subtask 1 span lightweight encoder ranking models, prompt-based frontier LLMs, fine-tuned Arabic LLMs, and ensemble methods. The CVPD submission reports that its best local encoder, MARBERT combined with Attentive Relevance Scoring, reaches 69.87% test accuracy, while the same study reports Gemini-2.5-flash at 87.60% under single-question prompting. A separate legal-reasoning study reports 92.7% test accuracy for a majority-vote ensemble of Gemini Flash 2.5, Gemini Pro 2.5, and GPT o3, which that paper identifies as third place overall in Task 1. QU-NLP reports 85.8% overall accuracy for a two-phase LoRA fine-tuning and RAG system over Fanar-1-9B, with 74.0% on beginner items and 97.6% on advanced items [2509.00457][2508.15796][2508.15854]. Subtask 2 elicited a somewhat different methodological profile because retrieval against authoritative source corpora was directly useful. The “Transformer Tafsir” submission reports that a three-stage hybrid RAG pipeline—BM25 retrieval, dense retrieval, and cross-encoder reranking—raises Fanar from 55% to 80% accuracy and Mistral from 69% to 79% on the general Islamic knowledge subtask. On the inheritance subtask, the same pipeline yields only modest gains, from 44% to 45% for Fanar and from 35% to 39% for Mistral, indicating that retrieval helps fact- and concept-centered Islamic QA more than it helps inheritance arithmetic and rule interaction [2509.23793]. ## 5. Methodological Tensions and Recurring Misreadings A persistent issue in the QIAS 2025 literature is that answer accuracy and genuine jurisprudential competence do not coincide perfectly. One participant study shows that inference strategy alone can dominate headline performance: gemini-2.5-flash rises from 68.65% under 50-question batched prompting to 87.60% under single-question prompts, with batched prompts introducing cross-question interference. The same study contrasts cloud LLMs with fixed-size local encoders and emphasizes the trade-off between peak accuracy and properties such as offline deployment, predictable latency, and privacy for sensitive family and financial data; in that framing, a lower-accuracy local model is positioned as an assistive system under human oversight rather than as an autonomous legal decision-maker [2509.00457]. The QIAS 2026 overview makes the same concern more explicit at the benchmark level: MCQ-style evaluation can confirm that a model selected a correct option, but it cannot by itself determine whether the internal reasoning was legally valid, whether heir blocking was understood correctly, or whether apparent success relied on heuristic answer matching. That retrospective critique is one of the main reasons the successor task moved from option selection to structured outputs with stage-wise evaluation [2606.13756]. ## 6. Legacy and Transition to Structured Reasoning The direct successor, QIAS 2026, recasts the problem as end-to-end Islamic inheritance reasoning from natural-language cases and replaces answer-only evaluation with the MAWARITH benchmark and the MIR-E metric. MAWARITH contains 12,500 Arabic inheritance cases annotated with intermediate reasoning steps and final answers, and MIR-E scores systems over heir identification and blocking, share assignment, adjustment type, and final distribution. Sixteen teams participated in that successor task, with leading systems spanning hybrid RAG, neuro-symbolic pipelines, prompting-only approaches, and QLoRA fine-tuning; the top reported MIR-E scores are 0.935 for CVPD, 0.931 for Simplicity, 0.916 for KMS, and 0.907 for QU-NLP [2606.13756]. Seen in that trajectory, ArabicNLP QIAS 2025 established the problem space, its MCQ formulation, and the empirical landscape of Arabic Islamic reasoning systems. It also exposed the main technical axes that now organize subsequent work: prompt sensitivity, retrieval grounding, Arabic-specific fine-tuning, privacy-preserving local deployment, and the unresolved gap between selecting a correct legal answer and producing a faithful, auditable legal derivation [2606.13756]. Source: https://www.emergentmind.com/topics/arabicnlp-qias-2025-challenge