Quran QA 2023 Shared Task Overview

• Quran QA 2023 Shared Task is a specialized evaluation setting focusing on retrieval and answer extraction across Quranic passages in a low-resource context.
• It employs advanced dual-encoder and cross-encoder architectures combined with ensemble and thresholding techniques to handle linguistic gaps between Modern Standard Arabic and Classical Arabic.
• Data augmentation and precise metrics such as MAP@10, MRR@10, and pAP@10 drive improved performance in both passage retrieval and extractive question answering.

Quran QA 2023 Shared Task is a Quranic question answering evaluation setting centered on two linked problems: passage retrieval and extractive answer ranking over Quran Passage Collection segments. In the task formulations reported by participating systems, Task A receives a Modern Standard Arabic question and returns a ranked list of up to 10 Qur’anic passages in Classical Arabic, or the null passage “-1” for zero-answer cases; Task B receives a question together with a single passage and returns a ranked list of up to 10 answer spans, or an empty list if no answer exists. The shared task is consistently described as low-resource and domain-specific, with performance constrained by the linguistic gap between Modern Standard Arabic questions and Classical Arabic source text, the small size of the annotated data, and the need to handle multi-answer and unanswerable cases (Elkomy et al., 2024, Basem et al., 9 Aug 2025).

1. Task formulation and evaluation

Task A, usually labeled Passage Retrieval, is a ranking problem over Qur’anic passages. The output is a ranked list of up to 10 Quran Passage Collection segments, with explicit support for the null passage “-1” when the question has no answer in the Quran. Task B, described as Ranking-based MRC or Answer Extraction, operates on a question–passage pair and returns up to 10 ranked answer spans, or an empty list for zero-answer instances. In the TCE formulation, full credit for zero-answer samples is granted only when the system abstains exactly: “-1” in Task A and an empty list in Task B (Elkomy et al., 2024).

Retrieval is commonly evaluated with Mean Average Precision at 10 and Mean Reciprocal Rank at 10. Using the notation reported for the shared task, with queries $Q$, total relevant passages $R_q$, precision at rank $k$ denoted $P_q(k)$, and relevance indicator $\delta_k$, the retrieval metrics are

$$\text{MAP@10} = \frac{1}{|Q|}\sum_{q\in Q}\frac{1}{\min(R_q,10)}\sum_{k=1}^{10}P_q(k)\,\delta_k$$

and

$$\text{MRR@10} = \frac{1}{|Q|}\sum_{q\in Q}\frac{1}{\mathrm{rank}_q},$$

where $\mathrm{rank}_q$ is the position of the first relevant passage, or contributes zero if none is found (Basem et al., 9 Aug 2025).

For Task B, the official metric is partial Average Precision at 10, written pAP@10. In the 2025 two-stage system, pAP@10 is described as generalizing MAP@10 by replacing the binary relevance indicator with a partial-credit term $\delta_k^{\mathrm{partial}}\in[0,1]$ based on span overlap. In the TCE formulation, the partial match is computed from token-level $F_1$ between a predicted span and the best matching gold span, so the metric rewards ranked answers that overlap the reference even when they do not exactly match it (Basem et al., 9 Aug 2025, Elkomy et al., 2024).

2. Data regime, splits, and augmentation

The defining constraint of Quran QA 2023 is data scarcity. One report states that the original shared-task dataset contains 251 questions, split into 174 training, 52 test, and 25 development queries (Basem et al., 2024). For Task A, TCE reports 174 distinct training questions paired with 972 question–passage relevance pairs, and 25 development questions with 160 pairs. For Task B, the same submission reports 992 training question–passage pairs with 1,179 question–passage–answer triplets, and 163 development pairs with 220 triplets; under a faithful split that removes or redistributes leaked samples, Task B contains 991 training pairs and 164 development pairs (Elkomy et al., 2024).

Several systems addressed this bottleneck through aggressive augmentation. One line of work re-examined the original 251 questions, expanded them to 629 through manual reformulation, and then paraphrased each question twice to produce 1,895 total questions. That expanded set was explicitly labeled by answer type: 976 single-answer questions, 535 multi-answer questions, and 384 zero-answer questions (Basem et al., 2024). A separate two-stage submission reports expanding the shared-task dataset from 251 to over 2,000 question variants by integrating external resources, manually paraphrasing each question twice for linguistic diversity, and cleaning and standardizing Modern Standard Arabic formulations. Its external sources included Tafsir literature, a thematic plant dataset, and the Arabic subset of TyDiQA (Basem et al., 9 Aug 2025).

A cross-language variant pursued augmentation in a different direction. It combined the 251 shared-task questions with 378 new questions extracted from domain-specific PDF resources to produce 629 Arabic questions, paraphrased them to 1,895 Arabic questions, translated all 1,895 into English using the Google Translate API, and used Pickthall’s English translation of the Quran as the retrieval collection (Oshallah et al., 29 Jan 2025). Across these efforts, the shared pattern is that augmentation was not limited to increasing sample count; it also targeted paraphrastic diversity, zero-answer coverage, and the Modern Standard Arabic–Classical Arabic mismatch.

3. Retrieval architectures

Early strong shared-task systems relied on transfer learning and ensembling. TCE used a pipelined fine-tuning strategy in Task A: TyDi-QA_AR, then QPC–Tafseer pairs, then the final Task A data. The models were AraBERTv0.2-base, CAMeLBERT-CA, and AraELECTRA, with both dual-encoder and cross-encoder variants. The dual-encoder was trained with STAR and hard negatives, while the cross-encoder packed $R_q$0 into a single input and trained on one positive plus three random negatives per question. Final predictions were stabilized with a self-ensemble over multiple random seeds, and zero-answer questions were handled with an explicit thresholding mechanism (Elkomy et al., 2024).

The cross-language approach replaced Arabic passage retrieval with retrieval over an English Quran translation. It pre-fine-tuned models on SQuAD v2, then used a cross-encoder architecture for passage retrieval over translated question–passage pairs. The reported model set included BERT-Medium, RoBERTa-Base, DeBERTa-v3-Base, ELECTRA-Large, Flan-T5, Bloom, and Falcon, with RoBERTa-Base giving the strongest MAP@10 and MRR in that setting (Oshallah et al., 29 Jan 2025).

The strongest 2025 shared-task retrieval system used an explicitly two-step ensemble. Four pre-trained Arabic transformers were fine-tuned on the enlarged data with a pointwise binary relevance objective: AraBERTv02-ARCD, AraELECTRA, CamelBERT-tydi-tafsir, and AraBERTv02-tydi-tafsir. A Cross-Encoder was then trained on the same data for high-fidelity pairwise relevance scoring. The final ranking merged outputs in two stages: dynamic-weight averaging of AraBERTv02-ARCD and AraELECTRA, followed by Reciprocal Rank Fusion among AraELECTRA, CamelBERT-tydi-tafsir, and AraBERTv02-tydi-tafsir. The second stage min–max normalized each model’s scores, applied an exponential decay factor to down-weight lower ranks, boosted scores above 0.8, and then combined model-specific weights with RRF through a geometric-mean fusion,

$R_q$1

where $R_q$2 is the normalized score of passage $R_q$3 under model $R_q$4 and $R_q$5 is its fusion weight (Basem et al., 9 Aug 2025).

4. Answer extraction and span ranking

Task B systems initially followed standard span-prediction pipelines. TCE attached start and end vectors to Arabic transformer encoders and trained span predictors under two objectives: First Answer Loss, which uses only the first annotated span, and Multi-Answer Loss, which sums log-likelihood across all annotated spans. Post-processing combined non-maximum suppression with rejection of uninformative short spans, and zero-answer handling used a threshold on the gap between the $R_q$6 null score and the highest non-null span score (Elkomy et al., 2024).

Later work found that pure fine-tuning saturated. In the two-stage 2025 system, QRCD v1.2, QUQA, and ARCD were merged into a unified MRC corpus, and AraBERTv02, CAMeLBERT, and BERT-large were fine-tuned to predict start and end span indices. Even after hyperparameter sweeps over learning rates $R_q$7 to $R_q$8, batch sizes 8–16, and sequence lengths up to 384, pure fine-tuning plateaued at approximately pAP@10 $R_q$9 (Basem et al., 9 Aug 2025).

The alternative was instruction-tuned few-shot prompting. The same system used Google’s Gemini and DeepSeek via few-shot prompting, with a three-shot prompt curated from QRCD v1.2 to cover a single correct span, multiple correct spans, and an unanswerable case. The prompt explicitly required literal quotations only, one quoted answer per line, and the fallback response “No answer found in the given text” when appropriate. Span candidates from both APIs were merged and deduplicated to produce the final ranked answer set (Basem et al., 9 Aug 2025).

A closely related study on extractive Quranic QA described a specialized Arabic prompt framework with the same three-shot structure, deterministic decoding ($k$0), and a post-processing pipeline consisting of subword alignment, overlap suppression, and semantic filtering. In that study, Gemini with an API-enhanced prompt reached pAP@10 $k$1, DeepSeek reached $k$2, and the best fine-tuned baseline, AraBERTv02 on merged data plus TyDi QA, reached $k$3 (Basem et al., 8 Aug 2025). This suggests that, within Quranic extractive QA, instruction-following LLMs were increasingly used not as open-ended generators but as constrained span selectors.

5. Reported results and competitive progression

Using the official Quran QA 2023 scripts, the 2025 two-stage framework reported the following top results on the shared task (Basem et al., 9 Aug 2025).

System	Retrieval	Extraction
TCE	MAP@10 $k$4, MRR@10 $k$5	pAP@10 $k$6
Two-stage ensemble retrieval + instruction-tuned extraction	MAP@10 $k$7, MRR@10 $k$8	pAP@10 $k$9

The same report states absolute gains of $P_q(k)$0 MAP@10, $P_q(k)$1 MRR@10, and $P_q(k)$2 pAP@10 over the next best leaderboard systems. Its ablation analysis attributes a large part of the retrieval improvement to data expansion: moving from 251 to 665 seed questions and then to 2,000 paraphrases contributed $P_q(k)$3 to $P_q(k)$4 absolute MAP@10 gain at each phase. Hyperparameter exploration improved base fine-tuned models, but did not match the increase delivered by instruction-tuned LLM extraction (Basem et al., 9 Aug 2025).

Outside the official Quran QA 2023 leaderboards, related experimental settings reported additional reference points. In the cross-language retrieval setup, RoBERTa-Base achieved MAP@10 $P_q(k)$5 and MRR $P_q(k)$6, while DeBERTa-v3-Base achieved Recall@10 $P_q(k)$7 and Precision@10 $P_q(k)$8 (Oshallah et al., 29 Jan 2025). In another expanded-dataset shared-task setting, AraBERT-base reached MAP@10 $P_q(k)$9 and MRR $\delta_k$0, with zero-answer precision and recall rising from $\delta_k$1 to $\delta_k$2 after the introduction of 384 unanswerable questions and a dedicated no-answer head (Basem et al., 2024). A later hybrid system on an expanded Quran QA 2022 dataset reported MAP@10 $\delta_k$3, MRR $\delta_k$4, and Recall@10 $\delta_k$5, indicating a methodological direction toward hybrid dense–sparse retrieval, semantic reranking, confidence gating, and query refinement rather than a direct continuation of the 2023 leaderboard itself (Salman et al., 11 Jun 2026).

6. Broader context, misconceptions, and subsequent directions

The shared task sits within a wider Quranic and Islamic QA landscape. A semantic search precursor trained embeddings over a corpus built from the Hafs ʿUthmāni manuscript and 30 Arabic plus 2 English tafsirs, aligned commentary to verse identifiers, indexed the resulting embeddings with HNSW in FAISS, and reported SNxLM performance of mean max-cosine $\delta_k$6 and recall@5 $\delta_k$7 on 200 manually curated topic queries (Shohoud et al., 2023). That work already described the retrieval component as the first stage in a retrieve-then-read or retrieval-augmented generation pipeline, which aligns closely with later two-stage shared-task systems.

A common simplification is to treat Quran QA as lexical verse lookup. The shared-task literature points in a different direction. Retrieval systems explicitly target the semantic gap between Modern Standard Arabic and Classical Arabic, use authentic Tafsir and domain-specific corpora, paraphrase questions to cover morphological and syntactic variability, and include zero-answer handling rather than forcing a verse for every query (Basem et al., 9 Aug 2025, Elkomy et al., 2024). Adjacent evaluation work reinforces this point. The Quran track of IslamicMMLU contains 2,013 multiple-choice questions across Ayah-Count Recall, Surah Identification, and Verse Retrieval, uses Jaccard-based filtering and hard-negative distractors, includes a “no verse” option for out-of-range verse indices, and reports model accuracies spanning $\delta_k$8 to $\delta_k$9 under a deterministic zero-shot prompt. Its failure analysis emphasizes hallucination on out-of-range verse positions, confusion caused by high lexical overlap, and weak boundary awareness (Abdelaal et al., 24 Mar 2026).

Another misconception is that higher automatic overlap necessarily implies faithful religious answering. A broader benchmark over Quranic Tafsir and Ahadith reported that, although automatic metrics such as ROUGE and BERTScore improved after fine-tuning, expert evaluation produced verdict consistency in the range $$\text{MAP@10} = \frac{1}{|Q|}\sum_{q\in Q}\frac{1}{\min(R_q,10)}\sum_{k=1}^{10}P_q(k)\,\delta_k$$0 to $$\text{MAP@10} = \frac{1}{|Q|}\sum_{q\in Q}\frac{1}{\min(R_q,10)}\sum_{k=1}^{10}P_q(k)\,\delta_k$$1 and contextual understanding in the range $$\text{MAP@10} = \frac{1}{|Q|}\sum_{q\in Q}\frac{1}{\min(R_q,10)}\sum_{k=1}^{10}P_q(k)\,\delta_k$$2 to $$\text{MAP@10} = \frac{1}{|Q|}\sum_{q\in Q}\frac{1}{\min(R_q,10)}\sum_{k=1}^{10}P_q(k)\,\delta_k$$3 (Qamar et al., 2024). This suggests that future Quran QA shared-task designs may need complementary evaluation criteria beyond retrieval rank and span overlap, especially for long-form, non-factoid, or jurisprudentially sensitive questions.

The future-work proposals in the literature are correspondingly diverse. They include multi-translation retrieval and multilingual models such as mT5 and XLM-R in cross-language pipelines (Oshallah et al., 29 Jan 2025), larger LMs and ranking-oriented MRC architectures instead of span-prediction plus post-processing (Elkomy et al., 2024), hybrid dense–sparse retrieval with confidence gating and AraT5-based query rewriting (Salman et al., 11 Jun 2026), and expert or human-in-the-loop evaluation for domains where lexical overlap can diverge from scholar-level judgment (Qamar et al., 2024). Taken together, these directions frame Quran QA 2023 not as a closed benchmark, but as a focal point in the ongoing development of low-resource, domain-specific Arabic QA over religious text.