Grounding Arabic LLMs in the Doha Historical Dictionary: Retrieval-Augmented Understanding of Quran and Hadith

Abstract: LLMs have achieved remarkable progress in many language tasks, yet they continue to struggle with complex historical and religious Arabic texts such as the Quran and Hadith. To address this limitation, we develop a retrieval-augmented generation (RAG) framework grounded in diachronic lexicographic knowledge. Unlike prior RAG systems that rely on general-purpose corpora, our approach retrieves evidence from the Doha Historical Dictionary of Arabic (DHDA), a large-scale resource documenting the historical development of Arabic vocabulary. The proposed pipeline combines hybrid retrieval with an intent-based routing mechanism to provide LLMs with precise, contextually relevant historical information. Our experiments show that this approach improves the accuracy of Arabic-native LLMs, including Fanar and ALLaM, to over 85\%, substantially reducing the performance gap with Gemini, a proprietary large-scale model. Gemini also serves as an LLM-as-a-judge system for automatic evaluation in our experiments. The automated judgments were verified through human evaluation, demonstrating high agreement (kappa = 0.87). An error analysis further highlights key linguistic challenges, including diacritics and compound expressions. These findings demonstrate the value of integrating diachronic lexicographic resources into retrieval-augmented generation frameworks to enhance Arabic language understanding, particularly for historical and religious texts. The code and resources are publicly available at: https://github.com/somayaeltanbouly/Doha-Dictionary-RAG.

• The paper introduces a retrieval-augmented generation pipeline grounded in 198,346 Doha Historical Dictionary records, combining BM25, multilingual embeddings, reranking, and intent-based prompting for Classical Arabic questions.
• Fine-tuned hybrid retrieval achieved up to 0.945 MRR and 0.652 Recall@10, while RAG increased Fanar accuracy from 54% to 89% and ALLaM accuracy from 56.85% to 86% on Qur'an and Hadith vocabulary.
• The results show that historical lexical evidence substantially improves Arabic LLMs, but diacritic-sensitive morphology, homographs, ambiguous entries, retrieval misses, and weak abstention remain key limitations.

Motivation and problem statement

Arabic LLMs perform poorly on Classical Arabic, particularly on Qur'anic and Hadith material, where archaic vocabulary, semantic drift over centuries, and diacritic-sensitive morphology produce anachronistic errors in models trained largely on contemporary web text. Prior RAG work for Arabic lexical knowledge, notably the Riyadh-dictionary study of Al-Rasheed et al., addressed only Modern Standard Arabic and left the diachronic dimension unexplored. The paper under review proposes a retrieval-augmented generation (RAG) framework grounded in the Doha Historical Dictionary of Arabic (DHDA), a structured lexicographic resource that documents word usage across more than ten centuries through dated citations (shawāhid), etymologies, inscriptions, and Semitic cognates. The central claim is that grounding generation in this diachronic evidence substantially closes the gap between mid-sized Arabic-native LLMs (Fanar, ALLaM) and a much larger proprietary model (Gemini 2.5 Pro) on questions about Qur'an and Hadith vocabulary.

Methodology

The pipeline comprises query preprocessing, hybrid retrieval, cross-encoder re-ranking, intent-based routing, and generation.

Query preprocessing combines rule-based noise removal and term weighting with a lightweight intent classifier: TF-IDF features fed to a Random Forest (200 estimators, confidence threshold 0.6) assign queries to nine categories (meaning, contextual meaning, author, date, source, morphology, etymology, inscriptions, other), plus a "Qur'anic first usage" category exploiting DHDA's Islamic content.

Retrieval uses BM25 as the sparse component and four multilingual embedders — Jina v3, BGE-m3, Arctic-Embed 2.0, Nomic Embed v2 — indexed exactly via FAISS IndexFlatL2 over a corpus of 198,346 documents built from DHDA entries. Diacritics are stripped from retrieval documents to match undiacritized user queries, but diacritized originals are preserved for the generator, a deliberate trade-off given diacritics' disambiguating role.

Re-ranking applies BAAI/bge-reranker-v2-m3, fine-tuned with binary cross-entropy on 10,000 template-generated positive/negative query–document pairs.

Intent-based routing conditions both the subset of retrieved entry fields passed to the LLM and the prompting strategy (zero-shot for extraction intents such as date or source; few-shot for interpretive intents such as meaning and morphology). Generation uses Fanar and ALLaM, with Gemini 2.5 Pro as an upper-bound reference; temperature is set to zero throughout.

Data construction

DHDA content was scraped in September 2025 and organized into structured records keyed by root, each containing lexical unit, morphological tagging, dating, citation, author, and bibliographic source. Retrieval documents concatenate only contextually meaningful fields (word, root, phrase, meaning, citation); dates and sources are excluded from retrieval to avoid ambiguity but forwarded to the generator. Training and evaluation data are produced by templated insertion of lexical fields into question/answer formats across eleven question types. All evaluation questions target words appearing in Qur'anic verses or Hadith citations. A notable concession: because questions are automatically generated from templates, they may not capture the diversity of natural user queries — the authors flag human-authored evaluation as needed future work, which limits external validity of the reported scores.

Evaluation methodology

Retrieval is assessed with MRR, MAP, and Recall@10 on 1,000 query–document pairs spanning multiple types ($k=10$). Answer correctness is scored 0–100% by Gemini 2.5 Pro acting as an automated judge. Judge reliability was validated against a single human evaluator on 200 stratified answers: exact score match in 83% of cases, agreement within one rubric category above 95%, mean signed difference of −0.63 points (negligible bias), MAE of 5.9, Pearson correlation 0.87, and quadratic weighted Cohen's $\kappa = 0.87$, characterized as "almost perfect" agreement. This validation is essential since the headline numbers rest entirely on automated judging; however, it involves one evaluator and one judge model, so judge-specific biases cannot be fully ruled out.

Retrieval results

BM25 alone outperformed all dense retrievers (MAP 0.522, MRR 0.677, R@10 0.586 versus best dense R@10 of 0.531 by Nomic v2), which the authors attribute to the headword- and citation-centric nature of dictionary queries plus dynamic term weighting. Re-ranking improved both families; fine-tuning the reranker yielded the strongest configurations:

Configuration	R@10	MRR	MAP
BM25 + FT reranker	0.647	0.936	0.611
Fusion (55/45) + FT reranker	0.652	0.945	0.609

Weighted RRF fusion slightly favored BM25 (55/45 split) consistent with its individual dominance. Notably, hybrid fusion did not beat BM25 + fine-tuned reranker on MAP; the two final configurations were retained deliberately to probe generator behavior under different retrieval conditions.

Generation results

The baseline gap is stark: on 1,000 meaning-based questions about Qur'anic/Hadith vocabulary, un-augmented Fanar scored 54% and ALLaM 56.85%, versus 88% for Gemini 2.5 Pro. With RAG in zero-shot mode, Fanar reached 89% (+35 points) and ALLaM 86% (+29 points), while Gemini rose to 96%.

Model	Baseline	RAG-ZS	RAG-FS
Fanar	54	89	89
ALLaM	56.85	86	78
Gemini 2.5 Pro	88	96	95

Two results deserve emphasis. First, the bold claim that RAG "dramatically narrows" the gap between mid-sized Arabic LLMs and much larger proprietary models is supported: augmented Fanar and ALLaM exceed 85% against Gemini's 96%. Second, few-shot prompting hurt ALLaM severely (86% → 78%), which the authors attribute to instruction-adherence degradation under longer contexts rather than example quality — a finding consistent with known long-context sensitivity of smaller LLMs. On the expanded 2,000-question set covering six types, few-shot Fanar and zero-shot ALLaM both averaged 87%.

Per-type analysis shows Fanar exceeding 90% on Author, Contextual Meaning, and Source of Citation (types with relatively unambiguous answers, since test citations come exclusively from the Qur'an and Hadith), but dropping to 72.2% on Part of Speech — reflecting the well-known diacritic sensitivity of Arabic POS identification. ALLaM was more uniform (79.5%–90.6%) and outperformed Fanar by 11 points on Historical Date, suggesting complementary strengths amenable to ensembling.

Error analysis

Most errors stem from confusing target words with diacritically distinct homographs or similar compound phrases, producing multi-answer responses where a single answer is required, or attributing citations to the wrong source (e.g., answering with the author of naffala instead of nafl). Some errors trace to genuine ambiguity within DHDA entries themselves (e.g., conflation of al-anāth and al-nisā' in the bakr min al-ināth entry), indicating that dictionary-side disambiguation is also implicated. Additional failure modes include retrieval misses leaving irrelevant context, weak abstention behavior — Gemini reliably reported "information not found" while Fanar and ALLaM sometimes did not — and a striking Fanar degeneration case in which the correct document was present yet the model emitted extended Chinese text before failing to answer. These cases show that retrieval quality is not the sole bottleneck; generation-side robustness and instruction adherence constrain performance independently.

Limitations and open questions

The paper concedes several constraints at specific points. Template-generated questions limit ecological validity; human-authored query sets remain unevaluated. Automated scoring rests on a single judge validated against a single human annotator. The retrieval corpus excludes dates and sources by design, so performance on those intents depends entirely on routing correctness. Few-shot sensitivity in ALLaM is explained but not resolved. Open questions left by the work include whether ensemble or voting-based retrieval configurations can exploit the observed complementarity between BM25-only and hybrid pipelines, whether dynamically adjusting the number of retrieved documents per query improves coverage without degrading precision, and how to build generators robust to diacritical variation and complex phrasal units.

Conclusion

The paper demonstrates that a historical, evidence-cited dictionary can serve as an effective non-parametric knowledge base for Arabic RAG. Hybrid retrieval with a fine-tuned cross-encoder achieves MRR up to 0.945 over 198K dictionary documents, and grounding Fanar or ALLaM in DHDA raises answer accuracy from the mid-50s to above 85%, approaching a far larger proprietary baseline. The residual error mass concentrates precisely where Classical Arabic is hardest — diacritics, homography, and compound expressions — making these the natural next targets for both modeling and resource curation.