Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Leveraging Domain Adaptation and Data Augmentation to Improve Qur'anic IR in English and Arabic (2312.02803v1)

Published 5 Dec 2023 in cs.CL and cs.AI

Abstract: In this work, we approach the problem of Qur'anic information retrieval (IR) in Arabic and English. Using the latest state-of-the-art methods in neural IR, we research what helps to tackle this task more efficiently. Training retrieval models requires a lot of data, which is difficult to obtain for training in-domain. Therefore, we commence with training on a large amount of general domain data and then continue training on in-domain data. To handle the lack of in-domain data, we employed a data augmentation technique, which considerably improved results in MRR@10 and NDCG@5 metrics, setting the state-of-the-art in Qur'anic IR for both English and Arabic. The absence of an Islamic corpus and domain-specific model for IR task in English motivated us to address this lack of resources and take preliminary steps of the Islamic corpus compilation and domain-specific LLM (LM) pre-training, which helped to improve the performance of the retrieval models that use the domain-specific LM as the shared backbone. We examined several LLMs (LMs) in Arabic to select one that efficiently deals with the Qur'anic IR task. Besides transferring successful experiments from English to Arabic, we conducted additional experiments with retrieval task in Arabic to amortize the scarcity of general domain datasets used to train the retrieval models. Handling Qur'anic IR task combining English and Arabic allowed us to enhance the comparison and share valuable insights across models and languages.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Cross-lingual transfer for low-resource Arabic language understanding. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 225–237, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  2. Detecting semantic-based similarity between verses of the quran with doc2vec. Procedia Computer Science, 189:351–358.
  3. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association.
  4. Ms marco: A human generated machine reading comprehension dataset.
  5. Arabic natural language processing for qur’anic research: A systematic review. Artificial Intelligence Review, 56(7):6801–6854.
  6. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, page 41–48, New York, NY, USA. Association for Computing Machinery.
  7. Bridging the lexical chasm: statistical approaches to answer-finding. In Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.
  8. Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4758–4781, Online. Association for Computational Linguistics.
  9. On the opportunities and risks of foundation models.
  10. mmarco: A multilingual version of the ms marco passage ranking dataset.
  11. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  12. Signature verification using a "siamese" time delay neural network. In Proceedings of the 6th International Conference on Neural Information Processing Systems, NIPS’93, page 737–744, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
  13. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.
  14. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  15. Pre-training with whole word masking for chinese bert. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3504–3514.
  16. “is whole word masking always better for Chinese BERT?”: Probing on Chinese grammatical error correction. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1–8, Dublin, Ireland. Association for Computational Linguistics.
  17. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  18. Fabio Henrique Kiyoiti dos Santos Tanaka and Claus Aranha. 2019. Data augmentation using gans.
  19. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans. Comput. Healthcare, 3(1).
  20. On the domain adaptation and generalization of pretrained language models: A survey.
  21. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. Association for Computational Linguistics.
  22. Christophe Van Gysel and Maarten de Rijke. 2018. Pytrec_eval. In The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM.
  23. Efficient natural language response suggestion for smart reply.
  24. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring.
  25. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92–104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  26. Resources for brewing beir: Reproducible reference models and an official leaderboard.
  27. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics.
  28. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert.
  29. Zero-shot neural passage retrieval via domain-targeted synthetic question generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1075–1088, Online. Association for Computational Linguistics.
  30. Rana Malhas and Tamer Elsayed. 2020. Ayatec: building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 19(6):1–21.
  31. Rana Malhas and Tamer Elsayed. 2022. Arabic machine reading comprehension on the holy qur’an using cl-arabert. Information Processing and Management, 59(6):103068.
  32. Rana R Malhas. 2023. ARABIC QUESTION ANSWERING ON THE HOLY QUR’AN. Ph.D. thesis.
  33. Tom M Mitchell. 1997. Machine learning, volume 1. McGraw-hill New York.
  34. Ensaf Hussein Mohamed and Eyad Mohamed Shokry. 2022. Qsst: A quranic semantic search tool based on word embedding. Journal of King Saud University - Computer and Information Sciences, 34(3):934–945.
  35. Vera Pavlova and Mohammed Makhlouf. 2023. BIOptimus: Pre-training an optimal biomedical language model with curriculum learning for named entity recognition. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 337–349, Toronto, Canada. Association for Computational Linguistics.
  36. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5835–5847, Online. Association for Computational Linguistics.
  37. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  38. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512–4525, Online. Association for Computational Linguistics.
  39. RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2825–2835, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  40. Maxim Romanov and Masoumeh Seydi. 2019. Openiti: a machine-readable corpus of islamicate texts. Zenodo, URL: https://doi. org/10.5281/zenodo, 3082464.
  41. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics.
  42. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3715–3734, Seattle, United States. Association for Computational Linguistics.
  43. Fast WordPiece tokenization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2089–2103, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  44. Curriculum learning: A survey.
  45. Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 296–310, Online. Association for Computational Linguistics.
  46. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS 2021) - Datasets and Benchmarks Track (Round 2).
  47. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).
  48. Representation learning with contrastive predictive coding.
  49. Attention is all you need.
  50. TSDAE: Using transformer-based sequential denoising auto-encoderfor unsupervised sentence embedding learning. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 671–688, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  51. GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2345–2360, Seattle, United States. Association for Computational Linguistics.
  52. A theoretical analysis of ndcg type ranking measures.
  53. Jason Wei and Kai Zou. 2019. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6382–6388, Hong Kong, China. Association for Computational Linguistics.
  54. Should you mask 15% in masked language modeling? In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2985–3000, Dubrovnik, Croatia. Association for Computational Linguistics.
  55. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  56. Unsupervised data augmentation for consistency training.
  57. Answering complex open-domain questions with multi-hop dense retrieval.
  58. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics.
  59. Qanet: Combining local convolution with global self-attention for reading comprehension.

Summary

  • The paper leverages domain adaptation and data augmentation to overcome data scarcity and improve retrieval on Qur'anic texts in English and Arabic.
  • It introduces a two-stage curriculum learning strategy and employs in-domain training with augmented Islamic corpora to refine neural IR models.
  • Performance metrics such as MRR@10, NDCG@5, and Recall@100 indicate significant improvements, narrowing the gap between English and Arabic retrieval.

Overview of Qur'anic Information Retrieval

Introduction to Challenges in Qur'anic IR

Recent advancements have bolstered capabilities in NLP, yet applying deep learning to Qur'anic Information Retrieval (IR) remains an underexplored area. The scarcity of domain-specific data for training retrieval models presents a significant hurdle. Qur'anic IR involves English and Arabic languages. English is abundant with databases and pre-trained LLMs (LMs), while Arabic, a low-resource language depending on context, fares better within the Islamic domain with existing Islamic corpora.

Islamic Domain LLM Development

To address the lack of specific resources for English IR tasks within the Islamic domain, this work begins with the compilation of an Islamic corpus and the pre-training of a domain-specific LM. The pre-training employs a two-stage curriculum learning method using an Islamic corpus and the continued adaptation of a general domain model. A masking technique initially predicts masked tokens or subtokens before moving to predict entire masked words in a sentence, increasing task complexity.

Neural IR Models and Augmentation Techniques

For testing, the Qur'anic Reading Comprehension Dataset (QRCD) was adapted to an IR format. Models were evaluated using metrics like MRR@10, NDCG@5, and Recall@100. To improve training, data augmentation leveraged verse-related content from Islamic exegesis texts by identifying strongly correlated verse pairs, creating a more extensive in-domain training set. Domain-specific models were parallelly trained on general domain data before in-domain training to effectively use limited data.

Tackling the Qur'anic IR Task in Arabic

The paper expands to Qur'anic IR in Arabic, firstly, by selecting appropriate Arabic LMs and, secondly, by experimenting with training techniques that proved successful in English. Among these, distillation from a bilingual teacher-student model and further pre-training on a machine-translated Natural Language Inference dataset showed promising improvements. The final phase included in-domain training to refine the retrieval models further, as demonstrated by improved performance metrics.

Conclusion and Future Directions

The paper highlights the importance of domain-specific data augmentation and adaptation of pre-trained models to the domain. Although retrieval models trained for English IR tasks outperformed Arabic ones, the gap was narrowed by transferring successful strategies from English and applying machine translation to augment data. Future work could include exploring multilingual extensions and evaluating real-world user queries to refine these models' effectiveness further.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets