Zero-Shot Cross-Lingual Reranking with Large Language Models for Low-Resource Languages (2312.16159v1)
Abstract: LLMs have shown impressive zero-shot capabilities in various document reranking tasks. Despite their successful implementations, there is still a gap in existing literature on their effectiveness in low-resource languages. To address this gap, we investigate how LLMs function as rerankers in cross-lingual information retrieval (CLIR) systems for African languages. Our implementation covers English and four African languages (Hausa, Somali, Swahili, and Yoruba) and we examine cross-lingual reranking with queries in English and passages in the African languages. Additionally, we analyze and compare the effectiveness of monolingual reranking using both query and document translations. We also evaluate the effectiveness of LLMs when leveraging their own generated translations. To get a grasp of the effectiveness of multiple LLMs, our study focuses on the proprietary models RankGPT-4 and RankGPT-3.5, along with the open-source model, RankZephyr. While reranking remains most effective in English, our results reveal that cross-lingual reranking may be competitive with reranking in African languages depending on the multilingual capability of the LLM.
- CIRAL: A Test Suite for CLIR in African Languages.
- Nima Asadi and Jimmy Lin. 2013. Effectiveness/Efficiency Tradeoffs for Candidate Generation in Multi-stage Retrieval Architectures. Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval.
- Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval.
- Dense Passage Retrieval for Open-Domain Question Answering. ArXiv, abs/2004.04906.
- Simple Yet Effective Neural Ranking and Reranking Baselines for Cross-Lingual Information Retrieval. ArXiv, abs/2304.01019.
- Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), pages 2356–2362.
- Fine-Tuning LLaMA for Multi-Stage Text Retrieval. ArXiv, abs/2310.08319.
- Zero-Shot Listwise Document Reranking with a Large Language Model. ArXiv, abs/2305.02156.
- Niklas Muennighoff. 2022. SGPT: GPT Sentence Embeddings for Semantic Search. ArXiv, abs/2202.08904.
- Text and Code Embeddings by Contrastive Pre-Training. ArXiv, abs/2201.10005.
- Large Dual Encoders Are Generalizable Retrievers. ArXiv, abs/2112.07899.
- Multi-Stage Document Ranking with BERT. ArXiv, abs/1910.14424.
- RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models. ArXiv, abs/2309.15088.
- RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze! ArXiv, abs/2312.02724.
- Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting. ArXiv, abs/2306.17563.
- Stephen E. Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Found. Trends Inf. Retr., 3:333–389.
- Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent. ArXiv, abs/2304.09542.
- Zephyr: Direct Distillation of LM Alignment. ArXiv, abs/2310.16944.
- GPT-NER: Named Entity Recognition via Large Language Models. ArXiv, abs/2304.10428.
- Language Models are Universal Embedders. ArXiv, abs/2310.08232.
- Large Language Models Are Human-Level Prompt Engineers. ArXiv, abs/2211.01910.
- Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis. ArXiv, abs/2304.04675.
- Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels. ArXiv, abs/2310.14122.
- Mofetoluwa Adeyemi (8 papers)
- Akintunde Oladipo (7 papers)
- Ronak Pradeep (26 papers)
- Jimmy Lin (208 papers)