Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Two-Stage Adaptation of Large Language Models for Text Ranking (2311.16720v3)

Published 28 Nov 2023 in cs.IR

Abstract: Text ranking is a critical task in information retrieval. Recent advances in pre-trained LLMs (PLMs), especially LLMs, present new opportunities for applying them to text ranking. While supervised fine-tuning (SFT) with ranking data has been widely explored to better align PLMs with text ranking goals, previous studies have focused primarily on encoder-only and encoder-decoder PLMs. Research on leveraging decoder-only LLMs for text ranking remains scarce. An exception to this is RankLLaMA, which uses direct SFT to explore LLaMA's potential for text ranking. In this work, we propose a two-stage progressive paradigm to better adapt LLMs to text ranking. First, we conduct continual pre-training (CPT) of LLMs on a large weakly-supervised corpus. Second, we perform SFT, and propose an improved optimization strategy building upon RankLLaMA. Our experimental results on multiple benchmarks show that our approach outperforms previous methods in both in-domain and out-domain scenarios.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. Qwen technical report. CoRR, abs/2309.16609.
  2. Yahoo! learning to rank challenge overview. In Proceedings of the Yahoo! Learning to Rank Challenge, held at ICML 2010, Haifa, Israel, June 25, 2010, volume 14 of JMLR Proceedings, pages 1–24. JMLR.org.
  3. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1597–1607. PMLR.
  4. Overview of the TREC 2020 deep learning track. CoRR, abs/2102.07662.
  5. Overview of the TREC 2019 deep learning track. CoRR, abs/2003.07820.
  6. Rethink training of BERT rerankers in multi-stage retrieval pipeline. In Proc. of ECIR.
  7. A deep relevance matching model for ad-hoc retrieval. In CIKM, pages 55–64. ACM.
  8. Learning-to-rank with BERT in tf-ranking. CoRR, abs/2004.08476.
  9. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  10. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  11. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
  12. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.
  13. Holistic evaluation of language models. CoRR, abs/2211.09110.
  14. Pretrained Transformers for Text Ranking: BERT and Beyond. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers.
  15. Fine-tuning llama for multi-stage text retrieval. CoRR, abs/2310.08319.
  16. Zero-shot listwise document reranking with a large language model. arXiv preprint arXiv:2305.02156.
  17. Niklas Muennighoff. 2022. SGPT: GPT sentence embeddings for semantic search. CoRR.
  18. MS MARCO: A human generated machine reading comprehension dataset. In Proc. of NeurIPS.
  19. Rodrigo Frassetto Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. CoRR, abs/1901.04085.
  20. Document ranking with a pretrained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 708–718. Association for Computational Linguistics.
  21. Multi-stage document ranking with BERT. CoRR, abs/1910.14424.
  22. OpenAI. 2023. Gpt-4 technical report.
  23. Tao Qin and Tie-Yan Liu. 2013. Introducing LETOR 4.0 datasets. CoRR, abs/1306.2597.
  24. Large language models are effective text rankers with pairwise ranking prompting. arXiv preprint arXiv:2306.17563.
  25. Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389.
  26. Improving passage retrieval with zero-shot question generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 3781–3797. Association for Computational Linguistics.
  27. Teven Le Scao et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100.
  28. Instruction distillation makes large language models efficient zero-shot rankers.
  29. Is chatgpt good at search? investigating large language models as re-ranking agent. arXiv preprint arXiv:2304.09542.
  30. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy. Association for Computational Linguistics.
  31. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
  32. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971.
  33. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288.
  34. Text embeddings by weakly-supervised contrastive pre-training.
  35. C-pack: Packaged resources to advance general chinese embedding.
  36. Hybrid retrieval and multi-stage text ranking solution at trec 2022 deep learning track. arXiv preprint arXiv:2308.12039.
  37. Baichuan 2: Open large-scale language models.
  38. Hlatr: enhance multi-stage text retrieval with hybrid list aware transformer reranking. arXiv preprint arXiv:2205.10569.
  39. Beyond yes and no: Improving zero-shot LLM rankers via scoring fine-grained relevance labels. CoRR, abs/2310.14122.
  40. Rankt5: Fine-tuning T5 for text ranking with ranking losses. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023, pages 2308–2313. ACM.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Longhui Zhang (9 papers)
  2. Yanzhao Zhang (18 papers)
  3. Dingkun Long (23 papers)
  4. Pengjun Xie (85 papers)
  5. Meishan Zhang (70 papers)
  6. Min Zhang (630 papers)
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets