Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ListT5: Listwise Reranking with Fusion-in-Decoder Improves Zero-shot Retrieval (2402.15838v3)

Published 24 Feb 2024 in cs.IR

Abstract: We propose ListT5, a novel reranking approach based on Fusion-in-Decoder (FiD) that handles multiple candidate passages at both train and inference time. We also introduce an efficient inference framework for listwise ranking based on m-ary tournament sort with output caching. We evaluate and compare our model on the BEIR benchmark for zero-shot retrieval task, demonstrating that ListT5 (1) outperforms the state-of-the-art RankT5 baseline with a notable +1.3 gain in the average NDCG@10 score, (2) has an efficiency comparable to pointwise ranking models and surpasses the efficiency of previous listwise ranking models, and (3) overcomes the lost-in-the-middle problem of previous listwise rerankers. Our code, model checkpoints, and the evaluation framework are fully open-sourced at \url{https://github.com/soyoung97/ListT5}.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (31)
  1. Ms marco: A human generated machine reading comprehension dataset.
  2. Overview of the trec 2020 deep learning track.
  3. Exaranker: Explanation-augmented neural ranker.
  4. Fid-light: Efficient and effective retrieval-augmented text generation.
  5. Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering.
  6. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20(4):422–446.
  7. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics.
  8. Fid-ex: Improving sequence-to-sequence models for extractive rationale generation.
  9. Lost in the middle: How language models use long contexts.
  10. Zero-shot listwise document reranking with a large language model.
  11. Keith McLuckie and Angus Barber. 1986. Tournament Sort, pages 68–86. Macmillan Education UK, London.
  12. Relational pooling for graph representations. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4663–4673. PMLR.
  13. Large dual encoders are generalizable retrievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9844–9855, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  14. Document ranking with a pretrained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708–718, Online. Association for Computational Linguistics.
  15. OpenAI. 2022. Introducing chatgpt. https://openai.com/blog/chatgpt.
  16. The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models.
  17. Rankvicuna: Zero-shot listwise document reranking with open-source large language models.
  18. Rankzephyr: Effective and robust zero-shot listwise reranking is a breeze!
  19. Large language models are effective text rankers with pairwise ranking prompting.
  20. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  21. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks.
  22. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389.
  23. In defense of cross-encoders for zero-shot retrieval.
  24. Instruction distillation makes large language models efficient zero-shot rankers.
  25. Is chatgpt good at search? investigating large language models as re-ranking agents.
  26. Found in the middle: Permutation self-consistency improves listwise ranking in large language models.
  27. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.
  28. Learning list-level domain-invariant representations for ranking.
  29. Dmitry Yarotsky. 2018. Universal approximations of invariant maps by neural networks.
  30. Coco-dr: Combating distribution shifts in zero-shot dense retrieval with contrastive and distributionally robust learning.
  31. Rankt5: Fine-tuning t5 for text ranking with ranking losses.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Soyoung Yoon (8 papers)
  2. Jiyeon Kim (22 papers)
  3. Yireun Kim (9 papers)
  4. Hyeongu Yun (7 papers)
  5. Seung-won Hwang (59 papers)
  6. Eunbi Choi (8 papers)
Citations (5)
View on Semantic Scholar

Summary

Enhancing Zero-shot Retrieval with ListT5: A Listwise Reranking Approach

Introduction

Traditional neural information retrieval systems, despite advancements, exhibit limitations in zero-shot or domain-agnostic tasks. Listwise reranking models have emerged as promising candidates to mitigate these issues. However, existing listwise approaches either suffer from computational inefficiency or are unable to effectively compare multiple passages due to the "lost in the middle" problem. To address these challenges, we introduce ListT5, a novel listwise reranking method based on Fusion-in-Decoder (FiD) architecture that simultaneously assesses multiple candidate passages, paving the way for more effective zero-shot retrieval.

ListT5 Approach

ListT5 leverages the FiD architecture, allowing it to efficiently process multiple passages as a unified input, overcoming the drawbacks seen in other listwise reranking solutions. By treating each passage with a unique identifier and concatenating these with a query, ListT5 enables effective comparative analysis without succumbing to positional biases. This method not only achieves notable improvements over existing approaches but does so with computational efficiency equivalent to pointwise models and better than other listwise rerankers.

Computational Efficiency

ListT5 significantly reduces computational overhead by introducing an innovative inference framework based on m-ary tournament sort with output caching, achieving an asymptotic cost lower than pairwise models and competitive with pointwise models. This efficiency does not compromise on the model's ability to process listwise inputs with smaller and more parameter-efficient architectures.

Robustness to Positional Bias

The FiD architecture inherently mitigates the "lost in the middle" problem encountered by LLM-based listwise rerankers. By eliminating positional encoding in favor of passage identifiers, ListT5 ensures robustness against biases towards passages located at the beginning or end of inputs, a critical advantage for accurately ranking passages of varying relevance.

Zero-shot Performance

In terms of zero-shot retrieval performance, ListT5 outperforms both pointwise and previous listwise reranking methods. It demonstrates superior efficiency and accuracy, validated through rigorous evaluations on the BEIR benchmark. This superior performance is attributed to its innovative reranking methodology and computational efficiency, marking a significant step forward in zero-shot retrieval capability.

Future Developments and Implications

ListT5's introduction marks a pivotal advancement in search and retrieval tasks, particularly in zero-shot settings. The model's ability to maintain computational efficiency while effectively ranking multiple passages presents a promising avenue for further exploration and development in AI-driven information retrieval systems. As we strive for models capable of understanding and retrieving information with minimal biases and inefficiencies, ListT5's novel approach offers a framework upon which future innovations can build.

Notably, while ListT5 addresses several limitations of existing reranking methods, areas such as early stopping in decoding processes and optimizing the tournament sort algorithm for better efficiency remain open for exploration. These potential enhancements underline the ongoing journey toward refining AI models for the complex demands of modern information retrieval.

In conclusion, the introduction of ListT5 to the domain of zero-shot retrieval undeniably represents a significant contribution to the field of neural information retrieval. By addressing efficiency, bias, and performance shortcomings of previous models, it sets a new benchmark for future research and practical applications in AI-driven search technologies.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets