Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

TPRF: A Transformer-based Pseudo-Relevance Feedback Model for Efficient and Effective Retrieval (2401.13509v2)

Published 24 Jan 2024 in cs.IR

Abstract: This paper considers Pseudo-Relevance Feedback (PRF) methods for dense retrievers in a resource constrained environment such as that of cheap cloud instances or embedded systems (e.g., smartphones and smartwatches), where memory and CPU are limited and GPUs are not present. For this, we propose a transformer-based PRF method (TPRF), which has a much smaller memory footprint and faster inference time compared to other deep LLMs that employ PRF mechanisms, with a marginal effectiveness loss. TPRF learns how to effectively combine the relevance feedback signals from dense passage representations. Specifically, TPRF provides a mechanism for modelling relationships and weights between the query and the relevance feedback signals. The method is agnostic to the specific dense representation used and thus can be generally applied to any dense retriever.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (32)
  1. Hiteshwar Kumar Azad and Akshay Deepak. 2019. Query expansion techniques for information retrieval: a survey. Information Processing & Management 56, 5 (2019), 1698–1735.
  2. Overview of the TREC 2019 Deep Learning Track. In Text REtrieval Conference, TREC.
  3. Overview of the TREC 2020 Deep Learning Track. In Text REtrieval Conference, TREC.
  4. Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2978–2988.
  5. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 4171–4186.
  6. A deep look into neural ranking models for information retrieval. Information Processing & Management 57, 6 (2020), 102067.
  7. Improving efficient neural ranking models with cross-architecture knowledge distillation. arXiv preprint arXiv:2010.02666 (2020).
  8. Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling. arXiv preprint arXiv:2104.06967 (2021).
  9. Billion-scale similarity search with gpus. IEEE Transactions on Big Data (2019).
  10. Pseudo relevance feedback with deep language models and dense retrievers: Successes and pitfalls. arXiv preprint arXiv:2108.11044 (2021).
  11. Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback: A Reproducibility Study. In Proceedings of the 44rd European Conference on Information Retrieval.
  12. Distilling dense representations for ranking using tightly-coupled teachers. arXiv preprint arXiv:2010.11386 (2020).
  13. In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021). 163–173.
  14. MS MARCO: A Human Generated Machine Reading Comprehension Dataset. In Workshop on Cognitive Computing at NIPS.
  15. Rethinking Query Expansion for BERT Reranking. In Advances in Information Retrieval-42nd European Conference on IR Research, ECIR 2020.
  16. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. arXiv preprint arXiv:2010.08191 (2020).
  17. Language Models Are Unsupervised Multitask Learners. In OpenAI blog.
  18. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21 (2020), 1–67.
  19. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  20. RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking. arXiv preprint arXiv:2110.07367 (2021).
  21. Attention is All You Need. In Proceedings of the 31st International Conference on Neural Information Processing Systems. 6000–6010.
  22. A Pseudo-Relevance Feedback Framework Combining Relevance Matching and Semantic Matching for Information Retrieval. Information Processing & Management 57, 6 (2020), 102342.
  23. Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval. arXiv preprint arXiv:2106.11251 (2021).
  24. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 38–45.
  25. Approximate Nearest Neighbor Negative Contrastive Learning For Dense Text Retrieval. arXiv preprint arXiv:2007.00808 (2020).
  26. XLNet: Generalized Autoregressive Pretraining for Language Understanding. Advances in Neural Information Processing Systems 32 (2019), 5753–5763.
  27. Pretrained Transformers for Text Ranking: BERT and Beyond. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 1154–1156.
  28. PGT: Pseudo Relevance Feedback Using a Graph-Based Transformer. In European Conference on Information Retrieval.
  29. Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management.
  30. RepBERT: Contextualized Text Embeddings for First-Stage Retrieval. arXiv preprint arXiv:2006.15498 (2020).
  31. BERT-QE: Contextualized Query Expansion for Document Re-ranking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings. 4718–4728.
  32. Contextualized query expansion via unsupervised chunk selection for text retrieval. Information Processing & Management 58, 5 (2021), 102672.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Chuting Yu (1 paper)
  2. Hang Li (277 papers)
  3. Ahmed Mourad (8 papers)
  4. Bevan Koopman (37 papers)
  5. Guido Zuccon (73 papers)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets