Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

GRM: Generative Relevance Modeling Using Relevance-Aware Sample Estimation for Document Retrieval (2306.09938v1)

Published 16 Jun 2023 in cs.IR

Abstract: Recent studies show that Generative Relevance Feedback (GRF), using text generated by LLMs, can enhance the effectiveness of query expansion. However, LLMs can generate irrelevant information that harms retrieval effectiveness. To address this, we propose Generative Relevance Modeling (GRM) that uses Relevance-Aware Sample Estimation (RASE) for more accurate weighting of expansion terms. Specifically, we identify similar real documents for each generated document and use a neural re-ranker to estimate their relevance. Experiments on three standard document ranking benchmarks show that GRM improves MAP by 6-9% and R@1k by 2-4%, surpassing previous methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. UMass at TREC 2004: Novelty and HARD. Computer Science Department Faculty Publication Series (2004), 189.
  2. ASK for information retrieval: Part I. Background and theory. Journal of documentation (1982).
  3. A Non-Factoid Question-Answering Taxonomy. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1196–1207.
  4. Inpars: Unsupervised dataset generation for information retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2387–2392.
  5. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020).
  6. Entity query feature expansion using knowledge base links. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval. 365–374.
  7. ExaRanker: Explanation-Augmented Neural Ranker. arXiv preprint arXiv:2301.10521 (2023).
  8. SPLADE: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2288–2292.
  9. Precise Zero-Shot Dense Retrieval without Relevance Labels. arXiv preprint arXiv:2212.10496 (2022).
  10. Samuel Huston and W Bruce Croft. 2014. Parameters learned in the comparison of retrieval models using term dependencies. Ir, University of Massachusetts (2014).
  11. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS) 20, 4 (2002), 422–446.
  12. Carlos Lassance and Stéphane Clinchant. 2023. Naver Labs Europe (SPLADE)@ TREC Deep Learning 2022. arXiv preprint arXiv:2302.12574 (2023).
  13. Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls. ArXiv abs/2108.11044 (2021).
  14. Improving query representations for dense retrieval with pseudo relevance feedback: A reproducibility study. In Advances in Information Retrieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10–14, 2022, Proceedings, Part I. Springer, 599–612.
  15. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2356–2362.
  16. In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021). 163–173.
  17. Query Expansion Using Contextual Clue Sampling with Language Models. arXiv preprint arXiv:2210.07093 (2022).
  18. IntenT5: Search Result Diversification using Causal Language Models. arXiv preprint arXiv:2108.04026 (2021).
  19. Streamlining Evaluation with ir-measures. In European Conference on Information Retrieval. Springer, 305–310.
  20. Adaptive Re-Ranking with a Corpus Graph. In 31st ACM International Conference on Information and Knowledge Management. https://doi.org/10.1145/3511808.3557231
  21. Generative and Pseudo-Relevant Feedback for Sparse, Dense and Learned Sparse Retrieval. arXiv preprint arXiv:2305.07477 (2023).
  22. Generative Relevance Feedback with Large Language Models. 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (2023).
  23. How deep is your learning: The DL-HARD annotated deep learning dataset. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2335–2341.
  24. CODEC: Complex Document and Entity Collection. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval.
  25. Conceptual language models for domain-specific retrieval. Information Processing & Management 46, 4 (2010), 448–469.
  26. Donald Metzler and W Bruce Croft. 2005. A markov random field model for term dependencies. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval. 472–479.
  27. Donald Metzler and W Bruce Croft. 2007. Latent concept expansion using markov random fields. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. 311–318.
  28. Ceqe: Contextualized embeddings for query expansion. In Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28–April 1, 2021, Proceedings, Part I 43. Springer, 467–482.
  29. Document Ranking with a Pretrained Sequence-to-Sequence Model. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings. 708–718.
  30. Visconde: Multi-document QA with GPT-3 and Neural Reranking. In Advances in Information Retrieval: 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2–6, 2023, Proceedings, Part II. Springer, 534–543.
  31. Stephen E Robertson and Steve Walker. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In SIGIR’94. Springer, 232–241.
  32. Joseph Rocchio. 1971. Relevance feedback in information retrieval. The Smart retrieval system-experiments in automatic document processing (1971), 313–323.
  33. Question rewriting for conversational question answering. In Proceedings of the 14th ACM international conference on web search and data mining. 355–363.
  34. Ellen M. Voorhees. 2004. Overview of the TREC 2004 Robust Track. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004). Gaithersburg, Maryland, 52–69.
  35. ColBERT-PRF: Semantic Pseudo-Relevance Feedback for Dense Passage and Document Retrieval. ACM Transactions on the Web (2022).
  36. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems.
  37. CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 10000–10014. https://aclanthology.org/2022.emnlp-main.679
  38. Chenyan Xiong and Jamie Callan. 2015. Query Expansion with Freebase. In Proceedings of the 2015 International Conference on The Theory of Information Retrieval (ICTIR ’15). Association for Computing Machinery, New York, NY, USA, 111–120. https://doi.org/10.1145/2808194.2809446
  39. Pretrained Transformers for Text Ranking: BERT and Beyond. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 1154–1156.
  40. Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 3592–3596.
  41. Chengxiang Zhai and John Lafferty. 2001. Model-based feedback in the language modeling approach to information retrieval. In Proceedings of the tenth international conference on Information and knowledge management. 403–410.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now