Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search (2303.06573v2)

Published 12 Mar 2023 in cs.IR

Abstract: Precisely understanding users' contextual search intent has been an important challenge for conversational search. As conversational search sessions are much more diverse and long-tailed, existing methods trained on limited data still show unsatisfactory effectiveness and robustness to handle real conversational search scenarios. Recently, LLMs have demonstrated amazing capabilities for text generation and conversation understanding. In this work, we present a simple yet effective prompting framework, called LLM4CS, to leverage LLMs as a text-based search intent interpreter to help conversational search. Under this framework, we explore three prompting methods to generate multiple query rewrites and hypothetical responses, and propose to aggregate them into an integrated representation that can robustly represent the user's real contextual search intent. Extensive automatic evaluations and human evaluations on three widely used conversational search benchmarks, including CAsT-19, CAsT-20, and CAsT-21, demonstrate the remarkable performance of our simple LLM4CS framework compared with existing methods and even using human rewrites. Our findings provide important evidence to better understand and leverage LLMs for conversational search.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. Open-domain question answering goes conversational via question rewriting. In NAACL-HLT, pages 520–534. Association for Computational Linguistics.
  2. Inpars: Data augmentation for information retrieval using large language models. CoRR, abs/2202.05144.
  3. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  4. Reinforced question rewriting for conversational question answering.
  5. Research frontiers in information retrieval: Report from the third strategic workshop on information retrieval in lorne (swirl 2018). In ACM SIGIR Forum, volume 52, pages 34–90. ACM New York, NY, USA.
  6. Zhuyun Dai and Jamie Callan. 2019. Deeper text understanding for IR with contextual neural language modeling. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pages 985–988. ACM.
  7. Dialog inpainting: Turning documents into dialogs. In International Conference on Machine Learning, pages 4558–4586. PMLR.
  8. Promptagator: Few-shot dense retrieval from 8 examples. In 11th International Conference on Learning Representations, ICLR 2023.
  9. Trec cast 2019: The conversational assistance track overview. In In Proceedings of TREC.
  10. Cast 2020: The conversational assistance track overview. In In Proceedings of TREC.
  11. Trec cast 2021: The conversational assistance track overview. In In Proceedings of TREC.
  12. Precise zero-shot dense retrieval without relevance labels. CoRR, abs/2212.10496.
  13. Inpars-v2: Large language models as efficient dataset generators for information retrieval. CoRR, abs/2301.01820.
  14. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. CoRR, abs/2306.02561.
  15. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval (SIGIR), pages 39–48. ACM.
  16. Sungdong Kim and Gangwoo Kim. 2022. Saving dense retriever from shortcut dependency in conversational search.
  17. Zero-shot query contextualization for conversational search. In Proceedings of the 45th International ACM SIGIR conference on research and development in Information Retrieval (SIGIR).
  18. Contextualized query embeddings for conversational search. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  19. Conversational question reformulation via sequence-to-sequence architectures and pretrained language models. arXiv preprint arXiv:2004.01909.
  20. Multi-stage conversational passage retrieval: An approach to fusing term importance estimation and neural query rewriting. ACM Transactions on Information Systems (TOIS), 39(4):1–29.
  21. Generative relevance feedback with large language models. CoRR, abs/2304.13157.
  22. Search-oriented conversational query editing. In ACL (Findings), volume ACL 2023 of Findings of ACL. Association for Computational Linguistics.
  23. Curriculum contrastive context denoising for few-shot conversational dense retrieval. In Proceedings of the 45th International ACM SIGIR conference on research and development in Information Retrieval (SIGIR).
  24. Convtrans: Transforming web search sessions for conversational dense retrieval. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  25. Learning denoised and interpretable session representation for conversational search. In Proceedings of the ACM Web Conference, pages 3193–3202.
  26. Generation-augmented retrieval for open-domain question answering. In ACL/IJCNLP (1), pages 4089–4100. Association for Computational Linguistics.
  27. ConvGQR: generative query reformulation for conversational search. In ACL, volume ACL 2023. Association for Computational Linguistics.
  28. Learning to relate to previous turns in conversational search. In 29th ACM SIGKDD Conference On Knowledge Discover and Data Mining (SIGKDD).
  29. Hongjin Qian and Zhicheng Dou. 2022. Explicit query rewriting for conversational dense retrieval. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  30. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.
  31. Large language models are strong zero-shot retriever. CoRR, abs/2304.14233.
  32. Is chatgpt good at search? investigating large language models as re-ranking agent. CoRR, abs/2304.09542.
  33. Lamda: Language models for dialog applications. CoRR, abs/2201.08239.
  34. Question rewriting for conversational question answering. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining (WSDM), pages 355–363.
  35. A comparison of question rewriting methods for conversational passage retrieval. In ECIR (2), volume 12657 of Lecture Notes in Computer Science, pages 418–424. Springer.
  36. Christophe Van Gysel and Maarten de Rijke. 2018. Pytrec_eval: An extremely fast python interface to trec_eval. In SIGIR. ACM.
  37. Query resolution for conversational search with limited supervision. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval (SIGIR), pages 921–930.
  38. Query2doc: Query expansion with large language models. CoRR, abs/2303.07678.
  39. Self-consistency improves chain of thought reasoning in language models. In 11th International Conference on Learning Representations, ICLR 2023.
  40. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  41. Chain of thought prompting elicits reasoning in large language models. Advances in neural information processing systems.
  42. Conqrr: Conversational query rewriting for retrieval with reinforcement learning.
  43. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
  44. Few-shot generative conversational query rewriting. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval (SIGIR), pages 1933–1936.
  45. Few-shot conversational dense retrieval. In Proceedings of the 44th International ACM SIGIR conference on research and development in Information Retrieval (SIGIR).
  46. Generate rather than retrieve: Large language models are strong context generators. In 11th International Conference on Learning Representations, ICLR 2023.
  47. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107.
  48. Large language models are built-in autoregressive search engines. In ACL (Findings), volume ACL 2023 of Findings of ACL. Association for Computational Linguistics.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Kelong Mao (23 papers)
  2. Zhicheng Dou (113 papers)
  3. Fengran Mo (35 papers)
  4. Jiewen Hou (1 paper)
  5. Haonan Chen (49 papers)
  6. Hongjin Qian (23 papers)
Citations (60)
View on Semantic Scholar
Reddit Logo Streamline Icon: https://streamlinehq.com

Reddit

  1. Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search (1 point, 0 comments)