Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting (2310.09716v2)

Published 15 Oct 2023 in cs.HC, cs.AI, cs.CL, and cs.IR

Abstract: Query rewriting plays a vital role in enhancing conversational search by transforming context-dependent user queries into standalone forms. Existing approaches primarily leverage human-rewritten queries as labels to train query rewriting models. However, human rewrites may lack sufficient information for optimal retrieval performance. To overcome this limitation, we propose utilizing LLMs as query rewriters, enabling the generation of informative query rewrites through well-designed instructions. We define four essential properties for well-formed rewrites and incorporate all of them into the instruction. In addition, we introduce the role of rewrite editors for LLMs when initial query rewrites are available, forming a "rewrite-then-edit" process. Furthermore, we propose distilling the rewriting capabilities of LLMs into smaller models to reduce rewriting latency. Our experimental evaluation on the QReCC dataset demonstrates that informative query rewrites can yield substantially improved retrieval performance compared to human rewrites, especially with sparse retrievers.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. Open-domain question answering goes conversational via question rewriting. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 520–534, Online. Association for Computational Linguistics.
  2. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.
  3. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  4. Reinforced question rewriting for conversational question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 357–370, Abu Dhabi, UAE. Association for Computational Linguistics.
  5. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184, Brussels, Belgium. Association for Computational Linguistics.
  6. Cast 2020: The conversational assistance track overview.
  7. Cast-19: A dataset for conversational information seeking. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1985–1988.
  8. A survey for in-context learning. arXiv preprint arXiv:2301.00234.
  9. Can you unpack that? learning to rewrite questions-in-context. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5918–5924, Hong Kong, China. Association for Computational Linguistics.
  10. Neural Approaches to Conversational Information Retrieval, volume 44. Springer Nature.
  11. Knowledge distillation: A survey. International Journal of Computer Vision, 129:1789–1819.
  12. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738.
  13. CGF: Constrained generation framework for query rewriting in conversational AI. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 475–483, Abu Dhabi, UAE. Association for Computational Linguistics.
  14. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301.
  15. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  16. Can large language models truly understand prompts? a case study with negated prompts. In Transfer Learning for Natural Language Processing Workshop, pages 52–62. PMLR.
  17. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535–547.
  18. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics.
  19. Kimiya Keyvan and Jimmy Xiangji Huang. 2022. How to approach ambiguous queries in conversational search: A survey of techniques, approaches, tools, and challenges. ACM Computing Surveys, 55(6):1–40.
  20. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39–48.
  21. Sungdong Kim and Gangwoo Kim. 2022. Saving dense retriever from shortcut dependency in conversational search. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10278–10287, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  22. Vaibhav Kumar and Jamie Callan. 2020. Making information seeking easier: An improved pipeline for conversational search. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3971–3980, Online. Association for Computational Linguistics.
  23. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466.
  24. Dynamic graph reasoning for conversational open-domain question answering. ACM Transactions on Information Systems (TOIS), 40(4):1–24.
  25. MMCoQA: Conversational question answering over text, tables, and images. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4220–4231, Dublin, Ireland. Association for Computational Linguistics.
  26. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  27. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2356–2362.
  28. Contextualized query embeddings for conversational search. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1004–1015, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  29. Conversational question reformulation via sequence-to-sequence architectures and pretrained language models. arXiv preprint arXiv:2004.01909.
  30. Multi-stage conversational passage retrieval: An approach to fusing term importance estimation and neural query rewriting. ACM Transactions on Information Systems (TOIS), 39(4):1–29.
  31. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965.
  32. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
  33. Teaching small language models to reason. arXiv preprint arXiv:2212.08410.
  34. Curriculum contrastive context denoising for few-shot conversational dense retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 176–186.
  35. What language reveals about perception: Distilling psychophysical knowledge from large language models. arXiv preprint arXiv:2302.01308.
  36. MetaICL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2791–2809, Seattle, United States. Association for Computational Linguistics.
  37. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  38. Convgqr: Generative query reformulation for conversational search. arXiv preprint arXiv:2305.15645.
  39. Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation. arXiv preprint arXiv:2305.16938.
  40. Large dual encoders are generalizable retrievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9844–9855, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  41. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  42. Exploiting simulated user feedback for conversational search: Ranking, rewriting, and beyond. arXiv preprint arXiv:2304.13874.
  43. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  44. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277.
  45. Hongjin Qian and Zhicheng Dou. 2022. Explicit query rewriting for conversational dense retrieval. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4725–4737, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  46. Filip Radlinski and Nick Craswell. 2017. A theoretical framework for conversational search. In Proceedings of the 2017 conference on conference human information interaction and retrieval, pages 117–126.
  47. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551.
  48. In-context retrieval-augmented language models. arXiv preprint arXiv:2302.00083.
  49. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389.
  50. Leading conversational search by suggesting useful questions. In Proceedings of the web conference 2020, pages 1160–1170.
  51. Distilling multi-step reasoning capabilities of large language models into smaller models via semantic decompositions. arXiv preprint arXiv:2212.00193.
  52. How does in-context learning help prompt tuning? arXiv preprint arXiv:2302.11521.
  53. Question rewriting for conversational question answering. In Proceedings of the 14th ACM international conference on web search and data mining, pages 355–363.
  54. Christophe Van Gysel and Maarten de Rijke. 2018. Pytrec_eval: An extremely fast python interface to trec_eval. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 873–876.
  55. Query resolution for conversational search with limited supervision. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 921–930.
  56. An in-depth investigation of user response simulation for conversational search. arXiv preprint arXiv:2304.07944.
  57. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.
  58. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846.
  59. CONQRR: Conversational query rewriting for retrieval with reinforcement learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10000–10014, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  60. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808.
  61. Asking clarification questions in knowledge-based question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1618–1629, Hong Kong, China. Association for Computational Linguistics.
  62. ASSIST: Towards label noise-robust dialogue state tracking. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2719–2731, Dublin, Ireland. Association for Computational Linguistics.
  63. MetaASSIST: Robust dialogue state tracking with meta learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1157–1169, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  64. Few-shot generative conversational query rewriting. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 1933–1936.
  65. Few-shot conversational dense retrieval. In Proceedings of the 44th International ACM SIGIR Conference on research and development in information retrieval, pages 829–838.
  66. McQueen: a benchmark for multimodal conversational query rewrite. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4834–4844, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  67. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Fanghua Ye (30 papers)
  2. Meng Fang (100 papers)
  3. Shenghui Li (8 papers)
  4. Emine Yilmaz (66 papers)
Citations (33)
View on Semantic Scholar