Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Query Understanding in the Age of Large Language Models (2306.16004v1)

Published 28 Jun 2023 in cs.IR and cs.AI

Abstract: Querying, conversing, and controlling search and information-seeking interfaces using natural language are fast becoming ubiquitous with the rise and adoption of large-LLMs (LLM). In this position paper, we describe a generic framework for interactive query-rewriting using LLMs. Our proposal aims to unfold new opportunities for improved and transparent intent understanding while building high-performance retrieval systems using LLMs. A key aspect of our framework is the ability of the rewriter to fully specify the machine intent by the search engine in natural language that can be further refined, controlled, and edited before the final retrieval phase. The ability to present, interact, and reason over the underlying machine intent in natural language has profound implications on transparency, ranking performance, and a departure from the traditional way in which supervised signals were collected for understanding intents. We detail the concept, backed by initial experiments, along with open questions for this interactive query understanding framework.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (91)
  1. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198 (2022).
  2. Efficient temporal keyword search over versioned text. In Proceedings of the 19th ACM international conference on Information and knowledge management. ACM, 699–708.
  3. Temporal index sharding for space-time efficiency in archive search. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval. 545–554.
  4. Explainable Information Retrieval: A Survey. arXiv preprint arXiv:2211.02405 (2022).
  5. Matches Made in Heaven: Toolkit and Large-Scale Datasets for Supervised Query Reformulation. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 4417–4425.
  6. Query Recommendation Using Query Logs in Search Engines.. In EDBT workshops, Vol. 3268. Springer, 588–596.
  7. Incorporating Relevance Feedback for Information-Seeking Retrieval using Few-Shot Document Re-Ranking. (2022). https://doi.org/10.48550/ARXIV.2210.10695
  8. Asking clarifying questions based on negative feedback in conversational search. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval. 157–166.
  9. InPars: Data Augmentation for Information Retrieval using Large Language Models. arXiv preprint arXiv:2202.05144 (2022).
  10. Improving language models by retrieving from trillions of tokens. (2021). https://doi.org/10.48550/ARXIV.2112.04426
  11. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  12. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.
  13. Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval. 335–336.
  14. Yi Chang and Hongbo Deng. 2020. Query understanding for search engines. Springer.
  15. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022).
  16. Deep reinforcement learning from human preferences. (2017). https://doi.org/10.48550/ARXIV.1706.03741
  17. Overview of the TREC 2009 Web Track. ([n. d.]).
  18. Promptagator: Few-shot dense retrieval from 8 examples. arXiv preprint arXiv:2209.11755 (2022).
  19. Can you unpack that? learning to rewrite questions-in-context. Can You Unpack That? Learning to Rewrite Questions-in-Context (2019).
  20. A study on the Interpretability of Neural Retrieval Models using DeepSHAP. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, Benjamin Piwowarski, Max Chevalier, Éric Gaussier, Yoelle Maarek, Jian-Yun Nie, and Falk Scholer (Eds.). ACM, 1005–1008. https://doi.org/10.1145/3331184.3331312
  21. Elias Frantar and Dan Alistarh. 2023. SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. (2023). https://doi.org/10.48550/ARXIV.2301.00774
  22. Precise Zero-Shot Dense Retrieval without Relevance Labels. (2022). https://doi.org/10.48550/ARXIV.2212.10496
  23. Medical information retrieval: introduction to the special issue. Information Retrieval Journal 19 (2016), 1–5.
  24. Knowledge Distillation: A Survey. International Journal of Computer Vision 129, 6 (mar 2021), 1789–1819. https://doi.org/10.1007/s11263-021-01453-z
  25. Context-and content-aware embeddings for query rewriting in sponsored search. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. 383–392.
  26. CGF: Constrained generation framework for query rewriting in conversational AI. In EMNLP 2022.
  27. Rethinking with Retrieval: Faithful Large Language Model Inference. arXiv preprint arXiv:2301.00303 (2022).
  28. Learning to rewrite queries. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. 1443–1452.
  29. The Curious Case of Neural Text Degeneration. (2019). https://doi.org/10.48550/ARXIV.1904.09751
  30. Unsupervised Dense Information Retrieval with Contrastive Learning. (2021). https://doi.org/10.48550/ARXIV.2112.09118
  31. Unbiased Learning-to-Rank with Biased Feedback. In WSDM. 781–789.
  32. Generating query substitutions. In Proceedings of the 15th international conference on World Wide Web. 387–396.
  33. Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP. arXiv preprint arXiv:2212.14024 (2022).
  34. Rare Query Expansion Through Generative Adversarial Networks in Search Advertising. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’18). Association for Computing Machinery, New York, NY, USA, 500–508. https://doi.org/10.1145/3219819.3219850
  35. Shiye Lei and Dacheng Tao. 2023. A Comprehensive Survey of Dataset Distillation. (2023). https://doi.org/10.48550/ARXIV.2301.05603
  36. Rationalizing Neural Predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, 107–117. https://doi.org/10.18653/v1/D16-1011
  37. Learnt Sparsity for Effective and Interpretable Document Ranking. ArXiv preprint abs/2106.12460 (2021). https://arxiv.org/abs/2106.12460
  38. Extractive Explanations for Interpretable Text Ranking. ACM Trans. Inf. Syst. (dec 2022). https://doi.org/10.1145/3576924
  39. Efficient Neural Ranking using Forward Indexes. In Proceedings of the ACM Web Conference 2022. 266–276.
  40. Fast Inference from Transformers via Speculative Decoding. (2022). https://doi.org/10.48550/ARXIV.2211.17192
  41. Query Rewriting in TaoBao Search. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (CIKM ’22). Association for Computing Machinery, New York, NY, USA, 3262–3271. https://doi.org/10.1145/3511808.3557068
  42. What Makes Good In-Context Examples for GPT-3?. In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures. Association for Computational Linguistics, Dublin, Ireland and Online, 100–114. https://doi.org/10.18653/v1/2022.deelio-1.10
  43. MedSearch: a specialized search engine for medical information retrieval. In Proceedings of the 17th ACM conference on Information and knowledge management. 143–152.
  44. Yuanhua Lv and ChengXiang Zhai. 2009. A comparative study of methods for estimating query language models with pseudo feedback. In Proceedings of the 18th ACM conference on Information and knowledge management. 1895–1898.
  45. Generative Relevance Feedback with Large Language Models. arXiv preprint arXiv:2304.13157 (2023).
  46. Entity linking and retrieval for semantic search. WSDM 10 (2014), 2556195–2556201.
  47. Rethinking search: making domain experts out of dilettantes. In ACM SIGIR Forum, Vol. 55. ACM New York, NY, USA, 1–27.
  48. WebGPT: Browser-assisted question-answering with human feedback. (2021). https://doi.org/10.48550/ARXIV.2112.09332
  49. Rodrigo Nogueira and Kyunghyun Cho. 2017. Task-oriented query reformulation with reinforcement learning. arXiv preprint arXiv:1704.04572 (2017).
  50. OpenAI. 2023. GPT-4 Technical Report. (2023). arXiv:cs.CL/2303.08774
  51. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27730–27744.
  52. Correcting for Selection Bias in Learning-to-Rank Systems. In Proceedings of The Web Conference 2020. 1863–1873.
  53. Improving Content Retrievability in Search with Controllable Query Generation. In Proceedings of the ACM Web Conference 2023 (WWW ’23). Association for Computing Machinery, New York, NY, USA, 3182–3192. https://doi.org/10.1145/3543507.3583261
  54. ExDocS: Evidence based Explainable Document Search. In ACM SIGIR Workshop on Causality in Search and Recommendation. ACM. https://csr21.github.io/polley-csr2021.pdf
  55. On Natural Language User Profiles for Transparent and Scrutable Recommendation. arXiv preprint arXiv:2205.09403 (2022).
  56. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02361 (2019).
  57. Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. arXiv preprint arXiv:1805.04655 (2018).
  58. Listwise Explanations for Ranking Models using Multiple Explainers. In Advances in Information Retrieval - 45th European Conference on IR Research, ECIR 2023, Dublin, Ireland, Proceedings, Part I (Lecture Notes in Computer Science). Springer.
  59. A Comprehensive Survey on Model Quantization for Deep Neural Networks. (2022). https://doi.org/10.48550/ARXIV.2205.07877
  60. Leading conversational search by suggesting useful questions. In Proceedings of the web conference 2020. 1160–1170.
  61. Rishiraj Saha Roy and Avishek Anand. 2020. Question Answering over Curated and Open Web Sources. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2432–2435.
  62. Exploiting Query Reformulations for Web Search Result Diversification. In Proceedings of the 19th International Conference on World Wide Web (WWW ’10). Association for Computing Machinery, New York, NY, USA, 881–890. https://doi.org/10.1145/1772690.1772780
  63. Proximal Policy Optimization Algorithms. (2017). https://doi.org/10.48550/ARXIV.1707.06347
  64. Prompting GPT-3 To Be Reliable. (2022). https://doi.org/10.48550/ARXIV.2210.09150
  65. Jaspreet Singh and Avishek Anand. 2019. EXS: Explainable Search Using Local Model Agnostic Interpretability. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM 2019, Melbourne, VIC, Australia, February 11-15, 2019, J. Shane Culpepper, Alistair Moffat, Paul N. Bennett, and Kristina Lerman (Eds.). ACM, 770–773. https://doi.org/10.1145/3289600.3290620
  66. Jaspreet Singh and Avishek Anand. 2020. Model agnostic interpretability of rankers via intent modelling. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 618–628.
  67. Discovering entities with just a little help from you. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. 1331–1340.
  68. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In proceedings of the 24th ACM international on conference on information and knowledge management. 553–562.
  69. LLaMA: Open and Efficient Foundation Language Models. (2023). arXiv:cs.CL/2302.13971
  70. Jan Trienes and Krisztian Balog. 2019. Identifying unclear questions in community question answering websites. In Advances in Information Retrieval: 41st European Conference on IR Research, ECIR 2019, Cologne, Germany, April 14–18, 2019, Proceedings, Part I 41. Springer, 276–289.
  71. Question rewriting for conversational question answering. In Proceedings of the 14th ACM international conference on web search and data mining. 355–363.
  72. Manisha Verma and Debasis Ganguly. 2019. LIRME: Locally Interpretable Ranking Model Explanation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, Benjamin Piwowarski, Max Chevalier, Éric Gaussier, Yoelle Maarek, Jian-Yun Nie, and Falk Scholer (Eds.). ACM, 1281–1284. https://doi.org/10.1145/3331184.3331377
  73. Query2doc: Query Expansion with Large Language Models. (2023). arXiv:cs.IR/2303.07678
  74. Learning to Rank with Selection Bias in Personal Search. In SIGIR. 115–124.
  75. Deep reinforced query reformulation for information retrieval. arXiv preprint arXiv:2007.07987 (2020).
  76. Zero-shot Clarifying Question Generation for Conversational Search. arXiv preprint arXiv:2301.12660 (2023).
  77. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021).
  78. CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning. arXiv preprint arXiv:2112.08558 (2021).
  79. An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA. (2021). https://doi.org/10.48550/ARXIV.2109.05014
  80. INVASE: Instance-wise Variable Selection using Neural Networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. https://openreview.net/forum?id=BJg_roAcK7
  81. Few-shot generative conversational query rewriting. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. 1933–1936.
  82. Using “annotator rationales” to improve machine learning for text categorization. In Human language technologies 2007: The conference of the North American chapter of the association for computational linguistics; proceedings of the main conference. 260–267.
  83. Retrieval-enhanced machine learning. arXiv preprint arXiv:2205.01230 (2022).
  84. Generating clarifying questions for information retrieval. In Proceedings of the web conference 2020. 418–428.
  85. Conversational information seeking. arXiv preprint arXiv:2201.08808 (2022).
  86. BERTScore: Evaluating Text Generation with BERT. (2019). https://doi.org/10.48550/ARXIV.1904.09675
  87. Active Example Selection for In-Context Learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 9134–9148. https://aclanthology.org/2022.emnlp-main.622
  88. Explain and Predict, and then Predict Again. In WSDM ’21, The Fourteenth ACM International Conference on Web Search and Data Mining, Virtual Event, Israel, March 8-12, 2021. ACM, 418–426. https://doi.org/10.1145/3437963.3441758
  89. Disentangling User Interest and Conformity for Recommendation with Causal Embedding. In Proceedings of the Web Conference 2021. 2980–2991.
  90. CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code. (2023). https://doi.org/10.48550/ARXIV.2302.05527
  91. Leveraging Summary Guidance on Medical Report Summarization. (2023). https://doi.org/10.48550/ARXIV.2302.04001
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now