Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

BASES: Large-scale Web Search User Simulation with Large Language Model based Agents (2402.17505v1)

Published 27 Feb 2024 in cs.IR and cs.CL

Abstract: Due to the excellent capacities of LLMs, it becomes feasible to develop LLM-based agents for reliable user simulation. Considering the scarcity and limit (e.g., privacy issues) of real user data, in this paper, we conduct large-scale user simulation for web search, to improve the analysis and modeling of user search behavior. Specially, we propose BASES, a novel user simulation framework with LLM-based agents, designed to facilitate comprehensive simulations of web search user behaviors. Our simulation framework can generate unique user profiles at scale, which subsequently leads to diverse search behaviors. To demonstrate the effectiveness of BASES, we conduct evaluation experiments based on two human benchmarks in both Chinese and English, demonstrating that BASES can effectively simulate large-scale human-like search behaviors. To further accommodate the research on web search, we develop WARRIORS, a new large-scale dataset encompassing web search user behaviors, including both Chinese and English versions, which can greatly bolster research in the field of information retrieval. Our code and data will be publicly released soon.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Out of one, many: Using language models to simulate human samples. CoRR, abs/2209.06899.
  2. Modeling the impact of short- and long-term behavior on search personalization. In The 35th International ACM SIGIR conference on research and development in Information Retrieval, SIGIR ’12, Portland, OR, USA, August 12-16, 2012, pages 185–194. ACM.
  3. Designing inclusive interfaces through user modeling and simulation. Int. J. Hum. Comput. Interact., 28(1):1–33.
  4. A click sequence model for web search. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pages 45–54. ACM.
  5. The Adaptive Web, Methods and Strategies of Web Personalization, volume 4321 of Lecture Notes in Computer Science. Springer.
  6. Overview of the TREC 2014 session track. In Proceedings of The Twenty-Third Text REtrieval Conference, TREC 2014, Gaithersburg, Maryland, USA, November 19-21, 2014, volume 500-308 of NIST Special Publication. National Institute of Standards and Technology (NIST).
  7. Chateval: Towards better llm-based evaluators through multi-agent debate. CoRR, abs/2308.07201.
  8. A dynamic bayesian network click model for web search ranking. In Proceedings of the 18th International Conference on World Wide Web, WWW 2009, Madrid, Spain, April 20-24, 2009, pages 1–10. ACM.
  9. Incorporating query reformulating behavior into web search evaluation. In CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021, pages 171–180. ACM.
  10. Tiangong-st: A new dataset with large-scale refined real-world web search sessions. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3-7, 2019, pages 2485–2488. ACM.
  11. A context-aware click model for web search. In WSDM ’20: The Thirteenth ACM International Conference on Web Search and Data Mining, Houston, TX, USA, February 3-7, 2020, pages 88–96. ACM.
  12. Click Models for Web Search. Synthesis Lectures on Information Concepts, Retrieval, and Services. Morgan & Claypool Publishers.
  13. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics.
  14. Georges Dupret and Benjamin Piwowarski. 2008. A user browsing model to predict search engine click data from past observations. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2008, Singapore, July 20-24, 2008, pages 331–338. ACM.
  15. User simulation for spoken dialogue systems: learning and evaluation. In INTERSPEECH 2006 - ICSLP, Ninth International Conference on Spoken Language Processing, Pittsburgh, PA, USA, September 17-21, 2006. ISCA.
  16. Christoph Hölscher and Gerhard Strube. 2000. Web search behavior of internet experts and newbies. Comput. Networks, 33(1-6):337–346.
  17. Defining a session on web search engines. J. Assoc. Inf. Sci. Technol., 58(6):862–871.
  18. Searching, browsing, and clicking in a search session: changes in user behavior by task and over time. In The 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’14, Gold Coast , QLD, Australia - July 06 - 11, 2014, pages 607–616. ACM.
  19. Contextual evaluation of query reformulations in a search session by user simulation. In 21st ACM International Conference on Information and Knowledge Management, CIKM’12, Maui, HI, USA, October 29 - November 02, 2012, pages 2635–2638. ACM.
  20. Surrealdriver: Designing generative driver agent simulation framework in urban contexts based on large language model. CoRR, abs/2309.13193.
  21. Rakesh Kochhar. 2021. Are you in the global middle class? find out with our income calculator. Https://www.pewresearch.org/short-reads/2021/07/21/are-you-in-the-global-middle-class-find-out-with-our-income-calculator/.
  22. Chunyan Liang. 2011. User profile for personalized web search. In Eighth International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2011, 26-28 July 2011, Shanghai, China, pages 1847–1850. IEEE.
  23. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 2356–2362. ACM.
  24. Robert R McCrae and Oliver P John. 1992. An introduction to the five-factor model and its applications. Journal of personality, 60(2):175–215.
  25. Jizhe Ning. 2021. Main data of the seventh national population census. National Bureau of Statistics of China.
  26. OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
  27. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST 2023, San Francisco, CA, USA, 29 October 2023- 1 November 2023, pages 2:1–2:22. ACM.
  28. A picture of search. In Proceedings of the 1st International Conference on Scalable Information Systems, Infoscale 2006, Hong Kong, May 30-June 1, 2006, volume 152 of ACM International Conference Proceeding Series, page 1. ACM.
  29. Sheldon M. Ross. 1997. Simulation (2. ed.). Statistical modeling and decision science. Academic Press.
  30. Jost Schatzmann and Steve J. Young. 2009. The hidden agenda user simulation model. IEEE Trans. Speech Audio Process., 17(4):733–747.
  31. Adaptive web search based on user profile constructed without any effort from users. In Proceedings of the 13th international conference on World Wide Web, WWW 2004, New York, NY, USA, May 17-20, 2004, pages 675–684. ACM.
  32. the World Bank. 2023. Education statistics (edstats). Https://gem-report-2023.unesco.org/.
  33. UNESCO. 2023. 2023 global education monitoring report. Https://gem-report-2023.unesco.org/.
  34. Peter-Paul Verbeek and Adriaan Slob. 2006. User behavior and technology development. Springer.
  35. A survey on large language model based autonomous agents. CoRR, abs/2308.11432.
  36. When large language model based agent meets user behavior analysis: A novel user simulation paradigm. CoRR, abs/2306.02552.
  37. The rise and potential of large language model based agents: A survey. CoRR, abs/2309.07864.
  38. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
  39. Cascade or recency: Constructing better evaluation metrics for session search. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 389–398. ACM.
  40. Agentcf: Collaborative learning with autonomous language agents for recommender systems. CoRR, abs/2310.09233.
  41. Dense text retrieval based on pretrained language models: A survey. ACM Transactions on Information Systems, 42(4):1–60.
  42. A survey of large language models. CoRR, abs/2303.18223.
  43. Group based personalized search by integrating search behaviour and friend network. In SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 92–101. ACM.
  44. Contrastive learning of user behavior sequence for context-aware document ranking. In CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021, pages 2780–2791. ACM.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now