Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
72 tokens/sec
GPT-4o
61 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration (2305.13626v2)

Published 23 May 2023 in cs.CL

Abstract: Conversational systems based on LLMs, such as ChatGPT, show exceptional proficiency in context understanding and response generation. However, despite their impressive capabilities, they still possess limitations, such as providing randomly-guessed answers to ambiguous queries or failing to refuse users' requests, both of which are considered aspects of a conversational agent's proactivity. This raises the question of whether LLM-based conversational systems are equipped to handle proactive dialogue problems. In this work, we conduct a comprehensive analysis of LLM-based conversational systems, specifically focusing on three aspects of proactive dialogue systems: clarification, target-guided, and non-collaborative dialogues. To trigger the proactivity of LLMs, we propose the Proactive Chain-of-Thought prompting scheme, which augments LLMs with the goal planning capability over descriptive reasoning chains. Empirical findings are discussed to promote future studies on LLM-based proactive dialogue systems.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Towards a human-like open-domain chatbot. CoRR, abs/2001.09977.
  2. Building and evaluating open-domain dialogue corpora with clarifying questions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, pages 4473–4484.
  3. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. CoRR, abs/2302.04023.
  4. Towards robust personalized dialogue generation via order-insensitive representation regularization. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7337–7345.
  5. PLACES: prompting language models for social conversation synthesis. In Findings of the Association for Computational Linguistics: EACL 2023, pages 814–838.
  6. Controllable mixed-initiative dialogue generation through prompting. CoRR, abs/2305.04147.
  7. Unidu: Towards A unified generative dialogue understanding framework. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL 2022, pages 442–455.
  8. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  9. Goal awareness for conversational AI: proactivity, non-collaborativity, and beyond. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, ACL 2023, pages 1–10.
  10. A survey on proactive dialogue systems: Problems, methods, and prospects. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, pages 6583–6591.
  11. PACIFIC: towards proactive conversational question answering over tabular and textual data in finance. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, pages 6970–6984.
  12. User satisfaction estimation with sequential dialogue act modeling in goal-oriented conversational systems. In WWW ’22: The ACM Web Conference 2022, pages 2998–3008.
  13. A unified multi-task learning framework for multi-goal conversational recommender systems. ACM Trans. Inf. Syst., 41(3):77:1–77:25.
  14. The second conversational intelligence challenge (convai2). CoRR, abs/1902.00098.
  15. Resper: Computationally modelling resisting strategies in persuasive conversations. In EACL.
  16. Improving language model negotiation with self-play and in-context learning from AI feedback. CoRR, abs/2305.10142.
  17. Abg-coqa: Clarifying ambiguity in conversational question answering. In 3rd Conference on Automated Knowledge Base Construction, AKBC 2021.
  18. Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2333–2343.
  19. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, ICML, volume 162, pages 9118–9147.
  20. Dialograph: Incorporating interpretable strategy-graph networks into negotiation dialogues. In ICLR.
  21. Dialogue state tracking with a language model using schema-driven prompting. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, pages 4937–4949.
  22. Prompted llms as chatbot modules for long open-domain conversation. CoRR, abs/2305.04533.
  23. Interacting with non-cooperative user: A new paradigm for proactive dialogue policy. In SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 212–222.
  24. End-to-end trainable non-collaborative dialog system. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, pages 8293–8302.
  25. Proactive conversational agents. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM 2023, pages 1244–1247. ACM.
  26. Multi-stage prompting for knowledgeable dialogue generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1317–1337.
  27. CINS: comprehensive instruction for few-shot learning in task-oriented dialog systems. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, pages 11076–11084.
  28. Dynamic knowledge routing network for target-guided open-domain conversation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, pages 8657–8664.
  29. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  30. Coqa: A conversational question answering challenge. Trans. Assoc. Comput. Linguistics, 7:249–266.
  31. Recipes for building an open-domain chatbot. In EACL.
  32. Teven Le Scao and Alexander M. Rush. 2021. How many data points is a prompt worth? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, pages 2627–2636.
  33. Otters: One-turn topic transitions for open-domain dialogue. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, pages 2492–2504.
  34. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 373–393.
  35. Target-guided open-domain conversation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pages 5624–5634.
  36. Stanford alpaca: An instruction-following llama model. GitHub repository.
  37. Lamda: Language models for dialog applications. CoRR, abs/2201.08239.
  38. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971.
  39. Chain-of-thought prompting for responding to in-depth dialogue questions with LLM. CoRR, abs/2305.11792.
  40. Dialogue planning via brownian bridge stochastic process for goal-directed proactive dialogue. CoRR, abs/2305.05290.
  41. Retrieval-free knowledge injection through multi-document traversal for dialogue models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, pages 6608–6619.
  42. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020.
  43. Persuasion for good: Towards a personalized persuasive dialogue system for social good. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pages 5635–5649.
  44. Emergent abilities of large language models. CoRR, abs/2206.07682.
  45. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903.
  46. Zero-shot information extraction via chatting with chatgpt. CoRR, abs/2302.10205.
  47. TOD-BERT: pre-trained natural language understanding for task-oriented dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, pages 917–929.
  48. Proactive human-machine conversation with explicit conversation goal. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pages 3794–3804.
  49. Topkg: Target-oriented dialog via global planning on knowledge graph. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, pages 745–755.
  50. React: Synergizing reasoning and acting in language models. CoRR, abs/2210.03629.
  51. Conversational information seeking. CoRR.
  52. Let’s negotiate! A survey of negotiation dialogue systems. CoRR.
  53. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020.
  54. Sgp-tod: Building task bots effortlessly via schema-guided llm prompting.
  55. DIALOGPT : Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL 2020, pages 270–278.
  56. Recent advances and challenges in task-oriented dialog systems. Science China Technological Sciences, 63(10):2011–2027.
  57. Is chatgpt equipped with emotional dialogue capabilities? CoRR, abs/2304.09582.
  58. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, volume 139, pages 12697–12706.
  59. Keyword-guided neural conversational model. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, pages 14568–14576.
  60. Augmenting non-collaborative dialog systems with explicit semantic and strategic dialog history. In 8th International Conference on Learning Representations, ICLR 2020.
  61. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, pages 3277–3287.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Yang Deng (113 papers)
  2. Lizi Liao (44 papers)
  3. Liang Chen (360 papers)
  4. Hongru Wang (62 papers)
  5. Wenqiang Lei (66 papers)
  6. Tat-Seng Chua (359 papers)
Citations (51)
View on Semantic Scholar