Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Improving Proactive Dialog Agents Using Socially-Aware Reinforcement Learning (2211.15359v2)

Published 25 Nov 2022 in cs.CL, cs.AI, cs.HC, and cs.LG

Abstract: The next step for intelligent dialog agents is to escape their role as silent bystanders and become proactive. Well-defined proactive behavior may improve human-machine cooperation, as the agent takes a more active role during interaction and takes off responsibility from the user. However, proactivity is a double-edged sword because poorly executed pre-emptive actions may have a devastating effect not only on the task outcome but also on the relationship with the user. For designing adequate proactive dialog strategies, we propose a novel approach including both social as well as task-relevant features in the dialog. Here, the primary goal is to optimize proactive behavior so that it is task-oriented - this implies high task success and efficiency - while also being socially effective by fostering user trust. Including both aspects in the reward function for training a proactive dialog agent using reinforcement learning showed the benefit of our approach for more successful human-machine cooperation.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. D. McMillan and R. Jaber, “Leaving the butler behind: The future of role reproduction in cui,” in CUI 2021-3rd Conference on Conversational User Interfaces, 2021, pp. 1–4.
  2. A. P. Chaves and M. A. Gerosa, “How should my chatbot interact? a survey on social characteristics in human–chatbot interaction design,” International Journal of Human–Computer Interaction, vol. 37, no. 8, pp. 729–758, 2021.
  3. R. Sarikaya, “The technology behind personal digital assistants: An overview of the system architecture and key components,” IEEE Signal Processing Magazine, vol. 34, no. 1, pp. 67–81, 2017.
  4. F. Nothdurft, S. Ultes, and W. Minker, “Finding appropriate interaction strategies for proactive dialogue systems-an open quest,” in Proceedings of the 2nd European and the 5th Nordic Symposium on Multimodal Communication, August 6-8, 2014, Tartu, Estonia, no. 110.   Linköping University Electronic Press, 2015, pp. 73–80.
  5. Z. Peng, Y. Kwon, J. Lu, Z. Wu, and X. Ma, “Design and evaluation of service robot’s proactivity in decision-making support process,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems.   ACM, 2019, p. 98.
  6. J. Baraglia, M. Cakmak, Y. Nagai, R. Rao, and M. Asada, “Initiative in robot assistance during collaborative task execution,” in The Eleventh ACM/IEEE International Conference on Human Robot Interaction.   IEEE Press, 2016, pp. 67–74.
  7. E. Horvitz, “Lumiere project: Bayesian reasoning for automated assistance,” Decision Theory & Adaptive Systems Group, Microsoft Research. Microsoft Corp. Redmond, WA, 1998.
  8. T. W. Bickmore and R. W. Picard, “Establishing and maintaining long-term human-computer relationships,” ACM Transactions on Computer-Human Interaction (TOCHI), vol. 12, no. 2, pp. 293–327, 2005.
  9. M. McTear, “Conversational ai: Dialogue systems, conversational agents, and chatbots,” Synthesis Lectures on Human Language Technologies, vol. 13, no. 3, pp. 1–251, 2020.
  10. S. Young, M. Gašić, B. Thomson, and J. D. Williams, “Pomdp-based statistical spoken dialog systems: A review,” Proceedings of the IEEE, vol. 101, no. 5, pp. 1160–1179, 2013.
  11. O. Lemon, “Adaptive natural language generation in dialogue using reinforcement learning,” in LONDIAL 2008 the 12th Workshop on the Semantics and Pragmatics of Dialogue, 2008, p. 149.
  12. S. Ultes, M. Kraus, A. Schmitt, and W. Minker, “Quality-adaptive spoken dialogue initiative selection and implications on reward modelling,” in Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2015, pp. 374–383.
  13. S. Ultes, “Improving interaction quality estimation with bilstms and the impact on dialogue policy learning,” in Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, 2019, pp. 11–20.
  14. M. Kraus, N. Wagner, N. Untereiner, and W. Minker, “Including social expectations for trustworthy proactive human-robot dialogue,” in Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization.   ACM, 2022.
  15. C. Meurisch, C. A. Mihale-Wilson, A. Hawlitschek, F. Giger, F. Müller, O. Hinz, and M. Mühlhäuser, “Exploring user expectations of proactive ai systems,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 4, no. 4, pp. 1–22, 2020.
  16. N. Yorke-Smith, S. Saadati, K. L. Myers, and D. N. Morley, “The design of a proactive personal agent for task management,” International Journal on Artificial Intelligence Tools, vol. 21, no. 01, p. 1250004, 2012.
  17. E. Horvitz, “Principles of mixed-initiative user interfaces,” in Proceedings of the SIGCHI conference on Human Factors in Computing Systems.   ACM, 1999, pp. 159–166.
  18. K. Christakopoulou, F. Radlinski, and K. Hofmann, “Towards conversational recommender systems,” in Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining.   ACM, 2016, pp. 815–824.
  19. W. Wu, Z. Guo, X. Zhou, H. Wu, X. Zhang, R. Lian, and H. Wang, “Proactive human-machine conversation with explicit conversation goals,” arXiv preprint arXiv:1906.05572, 2019.
  20. Y. Zhu, J.-Y. Nie, K. Zhou, P. Du, H. Jiang, and Z. Dou, “Proactive retrieval-based chatbots based on relevant knowledge and goals,” in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021, pp. 2000–2004.
  21. J. Tang, T. Zhao, C. Xiong, X. Liang, E. P. Xing, and Z. Hu, “Target-guided open-domain conversation,” arXiv preprint arXiv:1905.11553, 2019.
  22. J. Xu, H. Wang, Z. Niu, H. Wu, and W. Che, “Knowledge graph grounded goal planning for open-domain conversation generation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 05, 2020, pp. 9338–9345.
  23. C. L. Isbell and J. S. Pierce, “An IP continuum for adaptive interface design,” in Proc. of HCI International, 2005.
  24. T. B. Sheridan and W. L. Verplank, “Human and computer control of undersea teleoperators,” Massachusetts Inst of Tech Cambridge Man-Machine Systems Lab, Tech. Rep., 1978.
  25. A. Faulring, B. Myers, K. Mohnkern, B. Schmerl, A. Steinfeld, J. Zimmerman, A. Smailagic, J. Hansen, and D. Siewiorek, “Agent-assisted task management that reduces email overload,” in Proceedings of the 15th international conference on Intelligent user interfaces, 2010, pp. 61–70.
  26. D. Garlan and B. Schmerl, “The radar architecture for personal cognitive assistance,” International Journal of Software Engineering and Knowledge Engineering, vol. 17, no. 02, pp. 171–190, 2007.
  27. M. Kraus, N. Wagner, and W. Minker, “Effects of proactive dialogue strategies on human-computer trust,” in Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization, ser. UMAP ’20.   New York, NY, USA: Association for Computing Machinery, 2020, p. 107–116. [Online]. Available: https://doi.org/10.1145/3340631.3394840
  28. K. E. Schaefer, J. Y. Chen, J. L. Szalma, and P. A. Hancock, “A meta-analysis of factors influencing the development of trust in automation: Implications for understanding autonomy in future systems,” Human factors, vol. 58, no. 3, pp. 377–400, 2016.
  29. J. D. Lee and K. A. See, “Trust in automation: Designing for appropriate reliance,” Human factors, vol. 46, no. 1, pp. 50–80, 2004.
  30. M. Madsen and S. Gregor, “Measuring human-computer trust,” in 11th australasian conference on information systems, vol. 53.   Citeseer, 2000, pp. 6–8.
  31. B. M. Muir and N. Moray, “Trust in automation. part ii. experimental studies of trust and human intervention in a process control simulation,” Ergonomics, vol. 39, no. 3, pp. 429–460, 1996.
  32. J. D. Lee and N. Moray, “Trust, self-confidence, and operators’ adaptation to automation,” International journal of human-computer studies, vol. 40, no. 1, pp. 153–184, 1994.
  33. F. Pecune and S. Marsella, “A framework to co-optimize task and social dialogue policies using reinforcement learning,” in Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents, 2020, pp. 1–8.
  34. A. Jain, F. Pecune, Y. Matsuyama, and J. Cassell, “A user simulator architecture for socially-aware conversational agents,” in Proceedings of the 18th International Conference on Intelligent Virtual Agents, 2018, pp. 133–140.
  35. M. Kraus, N. Wagner, and W. Minker, “Prodial – an annotated proactive dialogue act corpus for conversational assistants using crowdsourcing,” in Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2022).   ELRA, 2022.
  36. M. Kraus, N. Wagner, Z. Callejas, and W. Minker, “The role of trust in proactive conversational assistants,” IEEE Access, vol. 9, pp. 112 821–112 836, 2021.
  37. R. R. McCrae and O. P. John, “An introduction to the five-factor model and its applications,” Journal of personality, vol. 60, no. 2, pp. 175–215, 1992.
  38. S. Kullback and R. A. Leibler, “On information and sufficiency,” The annals of mathematical statistics, vol. 22, no. 1, pp. 79–86, 1951.
  39. M. Kraus, N. Wagner, and W. Minker, “Modelling and predicting trust for developing proactive dialogue strategies in mixed-initiative interaction,” in Proceedings of the 2021 International Conference on Multimodal Interaction, 2021, pp. 131–140.
  40. N. Rach, W. Minker, and S. Ultes, “Interaction quality estimation using long short-term memories,” in Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, 2017, pp. 164–169.
  41. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015.
  42. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv e-prints, pp. arXiv–1412, 2014.
  43. P.-L. P. Rau, Y. Li, and J. Liu, “Effects of a social robot’s autonomy and group orientation on human decision-making,” Advances in Human-Computer Interaction, vol. 2013, p. 11, 2013.
  44. B. M. Muir, “Trust in automation: Part i. theoretical issues in the study of trust and human intervention in automated systems,” Ergonomics, vol. 37, no. 11, pp. 1905–1922, 1994.
  45. H. Ritschel, T. Baur, and E. André, “Adapting a robot’s linguistic style based on socially-aware reinforcement learning,” in 2017 26th ieee international symposium on robot and human interactive communication (ro-man).   IEEE, 2017, pp. 378–384.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Matthias Kraus (9 papers)
  2. Nicolas Wagner (10 papers)
  3. Ron Riekenbrauck (2 papers)
  4. Wolfgang Minker (18 papers)
Citations (6)
View on Semantic Scholar