Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

One Agent Too Many: User Perspectives on Approaches to Multi-agent Conversational AI (2401.07123v1)

Published 13 Jan 2024 in cs.HC and cs.CL

Abstract: Conversational agents have been gaining increasing popularity in recent years. Influenced by the widespread adoption of task-oriented agents such as Apple Siri and Amazon Alexa, these agents are being deployed into various applications to enhance user experience. Although these agents promote "ask me anything" functionality, they are typically built to focus on a single or finite set of expertise. Given that complex tasks often require more than one expertise, this results in the users needing to learn and adopt multiple agents. One approach to alleviate this is to abstract the orchestration of agents in the background. However, this removes the option of choice and flexibility, potentially harming the ability to complete tasks. In this paper, we explore these different interaction experiences (one agent for all) vs (user choice of agents) for conversational AI. We design prototypes for each, systematically evaluating their ability to facilitate task completion. Through a series of conducted user studies, we show that users have a significant preference for abstracting agent orchestration in both system usability and system performance. Additionally, we demonstrate that this mode of interaction is able to provide quality responses that are rated within 1% of human-selected answers.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Market and Markets, “Conversational ai market,” Jun 2020. [Online]. Available: https://www.marketsandmarkets.com/Market-Reports/conversational-ai-market-49043506.html
  2. S.-C. Lin, C.-H. Hsu, W. Talamonti, Y. Zhang, S. Oney, J. Mars, and L. Tang, “Adasa: A conversational in-vehicle digital assistant for advanced driver assistance features,” in Proceedings of the 31st ACM Symposium on User Interface Software and Technology (UIST 2018), ser. UIST-31.   Berlin, Germany: ACM, 2018.
  3. M. Luria, G. Hoffman, and O. Zuckerman, “Comparing social robot, screen and voice interfaces for smart-home control,” in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, ser. CHI ’17.   New York, NY, USA: ACM, 2017, pp. 580–628. [Online]. Available: http://doi.acm.org/10.1145/3025453.3025786
  4. A. Frangoul, “Here’s how robots are transforming takeout deliveries,” https://www.cnbc.com/2018/08/02/virtual-assistants-and-robotic-deliveries-are-transforming-takeout.html, August 2018.
  5. A. P. Chaves and M. A. Gerosa, “Single or multiple conversational agents?: An interactional coherence comparison,” in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ser. CHI ’18.   New York, NY, USA: ACM, 2018, pp. 191:1–191:13. [Online]. Available: http://doi.acm.org/10.1145/3173574.3173765
  6. G. Nealon, “Using facebook messenger and chatbots to grow your audience,” Jun 2018. [Online]. Available: https://www.forbes.com/sites/forbesagencycouncil/2018/06/04/using-facebook-messenger-and-chatbots-to-grow-your-audience/
  7. S. Roller, E. Dinan, N. Goyal, D. Ju, M. Williamson, Y. Liu, J. Xu, M. Ott, K. Shuster, E. M. Smith et al., “Recipes for building an open-domain chatbot,” arXiv preprint arXiv:2004.13637, 2020.
  8. C. Clarke, J. Peper, K. Krishnamurthy, W. Talamonti, K. Leach, W. Lasecki, Y. Kang, L. Tang, and J. Mars, “One agent to rule them all: Towards multi-agent conversational AI,” in Findings of the Association for Computational Linguistics: ACL 2022.   Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 3258–3267. [Online]. Available: https://aclanthology.org/2022.findings-acl.257
  9. A. Inc., “Amazon and leading technology companies announce the voice interoperability initiative,” 2019. [Online]. Available: https://press.aboutamazon.com/news-releases/news-release-details/amazon-and-leading-technology-companies-announce-voice
  10. M. Dubiel, M. Halvey, L. Azzopardi, and S. Daronnat, “Interactive evaluation of conversational agents: Reflections on the impact of search task design,” in Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval, ser. ICTIR ’20.   New York, NY, USA: Association for Computing Machinery, 2020, p. 85–88. [Online]. Available: https://doi.org/10.1145/3409256.3409814
  11. D. Novick, L. J. Hinojos, A. E. Rodriguez, A. Camacho, and M. Afravi, “Conversational interaction with multiple agents initiated via proxemics and gaze,” in Proceedings of the 6th International Conference on Human-Agent Interaction, ser. HAI ’18.   New York, NY, USA: Association for Computing Machinery, 2018, p. 356–358. [Online]. Available: https://doi.org/10.1145/3284432.3287185
  12. T. L. Saltsman, M. D. Seery, C. L. Kondrak, V. M. Lamarche, and L. Streamer, “Too many fish in the sea: A motivational examination of the choice overload experience,” Biological Psychology, vol. 145, pp. 17–30, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0301051118306021
  13. K. Sugisaki and A. Bleiker, “Usability guidelines and evaluation criteria for conversational user interfaces: A heuristic and linguistic approach,” in Proceedings of Mensch Und Computer 2020, ser. MuC ’20.   New York, NY, USA: Association for Computing Machinery, 2020, p. 309–319. [Online]. Available: https://doi.org/10.1145/3404983.3405505
  14. J. Weizenbaum, “Eliza—a computer program for the study of natural language communication between man and machine,” Commun. ACM, vol. 9, no. 1, pp. 36–45, Jan. 1966. [Online]. Available: http://doi.acm.org/10.1145/365153.365168
  15. J. C. R. Licklider, “Man-computer symbiosis,” IEEE Ann. Hist. Comput., vol. 14, no. 1, pp. 24–, Jan. 1992. [Online]. Available: http://dl.acm.org/citation.cfm?id=612400.612433
  16. E. Fast, B. Chen, J. Mendelsohn, J. Bassen, and M. S. Bernstein, “Iris,” Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ’18, 2018. [Online]. Available: http://dx.doi.org/10.1145/3173574.3174047
  17. J. Cassell, “Embodied conversational interface agents,” Commun. ACM, vol. 43, no. 4, pp. 70–78, Apr. 2000. [Online]. Available: http://doi.acm.org/10.1145/332051.332075
  18. S. Subramaniam, P. Aggarwal, G. B. Dasgupta, and A. Paradkar, “Cobots - a cognitive multi-bot conversational framework for technical support,” in Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, ser. AAMAS ’18.   Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems, 2018, pp. 597–604. [Online]. Available: http://dl.acm.org/citation.cfm?id=3237383.3237472
  19. M. Porcheron, J. E. Fischer, S. Reeves, and S. Sharples, “Voice interfaces in everyday life,” in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ser. CHI ’18.   New York, NY, USA: ACM, 2018, pp. 640:1–640:12. [Online]. Available: http://doi.acm.org/10.1145/3173574.3174214
  20. A. Cercas Curry, I. Papaioannou, A. Suglia, S. Agarwal, I. Shalyminov, X. Xinnuo, O. Dusek, A. Eshghi, I. Konstas, V. Rieser, and O. Lemon, “Alana v2: Entertaining and informative open-domain social dialogue using ontologies and entity linking,” in 1st Proceedings of Alexa Prize (Alexa Prize 2018), 2018.
  21. W. S. Lasecki, R. Wesley, J. Nichols, A. Kulkarni, J. F. Allen, and J. P. Bigham, “Chorus: a crowd-powered conversational assistant,” in Proceedings of the 26th annual ACM symposium on User interface software and technology.   ACM, 2013, pp. 151–162.
  22. A. Sciuto, A. Saini, J. Forlizzi, and J. I. Hong, “”hey alexa, what’s up?”: A mixed-methods studies of in-home conversational agent usage,” in Proceedings of the 2018 Designing Interactive Systems Conference, ser. DIS ’18.   New York, NY, USA: Association for Computing Machinery, 2018, p. 857–868. [Online]. Available: https://doi.org/10.1145/3196709.3196772
  23. H. Candello, F. Barth, E. Carvalho, and R. A. G. Cotia, “Understanding how visitors interact with voice-based conversational systems,” in Design, User Experience, and Usability. Design for Contemporary Interactive Environments, A. Marcus and E. Rosenzweig, Eds.   Cham: Springer International Publishing, 2020, pp. 40–55.
  24. S. Reeves, M. Porcheron, J. E. Fischer, H. Candello, D. McMillan, M. McGregor, R. J. Moore, R. Sikveland, A. S. Taylor, J. Velkovska, and M. Zouinar, “Voice-based conversational ux studies and design,” in Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, ser. CHI EA ’18.   New York, NY, USA: ACM, 2018, pp. W38:1–W38:8. [Online]. Available: http://doi.acm.org/10.1145/3170427.3170619
  25. N. Ali-Hasan, “Evaluating smartphone vocie assistants: A review of ux methods and challenges,” in Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, ser. CHI EA ’18.   New York, NY, USA: ACM, 2018. [Online]. Available: https://voiceux.files.wordpress.com/2018/03/ali-hasan.pdf
  26. E. H. R. R. Krithika Jagannath, “Intelligent agents in everyday settings: Leveraging a multi-methods’ approach,” in Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, ser. CHI EA ’18.   New York, NY, USA: ACM, 2018. [Online]. Available: https://voiceux.files.wordpress.com/2018/03/jagannath.pdf
  27. T.-H. K. Huang, W. S. Lasecki, A. Azaria, and J. P. Bigham, “” is there anything else i can help you with?” challenges in deploying an on-demand crowd-powered conversational agent,” in Fourth AAAI Conference on Human Computation and Crowdsourcing, 2016.
  28. M. Lee and S. Lee, “I Don’t Know Exactly but I Know a Little”: Exploring Better Responses of Conversational Agents with Insufficient Information.   New York, NY, USA: Association for Computing Machinery, 2021. [Online]. Available: https://doi.org/10.1145/3411763.3451812
  29. O. Täckström, D. Das, S. Petrov, R. McDonald, and J. Nivre, “Token and type constraints for cross-lingual part-of-speech tagging,” Transactions of the Association for Computational Linguistics, vol. 1, pp. 1–12, 2013.
  30. J. Hauswald, M. A. Laurenzano, Y. Zhang, C. Li, A. Rovinski, A. Khurana, R. G. Dreslinski, T. Mudge, V. Petrucci, L. Tang, and J. Mars, “Sirius: An open end-to-end voice and vision personal assistant and its implications for future warehouse scale computers,” SIGPLAN Not., vol. 50, no. 4, pp. 223–238, Mar. 2015. [Online]. Available: http://doi.acm.org/10.1145/2775054.2694347
  31. Venturebeat, “Google’s speech recognition technology now has a 4.9% word error rate,” 2017.
  32. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).   Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://www.aclweb.org/anthology/N19-1423
  33. D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. St. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, B. Strope, and R. Kurzweil, “Universal sentence encoder for English,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.   Brussels, Belgium: Association for Computational Linguistics, Nov. 2018, pp. 169–174. [Online]. Available: https://www.aclweb.org/anthology/D18-2029
  34. S. Arora, Y. Liang, and T. Ma, “A simple but tough-to-beat baseline for sentence embeddings,” 2017.
  35. J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Empirical Methods in Natural Language Processing (EMNLP), 2014, pp. 1532–1543. [Online]. Available: http://www.aclweb.org/anthology/D14-1162
  36. J. Brooke, “Sus: A quick and dirty usability scale,” 1996.
  37. L. Buzzard Software, “Sus-a,” 2015. [Online]. Available: https://itunes.apple.com/us/app/sus-a/id1026492557?mt=8
  38. W. H. Delone and E. R. McLean, “The delone and mclean model of information systems success: a ten-year update,” Journal of management information systems, vol. 19, no. 4, pp. 9–30, 2003.
  39. W. H. DeLone and E. R. McLean, “Information systems success: The quest for the dependent variable,” Information systems research, vol. 3, no. 1, pp. 60–95, 1992.
  40. Y. Kang, Y. Zhang, J. K. Kummerfeld, P. Hill, J. Hauswald, M. A. Laurenzano, L. Tang, and J. Mars, “Data collection for dialogue system: A startup perspective,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), 2018. [Online]. Available: http://aclweb.org/anthology/N18-3005
  41. L. V. Research, “Siri semester exam grade improves to C from D+,” 2018.
  42. ——, “Faceoff: Amazon echo show VS Google home part II,” 2017.
  43. M. Buhrmester, T. Kwang, and S. D. Gosling, “Amazon’s mechanical turk: A new source of inexpensive, yet high-quality, data?” Perspectives on psychological science, vol. 6, no. 1, pp. 3–5, 2011.
  44. A. Lab126, “Alexa voice service,” 2019. [Online]. Available: https://developer.amazon.com/docs/alexa-voice-service/api-overview.html
  45. Google, “Google assistant,” 2019. [Online]. Available: https://assistant.google.com/
  46. ——, “Google assistant library,” 2019. [Online]. Available: https://developers.google.com/assistant/sdk/reference/library/python/
  47. ——, “Google assistant service,” 2019. [Online]. Available: https://developers.google.com/assistant/sdk/reference/rpc/
  48. S. H. Inc., “Houndify,” 2019. [Online]. Available: https://www.houndify.com/faq
  49. ——, “Houndify python sdk,” 2019. [Online]. Available: https://www.houndify.com/sdks#python
  50. C. Inc., “Clinc conversational ai platform,” 2019. [Online]. Available: https://clinc.com/platform/
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Christopher Clarke (13 papers)
  2. Karthik Krishnamurthy (2 papers)
  3. Walter Talamonti (5 papers)
  4. Yiping Kang (8 papers)
  5. Lingjia Tang (15 papers)
  6. Jason Mars (21 papers)
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now