Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

LLM-based Smart Reply (LSR): Enhancing Collaborative Performance with ChatGPT-mediated Smart Reply System (2306.11980v5)

Published 21 Jun 2023 in cs.HC

Abstract: Interactive user interfaces have increasingly explored AI's role in enhancing communication efficiency and productivity in collaborative tasks. The emergence of LLMs such as ChatGPT has revolutionized conversational agents, employing advanced deep learning techniques to generate context-aware, coherent, and personalized responses. Consequently, LLM-based AI assistants provide a more natural and efficient user experience across various scenarios. In this paper, we study how LLM models can be used to improve work efficiency in collaborative workplaces. Specifically, we present an LLM-based Smart Reply (LSR) system utilizing the ChatGPT to generate personalized responses in professional collaborative scenarios while adapting to context and communication style based on prior responses. Our two-step process involves generating a preliminary response type (e.g., Agree, Disagree) to provide a generalized direction for message generation, thus reducing response drafting time. We conducted an experiment where participants completed simulated work tasks involving a Dual N-back test and subtask scheduling through Google Calendar while interacting with co-workers. Our findings indicate that the proposed LSR reduces overall workload, as measured by the NASA TLX, and improves work performance and productivity in the N-back task. We also provide qualitative analysis based on participants' experiences, as well as design considerations to provide future directions for improving such implementations.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.
  2. D. M. Katz, M. J. Bommarito, S. Gao, and P. Arredondo, “Gpt-4 passes the bar exam,” Philosophical Transactions of the Royal Society A, vol. 382, no. 2270, p. 20230254, 2024.
  3. G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023.
  4. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023.
  5. Q. Wu, G. Bansal, J. Zhang, Y. Wu, S. Zhang, E. Zhu, B. Li, L. Jiang, X. Zhang, and C. Wang, “Autogen: Enabling next-gen llm applications via multi-agent conversation framework,” arXiv preprint arXiv:2308.08155, 2023.
  6. S. Cai, S. Venugopalan, K. Tomanek, A. Narayanan, M. R. Morris, and M. P. Brenner, “Context-aware abbreviation expansion using large language models,” arXiv preprint arXiv:2205.03767, 2022.
  7. S. M. Goodman, E. Buehler, P. Clary, A. Coenen, A. Donsbach, T. N. Horne, M. Lahav, R. MacDonald, R. B. Michaels, A. Narayanan, et al., “Lampost: Design and evaluation of an ai-assisted email writing prototype for adults with dyslexia,” in Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 1–18, 2022.
  8. A. Yuan, A. Coenen, E. Reif, and D. Ippolito, “Wordcraft: story writing with large language models,” in 27th International Conference on Intelligent User Interfaces, pp. 841–852, 2022.
  9. N. N. Kamel and R. M. Davison, “Applying cscw technology to overcome traditional barriers in group interactions,” Information & Management, vol. 34, no. 4, pp. 209–219, 1998.
  10. R. P. Carasik and C. E. Grantham, “A case study of cscw in a dispersed organization,” in Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 61–66, 1988.
  11. G. Convertino, U. Farooq, M. B. Rosson, J. M. Carroll, and B. J. Meyer, “Supporting intergenerational groups in computer-supported cooperative work (cscw),” Behaviour & Information Technology, vol. 26, no. 4, pp. 275–285, 2007.
  12. D. Yadav, P. Malik, K. Dabas, and P. Singh, “Feedpal: Understanding opportunities for chatbots in breastfeeding education of women in india,” Proceedings of the ACM on Human-Computer Interaction, vol. 3, no. CSCW, pp. 1–30, 2019.
  13. I. Srba and M. Bielikova, “Askalot: community question answering as a means for knowledge sharing in an educational organization,” in Proceedings of the 18th ACM conference companion on computer supported cooperative work & social computing, pp. 179–182, 2015.
  14. F. Enembreck and J.-P. Barthes, “Personal assistant to improve cscw,” in The 7th International Conference on Computer Supported Cooperative Work in Design, pp. 329–335, IEEE, 2002.
  15. J. T. Hancock, M. Naaman, and K. Levy, “Ai-mediated communication: Definition, research agenda, and ethical considerations,” Journal of Computer-Mediated Communication, vol. 25, no. 1, pp. 89–100, 2020.
  16. Z. Zhang, B. Li, and L. Liu, “The impact of ai-based conversational agent on the firms’ operational performance: Empirical evidence from a call center,” Applied Artificial Intelligence, vol. 37, no. 1, p. 2157592, 2023.
  17. Kogan Page Publishers, 2016.
  18. Y. K. Dwivedi, N. Kshetri, L. Hughes, E. L. Slade, A. Jeyaraj, A. K. Kar, A. M. Baabdullah, A. Koohang, V. Raghavan, M. Ahuja, et al., ““so what if chatgpt wrote it?” multidisciplinary perspectives on opportunities, challenges and implications of generative conversational ai for research, practice and policy,” International Journal of Information Management, vol. 71, p. 102642, 2023.
  19. M. S. Teixeira, V. Maran, and M. Dragoni, “The interplay of a conversational ontology and ai planning for health dialogue management,” in Proceedings of the 36th annual ACM symposium on applied computing, pp. 611–619, 2021.
  20. P. Kulkarni, A. Mahabaleshwarkar, M. Kulkarni, N. Sirsikar, and K. Gadgil, “Conversational ai: An overview of methodologies, applications & future scope,” in 2019 5th International Conference On Computing, Communication, Control And Automation (ICCUBEA), pp. 1–7, IEEE, 2019.
  21. D. Cifci, G. P. Veldhuizen, S. Foersch, and J. N. Kather, “Ai in computational pathology of cancer: Improving diagnostic workflows and clinical outcomes?,” Annual Review of Cancer Biology, vol. 7, pp. 57–71, 2023.
  22. R. C. Li, M. Smith, J. Lu, A. Avati, S. Wang, W. G. Teuteberg, K. Shum, G. Hong, B. Seevaratnam, J. Westphal, et al., “Using ai to empower collaborative team workflows: two implementations for advance care planning and care escalation,” NEJM Catalyst Innovations in Care Delivery, vol. 3, no. 4, pp. CAT–21, 2022.
  23. J. Bidot, C. Goumopoulos, and I. Calemis, “Using ai planning and late binding for managing service workflows in intelligent environments,” in 2011 IEEE International Conference on Pervasive Computing and Communications (PerCom), pp. 156–163, IEEE, 2011.
  24. A. Bastola, J. Brinkley, H. Wang, and A. Razi, “Driving towards inclusion: Revisiting in-vehicle interaction in autonomous vehicles,” arXiv preprint arXiv:2401.14571, 2024.
  25. N. Jiang, X. Liu, H. Liu, E. T. K. Lim, C.-W. Tan, and J. Gu, “Beyond ai-powered context-aware services: the role of human–ai collaboration,” Industrial Management & Data Systems, 2022.
  26. O. C. Santos, M. Saneiro, J. G. Boticario, and M. C. Rodriguez-Sanchez, “Toward interactive context-aware affective educational recommendations in computer-assisted language learning,” New Review of Hypermedia and Multimedia, vol. 22, no. 1-2, pp. 27–57, 2016.
  27. P. Zaraté, J. P. Belaud, and G. Camilleri, “Collaborative decision making: Perspectives and challenges,” 2008.
  28. K. E. Schaefer, J. Oh, D. Aksaray, and D. Barber, “Integrating context into artificial intelligence: research from the robotics collaborative technology alliance,” Ai Magazine, vol. 40, no. 3, pp. 28–40, 2019.
  29. N. Kunnathuvalappil Hariharan, “Artificial intelligence and human collaboration in financial planning,” 2018.
  30. A. Bastola, M. Atik Enam, A. Bastola, A. Gluck, and J. Brinkley, “Multi-functional glasses for the blind and visually impaired: Design and development,” in Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 67, pp. 995–1001, SAGE Publications Sage CA: Los Angeles, CA, 2023.
  31. X. Chen, D. Zou, H. Xie, G. Cheng, and C. Liu, “Two decades of artificial intelligence in education,” Educational Technology & Society, vol. 25, no. 1, pp. 28–47, 2022.
  32. H. J. Wilson and P. R. Daugherty, “Collaborative intelligence: Humans and ai are joining forces,” Harvard Business Review, vol. 96, no. 4, pp. 114–123, 2018.
  33. J. M. Tien, “Internet of things, real-time decision making, and artificial intelligence,” Annals of Data Science, vol. 4, pp. 149–178, 2017.
  34. S. L. Epstein, “Wanted: collaborative intelligence,” Artificial Intelligence, vol. 221, pp. 36–45, 2015.
  35. T. H. Kung, M. Cheatham, A. Medenilla, C. Sillos, L. De Leon, C. Elepaño, M. Madriaga, R. Aggabao, G. Diaz-Candido, J. Maningo, et al., “Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models,” PLoS digital health, vol. 2, no. 2, p. e0000198, 2023.
  36. C. Regan, N. Iwahashi, S. Tanaka, and M. Oka, “Can generative agents predict emotion?,” arXiv preprint arXiv:2402.04232, 2024.
  37. E. Clark, T. August, S. Serrano, N. Haduong, S. Gururangan, and N. A. Smith, “All that’s ‘human’ is not gold: Evaluating human evaluation of generated text,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), (Online), pp. 7282–7296, Association for Computational Linguistics, Aug. 2021.
  38. A. Kannan, K. Kurach, S. Ravi, T. Kaufmann, A. Tomkins, B. Miklos, G. Corrado, L. Lukacs, M. Ganea, P. Young, and V. Ramavajjala, “Smart reply: Automated response suggestion for email,” 2016.
  39. M. X. Chen, B. N. Lee, G. Bansal, Y. Cao, S. Zhang, J. Lu, J. Tsay, Y. Wang, A. M. Dai, Z. Chen, T. Sohn, and Y. Wu, “Gmail smart compose: Real-time assisted writing,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery; Data Mining, KDD ’19, (New York, NY, USA), p. 2287–2295, Association for Computing Machinery, 2019.
  40. A. Kannan, K. Kurach, S. Ravi, T. Kaufmann, A. Tomkins, B. Miklos, G. Corrado, L. Lukacs, M. Ganea, P. Young, et al., “Smart reply: Automated response suggestion for email,” in Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 955–964, 2016.
  41. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  42. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  43. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al., “Sparks of artificial general intelligence: Early experiments with gpt-4,” arXiv preprint arXiv:2303.12712, 2023.
  44. M. Henderson, R. Al-Rfou, B. Strope, Y.-H. Sung, L. Lukács, R. Guo, S. Kumar, B. Miklos, and R. Kurzweil, “Efficient natural language response suggestion for smart reply,” arXiv preprint arXiv:1705.00652, 2017.
  45. J. M. Moosa, “How smart is the “smart reply” feature of google allo: From users’ viewpoint,” 2016.
  46. Z. Xiao, Q. V. Liao, M. Zhou, T. Grandison, and Y. Li, “Powering an ai chatbot with expert sourcing to support credible health information access,” in Proceedings of the 28th International Conference on Intelligent User Interfaces, pp. 2–18, 2023.
  47. H. Mieczkowski, J. T. Hancock, M. Naaman, M. Jung, and J. Hohenstein, “Ai-mediated communication: Language use and interpersonal effects in a referential communication task,” Proceedings of the ACM on Human-Computer Interaction, vol. 5, no. CSCW1, pp. 1–14, 2021.
  48. J. Hohenstein, R. F. Kizilcec, D. DiFranzo, Z. Aghajari, H. Mieczkowski, K. Levy, M. Naaman, J. Hancock, and M. F. Jung, “Artificial intelligence in communication impacts language and social relationships,” Scientific Reports, vol. 13, no. 1, p. 5487, 2023.
  49. S. Kim, S. Yun, H. Lee, M. Gubri, S. Yoon, and S. J. Oh, “Propile: Probing privacy leakage in large language models,” arXiv preprint arXiv:2307.01881, 2023.
  50. S.-L. Wamba-Taguimdje, S. Fosso Wamba, J. R. Kala Kamdjoug, and C. E. Tchatchouang Wanko, “Influence of artificial intelligence (ai) on firm performance: the business value of ai-based transformation projects,” Business Process Management Journal, vol. 26, no. 7, pp. 1893–1924, 2020.
  51. J. P. Shim, M. Warkentin, J. F. Courtney, D. J. Power, R. Sharda, and C. Carlsson, “Past, present, and future of decision support technology,” Decision support systems, vol. 33, no. 2, pp. 111–126, 2002.
  52. K. Wenker, “Who wrote this? how smart replies impact language and agency in the workplace,” Telematics and Informatics Reports, vol. 10, p. 100062, 2023.
  53. T. Eloundou, S. Manning, P. Mishkin, and D. Rock, “Gpts are gpts: An early look at the labor market impact potential of large language models,” arXiv preprint arXiv:2303.10130, 2023.
  54. J. Manyika, M. Chui, M. Miremadi, J. Bughin, K. George, P. Willmott, and M. Dewhurst, “A future that works: Ai, automation, employment, and productivity,” McKinsey Global Institute Research, Tech. Rep, vol. 60, pp. 1–135, 2017.
  55. L. Urquhart, A. Laffer, and D. Miranda, “Working with affective computing: Exploring uk public perceptions of ai enabled workplace surveillance,” arXiv preprint arXiv:2205.08264, 2022.
  56. E. Kasneci, K. Seßler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier, et al., “Chatgpt for good? on opportunities and challenges of large language models for education,” Learning and individual differences, vol. 103, p. 102274, 2023.
  57. A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al., “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,” arXiv preprint arXiv:2206.04615, 2022.
  58. N. Nawaz and A. M. Gomes, “Artificial intelligence chatbots are new recruiters,” IJACSA) International Journal of Advanced Computer Science and Applications, vol. 10, no. 9, 2019.
  59. C. Krishnan, A. Gupta, A. Gupta, and G. Singh, “Impact of artificial intelligence-based chatbots on customer engagement and business growth,” in Deep Learning for Social Media Data Analytics, pp. 195–210, Springer, 2022.
  60. K. M. Getchell, S. Carradini, P. W. Cardon, C. Fleischmann, H. Ma, J. Aritz, and J. Stapp, “Artificial intelligence in business communication: the changing landscape of research and teaching,” Business and Professional Communication Quarterly, vol. 85, no. 1, pp. 7–33, 2022.
  61. V. Pereira, E. Hadjielias, M. Christofi, and D. Vrontis, “A systematic literature review on the impact of artificial intelligence on workplace outcomes: A multi-process perspective,” Human Resource Management Review, vol. 33, no. 1, p. 100857, 2023.
  62. A. M. Owen, K. M. McMillan, A. R. Laird, and E. Bullmore, “N-back working memory paradigm: A meta-analysis of normative functional neuroimaging studies,” Human brain mapping, vol. 25, no. 1, pp. 46–59, 2005.
  63. S. G. Hart, “Nasa-task load index (nasa-tlx); 20 years later,” in Proceedings of the human factors and ergonomics society annual meeting, vol. 50, pp. 904–908, Sage publications Sage CA: Los Angeles, CA, 2006.
  64. R. Zhang, N. J. McNeese, G. Freeman, and G. Musick, “” an ideal human” expectations of ai teammates in human-ai teaming,” Proceedings of the ACM on Human-Computer Interaction, vol. 4, no. CSCW3, pp. 1–25, 2021.
  65. V. Braun and V. Clarke, “Using thematic analysis in psychology,” Qualitative research in psychology, vol. 3, no. 2, pp. 77–101, 2006.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Ashish Bastola (9 papers)
  2. Hao Wang (1120 papers)
  3. Judsen Hembree (1 paper)
  4. Pooja Yadav (14 papers)
  5. Zihao Gong (6 papers)
  6. Emma Dixon (4 papers)
  7. Abolfazl Razi (63 papers)
  8. Nathan McNeese (1 paper)
Citations (9)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets