Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Understanding Large-Language Model (LLM)-powered Human-Robot Interaction (2401.03217v1)

Published 6 Jan 2024 in cs.RO and cs.HC

Abstract: Large-LLMs hold significant promise in improving human-robot interaction, offering advanced conversational skills and versatility in managing diverse, open-ended user requests in various tasks and domains. Despite the potential to transform human-robot interaction, very little is known about the distinctive design requirements for utilizing LLMs in robots, which may differ from text and voice interaction and vary by task and context. To better understand these requirements, we conducted a user study (n = 32) comparing an LLM-powered social robot against text- and voice-based agents, analyzing task-based requirements in conversational tasks, including choose, generate, execute, and negotiate. Our findings show that LLM-powered robots elevate expectations for sophisticated non-verbal cues and excel in connection-building and deliberation, but fall short in logical communication and may induce anxiety. We provide design implications both for robots integrating LLMs and for fine-tuning LLMs for use with robots.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (77)
  1. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 (2022).
  2. Will You Rely on a Robot to Find a Treasure?. In Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction (Vienna, Austria) (HRI ’17). Association for Computing Machinery, New York, NY, USA, 71–72. https://doi.org/10.1145/3029798.3038394
  3. The effect of presence on human-robot interaction. In RO-MAN 2008-The 17th IEEE International Symposium on Robot and Human Interactive Communication. IEEE, 701–706.
  4. The benefits of interactions with physically present robots over video-displayed agents. International Journal of Social Robotics 3 (2011), 41–52.
  5. Measurement Instruments for the Anthropomorphism, Animacy, Likeability, Perceived Intelligence, and Perceived Safety of Robots. 1, 1 ([n. d.]), 71–81. https://doi.org/10.1007/s12369-008-0001-3
  6. Timothy W Bickmore and Rosalind W Picard. 2005. Establishing and maintaining long-term human-computer relationships. ACM Transactions on Computer-Human Interaction (TOCHI) 12, 2 (2005), 293–327.
  7. Language models for human-robot interaction. In ACM/IEEE International Conference on Human-Robot Interaction, March 13–16, 2023, Stockholm, Sweden. ACM Digital Library, 905–906.
  8. Saša Bodiroža. 2017. Gestures in human-robot interaction. Ph. D. Dissertation. Humboldt-Universität zu Berlin, Mathematisch-Naturwissenschaftliche Fakultät. https://doi.org/10.18452/17705
  9. Social robotics. Springer handbook of robotics (2016), 1935–1972.
  10. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  11. FurChat: An Embodied Conversational Agent using LLMs, Combining Open and Closed-Domain Dialogue with Facial Expressions. arXiv preprint arXiv:2308.15214 (2023).
  12. Designing persuasive robots: how robots might persuade people using vocal and nonverbal cues. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction. 293–300.
  13. Victoria Clarke and Virginia Braun. 2014. Thematic analysis. In Encyclopedia of critical psychology. Springer, 1947–1952.
  14. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168 [cs.LG]
  15. Developing and using a codebook for the analysis of interview data: An example from a professional development research project. Field methods 23, 2 (2011), 136–155.
  16. Embodiment in socially interactive robots. Foundations and Trends® in Robotics 7, 4 (2019), 251–356.
  17. PaLM-E: An Embodied Multimodal Language Model. arXiv:2303.03378 [cs.LG]
  18. OpenAGI: When LLM Meets Domain Experts. arXiv:2304.04370 [cs.AI]
  19. Google. 2023. Google Cloud Services—Speech to text. ”Accessed = 09-29-2023”.
  20. Design and evaluation of a peripheral robotic conversation companion. In Proceedings of the tenth annual ACM/IEEE international conference on human-robot interaction. 3–10.
  21. Laura Hoffmann and Nicole C. Krämer. 2013. Investigating the effects of physical and virtual embodiment in task-oriented and conversational contexts. International Journal of Human-Computer Studies 71, 7 (2013), 763–774. https://doi.org/10.1016/j.ijhcs.2013.04.007
  22. Parameter-Efficient Transfer Learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 2790–2799. https://proceedings.mlr.press/v97/houlsby19a.html
  23. LoRA: Low-Rank Adaptation of Large Language Models. CoRR abs/2106.09685 (2021). arXiv:2106.09685 https://arxiv.org/abs/2106.09685
  24. Chien-Ming Huang and Bilge Mutlu. 2013. Modeling and Evaluating Narrative Gestures for Humanlike Robots.. In Robotics: Science and Systems, Vol. 2. Citeseer.
  25. Between Reality and Delusion: Challenges of Applying Large Language Models to Companion Robots for Open-Domain Dialogues with Older Adults. https://doi.org/10.21203/rs.3.rs-2884789/v1
  26. Artificial empathy in social robots: An analysis of emotions in speech. In 2018 27th IEEE International symposium on robot and human interactive communication (RO-MAN). IEEE, 632–637.
  27. Survey of Hallucination in Natural Language Generation. ACM Comput. Surv. 55, 12, Article 248 (mar 2023), 38 pages. https://doi.org/10.1145/3571730
  28. Younbo Jung and Kwan Min Lee. 2004. Effects of physical embodiment on social presence of social robots. Proceedings of PRESENCE 2004 (2004), 80–87.
  29. Spill the Tea: When Robot Conversation Agents Support Well-Being for Older Adults. In Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction (Stockholm, Sweden) (HRI ’23). Association for Computing Machinery, New York, NY, USA, 178–182. https://doi.org/10.1145/3568294.3580067
  30. Structured and Unstructured Speech2Action Frameworks for Human-Robot Collaboration: A User Study. (2023).
  31. Tell me more! Assessing interactions with social robots from speech. Paladyn, Journal of Behavioral Robotics 12, 1 (2020), 136–159.
  32. The unboxing experience: Exploration and design of initial interactions between children and social robots. In Proceedings of the 2022 CHI conference on human factors in computing systems. 1–14.
  33. Developing Social Robots with Empathetic Non-Verbal Cues Using Large Language Models. arXiv:2308.16529 [cs.RO]
  34. Social robots for long-term interaction: a survey. International Journal of Social Robotics 5 (2013), 291–308.
  35. Jamy Li. 2015. The benefit of being physically present: A survey of experimental works comparing copresent robots, telepresent robots and virtual agents. International Journal of Human-Computer Studies 77 (2015), 23–37.
  36. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems 35 (2022), 1950–1965.
  37. Tailoring Large Language Models to Radiology: A Preliminary Approach to LLM Adaptation for a Highly Specialized Domain. In Machine Learning in Medical Imaging, Xiaohuan Cao, Xuanang Xu, Islem Rekik, Zhiming Cui, and Xi Ouyang (Eds.). Springer Nature Switzerland, Cham, 464–473.
  38. Arnold M Lund. 2001. Measuring usability with the use questionnaire12. Usability interface 8, 2 (2001), 3–6.
  39. The effects of eye design on the perception of social robots. In 2018 27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE, 1032–1037.
  40. Designing Vyo, a robotic Smart Home assistant: Bridging the gap between device and social agent. In 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE, 1019–1025.
  41. Comparing social robot, screen and voice interfaces for smart-home control. In Proceedings of the 2017 CHI conference on human factors in computing systems. 580–628.
  42. Re-Embodiment and Co-Embodiment: Exploration of social presence for robots and conversational agents. In Proceedings of the 2019 on Designing Interactive Systems Conference. 633–644.
  43. Reliability and Inter-rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice. Proceedings of the ACM on Human-Computer Interaction 3 (11 2019), 1–23. https://doi.org/10.1145/3359174
  44. Joseph Edward McGrath. 1984. Groups: Interaction and performance. Vol. 14. Prentice-Hall Englewood Cliffs, NJ.
  45. Jonathan Mumm and Bilge Mutlu. 2011. Human-Robot Proxemics: Physical and Psychological Distancing in Human-Robot Interaction. In Proceedings of the 6th International Conference on Human-Robot Interaction (Lausanne, Switzerland) (HRI ’11). Association for Computing Machinery, New York, NY, USA, 331–338. https://doi.org/10.1145/1957656.1957786
  46. Bilge Mutlu. 2021. The virtual and the physical: two frames of mind. iScience 24, 2 (2021), 101965. https://doi.org/10.1016/j.isci.2020.101965
  47. Conversational gaze mechanisms for humanlike robots. ACM Transactions on Interactive Intelligent Systems (TiiS) 1, 2 (2012), 1–33.
  48. Task structure and user attributes as elements of human-robot interaction design. In ROMAN 2006-The 15th IEEE International Symposium on Robot and Human Interactive Communication. IEEE, 74–79.
  49. Footing in human-robot conversations: how robots might shape participant roles using gaze cues. In Proceedings of the 4th ACM/IEEE international conference on Human robot interaction. 61–68.
  50. Creating Personalized Verbal Human-Robot Interactions Using LLM with the Robot Mini. In Proceedings of the 15th International Conference on Ubiquitous Computing & Ambient Intelligence (UCAmI 2023), José Bravo and Gabriel Urzáiz (Eds.). Springer Nature Switzerland, Cham, 148–159.
  51. Training language models to follow instructions with human feedback. arXiv:2203.02155 [cs.CL]
  52. Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback. arXiv:2302.12813 [cs.CL]
  53. Comparing a computer agent with a humanoid robot. In Proceedings of the ACM/IEEE international conference on Human-robot interaction. 145–152.
  54. Perceptions of explicitly vs. implicitly relayed commands between a robot and smart speaker. In 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 1012–1016.
  55. Leveraging robot embodiment to facilitate trust and smoothness. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 742–744.
  56. Aldebaran Robotics. 2023a. Animated Speech. ”Accessed = 09-29-2023”.
  57. Aldebaran Robotics. 2023b. Audio Device API. ”Accessed = 09-29-2023”.
  58. Soft Bank Robotics. 2023c. Pepper Robot. ”Accessed = 09-29-2023”.
  59. A bellboy robot: Study of the effects of robot behaviour on user engagement and comfort. International Journal of Human-Computer Studies 82 (2015), 83–95. https://doi.org/10.1016/j.ijhcs.2015.06.001
  60. Generation and evaluation of communicative robot gesture. International Journal of Social Robotics 4 (2012), 201–217.
  61. Evaluating the effect of saliency detection and attention manipulation in human-robot interaction. International Journal of Social Robotics 5 (2013), 139–152.
  62. How do you like me in this: User embodiment preferences for companion agents. In Intelligent Virtual Agents: 12th International Conference, IVA 2012, Santa Cruz, CA, USA, September, 12-14, 2012. Proceedings 12. Springer, 112–125.
  63. Poor Thing! Would You Feel Sorry for a Simulated Robot? A Comparison of Empathy toward a Physical and a Simulated Robot. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction (Portland, Oregon, USA) (HRI ’15). Association for Computing Machinery, New York, NY, USA, 125–132. https://doi.org/10.1145/2696454.2696471
  64. Generative Facial Expressions and Eye Gaze Behavior from Prompts for Multi-Human-Robot Interaction. In Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (¡conf-loc¿, ¡city¿San Francisco¡/city¿, ¡state¿CA¡/state¿, ¡country¿USA¡/country¿, ¡/conf-loc¿) (UIST ’23 Adjunct). Association for Computing Machinery, New York, NY, USA, Article 13, 3 pages. https://doi.org/10.1145/3586182.3616623
  65. Generate & Rank: A Multi-task Framework for Math Word Problems. arXiv:2109.03034 [cs.CL]
  66. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 11523–11530.
  67. I’m Sorry, Dave: I’m Afraid i Won’t Do That: Social Aspects of Human-Agent Conflict. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI ’09). Association for Computing Machinery, New York, NY, USA, 2099–2108. https://doi.org/10.1145/1518701.1519021
  68. Understanding the capabilities, limitations, and societal impact of large language models. arXiv preprint arXiv:2102.02503 (2021).
  69. Stanford alpaca: An instruction-following llama model.
  70. Chatgpt for robotics: Design principles and model abilities. Microsoft Auton. Syst. Robot. Res 2 (2023), 20.
  71. The role of physical embodiment in human-robot interaction. In ROMAN 2006-The 15th IEEE International Symposium on Robot and Human Interactive Communication. IEEE, 117–122.
  72. Embodiment and human-robot interaction: A task-based perspective. In RO-MAN 2007-The 16th IEEE International Symposium on Robot and Human Interactive Communication. IEEE, 872–877.
  73. Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv:2212.10560 [cs.CL]
  74. Building a hospitable and reliable dialogue system for android robots: a scenario-based approach with large language models. Advanced Robotics 37, 21 (2023), 1364–1381.
  75. Improved Trust in Human-Robot Collaboration With ChatGPT. IEEE Access 11 (2023), 55748–55754. https://doi.org/10.1109/ACCESS.2023.3282111
  76. Tom Ziemke. 2013. What’s that thing called embodiment? In Proceedings of the 25th Annual Cognitive Science Society. Psychology Press, 1305–1310.
  77. Zoom. 2023. Video Conferencing Platform. ”Accessed = 09-29-2023”.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Callie Y. Kim (2 papers)
  2. Christine P. Lee (6 papers)
  3. Bilge Mutlu (65 papers)
Citations (30)
View on Semantic Scholar