Coaching Copilot: Blended Form of an LLM-Powered Chatbot and a Human Coach to Effectively Support Self-Reflection for Leadership Growth (2405.15250v1)
Abstract: Chatbots' role in fostering self-reflection is now widely recognized, especially in inducing users' behavior change. While the benefits of 24/7 availability, scalability, and consistent responses have been demonstrated in contexts such as healthcare and tutoring to help one form a new habit, their utilization in coaching necessitating deeper introspective dialogue to induce leadership growth remains unexplored. This paper explores the potential of such a chatbot powered by recent LLMs in collaboration with professional coaches in the field of executive coaching. Through a design workshop with them and two weeks of user study involving ten coach-client pairs, we explored the feasibility and nuances of integrating chatbots to complement human coaches. Our findings highlight the benefits of chatbots' ubiquity and reasoning capabilities enabled by LLMs while identifying their limitations and design necessities for effective collaboration between human coaches and chatbots. By doing so, this work contributes to the foundation for augmenting one's self-reflective process with prevalent conversational agents through the human-in-the-loop approach.
- The Effect of Goal Moderation on the Achievement and Satisfaction of Physical Activity Goals. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 4 (2020), 116:1–116:18. https://doi.org/10.1145/3432209
- Guidelines for human-AI interaction. In Proceedings of the 2019 ACM CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 3. https://doi.org/10.1145/3290605.3300233
- Supporting Experts with a Multimodal Machine-Learning-Based Tool for Human Behavior Analysis of Conversational Videos. CoRR abs/2402.11145 (2024). https://doi.org/10.48550/ARXIV.2402.11145
- Riku Arakawa and Hiromu Yakura. 2019. REsCUE: A framework for REal-time feedback on behavioral CUEs using multimodal anomaly detection. In Proceedings of the 2019 ACM CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 572. https://doi.org/10.1145/3290605.3300802
- Riku Arakawa and Hiromu Yakura. 2020. INWARD: A computer-supported tool for video-reflection improves efficiency and effectiveness in executive coaching. In Proceedings of the 2020 ACM CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 1–13. https://doi.org/10.1145/3313831.3376703
- Chris Argyris. 1977. Double loop learning in organizations. Harvard business review 55, 5 (1977), 115–125.
- Godfrey T Barrett-Lennard. 1999. Carl Rogers’ helping system: Journey and substance. SAGE Publications, London, UK.
- Reviewing reflection: On the use of reflection in interactive system design. In Proceedings of the 2014 ACM Designing Interactive Systems Conference. ACM, New York, NY, 93–102. https://doi.org/10.1145/2598510.2598598
- Getting ready for adult healthcare: Designing a chatbot to coach adolescents with special health needs through the transitions of care. Journal of Pediatric Nursing 49 (2019), 85–91. https://doi.org/10.1016/j.pedn.2019.09.004
- LeadBoSki: A Smart Personal Assistant for Leadership Support in Video-Meetings. In Companion Publication of the 2021 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 19–22. https://doi.org/10.1145/3462204.3481764
- Revisiting reflection in HCI: Four design resources for technologies that support reflection. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 1 (2022), 2:1–2:27. https://doi.org/10.1145/3517233
- James Bridgeman and Andrea Giraldez-Hayes. 2023. Using artificial intelligence-enhanced video feedback for reflective practice in coach development: Benefits and potential drawbacks. Coaching: An International Journal of Theory, Research and Practice (2023), 1–18. https://doi.org/10.1080/17521882.2023.2228416
- Ethical Dilemmas, Mental Health, Artificial Intelligence, and LLM-Based Chatbots. In Proceedings of the 10th International Work-Conference on Bioinformatics and Biomedical Engineering, Vol. 13920. Springer, 313–326. https://doi.org/10.1007/978-3-031-34960-7_22
- “Listen to music, listen to yourself”: Design of a conversational agent to support self-awareness while listening to music. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 119:1–119:19. https://doi.org/10.1145/3544548.3581427
- Food diary coaching chatbot. In Proceedings of the 2018 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Wearable Computers. ACM, New York, NY, 1676–1680. https://doi.org/10.1145/3267305.3274191
- Understanding the Coach-Coachee-Client relationship: A conceptual framework for executive coaching. International Coaching Psychology Review 11, 1 (2016), 6–23.
- Fred D. Davis. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 13, 3 (1989), 319–340. https://doi.org/10.2307/249008
- Yogesh K. Dwivedi et al. 2023. Opinion paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management 71 (2023), 102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642
- The impact of a virtual teaching assistant (chatbot) on students' learning in Ghanaian higher education. International Journal of Educational Technology in Higher Education 19, 1 (2022). https://doi.org/10.1186/s41239-022-00362-6
- Gender Bias in Chatbot Design. In Proceedings of the 3rd International Workshop on Chatbot Research and Design. Springer, 79–93. https://doi.org/10.1007/978-3-030-39540-7_6
- Daniel C. Feldman and Melenie J. Lankau. 2005. Executive coaching: A review and agenda for future research. Journal of Management 31, 6 (2005), 829–848. https://doi.org/10.1177/0149206305279599
- Different chatbots for different purposes: Towards a typology of chatbots to understand interaction design. In Proceedings of the 2019 International Conference on Internet Science. Springer International Publishing, Cham, Switzerland, 145–156. https://doi.org/10.1007/978-3-030-17705-8_13
- Corey Ford and Nick Bryan-Kinns. 2023. Towards a reflection in creative experience questionnaire. In Proceedings of the 2023 ACM CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 763:1–763:16. https://doi.org/10.1145/3544548.3581077
- A chatbot-based coaching intervention for adolescents to promote life skills: Pilot study. JMIR Human Factors 7, 1 (2020), e16762. https://doi.org/10.2196/16762
- Anthony M. Grant. 2012. Making positive change: A randomized study comparing solution-focused vs. problem-focused coaching questions. Journal of Systemic Therapies 31, 2 (2012), 21–35. https://doi.org/10.1521/jsyt.2012.31.2.21
- Carolin Graßmann and Carsten C Schermuly. 2021. Coaching with artificial intelligence: Concepts and capabilities. Human Resource Development Review 20, 1 (2021), 106–126.
- Daniel P. Greenfield and William K. Hengen. 2004. Confidentiality in Coaching. Consulting to Management 15, 1, 9–14.
- Peter Hawkins. 2008. The coaching profession: some of the key challenges. Coaching: An International Journal of Theory, Research and Practice 1, 1 (2008), 28–38. https://doi.org/10.1080/17521880701878174
- Baek-Kyoo (Brian) Joo. 2005. Executive Coaching: A Conceptual Framework From an Integrative Review of Practice and Research. Human Resource Development Review 4, 4 (2005), 462–488. https://doi.org/10.1177/1534484305280866
- Pearl: A technology probe for machine-assisted reflection on personal data. In Proceedings of the 28th ACM International Conference on Intelligent User Interfaces. ACM, New York, NY, 902–918. https://doi.org/10.1145/3581641.3584054
- Towards the NESTORE e-Coach: A tangible and embodied conversational agent for older adults. In Proceedings of the 2018 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Wearable Computers. ACM, New York, NY, 1656–1663. https://doi.org/10.1145/3267305.3274188
- Richard R. Kilburg. 1997. Coaching and executive character: Core problems and basic approaches. Consulting Psychology Journal: Practice and Research 49, 4 (1997), 281–299. https://doi.org/10.1037/1061-4087.49.4.281
- Dyadic Mirror: Everyday Second-person Live-view for Empathetic Reflection upon Parent-child Interaction. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 3 (2020), 86:1–86:29. https://doi.org/10.1145/3411815
- Reflection Companion: A Conversational System for Engaging Users in Reflection on Physical Activity. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 2 (2018), 70:1–70:26. https://doi.org/10.1145/3214273
- Large Language Models are Zero-Shot Reasoners. In Proceedings of the 36th Annual Conference on Neural Information Processing Systems. Curran Associates, Inc., Red Hook, NY, 22199–22213.
- Interacting with educational chatbots: A systematic review. Education and Information Technologies 28, 1 (2023), 973–1018. https://doi.org/10.1007/s10639-022-11177-3
- Caring for Vincent: A Chatbot for Self-Compassion. In Proceedings of the 2019 ACM CHI Conference on Human Factors in Computing Systems. ACM, 702:1–702:13. https://doi.org/10.1145/3290605.3300932
- Exploring Design Opportunities for Reflective Conversational Agents to Reduce Compulsive Smartphone Use. In Proceedings of the 5th International Conference on Conversational User Interfaces. ACM, 37:1–37:6. https://doi.org/10.1145/3571884.3604305
- Incorporating a Reflective Thinking Promoting Mechanism into Artificial Intelligence-Supported English Writing Environments. Interactive Learning Environments 31, 9 (2021), 5614–5632. https://doi.org/10.1080/10494820.2021.2012812
- Doug MacKie. 2015. The effects of coachee readiness and core self-evaluations on leadership coaching outcomes: A controlled trial. Coaching: An International Journal of Theory, Research and Practice 8, 2 (2015), 120–136. https://doi.org/10.1080/17521882.2015.1019532
- Accompanying reflection processes by an AI-based StudiCoachBot: A study on rapport building in human-machine coaching using self disclosure. In Proceedings of the 23rd HCI International Conference - Late Breaking Papers. Springer, Cham, Switzerland, 439–457. https://doi.org/10.1007/978-3-030-90328-2_29
- Examining AI methods for micro-coaching dialogs. In Proceedings of the 2022 ACM CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 440:1–440:24. https://doi.org/10.1145/3491102.3501886
- Automated vs. human health coaching: Exploring participant and practitioner experiences. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 99:1–99:37. https://doi.org/10.1145/3449173
- Frode Moen and Einar Skaalvik. 2009. The effect from executive coaching on performance psychology. International Journal of Evidence Based Coaching & Mentoring 7, 2 (2009), 31–49.
- Informing design for reflection. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction. ACM, New York, NY, 1–10. https://doi.org/10.1145/2971485.2971494
- Jase Moussa-Inaty. 2015. Reflective Writing through the Use of Guiding Questions. International Journal of Teaching and Learning in Higher Education 27, 1 (2015), 104–113.
- Promoting wellbeing with sunny, a chatbot that facilitates positive messages within social groups. In Extended Abstracts of the 2020 ACM CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 1–8. https://doi.org/10.1145/3334480.3383062
- Training language models to follow instructions with human feedback. In Proceedings of the 36th Annual Conference on Neural Information Processing Systems. Curran Associates, Inc., Red Hook, NY, 27730–27744.
- Jonathan Passmore and David Tee. 2023. Can chatbots replace human coaches? Issues and dilemmas for the coaching profession, coaching clients and for organisations. The Coaching Psychologist 19, 1 (2023), 47–54. https://doi.org/10.53841/bpstcp.2023.19.1.47
- Exploring the challenges and opportunities in developing systems to improve alcohol use disorder through chatbot technology. In Extended Abstracts of the 2023 ACM CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 123:1–123:5. https://doi.org/10.1145/3544549.3585635
- Beyond behavior: The coach’s perspective on technology in health coaching. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 670. https://doi.org/10.1145/3290605.3300900
- Understanding how eHealth coaches tailor support for weight loss: Towards the design of person-centered coaching systems. In Proceedings of the 2022 ACM CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 285:1–285:16. https://doi.org/10.1145/3491102.3501864
- Trends, Challenges and Processes in Conversational Agent Design: Exploring Practitioners’ Views through Semi-Structured Interviews. In Proceedings of the 5th International Conference on Conversational User Interfaces. ACM, 13:1–13:10. https://doi.org/10.1145/3571884.3597143
- MeetingCoach: An Intelligent Dashboard for Supporting Effective & Inclusive Meetings. In Proceedings of the 2021 ACM CHI Conference on Human Factors in Computing Systems. ACM, 252:1–252:13. https://doi.org/10.1145/3411764.3445615
- Anselm L Strauss and Juliet M Corbin. 1990. Basics of qualitative research: Grounded theory procedures and techniques. Sage Publications, Newbury Park, CA.
- The potential use of the authenticity scale as an outcome measure in executive coaching. The Coaching Psychologist 7, 1 (2011), 16–25.
- Nicky Terblanche and Danie Cilliers. 2020. Factors that influence users’ adoption of being coached by an artificial intelligence coach. Philosophy of Coaching: An International Journal 5, 1 (2020), 61–70. https://doi.org/10.22316/poc/05.1.06
- Performance matters: Students’ perceptions of artificial intelligence coach adoption factors. Coaching: An International Journal of Theory, Research and Practice 16, 1 (2022), 100–114. https://doi.org/10.1080/17521882.2022.2094278
- User acceptance of information technology: Toward a unified view. MIS Quarterly 27, 3 (2003), 425–478. https://doi.org/10.2307/30036540
- SelVReflect: A guided VR experience fostering reflection on personal challenges. In Proceedings of the 2023 ACM CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 323:1–323:17. https://doi.org/10.1145/3544548.3580763
- Fridolin: Participatory Design and Evaluation of a Nutrition Chatbot for Older aAults. i-com 22, 1 (2023), 33–52. https://doi.org/10.1515/ICOM-2022-0042
- Virtual coaches: Background, theories, and future research directions. Business & Information Systems Engineering 64, 4 (2022), 515–528.
- Co-active coaching: New skills for coaching people toward success in work and life. Nicholas Brealey, Boston, MA.
- Robert Witherspoon and Randall P White. 1996. Executive coaching: A continuum of roles. Consulting Psychology Journal: Practice and Research 48, 2 (1996), 124–133. https://doi.org/10.1037/1061-4087.48.2.124
- Supporting autonomous motivation for physical activity with chatbots during the COVID-19 pandemic: Factorial experiment. JMIR Formative Research 7 (2023), e38500. https://doi.org/10.2196/38500
- A Script for Conversational Reflection Guidance: A Field Study on Developing Reflection Competence With Apprentices. IEEE Transactions on Learning Technologies 15, 5 (2022), 554–566. https://doi.org/10.1109/TLT.2022.3207226
- The authentic personality: A theoretical and empirical conceptualization and the development of the authenticity scale. Journal of Counseling Psychology 55, 3 (2008), 385–399. https://doi.org/10.1037/0022-0167.55.3.385
- The ”conversation” about loss: Understanding how chatbot technology was used in supporting people in grief. In Proceedings of the 2023 ACM CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 646:1–646:15. https://doi.org/10.1145/3544548.3581154
- Virtual avatar-based life coaching for children with autism spectrum disorder. Computer 53, 2 (2020), 26–34. https://doi.org/10.1109/MC.2019.2915979
Collections
Sign up for free to add this paper to one or more collections.
Summary
- The paper presents a blended coaching approach that integrates an LLM-powered chatbot with human coaching, validated by a two-week pilot study.
- The methodology involved setting coaching goals collaboratively using GPT-4-based interactions, while measuring engagement through message metrics.
- Results highlight chatbots' consistent engagement and scripted encouragement, yet emphasize the need for human coaches to deepen introspection.
Coaching Copilot: Blended Form of an LLM-Powered Chatbot and a Human Coach to Effectively Support Self-Reflection for Leadership Growth
This analysis covers the paper "Coaching Copilot: Blended Form of an LLM-Powered Chatbot and a Human Coach to Effectively Support Self-Reflection for Leadership Growth" (2405.15250). The research employs a novel approach that fuses LLM powered chatbots with human executive coaching to bolster individuals' self-reflection and leadership development.
Introduction to Blended Coaching Approach
The integration of chatbots in personal development, especially in contexts requiring deep introspection such as executive coaching, remains under-explored. This paper addresses this gap by integrating an LLM-powered chatbot with traditional executive coaching to enhance leadership growth. A two-week user paper involving ten coach-client pairs demonstrated chatbots' advantages such as 24/7 availability and consistent response capabilities while also showcasing the necessity for thoughtful design integration for successful human-chatbot collaboration.
Workshop with Professional Coaches
An initial workshop with eight professional coaches identified key areas for chatbot integration. Coaches appreciated the potential for chatbots to support text-based coaching between face-to-face sessions, reducing their burden associated with maintaining frequent client communication.
Through interactions with a GPT-4-based chatbot prototype, insights were gained into the practical roles of chatbots and human coaches. Coaches favored text-based complementary roles for chatbots, ensuring human presence remains crucial for navigating complex client dynamics.
Figure 1: The prototype interface to use an LLM-powered chatbot coach. In the user paper, this prototype was provided to clients and used at their own pace for two weeks to support their reflection to accomplish their professional goals. The presented conversation was derived from the use of a prototype by one of the authors to demonstrate its behavior.
Evaluation Study: Methodology and Interaction
The developed prototype facilitated interaction with the GPT-4 model to emulate a human-like coaching dialogue. Coaching goals were collaboratively set between coaches and clients before commencing the text-based chatbot interaction. Participants engaged in multiple sessions wherein they conversed with a chatbot coach, with results measured in terms of self-reflection and behavioral intention usage metrics.
Figure 2: The procedure of the user paper using the developed chatbot text coach as a supplement to the regular coaching.
Results: Chatbot Coach Efficacy
Figure 3: (Left) The number of messages each client sent to the chatbot coach per session. (Right) The total length of the messages per session in the number of characters. The blue area highlights indicate the 95\% confidence interval of the average value.
The number of messages and conversation lengths were consistent across sessions, indicating sustained engagement. Analysis of message exchanges pointed out that chatbots effectively leveraged acknowledgment techniques, keeping clients motivated toward goal achievement.
Figure 4: Example from one client's messages with the chatbot coach. It is observed that the chatbot often acknowledges the client's actions and asks questions that can further break down the problems they face.
During interviews, clients expressed appreciation for the convenience and sustained motivation facilitated by the chat-based interaction, which offered flexible engagement and action-promoting dialogue. The chatbot's scripted encouragement and logical progression helped clients clarify action steps, effectively maintaining engagement in their personal growth journey.
Limitations and Human Coach Intervention
Despite chatbots' ability to promote action steps, clients and coaches recognized their current limitations in fostering deeper introspection and posing more challenging questions. Such capabilities are crucial for executive coaching, emphasizing the necessity of a human coach's role in challenging clients' beliefs and behaviors for profound reflection.
The paper also highlighted the importance of accurately setting clients' goals and maintaining human involvement, ensuring motivation, and readiness.
Figure 5: The transition of the score of the participated clients' behavioral intention to use the chatbot coach. The blue area highlights the 95\% confidence interval of the average score.
Figure 6: The transitions of the scores of the three factors of the participated clients' authenticity. The blue area highlights the 95\% confidence interval of the average score.
Towards a Collaborative Framework
The paper identified key guidelines to enhance productive blending of human and LLM-powered chatbot coaches. The synergistic approach capitalizes on a chatbot's ubiquitous presence while addressing the limitations of LLMs in deepening reflection. Emphasizing the necessity of clear goal-setting and human coaches' commitment, the framework is presented visually below.
Figure 7: Guideline for the blended approach of human and LLM-powered chatbot coach for leadership growth in executive coaching, synthesized through our paper. The human coach is suggested to follow the steps to introduce the chatbot coach successfully and to augment clients' reflection: 1) introducing the chatbot as complementary text coaching, 2) conducting pre-session to foster a client's readiness before initiating chatbot-driven coaching, 3) maintaining a communication channel with the client during the text coaching period, and 4) monitoring client progress and identify moments to intervene.
Conclusion
The exploration of LLM-powered chatbot coaches as complements within executive coaching showcases their competency in promoting self-reflection and action planning, enhancing the effectiveness of the self-reflective process for leadership growth. This paper sheds light on the vast potential of LLM-integrated chatbots as powerful agents for single-loop learning and fostering continuous client engagement. The need for a blended coaching model, which fuses the strengths of both human and chatbot coaches, is deemed essential for navigating the limitations in double-loop learning and deep introspective dialogue. Future research directions include broadening the participant pool, quantifying outcomes in controlled settings, and exploring ethical implications, particularly concerning data privacy and potential biases. The findings offer valuable insights for advancing chatbot-based executive coaching and set a foundational direction for future research in utilizing LLM-powered conversational agents in HRD and broader self-reflection support systems within HCI.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.