Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 63 tok/s
Gemini 2.5 Pro 49 tok/s Pro
GPT-5 Medium 11 tok/s Pro
GPT-5 High 10 tok/s Pro
GPT-4o 83 tok/s Pro
Kimi K2 139 tok/s Pro
GPT OSS 120B 438 tok/s Pro
Claude Sonnet 4 38 tok/s Pro
2000 character limit reached

IMBUE: Improving Interpersonal Effectiveness through Simulation and Just-in-time Feedback with Human-Language Model Interaction (2402.12556v1)

Published 19 Feb 2024 in cs.HC and cs.CL

Abstract: Navigating certain communication situations can be challenging due to individuals' lack of skills and the interference of strong emotions. However, effective learning opportunities are rarely accessible. In this work, we conduct a human-centered study that uses LLMs to simulate bespoke communication training and provide just-in-time feedback to support the practice and learning of interpersonal effectiveness skills. We apply the interpersonal effectiveness framework from Dialectical Behavioral Therapy (DBT), DEAR MAN, which focuses on both conversational and emotional skills. We present IMBUE, an interactive training system that provides feedback 25% more similar to experts' feedback, compared to that generated by GPT-4. IMBUE is the first to focus on communication skills and emotion management simultaneously, incorporate experts' domain knowledge in providing feedback, and be grounded in psychology theory. Through a randomized trial of 86 participants, we find that IMBUE's simulation-only variant significantly improves participants' self-efficacy (up to 17%) and reduces negative emotions (up to 25%). With IMBUE's additional just-in-time feedback, participants demonstrate 17% improvement in skill mastery, along with greater enhancements in self-efficacy (27% more) and reduction of negative emotions (16% more) compared to simulation-only. The improvement in skill mastery is the only measure that is transferred to new and more difficult situations; situation specific training is necessary for improving self-efficacy and emotion reduction.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Selecting shots for demographic fairness in few-shot learning with large language models. arXiv preprint arXiv:2311.08472.
  2. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning.
  3. Large-scale analysis of counseling conversations: An application of natural language processing to mental health. Transactions of the Association for Computational Linguistics.
  4. Leveraging ai for democratic discourse: Chat interventions can improve online political conversations at scale. Proceedings of the National Academy of Sciences, 120(41):e2311627120.
  5. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351.
  6. Aaron T Beck. 1979. Cognitive therapy and the emotional disorders. Penguin.
  7. Aaron T Beck. 1996. Beyond belief: A theory of modes, personality, and psychopathology.
  8. Language models are few-shot learners. ArXiv, abs/2005.14165.
  9. Using large language models in psychology. Nature Reviews Psychology, 2:688 – 701.
  10. The faiss library.
  11. Alpacafarm: A simulation framework for methods that learn from human feedback. ArXiv, abs/2305.14387.
  12. Towards measuring the representation of subjective global opinions in language models. ArXiv, abs/2306.16388.
  13. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pages 226–231.
  14. Antonio Freitas and Peter Salovey. 2000. Regulating emotion in the short and long term. Psychological Inquiry, 11(3):178–179.
  15. Robert M Gagne. 1965. The analysis of instructional objectives for the design of instruction. Teaching machines and programmed learning II: Data and directions, pages 21–65.
  16. Knowledge-aware assessment of severity of suicide risk for early intervention. In WWW.
  17. Bias Runs Deep: Implicit reasoning biases in persona-assigned LLMs. In The Twelfth International Conference on Learning Representations.
  18. Peter Hartley. 2002. Interpersonal communication. Routledge.
  19. Fiona A L Henderson. 2016. Difficult conversations on the frontline. Managing the tensions between care and control: are communication skills enough? Ph.D. thesis, University of Essex.
  20. Systematic methodological review: developing a framework for a qualitative semi-structured interview guide. Journal of advanced nursing, 72(12):2954–2965.
  21. Identifying therapist conversational actions across diverse psychotherapeutic approaches. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology.
  22. Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of psychology.
  23. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  24. Gendered mental health stigma in masked language models. In EMNLP.
  25. Marsha Linehan. 2014. DBT Skills training manual. Guilford Publications.
  26. Training socially aligned language models in simulated human society. ArXiv, abs/2305.16960.
  27. Improving interpersonal communication by simulating audiences with language models. arXiv preprint arXiv:2311.00687.
  28. Clinicians’ strategies for managing their emotions during difficult healthcare conversations. Patient Education and Counseling, 99(9):1461–1466. Communication in Healthcare: Best papers from the International Conference on Communication in Healthcare, New Orleans, LA, USA, October 25-28, 2015.
  29. Mark Olfson. 2016. Building the mental health workforce capacity needed to treat adults with serious mental illnesses. Health Affairs, 35(6):983–990. PMID: 27269013.
  30. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22.
  31. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1–18.
  32. Pair: Prompt-aware margin ranking for counselor reflection scoring in motivational interviewing. In EMNLP.
  33. What makes a good counselor? learning to distinguish between high-quality and low-quality counseling conversations. In ACL.
  34. Robert Plutchik. 1980. A general psychoevolutionary theory of emotion.
  35. Marshall B Rosenberg and Deepak Chopra. 2015. Nonviolent communication: A language of life: Life-changing tools for healthy relationships. PuddleDancer Press.
  36. Whose opinions do language models reflect? ArXiv, abs/2303.17548.
  37. Modeling motivational interviewing strategies on an online peer-to-peer counseling platform. CSCW.
  38. Rehearsal: Simulating conflict to teach conflict resolution. arXiv preprint arXiv:2309.12309.
  39. Engagement patterns of peer-to-peer interactions on mental health platforms. In ICWSM.
  40. Human–AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support. Nature Machine Intelligence.
  41. A computational approach to understanding empathy expressed in text-based mental health support. In EMNLP.
  42. Cognitive reframing of negative thoughts through human-language model interaction. arXiv preprint arXiv:2305.02466.
  43. The science of self-report: Implications for research and practice. Psychology Press.
  44. Difficult conversations: How to discuss what matters most. Penguin.
  45. The effect of moderation on online mental health conversations. In ICWSM.
  46. A survey on large language model based autonomous agents. ArXiv, abs/2308.11432.
  47. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903.
  48. Expressive interviewing: A conversational system for coping with covid-19. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020.
  49. Role of technology in self-assessment and feedback among hospitalist physicians: Semistructured interviews and thematic analysis. Journal of Medical Internet Research, 22.
  50. Justine Zhang and Cristian Danescu-Niculescu-Mizil. 2020. Balancing objectives in counseling conversations: Advancing forwards or looking backwards. In ACL.
  51. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
  52. Navigating the grey area: Expressions of overconfidence and uncertainty in language models. ArXiv, abs/2302.13439.
  53. Sotopia: Interactive evaluation for social intelligence in language agents. ArXiv, abs/2310.11667.
  54. Can large language models transform computational social science? arXiv preprint arXiv:2305.03514.
  55. Can large language models transform computational social science? ArXiv, abs/2305.03514.
Citations (4)
View on Semantic Scholar

Summary

  • The paper introduces Imbue, an AI system that uses language model simulation and just-in-time feedback based on psychological frameworks to improve interpersonal communication and emotional regulation skills.
  • A randomized controlled trial shows Imbue increases self-efficacy in handling interpersonal situations by up to 27% and reduces negative emotional responses by 16% compared to simulation alone.
  • Imbue's technical contributions include generating feedback significantly closer to expert quality and integrating expert-annotated data, laying groundwork for more accessible, flexible AI-driven training solutions.

Imbue: Advancements in AI-Driven Interpersonal Communication Coaching

The paper "Imbue: Improving Interpersonal Effectiveness through Simulation and Just-in-time Feedback with Human-LLM Interaction" offers a methodologically robust exploration of improving communication skills and emotional regulation using AI-based systems. This paper taps into computational linguistics and human-computer interaction to address the ongoing challenge of accessible training for effective interpersonal communication.

Methodological Framework

The research leverages the DEAR MAN framework from Dialectical Behavioral Therapy (DBT) to provide a structured approach to interpersonal effectiveness. This includes not only teaching conversational strategies but also aiding in emotional regulation. This dual focus is crucial given that emotional disruptions can undermine communication abilities, even in skilled communicators.

Imbue, the proposed interactive system, simulates realistic communication scenarios with the aid of LMs, such as GPT-4, combined with human inputs to create specialized training. The system provides just-in-time feedback based on the DEAR MAN framework. This comprises two primary tasks: offering preparatory skills suggestions before interaction and providing feedback post-interaction.

Empirical Validation

The research is underpinned by a rigorous evaluation involving a randomized controlled trial with 86 participants. Participants engaged with two versions of the Imbue system: one focused solely on simulation and the other combining simulation with real-time feedback. Quantitative assessments revealed that the simulation coupled with feedback led to significant improvements in several key areas:

  • Participants’ self-efficacy in handling interpersonal situations increased by up to 27%, a notable gain compared to the simulation-only group's 17%.
  • Negative emotional responses were reduced by 16%, reinforcing the system's efficacy in both psychological and practical realms of communication.

Technical Contributions

The paper also provides several noteworthy contributions to the domain of human-AI interaction:

  1. Enhanced Feedback Mechanism: The system’s feedback was tailored to mimic expert feedback significantly closer than standard models, exceeding a baseline model by 24.8%.
  2. Data Collection and Annotation: A dataset annotated by psychology experts highlights the integration of expert domain knowledge into the simulation and feedback, enabling fine-grained adjustments.
  3. Training Outcomes Across Contexts: Notably, while skill mastery effectively transferred to new scenarios, emotional regulation and self-efficacy improvements were situation-specific, suggesting a need for continuous and context-specific training interventions.

Implications and Future Directions

From a theoretical perspective, this research advances the understanding of how LLMs can be harnessed for behavioral and therapeutic applications beyond traditional NLP tasks. Practically, Imbue's methodology promises more accessible and flexible communication training solutions, potentially reducing reliance on in-person therapies and enabling scalable learning environments.

Looking forward, opportunities for further exploration include expanding the cultural and situational adaptability of the model, refining feedback mechanisms to include more nuanced psychological insights, and extending training to cover a broader spectrum of interpersonal skills. Imbue could serve as a springboard for future AI-driven educational tools that tailor learning experiences to individual user needs, incorporating a feedback loop informed by continuous user interaction data.

The integration of AI in human-centric domains presents exciting potential for personalized learning experiences, and Imbue's structured approach lays foundational groundwork in this expanding area of research.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 5 posts and received 116 likes.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up to View