Chain-of-Interaction: Enhancing Large Language Models for Psychiatric Behavior Understanding by Dyadic Contexts (2403.13786v2)
Abstract: Automatic coding patient behaviors is essential to support decision making for psychotherapists during the motivational interviewing (MI), a collaborative communication intervention approach to address psychiatric issues, such as alcohol and drug addiction. While the behavior coding task has rapidly adapted machine learning to predict patient states during the MI sessions, lacking of domain-specific knowledge and overlooking patient-therapist interactions are major challenges in developing and deploying those models in real practice. To encounter those challenges, we introduce the Chain-of-Interaction (CoI) prompting method aiming to contextualize LLMs for psychiatric decision support by the dyadic interactions. The CoI prompting approach systematically breaks down the coding task into three key reasoning steps, extract patient engagement, learn therapist question strategies, and integrates dyadic interactions between patients and therapists. This approach enables LLMs to leverage the coding scheme, patient state, and domain knowledge for patient behavioral coding. Experiments on real-world datasets can prove the effectiveness and flexibility of our prompting method with multiple state-of-the-art LLMs over existing prompting baselines. We have conducted extensive ablation analysis and demonstrate the critical role of dyadic interactions in applying LLMs for psychotherapy behavior understanding.
- M. Freeman, “The world mental health report: transforming mental health for all,” World Psychiatry, vol. 21, pp. 391–392, 10 2022. [Online]. Available: https://onlinelibrary.wiley.com/doi/10.1002/wps.21018
- T. B. Moyers, T. Martin, J. K. Manuel, and W. R. Miller, “The motivational interviewing treatment integrity (miti) code: Version 2.0,” 2003.
- D. C. Atkins, M. Steyvers, Z. E. Imel, and P. Smyth, “Scaling up the evaluation of psychotherapy: evaluating motivational interviewing fidelity via statistical text classification,” Implementation Science, vol. 9, p. 49, 2014.
- M. Tanana, K. A. Hallgren, Z. E. Imel, D. C. Atkins, and V. Srikumar, “A comparison of natural language processing methods for automated coding of motivational interviewing,” Journal of Substance Abuse Treatment, vol. 65, pp. 43–50, 2016, motivational Interviewing in Substance Use Treatment. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0740547216000222
- X. Huang, L. Liu, K. Carey, J. Woolley, S. Scherer, and B. Borsari, “Modeling temporality of human intentions by domain adaptation,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, Eds. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 696–701. [Online]. Available: https://aclanthology.org/D18-1074
- J. Cao, M. Tanana, Z. Imel, E. Poitras, D. Atkins, and V. Srikumar, “Observing dialogue in therapy: Categorizing and forecasting behavioral codes,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez, Eds. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 5599–5611. [Online]. Available: https://aclanthology.org/P19-1563
- J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio, Eds. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186.
- L. Tavabi et al., “Multimodal automatic coding of client behavior in motivational interviewing,” in Proceedings of the 2020 International Conference on Multimodal Interaction, ser. ICMI ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 406–413.
- J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” 2023. [Online]. Available: https://arxiv.org/abs/2303.08774
- Ouyang et al., “Training language models to follow instructions with human feedback,” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. New Orleans, Louisiana: Curran Associates, Inc., 2022, pp. 27 730–27 744. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf
- H. Touvron et al., “Llama 2: Open foundation and fine-tuned chat models,” 2023. [Online]. Available: https://arxiv.org/abs/2307.09288
- K. Yang, T. Zhang, Z. Kuang, Q. Xie, S. Ananiadou, and J. Huang, “Mentallama: Interpretable mental health analysis on social media with large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2309.13567
- S. Ji, T. Zhang, L. Ansari, J. Fu, P. Tiwari, and E. Cambria, “MentalBERT: Publicly available pretrained language models for mental healthcare,” in Proceedings of the Thirteenth Language Resources and Evaluation Conference, N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, J. Odijk, and S. Piperidis, Eds. Marseille, France: European Language Resources Association, Jun. 2022, pp. 7184–7190.
- J. M. Liu, D. Li, H. Cao, T. Ren, Z. Liao, and J. Wu, “Chatcounselor: A large language models for mental health support,” 2023. [Online]. Available: https://arxiv.org/abs/2309.15461
- J. Wei et al., “Chain-of-thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 24 824–24 837. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf
- Y. Zhou, X. Geng, T. Shen, C. Tao, G. Long, J.-G. Lou, and J. Shen, “Thread of thought unraveling chaotic contexts,” 2023. [Online]. Available: https://arxiv.org/abs/2311.08734
- Y. Zhang, M. Zhu, Y. Gong, and R. Ding, “Optimizing science question ranking through model and retrieval-augmented generation,” International Journal of Computer Science and Information Technology, vol. 1, no. 1, pp. 124–130, 2023.
- Z. Jing, Y. Su, Y. Han, B. Yuan, H. Xu, C. Liu, K. Chen, and M. Zhang, “When large language models meet vector databases: A survey,” 2024. [Online]. Available: https://arxiv.org/abs/2402.01763
- E. Almazrouei et al., “The falcon series of open language models,” 2023. [Online]. Available: https://arxiv.org/abs/2311.16867
- A. Q. Jiang et al., “Mistral 7b,” 2023. [Online]. Available: https://arxiv.org/abs/2310.06825
- OpenAI, “Chatgpt: Optimizing language models for dialogue,” https://openai.com/blog/chatgpt/, 2022, accessed: 2023-07-24.
- B. Borsari et al., “In-session processes of brief motivational interventions in two trials with mandated college students.” Journal of consulting and clinical psychology, vol. 83, pp. 56–67, 2 2015.
- W. R. Miller, T. B. Moyers, D. Ernst, and P. Amrhein, “Manual for the motivational interviewing skill code (misc) version 2.1,” Substance Abuse and Addiction (CASAA), University of New Mexico, vol. 8, pp. 901–4, 2008.
- W. R. Miller and G. S. Rose, “Toward a theory of motivational interviewing.” The American psychologist, vol. 64, pp. 527–37, 9 2009.
- M. Magill and K. A. Hallgren, “Mechanisms of behavior change in motivational interviewing: do we understand how mi works?” Current opinion in psychology, vol. 30, pp. 1–5, 12 2019.
- K. Mishra, P. Priya, M. Burja, and A. Ekbal, “e-THERAPIST: I suggest you to cultivate a mindset of positivity and nurture uplifting thoughts,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 13 952–13 967. [Online]. Available: https://aclanthology.org/2023.emnlp-main.861
- T. Tran et al., “Multimodal analysis and assessment of therapist empathy in motivational interviews,” in Proceedings of the 25th International Conference on Multimodal Interaction, ser. ICMI ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 406–415.
- A. Gagneur, “Respiratory syncytial virus: Motivational interviewing: A powerful tool to address vaccine hesitancy,” Canada Communicable Disease Report, vol. 46, no. 4, p. 93, 2020.
- S. Rollnick, W. R. Miller, C. C. Butler, and M. S. Aloia, “Motivational interviewing in health care: Helping patients change behavior,” COPD: Journal of Chronic Obstructive Pulmonary Disease, vol. 5, pp. 203–203, 1 2008.
- S. A. Cole, D. Sannidhi, Y. T. Jadotte, and A. Rozanski, “Using motivational interviewing and brief action planning for adopting and maintaining positive health behaviors,” Progress in Cardiovascular Diseases, vol. 77, pp. 86–94, 2023.
- W. Wang, V. W. Zheng, H. Yu, and C. Miao, “A survey of zero-shot learning: Settings, methods, and applications,” ACM Trans. Intell. Syst. Technol., vol. 10, no. 2, jan 2019.
- W. Yin, J. Hay, and D. Roth, “Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan, Eds. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 3914–3923. [Online]. Available: https://aclanthology.org/D19-1404
- T. B. Brown et al., “Language models are few-shot learners,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS’20. Red Hook, NY, USA: Curran Associates Inc., 2020.
- X. Liu et al., “Large language models are few-shot health learners,” 2023. [Online]. Available: https://arxiv.org/abs/2305.15525
- T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” 2023. [Online]. Available: https://arxiv.org/abs/2205.11916
- K. Cobbe et al., “Training verifiers to solve math word problems,” 2021. [Online]. Available: https://arxiv.org/abs/2110.14168
- Y. Bai et al., “Constitutional ai: Harmlessness from ai feedback,” 2022. [Online]. Available: https://arxiv.org/abs/2212.08073
- P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” in Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf
- N. Stiennon et al., “Learning to summarize with human feedback,” in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 3008–3021. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/1f89885d556929e98d3ef9b86448f951-Paper.pdf
- G. Penedo et al., “The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only,” 2023. [Online]. Available: https://arxiv.org/abs/2306.01116
- T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 16 344–16 359. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf
- A. Wang et al., “Superglue: A stickier benchmark for general-purpose language understanding systems,” in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2019/file/4496bf24afe7fab6f046bf4923da8de6-Paper.pdf
- M. Chen et al., “Evaluating large language models trained on code,” 2021. [Online]. Available: https://arxiv.org/abs/2107.03374
- A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, “The curious case of neural text degeneration,” in International Conference on Learning Representations, 2020. [Online]. Available: https://openreview.net/forum?id=rygGQyrFvH
- X. Wang et al., “”my answer is c”: First-token probabilities do not match text answers in instruction-tuned language models,” 2024. [Online]. Available: https://arxiv.org/abs/2402.14499
- B. Xiao, C. Huang, Z. E. Imel, D. C. Atkins, P. Georgiou, and S. S. Narayanan, “A technology prototype system for rating therapist empathy from audio recordings in addiction counseling,” PeerJ Computer Science, vol. 2, p. e59, 4 2016.
- Z. L. Inbar and Elyoseph, “Suicide risk assessments through the eyes of chatgpt-3.5 versus chatgpt-4: Vignette study,” JMIR Ment Health, vol. 10, p. e51232, 9 2023.
- D. Can, P. G. Georgiou, D. C. Atkins, and S. S. Narayanan, “A case study: detecting counselor reflections in psychotherapy for addictions using linguistic features,” in Proc. Interspeech 2012, 2012, pp. 2254–2257.
- K. Singla et al., “Using prosodic and lexical information for learning utterance-level behaviors in psychotherapy.” Interspeech, vol. 2018, pp. 3413–3417, 9 2018.
- J. Gibson, D. Can, P. G. Georgiou, D. C. Atkins, and S. S. Narayanan, “Attention networks for modeling behaviors in addiction counseling,” in Interspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017, F. Lacerda, Ed. ISCA, 2017, pp. 3251–3255.
- J. Gibson, D. C. Atkins, T. A. Creed, Z. Imel, P. Georgiou, and S. Narayanan, “Multi-label multi-task deep learning for behavioral coding,” IEEE Transactions on Affective Computing, vol. 13, no. 1, pp. 508–518, 2022.
- K. Singla, Z. Chen, D. Atkins, and S. Narayanan, “Towards end-2-end learning for predicting behavior codes from spoken utterances in psychotherapy conversations,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds. Online: Association for Computational Linguistics, Jul. 2020, pp. 3797–3803. [Online]. Available: https://aclanthology.org/2020.acl-main.351
- D. Demszky, D. Yang, D. S. Yeager, C. J. Bryan, M. Clapper, S. Chandhok, J. C. Eichstaedt, C. Hecht, J. Jamieson, M. Johnson, M. Jones, D. Krettek-Cobb, L. Lai, N. JonesMitchell, D. C. Ong, C. S. Dweck, J. J. Gross, and J. W. Pennebaker, “Using large language models in psychology,” Nature Reviews Psychology, vol. 2, no. 11, pp. 688–701, 2023. [Online]. Available: https://doi.org/10.1038/s44159-023-00241-5
- T. He et al., “Towards a psychological generalist ai: A survey of current applications of large language models and future prospects,” 2023. [Online]. Available: https://arxiv.org/abs/2312.04578
- R. Lou, K. Zhang, and W. Yin, “A comprehensive survey on instruction following,” 2024. [Online]. Available: https://arxiv.org/abs/2303.10475
- R. Lou and W. Yin, “Toward zero-shot instruction following,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, N. Falk, S. Papi, and M. Zhang, Eds. St. Julian’s, Malta: Association for Computational Linguistics, Mar. 2024, pp. 50–60. [Online]. Available: https://aclanthology.org/2024.eacl-srw.5
- Y. Li, L. Zihan, K. Zhang, D. Ruilong, S. Jiang, and Y. Zhang, “Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge,” Cureus, vol. 15, no. 6, 2023. [Online]. Available: https://www.proquest.com/scholarly-journals/chatdoctor-medical-chat-model-fine-tuned-on-large/docview/2844019753/se-2
- T. Lai et al., “Psy-llm: Scaling up global mental health psychological services with ai-based large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2307.11991
- H. Qi et al., “Supervised learning and large language model benchmarks on mental health datasets: Cognitive distortions and suicidal risks in chinese social media,” 2023. [Online]. Available: https://arxiv.org/abs/2309.03564
- Z. Chen, Y. Lu, and W. Wang, “Empowering psychotherapy with large language models: Cognitive distortion detection through diagnosis of thought prompting,” in Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 4295–4304. [Online]. Available: https://aclanthology.org/2023.findings-emnlp.284
- Guangzeng Han (5 papers)
- Weisi Liu (7 papers)
- Xiaolei Huang (45 papers)
- Brian Borsari (1 paper)