2000 character limit reached
Bias in Emotion Recognition with ChatGPT (2310.11753v2)
Published 18 Oct 2023 in cs.RO and cs.CL
Abstract: This technical report explores the ability of ChatGPT in recognizing emotions from text, which can be the basis of various applications like interactive chatbots, data annotation, and mental health analysis. While prior research has shown ChatGPT's basic ability in sentiment analysis, its performance in more nuanced emotion recognition is not yet explored. Here, we conducted experiments to evaluate its performance of emotion recognition across different datasets and emotion labels. Our findings indicate a reasonable level of reproducibility in its performance, with noticeable improvement through fine-tuning. However, the performance varies with different emotion labels and datasets, highlighting an inherent instability and possible bias. The choice of dataset and emotion labels significantly impacts ChatGPT's emotion recognition performance. This paper sheds light on the importance of dataset and label selection, and the potential of fine-tuning in enhancing ChatGPT's emotion recognition capabilities, providing a groundwork for better integration of emotion analysis in applications using ChatGPT.
- K. Terada, A. Yamauchi, and A. Ito, “Artificial emotion expression for a robot by dynamic color change,” in 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication, pp. 314–321, IEEE, 2012.
- D. Löffler, N. Schmidt, and R. Tscharn, “Multimodal expression of artificial emotion in social robots using color, motion and sound,” in Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, pp. 334–343, 2018.
- B. Koptyra, A. Ngo, Ł. Radliński, and J. Kocoń, “Clarin-emo: Training emotion recognition models using human annotation and chatgpt,” in International Conference on Computational Science, pp. 365–379, Springer, 2023.
- Y. Zhang, Z. Wang, and J. Shang, “Clusterllm: Large language models as a guide for text clustering,” arXiv preprint arXiv:2305.14871, 2023.
- K. Yang, S. Ji, T. Zhang, Q. Xie, and S. Ananiadou, “On the evaluations of chatgpt and emotion-enhanced prompting for mental health analysis,” arXiv preprint arXiv:2304.03347, 2023.
- B. Lamichhane, “Evaluation of chatgpt for nlp-based mental health applications,” arXiv preprint arXiv:2303.15727, 2023.
- M. M. Amin, E. Cambria, and B. W. Schuller, “Will affective computing emerge from foundation models and general ai? a first evaluation on chatgpt,” arXiv preprint arXiv:2303.03186, 2023.
- C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang, “Is chatgpt a general-purpose natural language processing task solver?,” arXiv preprint arXiv:2302.06476, 2023.
- Q. Zhong, L. Ding, J. Liu, B. Du, and D. Tao, “Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert,” arXiv preprint arXiv:2302.10198, 2023.
- J. Kocoń, I. Cichecki, O. Kaszyca, M. Kochanek, D. Szydło, J. Baran, J. Bielaniewicz, M. Gruza, A. Janz, K. Kanclerz, et al., “Chatgpt: Jack of all trades, master of none,” Information Fusion, p. 101861, 2023.
- W. Zhao, Y. Zhao, X. Lu, S. Wang, Y. Tong, and B. Qin, “Is chatgpt equipped with emotional dialogue capabilities?,” arXiv preprint arXiv:2304.09582, 2023.
- S. S. Sohail, F. Farhat, Y. Himeur, M. Nadeem, D. Ø. Madsen, Y. Singh, S. Atalla, and W. Mansoor, “Decoding chatgpt: a taxonomy of existing research, current challenges, and possible future directions,” Journal of King Saud University-Computer and Information Sciences, p. 101675, 2023.
- OpenAI, “Chatgpt.” https://openai.com/blog/chatgpt. Accessed: 2023-08-05.
- S. Poria, N. Majumder, R. Mihalcea, and E. Hovy, “Emotion recognition in conversation: Research challenges, datasets, and recent advances,” IEEE Access, vol. 7, pp. 100943–100953, 2019.
- C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” Language resources and evaluation, vol. 42, pp. 335–359, 2008.
- S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea, “Meld: A multimodal multi-party dataset for emotion recognition in conversations,” arXiv preprint arXiv:1810.02508, 2018.
- S. M. Zahiri and J. D. Choi, “Emotion detection on tv show transcripts with sequence-based convolutional neural networks,” in Workshops at the thirty-second aaai conference on artificial intelligence, 2018.
- Y. Li, H. Su, X. Shen, W. Li, Z. Cao, and S. Niu, “Dailydialog: A manually labelled multi-turn dialogue dataset,” arXiv preprint arXiv:1710.03957, 2017.
- J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psychological measurement, vol. 20, no. 1, pp. 37–46, 1960.
- R. Mao, Q. Liu, K. He, W. Li, and E. Cambria, “The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection,” IEEE Transactions on Affective Computing, 2022.
- Z. Elyoseph, D. Hadar-Shoval, K. Asraf, and M. Lvovsky, “Chatgpt outperforms humans in emotional awareness evaluations,” Frontiers in Psychology, vol. 14, p. 1199058, 2023.
- K. Uludag, “Testing creativity of chatgpt in psychology: Interview with chatgpt,” Available at SSRN 4390872, 2023.
- K. He, R. Mao, Q. Lin, Y. Ruan, X. Lan, M. Feng, and E. Cambria, “A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics,” arXiv preprint arXiv:2310.05694, 2023.
- P. Ekman, “Are there basic emotions?,” 1992.
- R. Plutchik, “A general psychoevolutionary theory of emotion,” in Theories of emotion, pp. 3–33, Elsevier, 1980.
- Naoki Wake (34 papers)
- Atsushi Kanehira (24 papers)
- Kazuhiro Sasabuchi (29 papers)
- Jun Takamatsu (33 papers)
- Katsushi Ikeuchi (40 papers)