Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

LineConGraphs: Line Conversation Graphs for Effective Emotion Recognition using Graph Neural Networks (2312.03756v1)

Published 4 Dec 2023 in cs.CL, cs.AI, cs.HC, and cs.LG

Abstract: Emotion Recognition in Conversations (ERC) is a critical aspect of affective computing, and it has many practical applications in healthcare, education, chatbots, and social media platforms. Earlier approaches for ERC analysis involved modeling both speaker and long-term contextual information using graph neural network architectures. However, it is ideal to deploy speaker-independent models for real-world applications. Additionally, long context windows can potentially create confusion in recognizing the emotion of an utterance in a conversation. To overcome these limitations, we propose novel line conversation graph convolutional network (LineConGCN) and graph attention (LineConGAT) models for ERC analysis. These models are speaker-independent and built using a graph construction strategy for conversations -- line conversation graphs (LineConGraphs). The conversational context in LineConGraphs is short-term -- limited to one previous and future utterance, and speaker information is not part of the graph. We evaluate the performance of our proposed models on two benchmark datasets, IEMOCAP and MELD, and show that our LineConGAT model outperforms the state-of-the-art methods with an F1-score of 64.58% and 76.50%. Moreover, we demonstrate that embedding sentiment shift information into line conversation graphs further enhances the ERC performance in the case of GCN models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. G. Taylor, “Emotion recognition in human-computer interaction,” IEEE Signal processing magazine, vol. 18, no. 1, pp. 32–80, 2001.
  2. B. Schuller, “Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends,” Communications of the ACM, vol. 61, pp. 90–99, 04 2018.
  3. S. Poria, N. Majumder, R. Mihalcea, and E. Hovy, “Emotion recognition in conversation: Research challenges, datasets, and recent advances,” IEEE Access, vol. 7, pp. 100 943–100 953, 2019.
  4. W. Shen, J. Chen, X. Quan, and Z. Xie, “Dialogxl: All-in-one xlnet for multi-party conversation emotion recognition,” 2020.
  5. Q. Gao, B. Cao, X. Guan, T. Gu, X. Bao, J. Wu, B. Liu, and J. Cao, “Emotion recognition in conversations with emotion shift detection based on multi-task learning,” Knowledge-Based Systems, vol. 248, p. 108861, 2022.
  6. S. Latif, R. Rana, S. Khalifa, R. Jurdak, and B. W. Schuller, “Multitask learning from augmented auxiliary data for improving speech emotion recognition,” IEEE Transactions on Affective Computing, 2022.
  7. C. Lu, Y. Zong, W. Zheng, Y. Li, C. Tang, and B. W. Schuller, “Domain invariant feature learning for speaker-independent speech emotion recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 2217–2230, 2022.
  8. J. Han, Z. Zhang, Z. Ren, and B. Schuller, “Emobed: Strengthening monomodal emotion recognition via training with crossmodal emotion embeddings,” IEEE Transactions on Affective Computing, vol. 12, pp. 553–564, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:198229728
  9. J. Li, Y. Liu, X. Wang, and Z. Zeng, “Cfn-esa: A cross-modal fusion network with emotion-shift awareness for dialogue emotion recognition,” 2023.
  10. W. Zhao, Y. Zhao, and X. Lu, “Cauain: Causal aware interaction network for emotion recognition in conversations,” in Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, L. D. Raedt, Ed.   International Joint Conferences on Artificial Intelligence Organization, 7 2022, pp. 4524–4530, main Track. [Online]. Available: https://doi.org/10.24963/ijcai.2022/628
  11. S. Lim and J. Kim, “Sapbert: Speaker-aware pretrained bert for emotion recognition in conversation,” Algorithms, vol. 16, no. 1, 2023. [Online]. Available: https://www.mdpi.com/1999-4893/16/1/8
  12. Y. Liu, J. Zhao, J. Hu, R. Li, and Q. Jin, “DialogueEIN: Emotion interaction network for dialogue affective analysis,” in Proceedings of the 29th International Conference on Computational Linguistics.   Gyeongju, Republic of Korea: International Committee on Computational Linguistics, Oct. 2022, pp. 684–693. [Online]. Available: https://aclanthology.org/2022.coling-1.57
  13. L. Yuan, G. Huang, F. Li, X. Yuan, C.-M. Pun, and G. Zhong, “Rba-gcn: Relational bilevel aggregation graph convolutional network for emotion recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 2325–2337, 2023.
  14. J. Hu, Y. Liu, J. Zhao, and Q. Jin, “Mmgcn: Multimodal fusion via deep graph convolution network for emotion recognition in conversation,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 5666–5675.
  15. T. Kim and P. Vossen, “Emoberta: Speaker-aware emotion recognition in conversation with roberta,” CoRR, vol. abs/2108.12009, 2021. [Online]. Available: https://arxiv.org/abs/2108.12009
  16. D. Hazarika, S. Poria, R. Mihalcea, E. Cambria, and R. Zimmermann, “ICON: Interactive conversational memory network for multimodal emotion detection,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.   Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 2594–2604. [Online]. Available: https://aclanthology.org/D18-1280
  17. D. Zhang, L. Wu, C. Sun, S. Li, Q. Zhu, and G. Zhou, “Modeling both context- and speaker-sensitive dependence for emotion detection in multi-speaker conversations,” in Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19.   International Joint Conferences on Artificial Intelligence Organization, 7 2019, pp. 5415–5421. [Online]. Available: https://doi.org/10.24963/ijcai.2019/752
  18. N. Majumder, S. Poria, D. Hazarika, R. Mihalcea, A. Gelbukh, and E. Cambria, “Dialoguernn: An attentive rnn for emotion detection in conversations,” in Proceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, 2019, pp. 6818–6825.
  19. D. Ghosal, N. Majumder, S. Poria, N. Chhaya, and A. Gelbukh, “Dialoguegcn: A graph convolutional neural network for emotion recognition in conversation,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 154–164.
  20. P. Saxena, Y. J. Huang, and S. Kurohashi, “Static and dynamic speaker modeling based on graph neural network for emotion recognition in conversation,” in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop.   Association for Computational Linguistics, Jul. 2022, pp. 247–253.
  21. Y. Wang, J. Zhang, J. Ma, S. Wang, and J. Xiao, “Contextualized emotion recognition in conversation as sequence tagging,” in Proceedings of the 21th annual meeting of the special interest group on discourse and dialogue, 2020, pp. 186–195.
  22. T. Wang, Y. Hou, D. Zhou, and Q. Zhang, “A contextual attention network for multimodal emotion recognition in conversation,” in 2021 International Joint Conference on Neural Networks (IJCNN), 2021, pp. 1–7.
  23. G. Tu, J. Wen, C. Liu, D. Jiang, and E. Cambria, “Context- and sentiment-aware networks for emotion recognition in conversation,” IEEE Transactions on Artificial Intelligence, vol. 3, no. 5, pp. 699–708, 2022.
  24. J. Li, X. Wang, and Z. Zeng, “A dual-stream recurrence-attention network with global-local awareness for emotion recognition in textual dialogue,” 2023.
  25. K. S. X. G. H. G. S. L. J. P. Lingfei Wu, Yu Chen and B. Long, “Graph neural networks for natural language processing: A survey,” CoRR, vol. abs/2106.06090, 2021. [Online]. Available: https://arxiv.org/abs/2106.06090
  26. M. W. Morris and D. Keltner, “How emotions work: The social functions of emotional expression in negotiations,” Research in Organizational Behavior, vol. 22, pp. 1–50, 2000. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0191308500220029
  27. Y.-J. Choi, Y.-W. Lee, and B.-G. Kim, “Residual-based graph convolutional network for emotion recognition in conversation for smart internet of things,” Big Data, vol. 9, no. 4, pp. 279–288, 2021.
  28. W. Li, L. Zhu, R. Mao, and E. Cambria, “Skier: A symbolic knowledge integrated model for conversational emotion recognition,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
  29. K. Yang, T. Zhang, H. Alhuzali, and S. Ananiadou, “Cluster-level contrastive learning for emotion recognition in conversations,” IEEE Transactions on Affective Computing, 2023.
  30. Y. Zhang, J. Wang, Y. Liu, L. Rong, Q. Zheng, D. Song, P. Tiwari, and J. Qin, “A multitask learning model for multimodal sarcasm, sentiment and emotion recognition in conversations,” Information Fusion, vol. 93, pp. 282–301, 2023.
  31. J. Li, X. Wang, G. Lv, and Z. Zeng, “Graphmft: A graph network based multimodal fusion technique for emotion recognition in conversation,” Neurocomputing, vol. 550, p. 126427, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0925231223005507
  32. W. L. Hamilton, “Graph representation learning,” Synthesis Lectures on Artificial Intelligence and Machine Learning, vol. 14, no. 3, pp. 1–159.
  33. J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun, “Graph neural networks: A review of methods and applications,” AI Open, vol. 1, pp. 57–81, 2020.
  34. Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and S. Y. Philip, “A comprehensive survey on graph neural networks,” IEEE Transactions on neural networks and learning systems, vol. 32, no. 1, pp. 4–24, 2020.
  35. Z. Zhang, P. Cui, and W. Zhu, “Deep learning on graphs: A survey,” CoRR, vol. abs/1812.04202, 2018. [Online]. Available: http://arxiv.org/abs/1812.04202
  36. C. Navarretta, “Mirroring facial expressions and emotions in dyadic conversations,” in Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), 2016, pp. 469–474.
  37. C. T. Duong, T. D. Hoang, H. T. H. Dang, Q. V. H. Nguyen, and K. Aberer, “On node features for graph neural networks,” 2019.
  38. J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun, “Graph neural networks: A review of methods and applications,” 2021.
  39. Z. Zhang, P. Cui, and W. Zhu, “Deep learning on graphs: A survey,” 2020.
  40. F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, “The graph neural network model,” IEEE Transactions on Neural Networks, vol. 20, no. 1, pp. 61–80, 2009.
  41. T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” arXiv preprint arXiv:1609.02907, 2016.
  42. M. Chen, Z. Wei, Z. Huang, B. Ding, and Y. Li, “Simple and deep graph convolutional networks,” in Proceedings of the 37th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, H. D. III and A. Singh, Eds., vol. 119.   PMLR, 13–18 Jul 2020, pp. 1725–1735. [Online]. Available: https://proceedings.mlr.press/v119/chen20v.html
  43. P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” arXiv preprint arXiv:1710.10903, 2017.
  44. S. Brody, U. Alon, and E. Yahav, “How attentive are graph attention networks?” arXiv preprint arXiv:2105.14491, 2021.
  45. S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea, “Meld: A multimodal multi-party dataset for emotion recognition in conversations,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 527–536.
  46. C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” Language resources and evaluation, vol. 42, pp. 335–359, 2008.
  47. C.-C. Hsu, S.-Y. Chen, C.-C. Kuo, T.-H. Huang, and L.-W. Ku, “Emotionlines: An emotion corpus of multi-party conversations,” in Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018.
  48. M. Fey and J. E. Lenssen, “Fast graph representation learning with PyTorch Geometric,” in ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
  49. D. Loureiro, F. Barbieri, L. Neves, L. E. Anke, and J. Camacho-Collados, “Timelms: Diachronic language models from twitter,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2022, pp. 251–260.
  50. D. Hazarika, S. Poria, R. Mihalcea, E. Cambria, and R. Zimmermann, “Icon: Interactive conversational memory network for multimodal emotion detection,” in Proceedings of the 2018 conference on empirical methods in natural language processing, 2018, pp. 2594–2604.
  51. J. Wen, D. Jiang, G. Tu, C. Liu, and E. Cambria, “Dynamic interactive multiview memory network for emotion recognition in conversation,” Information Fusion, vol. 91, pp. 123–133, 2023.
  52. S. Padi, S. O. Sadjadi, R. D. Sriram, and D. Manocha, “Improved speech emotion recognition using transfer learning and spectrogram augmentation,” in Proceedings of the 2021 International Conference on Multimodal Interaction, ser. ICMI ’21.   New York, NY, USA: Association for Computing Machinery, 2021, p. 645–652.
  53. S. Padi, S. O. Sadjadi, D. Manocha, and R. D. Sriram, “Multimodal emotion recognition using transfer learning from speaker recognition and bert-based models,” in The Speaker and Language Recognition Workshop, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Gokul S Krishnan (7 papers)
  2. Sarala Padi (4 papers)
  3. Craig S. Greenberg (3 papers)
  4. Balaraman Ravindran (100 papers)
  5. Dinesh Manoch (1 paper)
  6. Ram D. Sriram (4 papers)
Citations (1)

Summary

We haven't generated a summary for this paper yet.