Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances (2407.21315v3)
Abstract: Emotion recognition in speech is a challenging multimodal task that requires understanding both verbal content and vocal nuances. This paper introduces a novel approach to emotion detection using LLMs, which have demonstrated exceptional capabilities in natural language understanding. To overcome the inherent limitation of LLMs in processing audio inputs, we propose SpeechCueLLM, a method that translates speech characteristics into natural language descriptions, allowing LLMs to perform multimodal emotion analysis via text prompts without any architectural changes. Our method is minimal yet impactful, outperforming baseline models that require structural modifications. We evaluate SpeechCueLLM on two datasets: IEMOCAP and MELD, showing significant improvements in emotion recognition accuracy, particularly for high-quality audio data. We also explore the effectiveness of various feature representations and fine-tuning strategies for different LLMs. Our experiments demonstrate that incorporating speech descriptions yields a more than 2% increase in the average weighted F1 score on IEMOCAP (from 70.111% to 72.596%).
- Seed-asr: Understanding diverse speech and contexts with llm-based speech recognition. Preprint, arXiv:2407.04675.
- Speech prefix-tuning with rnnt loss for improving llm predictions. Preprint, arXiv:2406.14701.
- Iemocap: interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42:335–359.
- AudioChatLlama: Towards general-purpose speech abilities for LLMs. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5522–5532, Mexico City, Mexico. Association for Computational Linguistics.
- Yumeng Fu. 2024. Ckerc : Joint large language models with commonsense knowledge for emotion recognition in conversation. Preprint, arXiv:2403.07260.
- Lanser: Language-model supported speech emotion recognition. arXiv preprint arXiv:2309.03978.
- Instructerc: Reforming emotion recognition in conversation with a retrieval multi-task llms framework. Preprint, arXiv:2309.11911.
- Audio-llm: Activating the capabilities of large language models to comprehend audio data. In Advances in Neural Networks – ISNN 2024, pages 133–142, Singapore. Springer Nature Singapore.
- An embarrassingly simple approach for llm with strong asr capacity. Preprint, arXiv:2402.08846.
- Meld: A multimodal multi-party dataset for emotion recognition in conversations. Preprint, arXiv:1810.02508.
- Large language model-based emotional speech annotation using context and acoustic feature for speech emotion recognition. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11026–11030.
- Secap: Speech emotion captioning with large language model. Preprint, arXiv:2312.10381.
- Bioserc: Integrating biography speakers supported by llms for erc tasks. Preprint, arXiv:2407.04279.
- When llms meets acoustic landmarks: An efficient approach to integrate speech into large language models for depression detection. Preprint, arXiv:2402.13276.
- Dialoguellm: Context and emotion knowledge-tuned large language models for emotion recognition in conversations. Preprint, arXiv:2310.11374.
- Zehui Wu (5 papers)
- Ziwei Gong (10 papers)
- Lin Ai (15 papers)
- Pengyuan Shi (7 papers)
- Kaan Donbekci (2 papers)
- Julia Hirschberg (37 papers)