Emo Pillars: Knowledge Distillation to Support Fine-Grained Context-Aware and Context-Less Emotion Classification (2504.16856v1)

Abstract: Most datasets for sentiment analysis lack context in which an opinion was expressed, often crucial for emotion understanding, and are mainly limited by a few emotion categories. Foundation LLMs like GPT-4 suffer from over-predicting emotions and are too resource-intensive. We design an LLM-based data synthesis pipeline and leverage a large model, Mistral-7b, for the generation of training examples for more accessible, lightweight BERT-type encoder models. We focus on enlarging the semantic diversity of examples and propose grounding the generation into a corpus of narratives to produce non-repetitive story-character-centered utterances with unique contexts over 28 emotion classes. By running 700K inferences in 450 GPU hours, we contribute with the dataset of 100K contextual and also 300K context-less examples to cover both scenarios. We use it for fine-tuning pre-trained encoders, which results in several Emo Pillars models. We show that Emo Pillars models are highly adaptive to new domains when tuned to specific tasks such as GoEmotions, ISEAR, IEMOCAP, and EmoContext, reaching the SOTA performance on the first three. We also validate our dataset, conducting statistical analysis and human evaluation, and confirm the success of our measures in utterance diversification (although less for the neutral class) and context personalization, while pointing out the need for improved handling of out-of-taxonomy labels within the pipeline.

• The paper presents an LLM-based methodology using Mistral-7b and narrative corpora to generate large synthetic datasets for fine-grained context-aware and context-less emotion classification.
• This approach enables fine-tuning of models like Emo Pillars, which achieve state-of-the-art performance on established emotion recognition benchmarks including GoEmotions, ISEAR, and IEMOCAP.
• The research provides a scalable method for overcoming dataset limitations in emotion recognition, suggesting LLMs' potential for knowledge distillation and enhancing model robustness across domains.

Fine-Grained Emotion Classification via Synthetic Dataset Generation

The paper "Emo Pillars: Knowledge Distillation to Support Fine-Grained Context-Aware and Context-Less Emotion Classification" presents a novel methodology employing LLMs for generating synthetic datasets aimed at enhancing emotion recognition systems. Recognizing limitations in existing sentiment analysis datasets—specifically their scarcity of context and restriction to a narrow range of emotion categories—the research introduces a comprehensive LLM-based data synthesis pipeline facilitated by Mistral-7b to produce training examples for BERT-type encoder models. The paper addresses both context-aware and context-less scenarios in emotion classification, enlarging semantic diversity and incorporating narrative contexts to enrich emotion understanding across 28 classes.

Methodology

The research utilizes a combination of LLMs with substantial manual intervention in prompt design to distill high-quality emotion-representative data effectively. The approach involves:

• Story Grounding and Diversity Enhancement: Utilizing story text corpora to base synthetic data, iterating over diverse character perspectives to generate varied emotional contexts and utterances.
• Contextual Awareness: Generating both context-rich and context-less examples through model iterations focused on semantic differentiation and context specificity, yielding datasets of 100K contextual and 300K context-less samples.
• Adaptive Modelling: Fine-tuning mid-sized encoder-based models with distilled data, resulting in multiple models termed Emo Pillars, which exhibit domain adaptability and state-of-the-art (SOTA) performance in emotion recognition tasks like GoEmotions, ISEAR, and IEMOCAP.

Results and Implications

The paper reports strong numerical results, particularly the adaptation and efficacy of the Emo Pillars models in novel domains, achieving SOTA scores in several established benchmarks. Statistical analysis and human evaluation demonstrate successful emotion diversification and context personalization, although improvements are required in handling out-of-taxonomy labels. The research showcases how synthetic data generation—augmented by narrative corpora and multi-perspective analysis—can overcome traditional limitations associated with sentiment analysis datasets and enhance model performance markedly across diverse emotion detection tasks.

Practical and Theoretical Impact

Practically, the methodology laid out in this paper offers a scalable solution to overcome dataset limitations, enhancing the training of emotion recognition models across various applications. Theoretically, it encourages further exploration of LLMs as tools for knowledge distillation, suggesting potential advancements in emotional intelligence benchmarking and model interpretability.

Future Directions

This research opens avenues for expanding emotion classification beyond linguistic constraints, potentially integrating multilingual synthetic data. It emphasizes refining neutral example generation and exploring the balance between explicitly and implicitly expressed emotions in training datasets. The introduction of interpretability techniques for better comprehension of model decisions and aspect-oriented emotion analysis are also recommended as future advancements.

In summary, this paper contributes significantly to the field of sentiment analysis and emotion recognition by demonstrating a practical and comprehensive approach to synthetic data generation using LLMs. The results highlight the importance of context and semantic diversity in improving model robustness and provide a framework that can be adapted for various emotion detection tasks across different domains.