Emotion Recognition in Conversations

• ERC is a task that predicts emotions in dialogue by analyzing conversation context, speaker dynamics, and multimodal cues.
• Recent methods employ advanced architectures like Transformers, Graph Neural Networks, and contrastive learning to enhance performance.
• ERC research tackles challenges such as label ambiguity, abrupt emotion shifts, and data imbalance to support empathetic AI systems.

Emotion Recognition in Conversations (ERC) is the computational task of automatically identifying the emotional state underlying each utterance in a dialogue, leveraging the context, speaker dynamics, and multivariate cues present in multi-turn interactions. ERC is central to affective computing and conversational AI, supporting applications from empathetic dialogue systems to emotion-aware agent modeling, and poses unique challenges in modeling context-dependence, speaker interactions, label ambiguity, and data imbalance (Bao et al., 2022, Pereira et al., 2022, Poria et al., 2019).

1. Problem Formulation and Research Challenges

ERC extends classic emotion classification to fully contextualized, speaker-aware utterance emotion prediction within natural dialogues. Given a conversation $$\mathcal{C} = [(u_1, s_1), (u_2, s_2), \dots, (u_N, s_N)]$$, where $u_i$ is the $i$th utterance and $s_i$ the speaker identity, the objective is to assign labels $y_i \in \mathcal{E}$ (from a fixed set of $K$ emotions) to each $u_i$, conditioned on conversation history and contextual cues.

Key research challenges include:

• Contextuality: Emotional interpretation of each turn is highly dependent on the preceding and sometimes following utterances. Local (few-turn) and global (long-range) context modeling are required for high accuracy (Jiao et al., 2020, Pereira et al., 2023).
• Speaker and Listener Modeling: Each speaker's emotional flow follows both intra- (self-continuity, inertia) and inter-speaker (influence, reaction) dependencies. Modeling evolving party states or ignoring speaker IDs affects performance (Bao et al., 2022, Li et al., 2020).
• Label Ambiguity and Subjectivity: Emotions may have overlapping cues (e.g., anger vs. frustration), and annotators may disagree, leading to label distributions rather than absolutes (Wu et al., 2022).
• Multiparty and Multimodal Complexity: Group dialogues, multimodal cues (text, audio, visual), and cross-linguistic settings significantly increase problem complexity (Ghosh et al., 2022, Van et al., 21 Dec 2024).
• Emotion Shift Detection: Abrupt changes in a speaker's state challenge traditional models attuned to inertia (Agarwal et al., 2021, Yang et al., 2021).
• Class Imbalance and Fine-Grained Emotions: Rare emotions (e.g., disgust, fear) and subtle distinctions (e.g., joy vs. surprise) are underrepresented, biasing models toward head classes (Van et al., 21 Dec 2024, Pereira et al., 2022).

2. Methodological Developments

ERC research evolved from hierarchical RNNs and CNNs to context-dependent utterance representations via Transformers and Graph Neural Networks (GNNs), with key advances in speaker-state tracking, multimodal fusion, and self-supervised or curriculum-driven regularization.

2.1 Speaker and Context Modeling Architectures

• Speaker-Aware RNNs/GRUs: DialogueRNN models three dynamic states (global, party/speaker, emotion), updating per-turn representations based on speaker identity and global context (Poria et al., 2019). Extensions (e.g., DialogueCRN, DialogueGCN) introduce relation-specific graph structures, further separating inter- and intra-speaker flows (Li et al., 2020).
• Graph-based and Transformer Approaches: Recent GNNs use conversation graphs, where each utterance is a node, with edges encoding sequential or speaker, intra/inter, or sentiment-shift relationships (Krishnan et al., 2023, Van et al., 21 Dec 2024). Hierarchical and mask-based Transformers segment self-, inter-, and global-attention for explicit modeling (Li et al., 2020, Bao et al., 2022).
• Metric and Contrastive Learning: Methods such as Emotion-Anchored Contrastive Learning (EACL) and Supervised Contrastive Learning (SSLCL) improve clusterability of emotion representations, especially for confusable pairs, via label-anchored or label-embedding contrastive losses (Yu et al., 29 Mar 2024, Shi et al., 2023).

2.2 Curriculum Learning and Data Augmentation

Hybrid Curriculum Learning (HCL) orchestrates training progression from easy (emotion-stable) to hard (high-shift, confusable) dialogues; at the utterance level, label targets interpolate from soft (label-similarity) to sharp (Yang et al., 2021). Curriculum strategies consistently raise F1 by up to 3 points, particularly for foundational GNN or RNN models.

Self-supervised pre-training on large unsupervised corpora, e.g., masked utterance completion (ConvCom), yields improved generalization and minority class F1 (Jiao et al., 2020).

2.3 Multimodal and Cross-Modal Fusion

State-of-the-art ERC models fuse text, audio, and visual features using mechanisms such as:

• Capsule-based representations, concatenating semantic and multimodal emotion vectors (EmoCaps) (Li et al., 2022).
• Joint vector-based cross-modal fusion, alternately propagating information between modalities while maintaining modality-specific cues (Shi et al., 28 May 2024).
• GNN hypergraphs and cross-modal attention, capturing both temporal and cross-modality dependencies (Van et al., 21 Dec 2024).

2.4 Handling Label Ambiguity and Uncertainty

Distribution-based ERC models treat each utterance's emotion as a sample from a Dirichlet distribution parameterized by the model, enabling calibrated uncertainty estimation and alignment with annotator disagreements. Maximizing the likelihood of observed soft-label distributions with a KL regularization term further sharpens confidence in ambiguous cases (Wu et al., 2022).

3. Benchmark Datasets and Evaluative Protocol

Benchmark datasets for ERC span scripted dyadic (IEMOCAP), multiparty TV scripts (MELD), open-domain chat (DailyDialog), multilingual corpora (M-MELD), and multimodal interactions (IEMOCAP, MELD) (Poria et al., 2019, Ghosh et al., 2022). Key dataset properties:

Dataset	Utterances	Emotions	Modality	Party Type
IEMOCAP	7,433	6	Text, Audio, Video	Dyadic
MELD	13,708	7	Text, Audio, Video	Multiparty
DailyDialog	102,979	7	Text-only	Dyadic
M-MELD	53,816	7	Text-only/Four Languages	Multiparty

Standard evaluation uses weighted/macro/micro F1 and accuracy. Many datasets are highly imbalanced (e.g., neutral dominates), necessitating weighted losses and macro F1 reporting (Van et al., 21 Dec 2024, Pereira et al., 2022).

4. Recent Results and Baseline Comparisons

Progressive methodological innovations have yielded steady SOTA gains. For example, on IEMOCAP (weighted F1):

• DialogueRNN: 62.75%
• DAG-ERC: 68.03%
• EmoCaps: 71.77% (Li et al., 2022)
• EACL: 70.41% (Yu et al., 29 Mar 2024)
• SpikEmo (multimodal, spiking SNN): 71.50% (Yu et al., 21 Nov 2024)
• ConxGNN (multi-scale GNN+hypergraph): 68.64% (Van et al., 21 Dec 2024)

On MELD, similar advances are observed. Specialized training strategies (HCL (Yang et al., 2021), SSLCL (Shi et al., 2023)), prompt-based methods (CISPER (Yi et al., 2022)), and uncertainty/ambiguity modeling with Dirichlet outputs (Wu et al., 2022) have further improved fine-grained and tail-class results.

5. Open Issues and Ongoing Research Directions

Despite tangible progress, fundamental challenges persist:

• Emotion Shifts: Improved emotion-shift modeling (explicit shift predictors (Agarwal et al., 2021), curriculum scheduling (Yang et al., 2021)) yields measurable accuracy gains on challenging utterances, but sudden transitions remain error-prone.
• Fine-grained and Multimodal Cues: Discriminating subtle emotion classes and leveraging visual/audio cues is a persistent weakness, with state-of-the-art models showing limited gains for classes distinguished primarily by prosody or facial affect (Bao et al., 2022, Shi et al., 28 May 2024, Van et al., 21 Dec 2024).
• Speaker-Agnostic and Multilingual ERC: Most methods depend on clean speaker identities and English-language data. New corpora (M-MELD (Ghosh et al., 2022)) and architectures disambiguating speakers without IDs (LineConGraphs (Krishnan et al., 2023)) remain active research fronts.
• Label Uncertainty, Subjectivity, and Real-time ERC: Bayesian/soft-label frameworks (Wu et al., 2022) and multi-annotator models are increasingly adopted for robust, uncertainty-calibrated emotion assignment. Online/streaming ERC and multimodal prompt learning are identified as promising future approaches (Pereira et al., 2022).
• Interpretable and Explainable Systems: Incorporation of logical reasoning, commonsense, explicit annotation of causes, and LLM-based generative ERC strategies are beginning to emerge (Augustine et al., 2022, Pereira et al., 2022).

6. Best Practices, Comparative Insights, and Limitations

Best-practice recommendations across multiple comparative reviews (Pereira et al., 2022, Poria et al., 2019):

• Fine-tune large pre-trained encoders (e.g., RoBERTa/BERT) rather than training from scratch.
• Explicitly model both intra- and inter-speaker context, either via dynamic party states, speaker-specific attention, or graph-based relations.
• Use data stratification, weighted or focal loss, and curriculum learning to handle class imbalance.
• Maintain soft-label targets or Dirichlet priors to reflect label subjectivity and improve calibration in ambiguous cases.
• Whenever possible, fuse multimodal cues and exploit external knowledge (e.g., commonsense) through tailored prompt or knowledge-augmented methods.
• Report weighted/macro F1 and provide ablation studies quantifying the effect of context, speaker, shift, and fusion components.

Major limitations highlighted include persistent confusions between semantically similar emotions, underperformance on rare/tail classes, and increased computational cost for complex architectures (multi-scale GNNs, spiking neural networks, multimodal prompt pipelines) (Van et al., 21 Dec 2024, Yu et al., 21 Nov 2024).

7. Outlook and Future Directions

ERC is converging toward architectures that unify adaptive speaker/context modeling, contrastive or metric learning, curriculum training, and principled uncertainty handling. Emerging trends include:

• Integration with LLMs for generative or in-context emotion reasoning (Pereira et al., 2022).
• Multi-label and continuous emotion modeling (e.g., VAD regression, mixed emotions).
• Real-time, streaming ERC for dynamic human–robot interaction.
• Explicit modeling of emotion causes, sarcasm, and conversation-level reasoning (Poria et al., 2019).
• Cross-lingual and cross-modal transfer learning leveraged by high-quality parallel corpora and multilingual representations (Ghosh et al., 2022).

As datasets and modeling approaches broaden, the need for interpretable, reliable, and context-preserving ERC frameworks will increase, establishing ERC as a core testbed for social and empathetic AI (Pereira et al., 2022, Poria et al., 2019).