---
title: Emotion Recognition in Conversations
url: https://www.emergentmind.com/topics/emotion-recognition-in-conversations-erc
type: topic
---

# Emotion Recognition in Conversations

Emotion Recognition in Conversations (ERC) is the computational task of automatically identifying the emotional state underlying each utterance in a dialogue, leveraging the context, speaker dynamics, and multivariate cues present in multi-turn interactions. ERC is central to affective computing and conversational AI, supporting applications from empathetic dialogue systems to emotion-aware agent modeling, and poses unique challenges in modeling context-dependence, speaker interactions, label ambiguity, and data imbalance [2206.03173], [2211.09172], [1905.02947].

## 1. Problem Formulation and Research Challenges

ERC extends classic emotion classification to fully contextualized, speaker-aware utterance emotion prediction within natural dialogues. Given a conversation $\mathcal{C} = [(u_1, s_1), (u_2, s_2), \dots, (u_N, s_N)]$, where $u_i$ is the $i$th utterance and $s_i$ the speaker identity, the objective is to assign labels $y_i \in \mathcal{E}$ (from a fixed set of $K$ emotions) to each $u_i$, conditioned on conversation history and contextual cues.

Key research challenges include:
- **Contextuality**: Emotional interpretation of each turn is highly dependent on the preceding and sometimes following utterances. Local (few-turn) and global (long-range) context modeling are required for high accuracy [2010.01908], [2304.08216].
- **Speaker and Listener Modeling**: Each speaker's emotional flow follows both intra- (self-continuity, inertia) and inter-speaker (influence, reaction) dependencies. Modeling evolving party states or ignoring speaker IDs affects performance [2206.03173], [2012.14781].
- **Label Ambiguity and Subjectivity**: Emotions may have overlapping cues (e.g., anger vs. frustration), and annotators may disagree, leading to label distributions rather than absolutes [2211.04834].
- **Multiparty and Multimodal Complexity**: Group dialogues, multimodal cues (text, audio, visual), and cross-linguistic settings significantly increase problem complexity [2203.16799], [2412.16444].
- **Emotion Shift Detection**: Abrupt changes in a speaker's state challenge traditional models attuned to inertia [2112.01938], [2112.11718].
- **Class Imbalance and Fine-Grained Emotions**: Rare emotions (e.g., disgust, fear) and subtle distinctions (e.g., joy vs. surprise) are underrepresented, biasing models toward head classes [2412.16444], [2211.09172].

## 2. Methodological Developments

ERC research evolved from hierarchical RNNs and CNNs to context-dependent utterance representations via Transformers and Graph Neural Networks (GNNs), with key advances in speaker-state tracking, multimodal fusion, and self-supervised or curriculum-driven regularization.

### 2.1 Speaker and Context Modeling Architectures

- **Speaker-Aware RNNs/GRUs**: DialogueRNN models three dynamic states (global, party/speaker, emotion), updating per-turn representations based on speaker identity and global context [1905.02947]. Extensions (e.g., DialogueCRN, DialogueGCN) introduce relation-specific graph structures, further separating inter- and intra-speaker flows [2012.14781].
- **Graph-based and Transformer Approaches**: Recent GNNs use conversation graphs, where each utterance is a node, with edges encoding sequential or speaker, intra/inter, or sentiment-shift relationships [2312.03756], [2412.16444]. Hierarchical and mask-based Transformers segment self-, inter-, and global-attention for explicit modeling [2012.14781], [2206.03173].
- **Metric and Contrastive Learning**: Methods such as Emotion-Anchored Contrastive Learning (EACL) and Supervised Contrastive Learning (SSLCL) improve clusterability of emotion representations, especially for confusable pairs, via label-anchored or label-embedding contrastive losses [2403.20289], [2310.16676].

### 2.2 Curriculum Learning and Data Augmentation

Hybrid Curriculum Learning (HCL) orchestrates training progression from easy (emotion-stable) to hard (high-shift, confusable) dialogues; at the utterance level, label targets interpolate from soft (label-similarity) to sharp [2112.11718]. Curriculum strategies consistently raise F1 by up to 3 points, particularly for foundational GNN or RNN models.

Self-supervised pre-training on large unsupervised corpora, e.g., masked utterance completion (ConvCom), yields improved generalization and minority class F1 [2010.01908].

### 2.3 Multimodal and Cross-Modal Fusion

State-of-the-art ERC models fuse text, audio, and visual features using mechanisms such as:
- Capsule-based representations, concatenating semantic and multimodal emotion vectors (EmoCaps) [2203.13504].
- Joint vector-based cross-modal fusion, alternately propagating information between modalities while maintaining modality-specific cues [2405.17900].
- GNN hypergraphs and cross-modal attention, capturing both temporal and cross-modality dependencies [2412.16444].

### 2.4 Handling Label Ambiguity and Uncertainty

Distribution-based ERC models treat each utterance's emotion as a sample from a Dirichlet distribution parameterized by the model, enabling calibrated uncertainty estimation and alignment with annotator disagreements. Maximizing the likelihood of observed soft-label distributions with a KL regularization term further sharpens confidence in ambiguous cases [2211.04834].

## 3. Benchmark Datasets and Evaluative Protocol

Benchmark datasets for ERC span scripted dyadic (IEMOCAP), multiparty TV scripts (MELD), open-domain chat (DailyDialog), multilingual corpora (M-MELD), and multimodal interactions (IEMOCAP, MELD) [1905.02947], [2203.16799]. Key dataset properties:

| Dataset      | Utterances | Emotions | Modality        | Party Type    |
|--------------|------------|----------|-----------------|---------------|
| IEMOCAP      | 7,433      | 6        | Text, Audio, Video | Dyadic        |
| MELD         | 13,708     | 7        | Text, Audio, Video | Multiparty    |
| DailyDialog  | 102,979    | 7        | Text-only       | Dyadic        |
| M-MELD       | 53,816     | 7        | Text-only/Four Languages | Multiparty   |

Standard evaluation uses weighted/macro/micro F1 and accuracy. Many datasets are highly imbalanced (e.g., neutral dominates), necessitating weighted losses and macro F1 reporting [2412.16444], [2211.09172].

## 4. Recent Results and Baseline Comparisons

Progressive methodological innovations have yielded steady SOTA gains. For example, on IEMOCAP (weighted F1):

- DialogueRNN: 62.75%
- DAG-ERC: 68.03%
- EmoCaps: 71.77% [2203.13504]
- EACL: 70.41% [2403.20289]
- SpikEmo (multimodal, spiking SNN): 71.50% [2411.13917]
- ConxGNN (multi-scale GNN+hypergraph): 68.64% [2412.16444]

On MELD, similar advances are observed. Specialized training strategies (HCL [2112.11718], SSLCL [2310.16676]), prompt-based methods (CISPER [2207.13254]), and uncertainty/ambiguity modeling with Dirichlet outputs [2211.04834] have further improved fine-grained and tail-class results.

## 5. Open Issues and Ongoing Research Directions

Despite tangible progress, fundamental challenges persist:

- **Emotion Shifts**: Improved emotion-shift modeling (explicit shift predictors [2112.01938], curriculum scheduling [2112.11718]) yields measurable accuracy gains on challenging utterances, but sudden transitions remain error-prone.
- **Fine-grained and Multimodal Cues**: Discriminating subtle emotion classes and leveraging visual/audio cues is a persistent weakness, with state-of-the-art models showing limited gains for classes distinguished primarily by prosody or facial affect [2206.03173], [2405.17900], [2412.16444].
- **Speaker-Agnostic and Multilingual ERC**: Most methods depend on clean speaker identities and English-language data. New corpora (M-MELD [2203.16799]) and architectures disambiguating speakers without IDs (LineConGraphs [2312.03756]) remain active research fronts.
- **Label Uncertainty, Subjectivity, and Real-time ERC**: Bayesian/soft-label frameworks [2211.04834] and multi-annotator models are increasingly adopted for robust, uncertainty-calibrated emotion assignment. Online/streaming ERC and multimodal prompt learning are identified as promising future approaches [2211.09172].
- **Interpretable and Explainable Systems**: Incorporation of logical reasoning, commonsense, explicit annotation of causes, and LLM-based generative ERC strategies are beginning to emerge [2207.07238], [2211.09172].

## 6. Best Practices, Comparative Insights, and Limitations

Best-practice recommendations across multiple comparative reviews [2211.09172], [1905.02947]:
- Fine-tune large pre-trained encoders (e.g., RoBERTa/BERT) rather than training from scratch.
- Explicitly model both intra- and inter-speaker context, either via dynamic party states, speaker-specific attention, or graph-based relations.
- Use data stratification, weighted or focal loss, and curriculum learning to handle class imbalance.
- Maintain soft-label targets or Dirichlet priors to reflect label subjectivity and improve calibration in ambiguous cases.
- Whenever possible, fuse multimodal cues and exploit external knowledge (e.g., commonsense) through tailored prompt or knowledge-augmented methods.
- Report weighted/macro F1 and provide ablation studies quantifying the effect of context, speaker, shift, and fusion components.

Major limitations highlighted include persistent confusions between semantically similar emotions, underperformance on rare/tail classes, and increased computational cost for complex architectures (multi-scale GNNs, spiking neural networks, multimodal prompt pipelines) [2412.16444], [2411.13917].

## 7. Outlook and Future Directions

ERC is converging toward architectures that unify adaptive speaker/context modeling, contrastive or metric learning, curriculum training, and principled uncertainty handling. Emerging trends include:
- Integration with LLMs for generative or in-context emotion reasoning [2211.09172].
- Multi-label and continuous emotion modeling (e.g., VAD regression, mixed emotions).
- Real-time, streaming ERC for dynamic human–robot interaction.
- Explicit modeling of emotion causes, sarcasm, and conversation-level reasoning [1905.02947].
- Cross-lingual and cross-modal transfer learning leveraged by high-quality parallel corpora and multilingual representations [2203.16799].

As datasets and modeling approaches broaden, the need for interpretable, reliable, and context-preserving ERC frameworks will increase, establishing ERC as a core testbed for social and empathetic AI [2211.09172], [1905.02947].

Source: https://www.emergentmind.com/topics/emotion-recognition-in-conversations-erc