---
title: Emotion Recognition in Conversation (ERC)
url: https://www.emergentmind.com/topics/emotion-recognition-in-conversation-erc
type: topic
---

# Emotion Recognition in Conversation (ERC)

Emotion Recognition in Conversation (ERC) constitutes a central research domain in affective computing and natural language processing, focusing on the automatic identification of emotions expressed in the utterances of multi-turn, multi-party dialogues. The overarching objective is to assign an emotion label—drawn from a defined set such as Ekman’s six basic emotions plus neutral, or task-oriented extensions—to each utterance within a conversational context, leveraging not only the immediate text but also the preceding (and sometimes following) dialogue history. ERC plays a pivotal role in the deployment of empathetic conversational agents, dialogue systems, and broader emotion-aware AI applications, and has catalyzed the development of a heterogeneous methodological landscape encompassing pre-trained transformer encoders, recurrent and graph neural architectures, distributional learning frameworks, logic-based models, and prompt-based large language models [2304.08216][2108.12009][2211.04834][2403.20289][2206.03173][2404.11141][2403.07260][2303.11117][2112.01938][2207.07238][2012.14781][2412.16444][2010.01908][2112.11718][2405.17900][2207.13254][2203.13504][2203.16799][2211.09172][1905.02947].

## 1. Formal Task Definition and Research Challenges

ERC formalizes as follows: given a dialogue $U = (u_1, u_2, ..., u_T)$, where each $u_t$ is an utterance produced by a speaker $s_t$, the task is to predict an emotion label $y_t$ for each $u_t$, such that $y_t \in E$, where $E$ is a fixed emotion set. The function $f: (U, S) \to Y$ aims to minimize a loss, typically cross-entropy, over utterances [2211.09172][2304.08216][1905.02947]. Core challenges include:

- **Context Dependence**: Emotional interpretation often requires modeling long-range contextual dependencies; predictions based solely on the isolated utterance are inadequate [2304.08216][2211.09172].
- **Speaker Dynamics**: Interplay between intra-speaker emotional inertia (self-dependence) and inter-speaker contagion (other-dependence) [2303.11117][2206.03173][2211.09172].
- **Emotion Shift and Ambiguity**: Emotion shifts are abrupt and difficult to detect; annotator subjectivity yields inherent label noise and uncertainty [2303.11117][2211.04834][2112.01938].
- **Fine-grained and Imbalanced Categories**: Emotion classes are unequally distributed; rare or semantically similar emotions (e.g., excitement versus happiness) are easily confused [2403.20289][2412.16444].
- **Multiparty and Multimodal Complexity**: Multispeaker interactions and multimodal signals (audio, visual) compound context modeling complexity [2412.16444][2112.01938][2203.13504].

## 2. Methodological Paradigms in ERC

The field encompasses a rich spectrum of architectures:

**2.1 Transformer-based Utterance Encoding and Context Integration**  
Contemporary ERC models frequently employ pre-trained transformer language models (e.g., RoBERTa, BERT) to produce utterance embeddings. Recent advances inject context directly into the input of the encoder by concatenating $k$ previous turns:  
\[
\text{Input sequence}: [\text{CLS}]~u_{t-k}~[\text{SEP}]~...~u_t~[\text{SEP}]
\]
This approach yields a context-aware [CLS] embedding, eliminating the need for separate context modules and simplifying the architecture [2304.08216]. CD-ERC exemplifies this design, establishing optimal performance with $k=3$ context turns and outperforming both context-independent RoBERTa baselines and elaborate graph/recurrent models on DailyDialog and EmoWOZ [2304.08216].

**2.2 Speaker Modeling and Intra/Inter-Speaker Dependencies**  
Effective ERC requires explicit modeling of speaker dependencies. EmoBERTa operationalizes this by prepending speaker identities to utterances, thereby enabling the transformer to learn both intra- and inter-speaker relations via self-attention alone [2108.12009]. SGED (Speaker-Guided Encoder-Decoder) introduces dynamic, attention-based speaker state tracking modules that compute distinct intra- and inter-speaker influences, which are fused and fed to a GRU-based decoder [2206.03173]. Masking mechanisms in hierarchical transformers further support fine-grained separation of speaker-dependent context [2012.14781].

**2.3 Graph Neural Networks, Hypergraphs, and Metric-Learning**  
ERC has seen significant innovation in graph-based representations: ConxGNN integrates a multi-scale heterogeneous graph with a hypergraph, capturing both local/global context and high-order cross-modal interactions, with a cross-modal attention fusion layer to align modalities (text, audio, visual). A class-balanced composite loss mitigates class imbalance and similar emotion confusion, achieving state-of-the-art weighted-F1 on IEMOCAP and MELD [2412.16444]. Separately, SentEmoContext frames ERC as a metric learning task, embedding context-aware utterances in a space structured by triplet loss, yielding the highest macro-F1 on DailyDialog with a lightweight, efficient pipeline [2404.11141].

**2.4 Uncertainty Estimation and Distributional Modeling**  
ERC models increasingly accommodate annotation subjectivity and label ambiguity by estimating the full distribution of possible emotions for each utterance. A Dirichlet prior network models the empirical distribution over annotator votes, optimized via a combination of negative log-likelihood and KL divergence, thus yielding well-calibrated uncertainty and improving not only accuracy but also the capacity to flag out-of-distribution or highly ambiguous utterances [2211.04834].

**2.5 Contrastive, Curriculum, and Prompt-based Learning**  
EACL leverages emotion label encodings as geometric anchors in a contrastive representation space, equipped with auxiliary losses to maximally separate similar emotions. Adaptation of anchor positions post-hoc fine-tunes the classifier for maximal separation, drastically reducing confusion between similar classes (e.g., happy/excited) [2403.20289]. Hybrid curriculum learning schedules both conversations and labels for model training—first exposing the model to conversations with few emotion shifts and soft-label distributions before increasing task entropy—thus enhancing robustness on difficult and minority-class utterances [2112.11718]. Prompt-based techniques, such as CISPER or LaERC-S, explicitly blend dialogue context and external commonsense into the input prompt for large language models, recasting ERC as a cloze or generative task and achieving state-of-the-art results [2207.13254][2403.07260].

## 3. Multimodal and Multilingual ERC

State-of-the-art ERC systems increasingly operate in the multimodal domain. Examples include:

- **Multimodal Fusion**: JFM-based architectures employ iterative cross-modal vector updates for deep fusion of text and audio representations, supervised by inter-class contrastive losses to enhance class separation and sample-balance [2405.17900]. 
- **Emotion Capsules and Emoformer**: EmoCaps fuses modality-specific emotion vectors with sentence embeddings into "emotion capsules," processed by context-aware modules (Bi-LSTM or DialogueRNN), achieving SOTA results on IEMOCAP and MELD [2203.13504].  
- **Emotion Shifts and Multimodal Gating**: Emotion shift modules, such as those described by [2112.01938], interleave emotion-shift gating with context-aware recurrent backbones, explicitly modeling abrupt emotional transitions.
- **Multilingual Corpora and Transfer**: M-MELD extends MELD to multiple languages, accompanied by discursive graph-based LSTM backends that exploit XLM-RoBERTa for zero-shot transfer and discourse parsing, generalizing across diverse linguistic regimes [2203.16799].

## 4. Evaluation Metrics, Benchmark Datasets, and Empirical Comparisons

ERC relies on macro-averaged and weighted F1 scores due to pronounced class imbalance. Datasets such as IEMOCAP, MELD, DailyDialog, and EmoryNLP span dyadic and multi-party dialogues, with 6–8 categorical emotion labels (see summary in [2211.09172][2304.08216][2412.16444][2112.01938]). Strict experimental setups—5-fold cross-validation, multiple seeds, class-balanced sampling—are standard. State-of-the-art models (e.g., EACL, ConxGNN) report significant advances:

| Model            | IEMOCAP wF1 | MELD wF1 | EmoryNLP wF1 | DailyDialog macro-F1 |
|------------------|-------------|----------|--------------|---------------------|
| DAG-ERC          | 68.03       | 63.65    | 39.02        | 59.33               |
| EmotionIC        | 69.61       | 66.32    | 40.25        | 54.19               |
| EACL             | 70.41       | 67.12    | 40.24        | —                   |
| ConxGNN          | 68.64       | 65.69    | —            | —                   |
| CD-ERC (k=3)     | —           | —        | —            | 51.23 (DailyDialog) |
| SentEmoContext   | —           | —        | —            | 57.71 (DailyDialog) |

Empirical ablations highlight that contextualization (window size $k$), explicit speaker modeling, anchor-based contrastive learning, and curriculum design drive consistent gains across datasets [2304.08216][2403.20289][2404.11141][2112.11718][2206.03173].

## 5. Subjectivity, Annotation, and Distributional Grounding

Annotation subjectivity remains a pivotal concern. Modeling strategies include:

- **Soft-Label and Distributional Learning**: Directly fitting empirical annotation distributions or modeling uncertainty via Bayesian approaches (Dirichlet prior) yields improved calibration and retention of ambiguous cases, as well as higher AUPR for uncertainty detection [2211.04834].
- **Logical/Relational Inference**: Probabilistic Soft Logic (PSL) formulas propagate emotion labels over dialogue structure and text similarity, using neural embeddings as soft evidences; this collective inference corrects noisy or ambiguous human annotations and delivers large gains on class-imbalanced corpora [2207.07238].
- **Few-Shot and Minority-Class Robustness**: Pre-training context-dependent encoders on unsupervised data—e.g., via the Conversation Completion (ConvCom) task—boosts minority-class recognition and generalization to low-resource settings [2010.01908][2404.11141].

## 6. Major Advances, Limitations, and Future Directions

Recent ERC models consistently:

- Leverage context-dependent utterance encoding, often by integrating conversational history directly into the transformer input [2304.08216][2108.12009][2207.13254].
- Explicitly model speaker identity and dynamic intra-/inter-speaker dependencies for improved emotion prediction, especially in multiparty dialogues [2108.12009][2206.03173][2012.14781].
- Address class imbalance, subjective annotation, and ambiguous utterances via contrastive/diversity-enhancing losses, distributional supervision, and logic-based relational models [2403.20289][2211.04834][2207.07238].
- Achieve new state-of-the-art performance through anchor-guided contrastive learning, multi-scale GNN/hypergraph fusion, curriculum learning, and prompt-based LLM tuning [2403.20289][2412.16444][2112.11718][2207.13254][2403.07260].

Open problems persist with multimodal fusion (especially for visual/facial cues), real-time and online ERC, continuous emotion representation, and scalable generalization to multilingual, code-mixed, or out-of-domain conversations [2412.16444][2112.01938][2203.16799][2211.09172].

**Future research directions** include hierarchical and dynamic graph construction, adaptive window/context scaling, cross-modal external knowledge infusion (e.g., commonsense graphs), and subjectivity-embracing architectures for more interpretable and uncertainty-aware emotion recognition [2211.09172][2403.07260][2211.04834][2412.16444][2207.13254][2112.11718].

Source: https://www.emergentmind.com/topics/emotion-recognition-in-conversation-erc