DialogueRNN: Speaker-Aware Emotion Detection

• DialogueRNN is an attentive recurrent architecture explicitly modeling speaker-specific states and global context for dynamic emotion recognition in conversations.
• It decomposes conversational emotion dynamics into three recurrent components—global context, party state, and emotion evolution—to capture nuanced inter-speaker influences.
• Empirical evaluations on datasets like IEMOCAP and AVEC demonstrate its effectiveness over prior models and establish it as a strong baseline for multi-party dialogue emotion analysis.

DialogueRNN is an attentive recurrent architecture for emotion detection in conversations that predicts an emotion label for each utterance while explicitly tracking the emotional or cognitive state of each party across dialogue time. Its central design choice is to treat speaker identity as a persistent dynamical variable rather than as incidental metadata: each participant has a recurrent party state, the conversation as a whole has a global recurrent context, and emotion evolution is modeled by a separate recurrent process. In this formulation, conversational emotion recognition is not reduced to classifying isolated utterances or a flat utterance sequence, but to modeling how speaker-specific states, context, and emotional trajectories interact over turns (Majumder et al., 2018).

1. Problem formulation and motivation

DialogueRNN is defined for a conversation $(u_1, u_2, \dots, u_N)$ with multiple parties $p_1, \dots, p_M)$, where each utterance $u_t$ is spoken by some party $p_{s(u_t)}$. The task is to predict an emotion label for each utterance, with example classes including happy, sad, neutral, angry, excited, and frustrated (Majumder et al., 2018).

The model was motivated by limitations in prior conversational emotion systems. Many earlier approaches treated a conversation as a flat sequence of utterances and did not differentiate between speakers. In the original comparison, c-LSTM encoded surrounding context but did not distinguish speakers, while CMN used separate memories for two speakers without deeply integrating speaker identity into a persistent recurrent party state. The original formulation therefore identified two missing capabilities in prior work: explicit, persistent speaker-specific states that evolve with the dialogue, and a mechanism for modeling inter-speaker emotional influence, such as one speaker’s anger leading to frustration in another (Majumder et al., 2018).

This task formulation was linked to applications including opinion mining over chat logs and social media threads, debates and argumentation mining, analysing customer feedback in live chat or call centers, and human–computer interaction or empathetic dialogue systems. The motivating assumption is that utterance emotion is not only a function of local lexical content, but also of speaker continuity, interlocutor influence, and long-range contextual dependencies (Majumder et al., 2018).

2. Architectural decomposition

DialogueRNN decomposes conversational emotion dynamics into three recurrent factors: speaker state, context, and emotion dynamics. The model introduces party states $q_i$ for participants, a global GRU to encode speaker-conditioned conversational context, and an emotion GRU to propagate emotion information across adjacent utterances (Majumder et al., 2018).

Component	State	Function
Global GRU	$g_t$	Encodes global conversation context
Party GRU	$q_{i,t}$	Tracks speaker-specific state
Emotion GRU	$e_t$	Produces emotion representation

This separation is one of the architecture’s defining properties. The global state captures the utterance stream together with speaker information; the party state tracks how an individual participant evolves across turns; and the emotion state acts as a dedicated emotional process that links current speaker state to previous emotional context. The authors interpret the global and party GRUs as an encoder, and the emotion GRU as a decoder that refines an emotion-specific representation (Majumder et al., 2018).

Utterance representation is modality-dependent. For text, the model uses a CNN following Kim (2014): 1D convolutions with filter sizes 3, 4, and 5, each with 50 feature maps, followed by max pooling over time, ReLU activations, concatenation, and a dense 100-dimensional layer. In multimodal settings, visual features are obtained from 3D CNNs and audio features from openSMILE; multimodal fusion is performed by simple concatenation (Majumder et al., 2018).

A common misconception is to regard DialogueRNN as merely a contextual RNN with attention. In the original formulation, attention is important, but the architectural novelty is the explicit party-state mechanism: the model’s defining distinction is that it keeps a separate evolving state for each participant and updates that state when the party speaks (Majumder et al., 2018).

3. State updates, attention, and learning objective

At time $t$, the utterance $u_t$ is spoken by party $p_1, \dots, p_M)$0 whose previous party state is $p_1, \dots, p_M)$1. The global state update combines the current utterance with the previous speaker state:

$p_1, \dots, p_M)$2

This yields a speaker-specific utterance representation plus context, so that the sequence $p_1, \dots, p_M)$3 can be queried for emotionally relevant context (Majumder et al., 2018).

The context vector used for speaker updating is computed by attention over past global states:

$p_1, \dots, p_M)$4

$p_1, \dots, p_M)$5

The current speaker’s party state is then updated as

$p_1, \dots, p_M)$6

The intended interpretation is that $p_1, \dots, p_M)$7 encodes what the speaker said, while $p_1, \dots, p_M)$8 encodes attended dialogue context from all preceding utterances (Majumder et al., 2018).

For listeners, the original paper presented two options. The default and empirically sufficient scheme leaves non-speaking parties unchanged:

$p_1, \dots, p_M)$9

An alternative listener GRU can update listeners using listener visual cues $u_t$0 and context $u_t$1,

$u_t$2

but this explicit listener update did not significantly improve performance and added parameters, so the simpler no-update scheme remained the main formulation (Majumder et al., 2018).

The emotion representation is produced by a dedicated emotion GRU:

$u_t$3

This state propagates emotional context across utterances and links the updated speaker state to prior emotional states, thereby indirectly integrating the influence of other parties through the attended context embedded in earlier turns (Majumder et al., 2018).

Classification uses a 2-layer MLP with ReLU and softmax:

$u_t$4

$u_t$5

$u_t$6

Training uses categorical cross-entropy with L2 regularization,

$u_t$7

optimized with Adam (Majumder et al., 2018).

4. Datasets, benchmarks, and model variants

The original DialogueRNN evaluation used two datasets. IEMOCAP consists of dyadic video conversations with 10 unique speakers, with train+val comprising 5810 utterances across 120 dialogues and test comprising 1623 utterances across 31 dialogues; each utterance has one of six emotions: happy, sad, neutral, angry, excited, and frustrated. AVEC, based on modified SEMAINE, contains human–agent interactions with utterance-level labels derived by averaging annotations over each utterance span; train+val contains 4368 utterances across 63 dialogues and test contains 1430 utterances across 32 dialogues. IEMOCAP was evaluated with per-class accuracy and F1 plus weighted average accuracy and F1, while AVEC was evaluated with MAE and Pearson correlation coefficient $u_t$8 for valence, arousal, expectancy, and power (Majumder et al., 2018).

On IEMOCAP text-only classification, CMN achieved weighted F1 of 56.13% and weighted accuracy of 56.56, whereas DialogueRNN achieved weighted F1 of 59.89% and weighted accuracy of 59.33. DialogueRNN surpassed CMN on five of six emotions by F1, including sad at 69.83 versus 62.41 and excited at 64.45 versus 60.25, but was slightly worse on frustrated at 59.46 versus 60.69 (Majumder et al., 2018).

On AVEC text-only regression, DialogueRNN improved over CMN on all four attributes. For valence, DialogueRNN reported MAE 0.188 and $u_t$9 versus CMN’s MAE 0.192 and $p_{s(u_t)}$0; for arousal, 0.201 and 0.36 versus 0.213 and 0.29; for expectancy, 0.188 and 0.32 versus 0.195 and 0.26; and for power, 8.19 and 0.31 versus 8.74 and $p_{s(u_t)}$1 (Majumder et al., 2018).

Several variants were introduced. DialogueRNN$p_{s(u_t)}$2 incorporated listener updates and was slightly worse overall than DialogueRNN. BiDialogueRNN added bidirectional modeling. DialogueRNN+Att applied attention over emotion states. BiDialogueRNN+Att combined bidirectionality with emotional attention and yielded the strongest reported results in the original study: on IEMOCAP text, weighted F1 of 62.75% versus 59.89% for DialogueRNN and 56.13% for CMN; on AVEC text, best MAE and $p_{s(u_t)}$3 for all four attributes, including valence MAE 0.165 and $p_{s(u_t)}$4 (Majumder et al., 2018).

In multimodal form, BiDialogueRNN+Att$p_{s(u_t)}$5, using simple concatenation of text, visual, and acoustic features, achieved IEMOCAP F1 of 62.9% and AVEC correlations of 0.37 for valence, 0.60 for arousal, 0.37 for expectancy, and 0.41 for power (Majumder et al., 2018).

Later evaluation on MELD placed DialogueRNN in a larger multi-party and multimodal setting derived from Friends. MELD contains 1,433 dialogues and about 13,000 utterances, with train/dev/test splits of 9,989/1,109/2,610 utterances and an average of roughly 2.6–3.0 speakers per dialogue. In that paper, DialogueRNN was described as “a strong baseline” capable of emotion recognition in multi-party dialogues by inter-party dependency modeling. Reported weighted F1 scores on MELD were 57.03 for text-only, 41.79 for audio-only, and 60.25 for text+audio in 7-class emotion classification; in 5-class emotion classification, text+audio reached 61.6; and in 3-class sentiment classification, text+audio reached 67.56, the best reported sentiment baseline in that study (Poria et al., 2018).

5. Interpretability, ablation results, and error profile

DialogueRNN was accompanied by qualitative and quantitative analyses of its attention behavior. In attention over past global states, the paper reported that DialogueRNN’s $p_{s(u_t)}$6 attention is sharper and more focused than CMN’s. In an example where a speaker’s emotion shifted from neutral to frustrated, the model attended to the immediately relevant turns involving both the speaker and the interlocutor, whereas CMN’s attention was more diffuse and led to misclassification (Majumder et al., 2018).

For BiDialogueRNN+Att, the model also defined emotional attention over the sequence of emotion representations:

$p_{s(u_t)}$7

Visualizations showed that utterances sometimes attend strongly to future turns, including turns where a partner becomes enraged, suggesting that the model can use anticipated emotional trajectory to interpret current emotion. The same analysis showed that attention often localizes around stretches where the speaker’s emotional state is stable, but can also assign nontrivial mass to distant past or future context when necessary (Majumder et al., 2018).

The long-term dependency analysis reported that while the model relies strongly on local context, about 18% of cases attend to utterances 20–40 turns away. One example involved the utterance “Horrible thing. I hated it.” being correctly interpreted as excited or humorous by attending to earlier happy statements, showing that lexical polarity alone is insufficient for utterance-level classification in dialogue (Majumder et al., 2018).

Error analysis identified several recurrent failure modes. Misclassifications were concentrated among related emotions, particularly happy versus excited and angry versus frustrated. Neutral produced many false positives because it is the majority class. Emotional shift points were substantially harder than non-shift regions: the correct prediction rate was approximately 47.5% at shift points versus approximately 69.2% in non-shift regions. This indicates that modeling fine-grained emotional transitions remains difficult even when party state and context are explicitly represented (Majumder et al., 2018).

The ablation study isolated the contribution of the main recurrent modules. Removing party state while retaining the emotion GRU produced F1 = 55.56%; keeping party state but removing the emotion GRU produced F1 = 57.38%; using both produced F1 = 59.89%. The reported interpretation was that both components are important, with party state contributing more strongly than the emotion GRU (Majumder et al., 2018).

6. Position in the literature and later comparative assessments

Within dialog modeling more broadly, DialogueRNN belongs to a line of recurrent architectures that moved from utterance-isolated or flat contextual models toward explicit modeling of speaker interaction. A conceptually related precursor is the dialog context language modeling work that proposed IDCLM and ESIDCLM for two-speaker dialogue, distinguishing same-speaker continuation from cross-speaker response and introducing an external dialog-level state. That work did not model emotion, but it already encoded the core idea that dialog interaction should not be treated as ordinary document context; DialogueRNN can be understood as a more explicit realization of speaker-aware recurrent state tracking for utterance-level affective inference (Liu et al., 2017).

In the ERC literature, DialogueRNN was described as the state of the art when later systems were introduced. DialogueGCN characterized it as a recurrent model with three GRUs for speaker states, global context, and emotion state, and argued that recurrent speaker-level encoding suffers from long-term context propagation issues in long conversations. DialogueGCN further argued that attention-based pooling in DialogueRNN variants does not explicitly encode speaker information and relative position in the way graph edge types can. Empirically, DialogueGCN reported weighted F1 of 64.18% versus DialogueRNN’s 62.75% on IEMOCAP, lower MAE on all four AVEC dimensions, and MELD F1 of 58.10 versus 57.03 (Ghosal et al., 2019).

BiERU positioned DialogueRNN as a representative party-dependent architecture and used the same CNN utterance representation pipeline for fair comparison. Its critique focused on complexity, parameter cost, sensitivity to context and emotion shifts, and the bookkeeping burden of party-dependent modeling in multi-party dialogue. On IEMOCAP, BiERU-lc reported weighted accuracy of 66.09% and F1 of 64.59% versus DialogueRNN’s 63.40% and 62.75%; on AVEC it matched or exceeded DialogueRNN in Pearson $p_{s(u_t)}$8 on all dimensions except a tie on power; and on MELD it reported weighted average accuracy of 60.9% versus DialogueRNN’s 56.1%. The same comparison also emphasized parameter efficiency, reporting approximately 1M parameters for DialogueRNN versus approximately 0.5M for BiERU on IEMOCAP, and approximately 2.9M versus approximately 0.6M on MELD (Li et al., 2020).

These later comparisons clarify both the importance and the limits of DialogueRNN’s design. Its explicit party-state mechanism established speaker-conditioned recurrence as a central paradigm for conversational affect modeling, and MELD results supported the claim that speaker-specific modeling improves over speaker-agnostic baselines such as bcLSTM in multi-party settings (Poria et al., 2018). At the same time, later graph-based and party-ignorant models showed that speaker-aware recurrence is not the only viable route to strong conversational emotion recognition, especially in long, sparse, or heavily multi-party dialogues. A plausible implication is that DialogueRNN’s enduring contribution lies less in being a final architecture than in formalizing a durable decomposition—global context, party-specific state, and emotion dynamics—that subsequent ERC models either extend, replace, or explicitly argue against.