---
title: Emotion Transcription in Conversation (ETC)
url: https://www.emergentmind.com/topics/emotion-transcription-in-conversation-etc
type: topic
---

# Emotion Transcription in Conversation (ETC)

Searching arXiv for the cited ETC/ERC papers to ground the article in the current literature.
I’ll look up the relevant arXiv records now.
Emotion Transcription in Conversation (ETC) is a dialogue-emotion task in which a model generates a natural-language description of a speaker’s emotional state for each utterance, conditioned on the conversational context, rather than assigning only a categorical label [2603.07138]. It is positioned as a response to the limitations of conventional Emotion Recognition in Conversation (ERC), where categorical or dimensional annotations can fail to represent subtle, mixed, context-dependent, or culturally specific affective states [2603.07138]. Within the broader ERC literature, ETC also intersects with work on context-sensitive speaker modeling, commonsense inference, multimodal fusion, and transfer learning, because all of these lines of research address the underlying problem of inferring emotion from conversational structure rather than from isolated utterances alone [2403.07260].

## 1. Conceptual definition and relation to ERC

ETC is defined as generating “natural language descriptions that accurately reflect speakers' emotional states within conversational contexts” [2603.07138]. In the ETC formulation, the relevant question is not only which emotion category applies to an utterance, but “How would we describe the speaker’s internal emotional state in natural language, given the conversational context?” [2603.07138]. The task therefore changes the output space from a fixed inventory of labels to free-form text.

Traditional ERC, by contrast, is formulated as utterance-level emotion prediction in a dialogue. One formulation represents a conversation as a sequence \(U=[(u_1,p_1),(u_2,p_2),\dots,(u_n,p_n)]\), with the goal of predicting an emotion label \(e_i\) for each utterance \(u_i\) [2403.07260]. A multimodal variant writes the conversation as \(C = [(u_1,y_1),(u_2,y_2),\dots,(u_K,y_K)]\), where each utterance \(u_k=\{S_k,T_k\}\) includes speech and transcript modalities [2501.11468]. ETC preserves the context dependence central to ERC, but replaces label prediction with context-to-text generation [2603.07138].

The ETC paper identifies three explicit differences from traditional ERC: output format, expressiveness, and evaluation [2603.07138]. ERC outputs a label; ETC outputs a natural-language description. ERC is constrained by a fixed label inventory; ETC can express mixed, nuanced, and implicit emotions. ERC is usually evaluated with classification metrics; ETC is evaluated as a generation task using text-generation metrics and a fine-grained faithfulness score [2603.07138]. This suggests ETC is not merely a relabeling of ERC, but a reformulation that treats emotional understanding as descriptive inference.

## 2. Motivation: expressive emotion representation in dialogue

The ETC benchmark is motivated by the claim that existing ERC methods “predominantly employ categorical or dimensional emotion annotations, which often fail to adequately represent complex, subtle, or culturally specific emotional nuances” [2603.07138]. The paper emphasizes that real emotions in dialogue are frequently subtle, mixed or blended, context-dependent, and sometimes culturally specific [2603.07138]. Because ETC uses free-form descriptions, it can represent “what the speaker feels,” “why they feel it,” “how several emotions coexist,” and whether the state is tied to intention, empathy, relief, or shock [2603.07138].

This concern is consistent with earlier ERC research that treats conversation as a context-sensitive inference problem. TL-ERC argues that an utterance’s emotion often depends on earlier utterances, speaker identity, self-influence, inter-speaker influence, conversation topic, and latent pragmatic factors [1910.04980]. DialogueTRM likewise distinguishes individual context and conversational context, formalizing the difference between same-speaker history and all-speaker history [2010.07637]. LaERC-S, described in the provided details as CKERC, further argues that emotion should be inferred from the speaker’s latent mental state, intended effect, and likely reactions from others, rather than from text alone [2403.07260].

A common misconception is that ETC is simply a more verbose form of classification. The ETC paper argues otherwise by showing that emotion transcriptions differ lexically from utterances themselves: utterances tend to contain event or action words, while emotion transcriptions tend to contain internal feeling words and nuanced descriptors [2603.07138]. This supports the view that ETC is not paraphrasing dialogue turns, but modeling the speaker’s internal state.

## 3. Formal task specification and dataset design

The ETC paper formalizes a dialogue as
\[
D = ((u_1, s_1), (u_2, s_2), \dots, (u_N, s_N)),
\]
where \(u_i\) is the \(i\)-th utterance and \(s_i\) is the speaker of that utterance, with \(N = 10\) in the dataset [2603.07138]. For any position \(n\), the context up to that utterance is
\[
C_n = ((u_1, s_1), (u_2, s_2), \dots, (u_n, s_n)),
\]
and the model \(\mathcal{M}\) is trained to generate
\[
e_n = \mathcal{M}(C_n),
\]
where \(e_n\) is a natural-language description of the emotional state of speaker \(s_n\) at the time of uttering \(u_n\) [2603.07138]. The task is therefore explicitly a context-to-text generation problem.

The dataset is a Japanese text-based dialogue corpus collected using CrowdWorks [2603.07138]. It uses a speaker-listener setup inspired by EmpatheticDialogues, with one participant assigned the Speaker role and the other the Listener role [2603.07138]. Each dialogue contains 5 turns per person, for 10 utterances total, with alternating turns beginning from the Speaker [2603.07138]. Speakers were given one of 32 emotion labels, described as Japanese translations of the labels used in EmpatheticDialogues, and were instructed to talk about a personal experience related to that emotion [2603.07138].

After each utterance, both participants wrote a free-form natural-language description of their own emotional state at that moment [2603.07138]. The paper defines an emotion transcription as “a verbalization of the internal emotional state or intention a participant held at the time of their utterance” [2603.07138]. The resulting corpus contains 1,002 dialogues, 10,020 utterances / emotion transcriptions, and 199 unique crowdworkers [2603.07138]. The average utterance length is 42.72 characters, with 44.64 for Speakers and 40.80 for Listeners, while the average emotion transcription length is 28.89 characters, with 28.92 for Speakers and 28.86 for Listeners [2603.07138].

The dialogues were roughly balanced across the 32 emotion prompts, with about 30–32 dialogues per emotion label [2603.07138]. This suggests the corpus was designed to diversify affective starting conditions while preserving a fixed interactional structure.

## 4. Annotation scheme, corpus analysis, and empirical properties

Because free-form transcriptions are difficult to analyze quantitatively, the ETC dataset additionally annotates each transcription with emotion categories [2603.07138]. The category inventory contains 7 labels: joy, sadness, fear, anger, surprise, disgust, and neutral, corresponding to Ekman’s six basic emotions plus neutral [2603.07138]. Annotation is multi-label rather than single-label, so one transcription may express multiple emotions [2603.07138].

Each transcription was labeled by 3 annotators who saw the full dialogue context and the transcription [2603.07138]. For each of the 7 categories, annotators answered whether it was present. Final labels were assigned by majority vote: a label was assigned if 2 or more of 3 annotators agreed, and if no category reached agreement, the transcription was labeled neutral only [2603.07138]. The paper reports an overall Fleiss’ kappa of \(\kappa = 0.533\), interpreted as moderate agreement [2603.07138].

The corpus statistics show a strong neutral component together with substantial non-neutral affect. The overall distribution is Neutral 50.3%, Joy 25.0%, Sadness 9.8%, Fear 6.2%, Anger 3.2%, Surprise 3.5%, and Disgust 4.8% [2603.07138]. Speakers show more labeled emotion than listeners, while surprise appears more often for listeners than speakers [2603.07138]. About 5.6% of non-neutral transcriptions contain multiple emotion labels [2603.07138]. This multi-emotion rate provides direct evidence for the ETC claim that dialogue emotions can be mixed or overlapping.

| Aspect | ETC dataset property |
|---|---|
| Dialogue structure | 5 turns per person, 10 utterances total |
| Emotion transcription | Free-form natural-language description after each utterance |
| Category annotation | 7-label multi-label annotation with majority vote |
| Corpus size | 1,002 dialogues; 10,020 utterances / emotion transcriptions |
| Inter-annotator agreement | Fleiss’ kappa \(\kappa = 0.533\) |
| Multi-emotion transcriptions | About 5.6% of non-neutral transcriptions |

The annotation design also preserves a bridge back to ERC. Because each transcription is associated with category labels, the dataset supports both natural-language generation and conventional classification-style analysis [2603.07138]. A plausible implication is that ETC can function both as a richer benchmark and as a resource for studying the relation between free-form affect descriptions and standard emotion inventories.

## 5. Benchmark models and evaluation methodology

The ETC benchmark evaluates GPT-4.1 and Llama-3.1-Swallow-8B-Instruct-v0.3 in zero-shot and 4-shot settings, and additionally evaluates supervised fine-tuning of Llama-3.1 on the ETC dataset [2603.07138]. The dataset split is 80% train, 10% validation, and 10% test [2603.07138]. Fine-tuning was performed on 2 NVIDIA A100 80GB GPUs with batch size 8 for 2 epochs, using 4-bit quantization and PagedAdamW8bit, with validation every 200 steps [2603.07138]. The learning rate was selected from \(\{1e{-}5, 5e{-}5\}\), warm-up steps from \(\{300, 700\}\), and the best setting was \(1e{-}5\) with 300 warm-up steps [2603.07138].

Evaluation uses BLEU, ROUGE, and BERTScore, together with a fine-grained faithfulness evaluation inspired by FActScore [2603.07138]. In the faithfulness protocol, the emotion transcription is decomposed into atomic units, each containing one piece of emotional information, and each unit is judged as Supported, Not Supported, or Neutral [2603.07138]. Precision is the proportion of predicted atomic units supported by the ground truth, Recall is the proportion of ground-truth atomic units supported by the prediction, and F1 is the harmonic mean of Precision and Recall [2603.07138]. If the ground-truth transcription has no emotional description, it is excluded from this metric; if the model outputs nothing while the reference has emotion content, both Precision and Recall are set to 0 [2603.07138].

The ETC results show that fine-tuning Llama-3.1 on the dataset improves performance across all automatic metrics [2603.07138]. The fine-tuned model is best on BLEU, ROUGE, BERTScore, Precision, and F1, with a best reported F1 of 14.29 [2603.07138]. The fine-grained evaluation further shows that zero-shot models, especially GPT-4.1, tend to generate more atomic units, which increases Recall but lowers Precision [2603.07138]. The reported average atomic-unit counts are 1.35 for ground truth, 1.39 for fine-tuned Llama-3.1, 1.99 for Llama-3.1 zero-shot, and 3.39 for GPT-4.1 zero-shot [2603.07138]. This suggests that ETC models must balance informativeness against overgeneration.

A central finding is that current models still struggle with implicit emotional states [2603.07138]. The paper presents a case in which the literal dialogue concerns a negative event, but the true emotion transcription is happiness because of the partner’s empathetic response; most models focus on the explicit negative cues and miss the emergent positive emotion [2603.07138]. This is consistent with the ETC claim that affect in dialogue can emerge from interaction rather than from lexical polarity alone.

## 6. Technical antecedents in ERC: speaker modeling, multimodality, and transfer

Although ETC is defined as a generation task, several ERC lines of work provide technical antecedents for the kinds of inference ETC requires.

LaERC-S, described in the provided details as CKERC, uses LLM-generated commonsense conditioned on dialogue history to improve utterance-level emotion prediction [2403.07260]. It centers on ATOMIC relations in the mental-state category, specifically xIntent, xReact, and oReact, and uses Llama-2-7B-Chat to generate commonsense descriptions from the historical utterances preceding a target utterance [2403.07260]. The framework is trained in two stages: speaker commonsense identification with loss
\[
L_{cp}=\sum_1^n -\log{P(t_i,input_i)},
\]
followed by ERC fine-tuning with
\[
L_{e}=\sum_1^n -\log{P(e_i,input_i)}.
\]
It reports weighted-F1 scores of 72.40 on IEMOCAP, 42.08 on EmoryNLP, and 69.27 on MELD, compared with InstructERC scores of 71.39, 41.37, and 69.15 respectively, for an average weighted-F1 of 61.25 versus 60.64 [2403.07260]. In ETC terms, this work is relevant because it treats emotion as an inference over mental state, speaker intention, and listener reaction rather than as a surface-text label.

MERITS-L extends the ETC-related logic to multimodal ERC by exploiting transcribed emotional speech [2501.11468]. It first uses Whisper-large-v3 to transcribe MSP-PODCAST, then prompts GPT-3.5 Turbo for positive, negative, or neutral pseudo-labels, and fine-tunes RoBERTa-large on the resulting pseudo-labeled transcripts [2501.11468]. The downstream model combines text embeddings with CARE speech embeddings through a three-stage hierarchical process: utterance-level unimodal classification, conversation-level contextual modeling with a Bi-GRU with self-attention, and co-attention-based multimodal fusion [2501.11468]. On IEMOCAP, MELD, and CMU-MOSI, the final Audio+Text Stage III model reports weighted F1 of 86.48, 66.02, and 86.81 respectively [2501.11468]. The paper further reports that GPT-3.5-supervised RoBERTa improves over plain pre-trained RoBERTa by +8.22% on IEMOCAP, +5.01% on MELD, and +21.9% on CMU-MOSI [2501.11468]. This indicates that transcript-based supervision can be a strong intermediate representation for conversational emotion modeling.

DialogueTRM approaches the same problem from multimodal context modeling [2010.07637]. It represents conversational input as
\[
X=\{\! x^\lambda_\tau = \langle x^\lambda_{\tau,(t)}, x^\lambda_{\tau,(v)}, x^\lambda_{\tau,(a)} \rangle \mid \tau\in[1,L],\lambda\in[1,N]\},
\]
distinguishes individual context \(\varphi(x_i^j,X,K)\) from conversational context \(\phi(x_i^j,X,K)\), and uses a Hierarchical Transformer plus Multi-Grained Interactive Fusion [2010.07637]. Text is modeled with a context-dependent sequential Transformer, while visual and acoustic modalities use a context-free setting that behaves effectively as feed-forward [2010.07637]. DialogueTRM reports improvements over prior SOTA of +4.3% ACC and +7.1% F1 on IEMOCAP, and +10.4% ACC and +4.5% F1 on MELD [2010.07637]. For ETC, the significance lies in its explicit treatment of modality-specific temporal behavior and speaker-specific versus full-conversation context.

TL-ERC provides a transfer-learning perspective [1910.04980]. It pretrains a Hierarchical Recurrent Encoder-Decoder on large conversational corpora and transfers the context encoder to an ERC classifier with BERT-based utterance representations [1910.04980]. On IEMOCAP, weighted F-score rises from 53.8 with random initialization and 55.1 with BERT only to 58.8 with TL-ERC + Ubuntu context and 58.5 with TL-ERC + Cornell context [1910.04980]. The method is also evaluated under 10%, 25%, 50%, and 100% data conditions and is reported to be more robust in low-data settings [1910.04980]. This suggests that conversational pretraining may be especially relevant for ETC, where supervision is richer but harder to scale than categorical labels.

## 7. Research significance, limitations, and open directions

ETC reframes dialogue emotion understanding as generation of internal-state descriptions rather than classification over a fixed label set [2603.07138]. The benchmark demonstrates that supervised fine-tuning improves performance, but also that performance remains low overall and that implicit emotions remain difficult even for strong models [2603.07138]. The ETC paper therefore positions the task as a difficult benchmark rather than a solved extension of ERC.

Several limitations are explicit in the ETC benchmark. The current dataset is Japanese-only, so generalization to other languages and cultures remains open [2603.07138]. The authors also point to the need for better modeling of implicit emotion, advanced prompting and training such as chain-of-thought prompting, RLHF, and contrastive learning, speaker modeling through speaker embeddings, personality traits, and dynamic speaker state, greater use of human evaluation, and multimodal ETC with audio and visual cues [2603.07138]. These proposed directions align with the ERC literature summarized above: speaker characteristics in LaERC-S [2403.07260], transcript-based multimodal learning in MERITS-L [2501.11468], differentiated modality fusion in DialogueTRM [2010.07637], and conversational transfer in TL-ERC [1910.04980].

A further misconception is that ETC supersedes ERC. The ETC paper instead makes the dataset usable for conventional ERC-style classification experiments by adding emotion category labels to each transcription [2603.07138]. ETC is therefore better understood as an expanded formulation that can connect back to ERC quantitatively while preserving a richer descriptive target. A plausible implication is that ETC may serve as a testbed for studying whether models that can describe emotional states in natural language also acquire more faithful latent representations for downstream ERC.

In that sense, ETC occupies a distinctive position in dialogue affect analysis. It retains ERC’s emphasis on utterance-level prediction in context, but demands a representation of emotion that is compositional, context-sensitive, interaction-aware, and linguistically explicit [2603.07138]. The benchmark’s broader significance lies in making those requirements measurable.

Source: https://www.emergentmind.com/topics/emotion-transcription-in-conversation-etc