IEMOCAP: Multimodal Emotion Benchmark

• IEMOCAP is a multimodal dataset of five dyadic sessions that combine scripted and improvised interactions with aligned speech, text, video, and motion-capture streams.
• It supports diverse tasks in speech, multimodal, and conversational emotion recognition using both discrete labels and continuous ratings of valence, dominance, and activation.
• Evaluation protocols and data splits vary widely, underscoring challenges in reproducibility, class imbalance, and cross-modal alignment.

IEMOCAP, short for the Interactive Emotional Dyadic Motion Capture database, is a multimodal corpus of acted dyadic interaction that has become a central benchmark for speech emotion recognition (SER), multimodal emotion recognition (MER), and emotion recognition in conversation (ERC). It is described as a corpus of five dyadic interaction sessions, each with a unique male–female speaker pair, for a total of 10 speakers, and it combines scripted and improvised interactions annotated at the level of speaker turns with both discrete categorical labels and continuous dimensional ratings such as valence, dominance, and activation (Antoniou et al., 2023). Because it provides aligned speech, text transcripts, video, and motion-capture streams, the dataset has supported a wide range of modeling paradigms, from speech-only classifiers to tri-modal Transformer pipelines and dialogue-context models (Tripathi et al., 2018, Shayaninasab et al., 2024).

1. Corpus design and multimodal composition

IEMOCAP is routinely described as containing about 12 hours of multimodal or audiovisual interaction data produced by 10 professional actors arranged into 5 sessions, with one male and one female actor in each session (Gao et al., 20 May 2025, Chaudhari et al., 2024). The interactions include both scripted and improvised dialogues, and several studies describe utterances as being segmented at the level of speaker turns, with conversational segments typically spanning about 3 to 15 seconds (Antoniou et al., 2023, Chaudhari et al., 2024). A recent missing-modality study reports 10,039 utterances in the full corpus before the standard four-class reduction (Sun et al., 2024).

The dataset is unusual among affective computing benchmarks because it exposes several synchronized modalities. In addition to speech/audio and human-annotated transcripts, it includes video and motion capture channels for face expressions, hand movements, and head rotation (Tripathi et al., 2018). This multimodal design has made IEMOCAP suitable not only for utterance-level classification but also for feature alignment, cross-modal fusion, and conversational-context modeling. A plausible implication is that the dataset’s enduring importance comes less from raw scale than from the density of aligned supervisory signals across modalities.

2. Annotation scheme and label configurations

Each conversational segment is annotated for perceived emotion by 3 annotators, who assign both discrete categorical labels and continuous dimensional ratings along valence, dominance, and activation (Antoniou et al., 2023). Other descriptions emphasize that utterances are often filtered by agreement criteria before benchmark construction: some studies keep only utterances where at least two annotators agreed, while others retain only utterances with a majority label (Tripathi et al., 2018, Cho et al., 2019). These filtering steps are not incidental; they materially define the benchmark actually being used.

The most common SER formulation is a 4-way classification problem in which all segments outside {neutral, happy, sad, angry, excited} are discarded and happy and excited are merged because of their “expressive closeness” (Antoniou et al., 2023). Under that convention, one paper reports $N = 5531$ samples with class counts {1708, 1636, 1103, 1084} for {neutral, happy, sad, angry} (Antoniou et al., 2023). A recent multimodal Transformer study reports a closely related subset of 5,532 rows after its own preprocessing and likewise merges excited into happiness because of emotional similarity and low frequency (Shayaninasab et al., 2024).

Common configuration	Label handling	Reported size
Standard 4-class SER setup	Discard labels not in {neutral, happy, sad, angry, excited}; merge happy + excited	N = 5531 (Antoniou et al., 2023)
Recent multimodal 4-class setup	Select neutral, anger, sadness, happiness; merge excited into happiness	5532 rows (Shayaninasab et al., 2024)
Alternative branch criticized in the literature	Use {neutral, sad, angry, excited} and drop happy	Separate incompatible branch (Antoniou et al., 2023)

A major source of incompatibility is the handling of happy versus excited. The literature review in (Antoniou et al., 2023) argues that using the merged happy+excited class and reporting that setting as the primary benchmark is necessary for fair comparison, because alternative choices create separate evaluation branches that are not directly comparable.

3. Evaluation protocols and reproducibility

A defining feature of IEMOCAP benchmarking is that the corpus has no established nominal test set, so each study must specify its own split protocol (Antoniou et al., 2023). Two Speaker Independent (SI) protocols recur most often. The first is 5-fold cross-validation, also called leave-one-session-out, where four sessions train and one session tests. The second is 10-fold cross-validation, also called leave-one-speaker-out, where one speaker is held out for test and the remaining speakers are used for training and validation (Antoniou et al., 2023). In SER papers, performance is commonly summarized with Weighted Accuracy (WA), defined as the percentage of correct predictions, and Unweighted Accuracy (UA), defined as the average recall across classes (Antoniou et al., 2023). Other studies use accuracy, precision, recall, F1-score, average recall (AvRec), or average AUC (AvAUC) depending on the task formulation (Shayaninasab et al., 2024, Pepino et al., 2024).

The absence of a single canonical split has produced a large amount of protocol heterogeneity. Some work uses sessions 1–4 for training and session 5 for testing (Liang et al., 2023, Jiao et al., 2019); one multimodal Transformer study performs final evaluation with 5-fold session-based cross-validation, but also uses sessions 1–3 for training, session 4 for validation, and session 5 for testing during model selection (Shayaninasab et al., 2024). ERC-oriented work has introduced dialogue-level splits, for example 80% train, 10% validation, and 10% test, with all utterances from a dialogue kept in the same partition (Li et al., 26 Mar 2025). These differences mean that benchmark numbers on IEMOCAP are inseparable from the protocol used to obtain them.

The reproducibility literature identifies three recurrent failure modes. First, Speaker Dependent (SD) random splits can leak speaker identity across train and test partitions and are explicitly criticized as inflating performance (Antoniou et al., 2023). Second, some studies evaluate only the improvised subset, while others use the full dataset; without explicit disclosure, the resulting numbers are difficult to contextualize (Antoniou et al., 2023). Third, scripted material can induce severe lexical leakage for text-based models. A fusion study shows that the usual speaker-based split can still be overly optimistic for transcript-based systems because the same scripts may appear in both train and test folds, and therefore recommends a stricter speaker-and-script split or discarding scripted data altogether (Pepino et al., 2024). A separate analysis of dataset composition reaches a similar conclusion, finding that the scripted portion of IEMOCAP yields high within-dataset accuracy but poor cross-dataset generalization because of strong sentence overlap and lexical redundancy (Sutherland et al., 2021).

4. Modality instantiation and preprocessing practice

Although IEMOCAP is a fixed corpus, its modalities are operationalized differently across tasks. For text, multimodal work commonly reads each conversational turn from the transcript files and tokenizes it into sentence-level examples (Shayaninasab et al., 2024). Several text-based and multimodal studies state explicitly that they use human transcriptions rather than ASR output, which is important because it turns the textual branch into an analysis of the information content available in the reference transcript itself (Pepino et al., 2024, Cho et al., 2019).

For speech, the utterance is the basic unit in most SER work. In the 2024 Transformer-based MER pipeline, audio segments are already available at the utterance level; preprocessing consists mainly of matching utterances to labels, standardizing audio to 16 kHz, and truncating utterances longer than 10 seconds (Shayaninasab et al., 2024). Other pipelines operate on handcrafted frame-level descriptors, log-mel features, MFSC maps, or self-supervised speech representations, but the common assumption is still utterance-level prediction from a variable-length acoustic sequence (Lu et al., 2019, Boigne et al., 2020).

For video, the same multimodal Transformer study reconstructs one clip per utterance using transcript timing information, converts clips to MP4, removes audio, applies random horizontal mirroring with 50% probability, samples frames at 30 fps, and normalizes pixel values (Shayaninasab et al., 2024). This setup reflects a frequent pattern in IEMOCAP work: the video modality is aligned to utterances through transcript time stamps rather than treated as an independently segmented stream.

IEMOCAP’s motion-capture channels provide an additional representation not present in many later benchmarks. One multimodal deep-learning study decomposes MoCap into face (165 dimensions), hands (18 dimensions), and head rotation (6 dimensions), samples the values over each utterance, splits them into 200 partitions, averages within each partition, and concatenates the result into a (200, 189) representation (Tripathi et al., 2018). This MoCap formulation allows models to exploit facial and bodily movement without full video processing. A plausible implication is that IEMOCAP’s multimodal richness resides not only in raw sensory coverage but also in the availability of several alternative visual abstractions.

5. Benchmark roles and representative model families

IEMOCAP has supported several generations of SER baselines. On the improvised subset with four emotions, a CNN + BiLSTM operating on spectrograms reported 64.5% WA and 61.7% UA under 10-fold cross-validation (Etienne et al., 2018). A later speech-only approach based on pre-trained end-to-end ASR features from an RNN-T encoder and a bi-LSTM + multi-head self-attention decoder reported 71.7% WA and 72.6% UA under 10-fold leave-one-speaker-out cross validation (Lu et al., 2019). Self-supervised transfer learning further strengthened the benchmark: using the full four-class subset of 5,531 utterances, a wav2vec acoustic model reached 64.3% UA, while a multimodal wav2vec + BERT model aligned by an attention-based recurrent architecture reached 73.9% UA (Boigne et al., 2020). In a later cross-corpus setup with supervised contrastive learning, a WavLM-based system reported 77.41% UA on IEMOCAP under 5-fold cross-validation (minjie, 2024).

Multimodal studies use IEMOCAP to test whether emotional information is complementary across channels. An early deep multimodal system combining speech, text, and motion capture via late fusion reported 71.04% accuracy on a session-based train/test split (Tripathi et al., 2018). A more recent Transformer-based MER pipeline fine-tuned BERT, wav2vec 2.0, and VideoMAE, extracted the final-layer CLS representations of dimension 768, and found that early fusion of the three modality embeddings with an SVM achieved 75.42% accuracy on the four-class IEMOCAP task; in that study, anger was the strongest class and neutral the weakest (Shayaninasab et al., 2024). Another tri-modal architecture, GCM-Net, reported 85.66% accuracy and 72.69% F1-score on a four-class utterance-level setting, using graph-based recalibration, cross-modal attention, and a ConvXGB classifier (Chaudhari et al., 2024).

IEMOCAP is also a standard ERC benchmark rather than only an utterance-isolated MER corpus. In GatedxLSTM, the dataset is filtered to 151 dialogues and 4,490 utterances in four classes, with the model using the current speaker’s and interlocutor’s audio and text plus limited preceding context under a dialogue-level split. That work reports 76.34 ± 1.31% weighted accuracy and 75.97 ± 1.38% weighted F1 (Li et al., 26 Mar 2025). The breadth of these settings shows that IEMOCAP functions simultaneously as a speech corpus, a multimodal utterance benchmark, and a conversation-level affect dataset.

6. Dataset bias, limitations, and later extensions

The same properties that made IEMOCAP influential also make it methodologically difficult. The corpus is repeatedly described as small relative to newer SER resources; one review contrasts it with MSP-Podcast, cited as 27 hours, and argues that IEMOCAP’s limited size and class imbalance create substantial risks of overfitting and class bias (Antoniou et al., 2023). The imbalance is especially important in the standard four-class setup, where most samples come from neutral and happy (Antoniou et al., 2023). This has motivated methods based on augmentation, class-weighting, focal-style objectives, and distribution smoothing. For example, PDDS treats IEMOCAP as not only imbalanced across clear classes but also uneven across ambiguous emotion pairs, and reports gains of 0.2%–4.8% WA and 1.5%–5.9% UA across several models (Liang et al., 2023).

Text-based performance on IEMOCAP is a recurring point of controversy. Transcript models can be very strong within the corpus—one hierarchical text-only GRU reports 82.1 WA / 80.6 UWA under a sessions 1–4 train, session 5 test protocol (Jiao et al., 2019)—but the dataset-composition literature argues that such gains may partly reflect memorization of scripted lexical material rather than robust emotion understanding (Sutherland et al., 2021, Pepino et al., 2024). This does not imply that text is uninformative; rather, it indicates that IEMOCAP’s scripted structure can confound the interpretation of text-dominant or multimodal results unless script overlap is controlled explicitly.

The corpus has also been extended beyond its original annotation design. PA-IEMOCAP adds Big Five personality traits at the conversation level, producing 302 personality profiles for 151 dyadic interactions and reporting ICC values from 0.89 to 0.97, with an average of 0.92 (Gao et al., 20 May 2025). In parallel, IEMOCAP has become a testbed for robustness to missing modalities and for data compression. CM-ARR evaluates six incomplete-modality settings over text, speech, and video and reports average gains of 2.11% WAR and 2.12% UAR over the best baseline on IEMOCAP (Sun et al., 2024). A dataset-distillation study treats IEMOCAP itself as the source dataset to be compressed into a conditional generative model, reporting 94–97% storage reduction and about 95% downstream training-time reduction while remaining close to full-data performance in UAR (Ritter-Gutierrez et al., 2024). These extensions indicate that IEMOCAP is no longer only a benchmark for classifying four acted emotions; it is also an experimental substrate for studying evaluation design, multimodal robustness, speaker attributes, and efficient representation learning.