---
title: 'IEMOCAP: Multimodal Emotion Benchmark'
url: https://www.emergentmind.com/topics/iemocap-dataset
type: topic
---

# IEMOCAP: Multimodal Emotion Benchmark

IEMOCAP, short for the **Interactive Emotional Dyadic Motion Capture** database, is a multimodal corpus of acted dyadic interaction that has become a central benchmark for speech emotion recognition (SER), multimodal emotion recognition (MER), and emotion recognition in conversation (ERC). It is described as a corpus of **five dyadic interaction sessions**, each with a unique **male–female speaker pair**, for a total of **10 speakers**, and it combines **scripted** and **improvised** interactions annotated at the level of speaker turns with both **discrete categorical labels** and **continuous dimensional ratings** such as **valence, dominance, and activation** [2304.00860]. Because it provides aligned **speech**, **text transcripts**, **video**, and **motion-capture** streams, the dataset has supported a wide range of modeling paradigms, from speech-only classifiers to tri-modal Transformer pipelines and dialogue-context models [1804.05788, 2402.07327].

## 1. Corpus design and multimodal composition

IEMOCAP is routinely described as containing about **12 hours** of multimodal or audiovisual interaction data produced by **10 professional actors** arranged into **5 sessions**, with one male and one female actor in each session [2505.13978, 2410.12828]. The interactions include both **scripted** and **improvised** dialogues, and several studies describe utterances as being segmented at the level of **speaker turns**, with conversational segments typically spanning about **3 to 15 seconds** [2304.00860, 2410.12828]. A recent missing-modality study reports **10,039 utterances** in the full corpus before the standard four-class reduction [2407.09029].

The dataset is unusual among affective computing benchmarks because it exposes several synchronized modalities. In addition to **speech/audio** and **human-annotated transcripts**, it includes **video** and **motion capture** channels for **face expressions**, **hand movements**, and **head rotation** [1804.05788]. This multimodal design has made IEMOCAP suitable not only for utterance-level classification but also for feature alignment, cross-modal fusion, and conversational-context modeling. A plausible implication is that the dataset’s enduring importance comes less from raw scale than from the density of aligned supervisory signals across modalities.

## 2. Annotation scheme and label configurations

Each conversational segment is annotated for perceived emotion by **3 annotators**, who assign both **discrete categorical labels** and **continuous dimensional ratings** along **valence, dominance, and activation** [2304.00860]. Other descriptions emphasize that utterances are often filtered by agreement criteria before benchmark construction: some studies keep only utterances where **at least two annotators agreed**, while others retain only utterances with a **majority** label [1804.05788, 1911.00432]. These filtering steps are not incidental; they materially define the benchmark actually being used.

The most common SER formulation is a **4-way classification problem** in which all segments outside `{neutral, happy, sad, angry, excited}` are discarded and **happy** and **excited** are merged because of their **“expressive closeness”** [2304.00860]. Under that convention, one paper reports **\(N = 5531\)** samples with class counts `{1708, 1636, 1103, 1084}` for `{neutral, happy, sad, angry}` [2304.00860]. A recent multimodal Transformer study reports a closely related subset of **5,532 rows** after its own preprocessing and likewise merges **excited** into **happiness** because of emotional similarity and low frequency [2402.07327].

| Common configuration | Label handling | Reported size |
|---|---|---|
| Standard 4-class SER setup | Discard labels not in `{neutral, happy, sad, angry, excited}`; merge `happy + excited` | `N = 5531` [2304.00860] |
| Recent multimodal 4-class setup | Select `neutral, anger, sadness, happiness`; merge `excited` into `happiness` | `5532 rows` [2402.07327] |
| Alternative branch criticized in the literature | Use `{neutral, sad, angry, excited}` and drop `happy` | Separate incompatible branch [2304.00860] |

A major source of incompatibility is the handling of **happy** versus **excited**. The literature review in [2304.00860] argues that using the merged **happy+excited** class and reporting that setting as the primary benchmark is necessary for fair comparison, because alternative choices create separate evaluation branches that are not directly comparable.

## 3. Evaluation protocols and reproducibility

A defining feature of IEMOCAP benchmarking is that the corpus has **no established nominal test set**, so each study must specify its own split protocol [2304.00860]. Two **Speaker Independent (SI)** protocols recur most often. The first is **5-fold cross-validation**, also called **leave-one-session-out**, where four sessions train and one session tests. The second is **10-fold cross-validation**, also called **leave-one-speaker-out**, where one speaker is held out for test and the remaining speakers are used for training and validation [2304.00860]. In SER papers, performance is commonly summarized with **Weighted Accuracy (WA)**, defined as the **percentage of correct predictions**, and **Unweighted Accuracy (UA)**, defined as the **average recall across classes** [2304.00860]. Other studies use **accuracy**, **precision**, **recall**, **F1-score**, **average recall (AvRec)**, or **average AUC (AvAUC)** depending on the task formulation [2402.07327, 2403.18635].

The absence of a single canonical split has produced a large amount of protocol heterogeneity. Some work uses **sessions 1–4 for training and session 5 for testing** [2302.08650, 1904.04446]; one multimodal Transformer study performs final evaluation with **5-fold session-based cross-validation**, but also uses **sessions 1–3 for training, session 4 for validation, and session 5 for testing** during model selection [2402.07327]. ERC-oriented work has introduced **dialogue-level** splits, for example **80% train, 10% validation, and 10% test**, with all utterances from a dialogue kept in the same partition [2503.20919]. These differences mean that benchmark numbers on IEMOCAP are inseparable from the protocol used to obtain them.

The reproducibility literature identifies three recurrent failure modes. First, **Speaker Dependent (SD)** random splits can leak speaker identity across train and test partitions and are explicitly criticized as inflating performance [2304.00860]. Second, some studies evaluate only the **improvised** subset, while others use the full dataset; without explicit disclosure, the resulting numbers are difficult to contextualize [2304.00860]. Third, scripted material can induce severe lexical leakage for text-based models. A fusion study shows that the usual speaker-based split can still be overly optimistic for transcript-based systems because the same scripts may appear in both train and test folds, and therefore recommends a stricter **speaker-and-script** split or discarding scripted data altogether [2403.18635]. A separate analysis of dataset composition reaches a similar conclusion, finding that the scripted portion of IEMOCAP yields high within-dataset accuracy but poor cross-dataset generalization because of strong sentence overlap and lexical redundancy [2103.03700].

## 4. Modality instantiation and preprocessing practice

Although IEMOCAP is a fixed corpus, its modalities are operationalized differently across tasks. For **text**, multimodal work commonly reads each conversational turn from the transcript files and tokenizes it into sentence-level examples [2402.07327]. Several text-based and multimodal studies state explicitly that they use **human transcriptions rather than ASR output**, which is important because it turns the textual branch into an analysis of the information content available in the reference transcript itself [2403.18635, 1911.00432].

For **speech**, the utterance is the basic unit in most SER work. In the 2024 Transformer-based MER pipeline, audio segments are already available at the utterance level; preprocessing consists mainly of matching utterances to labels, standardizing audio to **16 kHz**, and truncating utterances longer than **10 seconds** [2402.07327]. Other pipelines operate on handcrafted frame-level descriptors, log-mel features, MFSC maps, or self-supervised speech representations, but the common assumption is still utterance-level prediction from a variable-length acoustic sequence [1911.09762, 2011.05585].

For **video**, the same multimodal Transformer study reconstructs one clip per utterance using transcript timing information, converts clips to **MP4**, removes audio, applies random horizontal mirroring with **50% probability**, samples frames at **30 fps**, and normalizes pixel values [2402.07327]. This setup reflects a frequent pattern in IEMOCAP work: the video modality is aligned to utterances through transcript time stamps rather than treated as an independently segmented stream.

IEMOCAP’s **motion-capture** channels provide an additional representation not present in many later benchmarks. One multimodal deep-learning study decomposes MoCap into **face** (**165 dimensions**), **hands** (**18 dimensions**), and **head rotation** (**6 dimensions**), samples the values over each utterance, splits them into **200 partitions**, averages within each partition, and concatenates the result into a **(200, 189)** representation [1804.05788]. This MoCap formulation allows models to exploit facial and bodily movement without full video processing. A plausible implication is that IEMOCAP’s multimodal richness resides not only in raw sensory coverage but also in the availability of several alternative visual abstractions.

## 5. Benchmark roles and representative model families

IEMOCAP has supported several generations of SER baselines. On the **improvised** subset with four emotions, a **CNN + BiLSTM** operating on spectrograms reported **64.5% WA** and **61.7% UA** under **10-fold cross-validation** [1802.05630]. A later speech-only approach based on **pre-trained end-to-end ASR features** from an **RNN-T** encoder and a **bi-LSTM + multi-head self-attention** decoder reported **71.7% WA** and **72.6% UA** under **10-fold leave-one-speaker-out cross validation** [1911.09762]. Self-supervised transfer learning further strengthened the benchmark: using the full four-class subset of **5,531 utterances**, a **wav2vec** acoustic model reached **64.3% UA**, while a multimodal **wav2vec + BERT** model aligned by an attention-based recurrent architecture reached **73.9% UA** [2011.05585]. In a later cross-corpus setup with supervised contrastive learning, a **WavLM-based** system reported **77.41% UA** on IEMOCAP under **5-fold cross-validation** [2411.19803].

Multimodal studies use IEMOCAP to test whether emotional information is complementary across channels. An early deep multimodal system combining **speech**, **text**, and **motion capture** via late fusion reported **71.04%** accuracy on a session-based train/test split [1804.05788]. A more recent Transformer-based MER pipeline fine-tuned **BERT**, **wav2vec 2.0**, and **VideoMAE**, extracted the final-layer **CLS** representations of dimension **768**, and found that **early fusion** of the three modality embeddings with an **SVM** achieved **75.42% accuracy** on the four-class IEMOCAP task; in that study, **anger** was the strongest class and **neutral** the weakest [2402.07327]. Another tri-modal architecture, **GCM-Net**, reported **85.66% accuracy** and **72.69% F1-score** on a four-class utterance-level setting, using graph-based recalibration, cross-modal attention, and a ConvXGB classifier [2410.12828].

IEMOCAP is also a standard ERC benchmark rather than only an utterance-isolated MER corpus. In **GatedxLSTM**, the dataset is filtered to **151 dialogues** and **4,490 utterances** in four classes, with the model using the current speaker’s and interlocutor’s **audio** and **text** plus limited preceding context under a **dialogue-level** split. That work reports **76.34 ± 1.31% weighted accuracy** and **75.97 ± 1.38% weighted F1** [2503.20919]. The breadth of these settings shows that IEMOCAP functions simultaneously as a speech corpus, a multimodal utterance benchmark, and a conversation-level affect dataset.

## 6. Dataset bias, limitations, and later extensions

The same properties that made IEMOCAP influential also make it methodologically difficult. The corpus is repeatedly described as **small** relative to newer SER resources; one review contrasts it with **MSP-Podcast**, cited as **27 hours**, and argues that IEMOCAP’s limited size and class imbalance create substantial risks of overfitting and class bias [2304.00860]. The imbalance is especially important in the standard four-class setup, where most samples come from **neutral** and **happy** [2304.00860]. This has motivated methods based on augmentation, class-weighting, focal-style objectives, and distribution smoothing. For example, **PDDS** treats IEMOCAP as not only imbalanced across clear classes but also uneven across **ambiguous emotion pairs**, and reports gains of **0.2%–4.8% WA** and **1.5%–5.9% UA** across several models [2302.08650].

Text-based performance on IEMOCAP is a recurring point of controversy. Transcript models can be very strong within the corpus—one hierarchical text-only GRU reports **82.1 WA / 80.6 UWA** under a **sessions 1–4 train, session 5 test** protocol [1904.04446]—but the dataset-composition literature argues that such gains may partly reflect memorization of scripted lexical material rather than robust emotion understanding [2103.03700, 2403.18635]. This does not imply that text is uninformative; rather, it indicates that IEMOCAP’s scripted structure can confound the interpretation of text-dominant or multimodal results unless script overlap is controlled explicitly.

The corpus has also been extended beyond its original annotation design. **PA-IEMOCAP** adds **Big Five personality traits** at the conversation level, producing **302 personality profiles** for **151 dyadic interactions** and reporting **ICC** values from **0.89 to 0.97**, with an average of **0.92** [2505.13978]. In parallel, IEMOCAP has become a testbed for robustness to missing modalities and for data compression. **CM-ARR** evaluates six incomplete-modality settings over **text**, **speech**, and **video** and reports average gains of **2.11% WAR** and **2.12% UAR** over the best baseline on IEMOCAP [2407.09029]. A dataset-distillation study treats IEMOCAP itself as the source dataset to be compressed into a conditional generative model, reporting **94–97%** storage reduction and about **95%** downstream training-time reduction while remaining close to full-data performance in **UAR** [2406.02963]. These extensions indicate that IEMOCAP is no longer only a benchmark for classifying four acted emotions; it is also an experimental substrate for studying evaluation design, multimodal robustness, speaker attributes, and efficient representation learning.

Source: https://www.emergentmind.com/topics/iemocap-dataset