---
title: 'MSP-Conversation Corpus: Dynamic SER'
url: https://www.emergentmind.com/topics/msp-conversation-corpus
type: topic
---

# MSP-Conversation Corpus: Dynamic SER

Searching arXiv for the MSP-Conversation paper and closely related corpus papers for citation support.
MSP-Conversation is a large-scale, naturalistic speech emotion corpus designed for time-continuous emotion recognition in conversational settings. It contains 310 conversations totaling **77 h 26 min** of audio from public podcasts, with dense, frame-level traces of **valence**, **arousal**, and **dominance**, each mapped to the range \([-100, 100]\). The corpus also provides detailed **human speaker diarization** with explicit marking of overlapping speech, and it overlaps with **12,555** speaking turns from MSP-Podcast, enabling direct comparison between **in-context** continuous annotations and **out-of-context** utterance-level annotations. The corpus is positioned as a resource for dynamic SER, context-aware affect modeling, and multi-speaker interaction analysis in naturalistic conversational audio [2603.22536].

## 1. Scope, scale, and corpus identity

MSP-Conversation is explicitly defined as a corpus for **naturalistic, time-continuous emotion recognition**. Its central design choice is to model emotional expression as a temporally evolving process rather than as a static label attached to an isolated turn. The corpus therefore combines long conversational context, continuous affective annotation, and diarization in a way intended to support analysis of emotional dynamics over seconds to minutes [2603.22536].

| Component | Value |
|---|---|
| Conversations | 310 |
| Total duration | 77 h 26 min |
| Annotation targets | Valence, arousal, dominance |
| Label range | \([-100, 100]\) |
| Overlap with MSP-Podcast | 12,555 speaking turns |

The audio is sourced from publicly available podcasts and was selected to preserve **natural, spontaneous speech** rather than laboratory prompts or acted scenarios. Conversations are typically **10–20 minutes** long, then split into **3–7 minute conversation parts** for annotation. The resulting corpus emphasizes conversational context, multi-speaker interaction, and continuous affect trajectories rather than isolated sentence-level judgments [2603.22536].

The partitioning scheme is **speaker- and podcast-independent**. The released duration split is **41 h 22 min** for Train, **12 h 36 min** for Development, **14 h 17 min** for Test1, and **9 h 9 min** for Test2. Test2 is described as a **blind evaluation set**, and the partition design is intended to support controlled benchmarking [2603.22536].

## 2. Corpus construction and partitioning logic

The corpus was built from the same public podcast pool used for MSP-Podcast, but it is not a repackaging of that dataset. The selection process began by inspecting MSP-Podcast turns from each podcast to identify segments with **extreme** or **highly variable** values for valence, arousal, and dominance. As the corpus grew, selection also sought to balance emotional content across the three affective dimensions. Candidate podcasts were then reviewed manually, undesirable topics were filtered out, and start and end times were chosen manually so that each retained segment formed a natural conversational unit [2603.22536].

A single conversation was chosen per podcast, with boundaries selected to contain many emotionally rich turns while preserving discourse continuity. Each conversation was then segmented into **3–7 minute** parts at natural breaks to reduce annotator fatigue. This workflow establishes a two-level structure: long-form conversational units for discourse continuity, and shorter annotation units for practical rating [2603.22536].

The corpus statistics given for the released partitions are as follows.

| Partition | Duration | Share |
|---|---:|---:|
| Train | 41 h 22 min | 53.4% |
| Development | 12 h 36 min | 16.3% |
| Test1 | 14 h 17 min | 18.5% |
| Test2 | 9 h 9 min | 11.8% |

This partitioning has methodological importance because the corpus is meant to support model comparison under **speaker-independent** conditions and without podcast leakage across splits. A plausible implication is that improvements on the benchmark are less likely to be driven by speaker memorization or source-specific artifacts than in randomly partitioned conversational data.

## 3. Continuous affect annotation methodology

MSP-Conversation uses the **core affect** dimensions **valence**, **arousal**, and **dominance**. Each attribute is annotated separately on a continuous scale mapped to \([-100, 100]\), where 0 is neutral, \(-100\) denotes extreme negative, calm, or weak, and \(100\) denotes extreme positive, excited, or strong. Annotation was carried out with **CARMA** in MATLAB using a **joystick**, whose position controlled the current affective value in real time [2603.22536].

Each conversation part was annotated in **three separate passes**, one per attribute, by **at least 6 different annotators** per attribute. Workers never annotated the same part-attribute pair twice. To reduce fatigue, sessions were capped at **1 hour**, and annotation order was randomized at the part level while attributes were assigned in blocks of 10. The annotator pool comprised **26 workers** from the University of Texas at Dallas community, all with extensive experience living in the US in order to reduce cultural variation. Training included annotation of **9 SEMAINE conversations**, followed by review using four CARMA intraclass correlation coefficient metrics: single rater agreement, average rater agreement, single rater consistency, and average rater consistency [2603.22536].

Quality control was applied after deployment. One worker who produced very noisy traces and another with low agreement were removed, and their annotations were replaced until each part had at least 6 usable annotations. The released labels were derived by first resampling each trace to **10 Hz**. CARMA produces irregularly sampled joystick traces with an average sampling rate of about **33 Hz**; resampling was done with a **median filter** using a **120 ms** window and **100 ms** shift. When a window had no samples, the previous value was repeated if available; otherwise the value was set to **0** [2603.22536].

The corpus then applies a standard **reaction lag correction** with a fixed delay \(T = 3\) seconds. If \(y(t)\) is the averaged trace and \(L\) is the recording length, the corrected trace is

$$
\tilde{y}(t)=
\begin{cases}
y(t+T) & t \leq L-T,\\
y(L) & t > L-T.
\end{cases}
$$

This backward shift is intended to align ratings with the audio events that elicited them [2603.22536].

The processed traces exhibit a slight positive and high-activation bias at corpus level. The reported means are **10.7** for valence, **23.6** for arousal, and **29.1** for dominance; the corresponding standard deviations are **18.3**, **12.9**, and **11.0**. Valence spans the widest range, while dominance is the narrowest. The paper reports that valence also has the highest inter-annotator consistency, with full-corpus Cronbach’s alpha values of **0.849** for valence, **0.758** for arousal, and **0.745** for dominance [2603.22536].

The reliability statistic is given as

$$
\alpha = \frac{k c}{\bar{\sigma}^2 + (k-1)c},
$$

where \(k\) is the number of items, \(c\) is the mean covariance between annotators across frames, and \(\bar{\sigma}^2\) is the mean annotator variance. Per-partition values show lower consistency in Test2: **0.736**, **0.633**, and **0.543** for valence, arousal, and dominance, respectively. The paper also defines an annotator-specific reliability difference,

$$
\Delta \alpha(a) = \alpha_{\text{incl}(a)} - \alpha_{\text{excl}(a)},
$$

to quantify whether including annotator \(a\) increases or decreases consensus. This suggests a route toward reliability-weighted gold labels beyond simple averaging [2603.22536].

## 4. Conversational structure, diarization, and overlap

MSP-Conversation separates **continuous affect annotation** from **speaker diarization**. Emotion is annotated over the entire conversation part, whereas diarization marks who is speaking when. The diarization layer was produced in **ELAN**, with one tier per speaker and explicit support for **overlapping speech**. Annotators marked speech segments, speaker-specific non-linguistic vocalizations such as laughter, crying, and interjections, and implicitly defined non-speech intervals when no speaker tier was active [2603.22536].

The first **89 conversations** were diarized completely manually. For the remaining **221**, annotators started from automatic guidance produced with **pyannote.audio** and then manually corrected it. The resulting diarization is described as fine-grained and includes overlap durations as well as silent or background-only regions [2603.22536].

The corpus is predominantly multi-party. About **94%** of conversations contain more than one speaker, and **40.6%** contain more than two speakers. The largest single group is two-speaker conversations at **53.9%**, and three-speaker conversations account for **19.0%**. Overlapped speech occupies **3 h 25 min**, or **4.8%** of the corpus as a whole. Non-speech audio occupies **5 h 58 min**, or **7.7%** of the corpus [2603.22536].

Partition-level overlap is not uniform. Overlap is **4.2%** in Train, **2.7%** in Development, **5.5%** in Test1, and **9.2%** in Test2. Non-speech audio is **8.4%** in Train, **7.2%** in Development, **9.2%** in Test1, and **3.2%** in Test2. The paper also reports apparent gender statistics derived from partial manual annotation and mapping to MSP-Podcast speakers: speakers are **50.4% female**, **43.4% male**, and **7.6% unknown** corpus-wide, while speech duration is **46.0%** female, **50.1%** male, and **3.9%** unknown [2603.22536].

These structural properties matter for modeling. The presence of explicit overlap labels, multi-speaker tiers, and long-context conversation parts makes the corpus suitable not only for single-stream dynamic SER but also for diarization-aware and interaction-aware affect modeling.

## 5. Relationship to MSP-Podcast and benchmark experiments

A defining feature of MSP-Conversation is its controlled overlap with **MSP-Podcast**. The shared material consists of **12,555** speaking turns that appear as isolated segments in MSP-Podcast and as embedded spans inside MSP-Conversation conversations. This overlap is used in two ways: it supports **speaker-independent partitions** by reusing MSP-Podcast speaker identities and metadata, and it enables direct comparison between **utterance-level, out-of-context** labels and **aggregated sentence-level, in-context** labels derived from the continuous traces [2603.22536].

The aggregation procedure is straightforward. For each overlapping speaking turn, the corresponding time range is located inside the MSP-Conversation part; frames from each annotator’s continuous trace are extracted for that interval; a **mean** is taken over time for each annotator; and those values are then averaged across annotators to produce one scalar per attribute. Pearson correlation between these aggregated labels and the original MSP-Podcast labels is **0.461** for valence, **0.530** for arousal, and **0.436** for dominance across all 12,555 overlapping turns. The paper interprets these values as evidence that **in-context** and **out-of-context** annotations are related but not identical [2603.22536].

The baseline dynamic SER experiments use **WavLM-large** as the speech representation backbone. The attention encoder is first fine-tuned on **MSP-Podcast v2.0** in a multi-task setting for valence, arousal, and dominance, and is then reused as a fixed feature extractor for MSP-Conversation. WavLM frames are mean-pooled over **120 ms** windows with **100 ms** steps, matching the label rate of **10 Hz**. A three-layer fully connected adaptation network with 512-unit layers, ReLU activations, and **0.5** dropout is followed by one of three heads: a deep **linear head**, a **2-layer BiLSTM** with hidden size **256**, or a **Transformer decoder** with **6** layers and **8** attention heads [2603.22536].

Training operates on conversation parts, which are further segmented into chunks of at most **1 minute** with **5 s overlap**. At inference time, predictions from overlapping chunks are truncated at the midpoint and concatenated to recover a full trace. The objective is the **CCC loss**,

$$
\rho_c = \frac{2\rho_{y,\hat{y}}\sigma_y \sigma_{\hat{y}}}{\sigma_y^2 + \sigma_{\hat{y}}^2 + (\mu_y - \mu_{\hat{y}})^2},
$$

with

$$
\mathcal{L}_{\mathrm{CCC}}(y,\hat{y}) = 1 - \rho_c(y,\hat{y}).
$$

For multi-task learning, the loss is the equally weighted average of the three attribute-specific CCC losses. Optimization uses **Adam**, learning rate **1e-4**, and batch size **64** one-minute segments; the best epoch is selected on Development CCC [2603.22536].

| Partition | Attribute | Best CCC |
|---|---|---:|
| Test1 | Valence | 0.676 |
| Test1 | Arousal | 0.669 |
| Test1 | Dominance | 0.525 |
| Test2 | Valence | 0.589 |
| Test2 | Arousal | 0.488 |
| Test2 | Dominance | 0.412 |

The strongest results are generally obtained by the **BiLSTM head**, especially for arousal and dominance. On **Test1**, the best scores are **0.676** for valence, **0.669** for arousal, and **0.525** for dominance. On **Test2**, the best reported values are **0.589**, **0.488**, and **0.412**, respectively. The paper notes that **single-task** training is stronger for arousal, whereas **multi-task** training helps dominance. It also notes that Test2 is harder, consistent with its lower inter-evaluator agreement, higher overlap, and different gender distribution [2603.22536].

## 6. Research significance, comparative context, and limitations

MSP-Conversation is designed for research problems that require affect to be modeled as a dynamic, context-sensitive process. The paper identifies **dynamic SER**, **context-aware emotion modeling**, **conversation-level affect modeling**, **diarization-aware SER**, **group interaction analysis**, and **label methodology research** as primary use cases. Because conversation parts average roughly five minutes and many source conversations last **10–20 minutes**, the corpus supports analyses of both short-term phenomena such as sudden emotional shifts and longer-term processes such as emotion propagation or contagion [2603.22536].

In the broader landscape of conversational corpora, MSP-Conversation occupies a specific niche. Unlike **CANDOR**, which is a multimodal corpus of **1,656** dyadic video conversations totaling **850+ hours** with audio, video, transcripts, and post-conversation surveys, MSP-Conversation is audio-only and centered on continuous affect traces rather than broad conversational psychology [2203.00674]. Unlike **MyST**, which contains **393 hours** of children’s spontaneous educational dialogue organized as strict turn-based tutoring sessions, MSP-Conversation is built from public podcasts and emphasizes naturalistic multi-party conversation with overlapping speech and time-continuous emotion annotation [2309.13347]. These contrasts are structural rather than evaluative: each corpus targets a different layer of conversation research.

The limitations reported for MSP-Conversation are consequential. The corpus is **audio-only** and does not include video or transcripts; it is **English only**; annotators are US-based; emotional labels and diarization remain imperfect despite training and agreement monitoring; and diarization used **one diarizer per conversation** without cross-check. The paper also notes a trade-off between naturalness and coverage: podcast speech provides realistic data but fewer clear, extreme emotional events than acted corpora, which may bias models toward subtle, near-neutral states. Test2 is explicitly described as more difficult because it has lower agreement, more overlap, and a different gender distribution [2603.22536].

Within those constraints, MSP-Conversation establishes a benchmark for continuous, context-sensitive affect modeling in real conversational speech. Its overlap with MSP-Podcast, its 10 Hz valence-arousal-dominance traces, and its explicit multi-speaker diarization make it a corpus for studying how emotion is perceived and modeled not as isolated labels on turns, but as a temporally evolving property of conversation [2603.22536].

Source: https://www.emergentmind.com/topics/msp-conversation-corpus