MSP-Conversation Corpus: Dynamic SER

Updated 5 July 2026

MSP-Conversation is a naturalistic, time-continuous speech emotion corpus built for dynamic SER and context-aware affect analysis.
The corpus comprises 310 podcast conversations, totaling over 77 hours of audio, with continuous annotations for valence, arousal, and dominance using a standardized scale.
It offers detailed speaker diarization and controlled data splits, enabling robust evaluation of in-context versus out-of-context emotion recognition methods.

Searching arXiv for the MSP-Conversation paper and closely related corpus papers for citation support. MSP-Conversation is a large-scale, naturalistic speech emotion corpus designed for time-continuous emotion recognition in conversational settings. It contains 310 conversations totaling 77 h 26 min of audio from public podcasts, with dense, frame-level traces of valence, arousal, and dominance, each mapped to the range $[-100, 100]$ . The corpus also provides detailed human speaker diarization with explicit marking of overlapping speech, and it overlaps with 12,555 speaking turns from MSP-Podcast, enabling direct comparison between in-context continuous annotations and out-of-context utterance-level annotations. The corpus is positioned as a resource for dynamic SER, context-aware affect modeling, and multi-speaker interaction analysis in naturalistic conversational audio (Martinez-Lucas et al., 23 Mar 2026).

1. Scope, scale, and corpus identity

MSP-Conversation is explicitly defined as a corpus for naturalistic, time-continuous emotion recognition. Its central design choice is to model emotional expression as a temporally evolving process rather than as a static label attached to an isolated turn. The corpus therefore combines long conversational context, continuous affective annotation, and diarization in a way intended to support analysis of emotional dynamics over seconds to minutes (Martinez-Lucas et al., 23 Mar 2026).

Component	Value
Conversations	310
Total duration	77 h 26 min
Annotation targets	Valence, arousal, dominance
Label range	$[-100, 100]$
Overlap with MSP-Podcast	12,555 speaking turns

The audio is sourced from publicly available podcasts and was selected to preserve natural, spontaneous speech rather than laboratory prompts or acted scenarios. Conversations are typically 10–20 minutes long, then split into 3–7 minute conversation parts for annotation. The resulting corpus emphasizes conversational context, multi-speaker interaction, and continuous affect trajectories rather than isolated sentence-level judgments (Martinez-Lucas et al., 23 Mar 2026).

The partitioning scheme is speaker- and podcast-independent. The released duration split is 41 h 22 min for Train, 12 h 36 min for Development, 14 h 17 min for Test1, and 9 h 9 min for Test2. Test2 is described as a blind evaluation set, and the partition design is intended to support controlled benchmarking (Martinez-Lucas et al., 23 Mar 2026).

2. Corpus construction and partitioning logic

The corpus was built from the same public podcast pool used for MSP-Podcast, but it is not a repackaging of that dataset. The selection process began by inspecting MSP-Podcast turns from each podcast to identify segments with extreme or highly variable values for valence, arousal, and dominance. As the corpus grew, selection also sought to balance emotional content across the three affective dimensions. Candidate podcasts were then reviewed manually, undesirable topics were filtered out, and start and end times were chosen manually so that each retained segment formed a natural conversational unit (Martinez-Lucas et al., 23 Mar 2026).

A single conversation was chosen per podcast, with boundaries selected to contain many emotionally rich turns while preserving discourse continuity. Each conversation was then segmented into 3–7 minute parts at natural breaks to reduce annotator fatigue. This workflow establishes a two-level structure: long-form conversational units for discourse continuity, and shorter annotation units for practical rating (Martinez-Lucas et al., 23 Mar 2026).

The corpus statistics given for the released partitions are as follows.

Partition	Duration	Share
Train	41 h 22 min	53.4%
Development	12 h 36 min	16.3%
Test1	14 h 17 min	18.5%
Test2	9 h 9 min	11.8%

This partitioning has methodological importance because the corpus is meant to support model comparison under speaker-independent conditions and without podcast leakage across splits. A plausible implication is that improvements on the benchmark are less likely to be driven by speaker memorization or source-specific artifacts than in randomly partitioned conversational data.

3. Continuous affect annotation methodology

MSP-Conversation uses the core affect dimensions valence, arousal, and dominance. Each attribute is annotated separately on a continuous scale mapped to $[-100, 100]$ , where 0 is neutral, $-100$ denotes extreme negative, calm, or weak, and $100$ denotes extreme positive, excited, or strong. Annotation was carried out with CARMA in MATLAB using a joystick, whose position controlled the current affective value in real time (Martinez-Lucas et al., 23 Mar 2026).

Each conversation part was annotated in three separate passes, one per attribute, by at least 6 different annotators per attribute. Workers never annotated the same part-attribute pair twice. To reduce fatigue, sessions were capped at 1 hour, and annotation order was randomized at the part level while attributes were assigned in blocks of 10. The annotator pool comprised 26 workers from the University of Texas at Dallas community, all with extensive experience living in the US in order to reduce cultural variation. Training included annotation of 9 SEMAINE conversations, followed by review using four CARMA intraclass correlation coefficient metrics: single rater agreement, average rater agreement, single rater consistency, and average rater consistency (Martinez-Lucas et al., 23 Mar 2026).

Quality control was applied after deployment. One worker who produced very noisy traces and another with low agreement were removed, and their annotations were replaced until each part had at least 6 usable annotations. The released labels were derived by first resampling each trace to 10 Hz. CARMA produces irregularly sampled joystick traces with an average sampling rate of about 33 Hz; resampling was done with a median filter using a 120 ms window and 100 ms shift. When a window had no samples, the previous value was repeated if available; otherwise the value was set to 0 (Martinez-Lucas et al., 23 Mar 2026).

The corpus then applies a standard reaction lag correction with a fixed delay $T = 3$ seconds. If $y(t)$ is the averaged trace and $L$ is the recording length, the corrected trace is

$\tilde{y}(t)= \begin{cases} y(t+T) & t \leq L-T,\ y(L) & t > L-T. \end{cases}$

This backward shift is intended to align ratings with the audio events that elicited them (Martinez-Lucas et al., 23 Mar 2026).

The processed traces exhibit a slight positive and high-activation bias at corpus level. The reported means are 10.7 for valence, 23.6 for arousal, and 29.1 for dominance; the corresponding standard deviations are 18.3, 12.9, and 11.0. Valence spans the widest range, while dominance is the narrowest. The paper reports that valence also has the highest inter-annotator consistency, with full-corpus Cronbach’s alpha values of 0.849 for valence, 0.758 for arousal, and 0.745 for dominance (Martinez-Lucas et al., 23 Mar 2026).

The reliability statistic is given as

$\alpha = \frac{k c}{\bar{\sigma}^2 + (k-1)c},$

where $[-100, 100]$ 0 is the number of items, $[-100, 100]$ 1 is the mean covariance between annotators across frames, and $[-100, 100]$ 2 is the mean annotator variance. Per-partition values show lower consistency in Test2: 0.736, 0.633, and 0.543 for valence, arousal, and dominance, respectively. The paper also defines an annotator-specific reliability difference,

$[-100, 100]$ 3

to quantify whether including annotator $[-100, 100]$ 4 increases or decreases consensus. This suggests a route toward reliability-weighted gold labels beyond simple averaging (Martinez-Lucas et al., 23 Mar 2026).

4. Conversational structure, diarization, and overlap

MSP-Conversation separates continuous affect annotation from speaker diarization. Emotion is annotated over the entire conversation part, whereas diarization marks who is speaking when. The diarization layer was produced in ELAN, with one tier per speaker and explicit support for overlapping speech. Annotators marked speech segments, speaker-specific non-linguistic vocalizations such as laughter, crying, and interjections, and implicitly defined non-speech intervals when no speaker tier was active (Martinez-Lucas et al., 23 Mar 2026).

The first 89 conversations were diarized completely manually. For the remaining 221, annotators started from automatic guidance produced with pyannote.audio and then manually corrected it. The resulting diarization is described as fine-grained and includes overlap durations as well as silent or background-only regions (Martinez-Lucas et al., 23 Mar 2026).

The corpus is predominantly multi-party. About 94% of conversations contain more than one speaker, and 40.6% contain more than two speakers. The largest single group is two-speaker conversations at 53.9%, and three-speaker conversations account for 19.0%. Overlapped speech occupies 3 h 25 min, or 4.8% of the corpus as a whole. Non-speech audio occupies 5 h 58 min, or 7.7% of the corpus (Martinez-Lucas et al., 23 Mar 2026).

Partition-level overlap is not uniform. Overlap is 4.2% in Train, 2.7% in Development, 5.5% in Test1, and 9.2% in Test2. Non-speech audio is 8.4% in Train, 7.2% in Development, 9.2% in Test1, and 3.2% in Test2. The paper also reports apparent gender statistics derived from partial manual annotation and mapping to MSP-Podcast speakers: speakers are 50.4% female, 43.4% male, and 7.6% unknown corpus-wide, while speech duration is 46.0% female, 50.1% male, and 3.9% unknown (Martinez-Lucas et al., 23 Mar 2026).

These structural properties matter for modeling. The presence of explicit overlap labels, multi-speaker tiers, and long-context conversation parts makes the corpus suitable not only for single-stream dynamic SER but also for diarization-aware and interaction-aware affect modeling.

5. Relationship to MSP-Podcast and benchmark experiments

A defining feature of MSP-Conversation is its controlled overlap with MSP-Podcast. The shared material consists of 12,555 speaking turns that appear as isolated segments in MSP-Podcast and as embedded spans inside MSP-Conversation conversations. This overlap is used in two ways: it supports speaker-independent partitions by reusing MSP-Podcast speaker identities and metadata, and it enables direct comparison between utterance-level, out-of-context labels and aggregated sentence-level, in-context labels derived from the continuous traces (Martinez-Lucas et al., 23 Mar 2026).

The aggregation procedure is straightforward. For each overlapping speaking turn, the corresponding time range is located inside the MSP-Conversation part; frames from each annotator’s continuous trace are extracted for that interval; a mean is taken over time for each annotator; and those values are then averaged across annotators to produce one scalar per attribute. Pearson correlation between these aggregated labels and the original MSP-Podcast labels is 0.461 for valence, 0.530 for arousal, and 0.436 for dominance across all 12,555 overlapping turns. The paper interprets these values as evidence that in-context and out-of-context annotations are related but not identical (Martinez-Lucas et al., 23 Mar 2026).

The baseline dynamic SER experiments use WavLM-large as the speech representation backbone. The attention encoder is first fine-tuned on MSP-Podcast v2.0 in a multi-task setting for valence, arousal, and dominance, and is then reused as a fixed feature extractor for MSP-Conversation. WavLM frames are mean-pooled over 120 ms windows with 100 ms steps, matching the label rate of 10 Hz. A three-layer fully connected adaptation network with 512-unit layers, ReLU activations, and 0.5 dropout is followed by one of three heads: a deep linear head, a 2-layer BiLSTM with hidden size 256, or a Transformer decoder with 6 layers and 8 attention heads (Martinez-Lucas et al., 23 Mar 2026).

Training operates on conversation parts, which are further segmented into chunks of at most 1 minute with 5 s overlap. At inference time, predictions from overlapping chunks are truncated at the midpoint and concatenated to recover a full trace. The objective is the CCC loss,

$[-100, 100]$ 5

with

$[-100, 100]$ 6

For multi-task learning, the loss is the equally weighted average of the three attribute-specific CCC losses. Optimization uses Adam, learning rate 1e-4, and batch size 64 one-minute segments; the best epoch is selected on Development CCC (Martinez-Lucas et al., 23 Mar 2026).

Partition	Attribute	Best CCC
Test1	Valence	0.676
Test1	Arousal	0.669
Test1	Dominance	0.525
Test2	Valence	0.589
Test2	Arousal	0.488
Test2	Dominance	0.412

The strongest results are generally obtained by the BiLSTM head, especially for arousal and dominance. On Test1, the best scores are 0.676 for valence, 0.669 for arousal, and 0.525 for dominance. On Test2, the best reported values are 0.589, 0.488, and 0.412, respectively. The paper notes that single-task training is stronger for arousal, whereas multi-task training helps dominance. It also notes that Test2 is harder, consistent with its lower inter-evaluator agreement, higher overlap, and different gender distribution (Martinez-Lucas et al., 23 Mar 2026).

6. Research significance, comparative context, and limitations

MSP-Conversation is designed for research problems that require affect to be modeled as a dynamic, context-sensitive process. The paper identifies dynamic SER, context-aware emotion modeling, conversation-level affect modeling, diarization-aware SER, group interaction analysis, and label methodology research as primary use cases. Because conversation parts average roughly five minutes and many source conversations last 10–20 minutes, the corpus supports analyses of both short-term phenomena such as sudden emotional shifts and longer-term processes such as emotion propagation or contagion (Martinez-Lucas et al., 23 Mar 2026).

In the broader landscape of conversational corpora, MSP-Conversation occupies a specific niche. Unlike CANDOR, which is a multimodal corpus of 1,656 dyadic video conversations totaling 850+ hours with audio, video, transcripts, and post-conversation surveys, MSP-Conversation is audio-only and centered on continuous affect traces rather than broad conversational psychology (Reece et al., 2022). Unlike MyST, which contains 393 hours of children’s spontaneous educational dialogue organized as strict turn-based tutoring sessions, MSP-Conversation is built from public podcasts and emphasizes naturalistic multi-party conversation with overlapping speech and time-continuous emotion annotation (Pradhan et al., 2023). These contrasts are structural rather than evaluative: each corpus targets a different layer of conversation research.

The limitations reported for MSP-Conversation are consequential. The corpus is audio-only and does not include video or transcripts; it is English only; annotators are US-based; emotional labels and diarization remain imperfect despite training and agreement monitoring; and diarization used one diarizer per conversation without cross-check. The paper also notes a trade-off between naturalness and coverage: podcast speech provides realistic data but fewer clear, extreme emotional events than acted corpora, which may bias models toward subtle, near-neutral states. Test2 is explicitly described as more difficult because it has lower agreement, more overlap, and a different gender distribution (Martinez-Lucas et al., 23 Mar 2026).

Within those constraints, MSP-Conversation establishes a benchmark for continuous, context-sensitive affect modeling in real conversational speech. Its overlap with MSP-Podcast, its 10 Hz valence-arousal-dominance traces, and its explicit multi-speaker diarization make it a corpus for studying how emotion is perceived and modeled not as isolated labels on turns, but as a temporally evolving property of conversation (Martinez-Lucas et al., 23 Mar 2026).