iMiGUE-Speech: A Speech Affective Corpus
- The paper introduces a novel speech-centered extension of iMiGUE that aligns audio with time-stamped micro-gesture annotations for affective analysis.
- It applies advanced processing pipelines using ffmpeg, pyannote.audio, Whisper Large, and Montreal Forced Aligner to deliver speaker diarization, word-level alignment, and role separation.
- Affective benchmarks using Wav2Vec 2.0, WavLM, and RoBERTa-large reveal distinct outcome-driven emotional patterns and integrative multimodal research opportunities.
Searching arXiv for iMiGUE-Speech and closely related iMiGUE papers to ground the article. iMiGUE-Speech is a speech-centered extension of the original iMiGUE dataset that reprocesses Grand Slam tennis post-match press-conference videos into a spontaneous affective corpus for speech-based and multimodal analysis. The extension preserves the source material of iMiGUE and enriches it with extracted audio, speaker-role separation between interviewer and interviewee, English transcripts, segment-level and word-level time alignments, and role-specific audio segmentation. Its stated purpose is to support affective analysis under naturally occurring, outcome-driven emotional conditions rather than acted or laboratory-elicited settings, while remaining synchronizable with the original micro-gesture annotations (Kakouros et al., 25 Feb 2026).
1. Origin, scope, and spontaneous-affect setting
iMiGUE-Speech inherits its recordings from the original iMiGUE collection: 359 Grand Slam tennis post-match press-conference videos from 72 players, balanced by gender at 36 male and 36 female, representing 28 countries. The underlying corpus already contains 18,499 clip-level micro-gesture annotations across 32 categories and a video-level binary emotional state label derived from match outcome, namely positive versus negative. iMiGUE-Speech does not introduce a new recording campaign; it reprocesses those interviews to make the speech modality directly usable for affective computing (Kakouros et al., 25 Feb 2026).
The defining characteristic of the corpus is its emphasis on spontaneous affect. The interviews occur after real wins and losses, so the affective state is tied to consequential outcomes rather than scripted portrayals. The setting is therefore semi-structured rather than fully unconstrained: journalists ask questions, athletes answer, and emotional expression is shaped by fatigue, pressure, social norms, and public self-presentation. This distinguishes the resource from acted emotional speech corpora and makes it particularly relevant for studying subtle prosodic cues such as pitch, intensity, speech rate, pausing, and answer-to-answer variability (Kakouros et al., 25 Feb 2026).
A common misunderstanding is to treat “spontaneous” as equivalent to “unmediated.” The corpus is spontaneous in the sense that the affect arises naturally from competition outcomes, but it remains embedded in a public-facing press-conference format. This suggests that the dataset is especially suitable for studying affect under social regulation rather than unfiltered private emotional expression.
2. Processing pipeline and annotation layers
The processing pipeline begins with audio extraction and standardization. Audio is extracted from each original video and normalized with ffmpeg into 1-channel PCM at a fixed sampling rate. pyannote.audio is then applied for speaker diarization, overlap detection, and voice activity detection, producing anonymous diarization labels such as SPEAKER_00 and SPEAKER_01, overlap intervals, and speech-region boundaries. Automatic transcripts are generated with Whisper Large, with the language flag forced to English “to ensure consistent English transcripts even in the presence of accents or background noise.” These outputs are merged into a Praat-compatible TextGrid containing separate tiers for diarization, overlap, VAD, and segment-level transcripts (Kakouros et al., 25 Feb 2026).
Word-level forced alignment is added with the Montreal Forced Aligner, using raw audio together with Whisper transcripts. The output is a separate TextGrid with time-stamped word boundaries and, optionally, phone boundaries. The paper does not provide alignment confidence filtering, lexicon adaptation, or pronunciation-model details beyond the software name and the input/output description (Kakouros et al., 25 Feb 2026).
Speaker-role separation is a further layer specific to the interview setting. Because the original dataset lacks turn annotations, the athlete is inferred as the diarized speaker with the longest cumulative speaking duration in a recording. The rationale given is that athletes typically produce longer answers, whereas journalists ask shorter questions. Athlete-attributed segments are exported as separate clips in temporal order, such as segment_001, segment_002, and so forth; non-athlete speech is retained separately as journalist or interviewer speech. The result is a role-disentangled release with two disjoint speech sets per recording (Kakouros et al., 25 Feb 2026).
| Layer | Contents | Tool or format |
|---|---|---|
| Audio layer | Extracted and standardized audio | ffmpeg |
| Segment layer | Diarization, overlap, VAD, segment transcripts | pyannote.audio, Whisper Large, TextGrid |
| Word layer | Word-level and optional phone boundaries | Montreal Forced Aligner, TextGrid |
| Role layer | Athlete/journalist separation and segment IDs | Longest-speaking-speaker heuristic |
The metadata explicitly listed in the paper therefore include extracted audio, diarization labels, overlap segments, VAD regions, segment-level English ASR transcripts, a unified segment-level TextGrid, word-level alignment, a separate word/phone-level TextGrid, role-identification metadata, speaker-specific audio clips, and sequential segment IDs (Kakouros et al., 25 Feb 2026).
3. Label structure and benchmark formulation
The corpus combines inherited labels from iMiGUE with new speech-oriented benchmark tasks. At the source-dataset level, emotional state is binary and video-level, defined by match outcome as positive or negative. For the speech extension, the paper introduces two evaluation tasks for comparative assessment: speech emotion recognition and transcript-based sentiment analysis. The speech analysis covers dimensional emotion variables—arousal, dominance, and valence—and categorical emotions—Angry, Sad, Happy, Surprise, Fear, Disgust, Contempt, and Neutral. The transcript task assigns Positive versus Negative polarity (Kakouros et al., 25 Feb 2026).
An important caveat is that the paper does not present iMiGUE-Speech as a conventional supervised benchmark with official segment-level gold labels for those dimensional or categorical affective targets. Instead, the reported analyses apply pre-trained affect models to athlete speech segments, aggregate the resulting predictions to the session level, and compare winner versus loser patterns. The paper also does not state an official train/validation/test partition for the speech release, nor does it provide counts such as total segmented athlete clips, total journalist clips, or total segmented audio duration (Kakouros et al., 25 Feb 2026).
The benchmark models are specified only at a high level. Dimensional speech emotion recognition uses Wav2Vec 2.0, described as a large-scale model pre-trained on broadly sourced speech data and subsequently fine-tuned on MSP-Podcast for dimensional emotion recognition. Categorical speech emotion recognition uses WavLM, described as fine-tuned for categorical emotion recognition and consistent with the Odyssey 2024 SER benchmark framework. Transcript sentiment uses RoBERTa-large, fine-tuned previously on a diverse aggregation of English sentiment datasets. The paper does not report exact checkpoints, head architectures, optimizer settings, or loss definitions for the dataset-specific use of these models (Kakouros et al., 25 Feb 2026).
Aggregation is the methodological centerpiece of the benchmark section. For dimensional estimates, “Inter Level” is computed by averaging segment-level predictions within each session and then averaging those session values within the Win and Lose groups. “Intra Variability” is computed as the standard deviation of segment-level predictions within each session and then averaged within the Win and Lose groups. For categorical emotion, the dominant emotion category for each interview serves as the session-level label. For transcript sentiment, session-level polarity is based on the average sentiment score across all textual segments (Kakouros et al., 25 Feb 2026).
| Task | Representation/model | Session-level aggregation |
|---|---|---|
| Dimensional SER | Wav2Vec 2.0 fine-tuned on MSP-Podcast | Mean and standard deviation within interview |
| Categorical SER | WavLM fine-tuned for categorical SER | Dominant emotion category per interview |
| Transcript sentiment | RoBERTa-large fine-tuned on English sentiment data | Average sentiment score per interview |
Because the paper reports descriptive group statistics rather than standard supervised metrics, it does not provide accuracy, UAR, F1, CCC, Pearson correlation, or MAE for these speech tasks. This suggests that the benchmark section is better interpreted as a demonstration of dataset utility than as a leaderboard-style challenge definition (Kakouros et al., 25 Feb 2026).
4. Reported affective patterns
The dimensional results show consistent winner–loser separation. For losers, the reported values are arousal 0.3468 with intra-variability 0.0787, dominance 0.4139 with intra-variability 0.0696, and valence 0.5025 with intra-variability 0.1166. For winners, the corresponding values are arousal 0.3690 with intra-variability 0.0754, dominance 0.4342 with intra-variability 0.0655, and valence 0.5361 with intra-variability 0.1202. Winners therefore exhibit higher inter-level arousal, dominance, and valence, with the largest mean differences appearing in valence and dominance (Kakouros et al., 25 Feb 2026).
The paper’s prose states that losers display slightly larger valence variability, but the table shows the opposite: winner valence variability is 0.1202, whereas loser valence variability is 0.1166. That inconsistency is internal to the paper and should be read as a reporting discrepancy rather than a resolved empirical conclusion (Kakouros et al., 25 Feb 2026).
The categorical emotion analysis is dominated by sadness in both groups, but to different degrees. For losers, the dominant-emotion proportions are Sad 85.2, Happy 5.9, Fear 2.0, and Neutral 6.9, with Angry, Surprise, Disgust, and Contempt all at 0.0. For winners, the proportions are Sad 64.7, Happy 15.5, Fear 0.8, and Neutral 19.0, with Angry, Surprise, Disgust, and Contempt likewise at 0.0. The authors attribute the absence of overtly hostile or highly animated dominant categories to the social norms of press interviews (Kakouros et al., 25 Feb 2026).
Transcript-based sentiment is much less polarized than the acoustic emotion outputs. Losers are classified as 12.9% Negative and 87.1% Positive at the session level, while winners are 4.6% Negative and 95.4% Positive. Both groups are therefore mostly Positive in text, even though the acoustic analyses suggest much stronger sadness prevalence, particularly for losers (Kakouros et al., 25 Feb 2026).
The main substantive conclusion drawn in the paper is that acoustic and linguistic modalities capture different aspects of spontaneous affect. The speech-emotion models reveal stronger outcome-related differences, especially in valence and dominance, whereas transcript sentiment remains predominantly positive even after losses. This suggests that, in these interviews, spontaneous affect is encoded not only in lexical content but also in prosodic and paralinguistic realization (Kakouros et al., 25 Feb 2026).
5. Relation to the broader iMiGUE ecosystem
A major scientific value of iMiGUE-Speech lies in its synchronous compatibility with the original micro-gesture annotations. Because speech segments and word-level alignments are time-stamped within the same recordings, they can be paired with clip-level micro-gesture labels from iMiGUE. The paper explicitly identifies applications such as joint modeling of vocal prosody and micro-gestures for emotion or stress recognition, temporal synchronization analyses between words, prosodic peaks, and body micro-movements, multimodal outcome prediction from speech, text, and gesture, turn-level studies of speech–gesture co-variation, and fine-grained alignment of gestures with individual words or syllables (Kakouros et al., 25 Feb 2026).
This multimodal opportunity is nontrivial because the original iMiGUE benchmark is already established in micro-gesture recognition. In the 3rd MiGA Challenge at IJCAI 2025, “MM-Gesture” frames iMiGUE as 32-way cross-subject micro-gesture classification and reports 73.213% top-1 accuracy using joint, limb, RGB, Taylor-series video, optical-flow video, and depth video within a late-fused multimodal system (Gu et al., 11 Jul 2025). A later challenge solution adds a self-supervised RGB branch pretrained on 120K unlabeled clips and reports 74.419% top-1 accuracy on the iMiGUE test set (Liu et al., 8 Jun 2026).
iMiGUE-Speech therefore extends an already active visual benchmark into a speech-aligned multimodal resource. The speech paper itself does not define a joint audio-visual training task over those synchronized modalities, and it does not report speech-conditioned micro-gesture recognition or gesture-conditioned speech analysis. This suggests that the release should be understood primarily as infrastructure for cross-modal research rather than as a completed multimodal benchmark protocol (Kakouros et al., 25 Feb 2026).
6. Limitations, caveats, and research uses
Several limitations are explicit. The domain is narrow: the corpus is drawn entirely from tennis post-match press conferences, so affect is tied to sports outcomes, public-speaking conventions, and interview structure. The speech is spontaneous, but not unconstrained. Athletes are public figures who may regulate emotional display, and winner–loser differences are potentially confounded by player identity, fatigue, media pressure, tournament stage, and personality (Kakouros et al., 25 Feb 2026).
The textual modality is also constrained. Transcripts are automatic rather than manually corrected, and transcription is forced to English despite the international speaker pool. This introduces possible errors for accented speech and multilingual lexical content. Similarly, the athlete-role heuristic assumes that the longest-speaking diarized speaker is the interviewee; the paper presents this as a practical inference rather than a guaranteed identity assignment (Kakouros et al., 25 Feb 2026).
The privacy position is nuanced. The original iMiGUE is described as identity-free in the visual sense, with faces masked or removed, but iMiGUE-Speech preserves voice, which is itself a biometric signal. A common misconception is therefore to treat the speech extension as de-identified simply because the video benchmark is identity-free. The paper does not resolve that tension in depth, and this suggests that voice-based privacy requires separate consideration from face masking (Kakouros et al., 25 Feb 2026).
The paper states that the extended dataset is available at https://github.com/CV-AC/imigue-speech, but it does not specify a license in the provided text. Users are therefore expected to consult the repository for terms of use and any restrictions inherited from the underlying public press-conference material (Kakouros et al., 25 Feb 2026).
Within those limits, the paper explicitly positions iMiGUE-Speech as useful for speech-based affective and paralinguistic analysis, dialogue and interaction analysis, text-based affective and semantic analysis, question/answer identification, turn-taking analysis, disfluency studies, ASR evaluation, prosody research, speech–gesture alignment, and multimodal emotion analysis. Its distinctive contribution is not a highly parameterized benchmark with exhaustive supervision, but a time-aligned spontaneous corpus that makes the speech modality operational within the iMiGUE ecosystem (Kakouros et al., 25 Feb 2026).