CMU-MOSI Dataset

• CMU-MOSI is a multimodal dataset comprising segmented vlog videos with synchronized text, audio, and visual data to enable fine-grained sentiment analysis.
• It features rigorous annotation protocols, including subjectivity segmentation and 7-point sentiment intensity ratings, ensuring robust cross-modal evaluation.
• Designed with speaker-independent splits and detailed alignment, the dataset underpins advances in representation learning and multimodal fusion techniques.

The CMU Multimodal Opinion-level Sentiment Intensity (CMU-MOSI) dataset is a benchmark corpus for research on automatic sentiment and subjectivity analysis in online opinion videos. Designed for multimodal machine learning, CMU-MOSI provides tightly aligned text, audio, and visual features, with fine-grained human annotations capturing both the presence of subjective content and the intensity of expressed opinions. The dataset underpins advances in multimodal sentiment analysis, robust cross-modal representation learning, and evaluation of sequential and fusion-based machine learning architectures (Zadeh et al., 2016, Huang et al., 3 Nov 2025).

1. Dataset Composition and Structure

CMU-MOSI consists exclusively of “vlog”-style YouTube videos: 3,702 annotated clips drawn from 93 distinct videos (89 unique speakers; 41 female, 48 male), with an average of 23.2 opinion segments per video. Each clip is segmented based on speaker pauses and observable opinion shifts, resulting in 2,199 segments containing at least one subjective (“private-state”) expression and 1,503 with only objective content. Mean segment duration is 4.2 seconds (4,200 ms), averaging 12 words per segment for a total segment vocabulary of 3,107 unique tokens—approximately 1,900 occurring ten or more times (Zadeh et al., 2016).

Speaker-independent partitioning is a core experimental protocol: speakers are exclusively segregated across training, validation, and test folds to prevent idiosyncratic speaking styles from contaminating evaluation. Standard splits employ 5-fold cross-validation on the speaker level, with further internal hyperparameter tuning via 4-fold splits of the training data (Zadeh et al., 2016). A typical split reported in recent literature is 1,284/229/686 for train/validation/test utterances, respectively (Huang et al., 3 Nov 2025).

2. Annotation Protocols

a) Subjectivity Segmentation

Annotations follow the private-state framework of Wiebe et al. (2005). Four annotation passes yield high-fidelity subjective segment boundaries:

• Initial span marking by a primary coder based on detailed guidelines.
• Review and error correction by a secondary coder.
• Reconciliation: intersecting the spans, extending partial overlaps as needed.
• Merge step: merging multi-utterance extensions expressing the same opinion.

The subjective/objective label for each segment is converted to a binary sequence for inter-annotator agreement, reported via Krippendorff’s $α = 0.68$ (Zadeh et al., 2016).

b) Sentiment Intensity Scoring

Each subjective segment receives a 7-point sentiment rating, $y_i ∈ \{-3, -2, -1, 0, +1, +2, +3\}$, via Amazon Mechanical Turk. Only “Master” workers (approval ≥ 95%) annotate; each segment is judged by five independent raters. Annotators rate the sentiment expressed by the speaker, not personal reaction, with an “uncertain” option available but rarely used. The mean across the five raters produces a real-valued target in $[-3, +3]$. Inter-annotator agreement for this scale is $α = 0.77$ (Zadeh et al., 2016).

3. Modalities, Feature Extraction, and Alignment

Every opinion segment is tri-modal, with synchronized and time-aligned streams:

Text:

• Manual four-stage transcription: expert creation, review, forced-alignment (via P2FA), and manual correction in Praat.
• Transcripts include word- and phoneme-level timestamps supporting fine-grained cross-modal analysis.
• Baseline features: unigrams/bigrams (after discarding infrequent n-grams); contemporary systems use BERT-derived embeddings (Huang et al., 3 Nov 2025).

Audio:

• Automatically extracted at 1 kHz with the COVAREP toolkit, yielding over 32 low-level descriptors: F0, NAQ, peak/energy slopes, 13 MFCCs.
• Recent work also incorporates Librosa features (additional MFCCs, chroma vectors) (Huang et al., 3 Nov 2025).

Visual:

• Frame rate: 30 fps.
• Automatic: 68 facial landmarks, 16 FACS Action Units, six prototypical expression classifiers, head pose, gaze.
• Manual: Four binary gestures (smile, frown, head nod, head shake); labeled for every opinion segment with 80.8% agreement rate between coders (Zadeh et al., 2016).
• Recent pipelines utilize FACET and OpenFace for 17 action units, head/eye metrics (Huang et al., 3 Nov 2025).

Data Format:

Files are distributed per modality: CSV tables for audio/visual descriptors, plain text for transcripts, JSON indices for alignment. Full video files are restricted to Creative Commons-labeled source material; otherwise, access is negotiated upon request (Zadeh et al., 2016).

4. Baseline Models and Quantitative Results

Published baselines (as of 2016) utilize linear ν-SVR and C-SVM (LIBSVM), and small deep neural networks, always on speaker-independent splits. Core benchmarks are:

Sentiment Intensity Regression:

• Metrics: Mean Absolute Error (MAE), Pearson correlation.
• Baseline performances (test MAE, correlation):
  • Random: 1.88, ~0.00
  • Visual (gestures): 1.24, 0.36
  • Verbal (bag-of-words): 1.18, 0.46
  • Early fusion: 1.14, 0.49
  • Multimodal Dictionary: 1.10, 0.53
  • Human (Turkers): 0.61, 0.83

Subjectivity Classification:

• Metric: Accuracy (subjective vs. objective).
• Linear SVM: Random baseline 59%, acoustic 57%, visual 61%, verbal 65%, multimodal dictionary 71%.
• Deep neural net: acoustic 51%, visual 53%, verbal 57%, multimodal dictionary 66% (Zadeh et al., 2016).

Recent advances, such as the Double Information Bottleneck (DIB), report higher performance:

• Acc-7 (7-class): 47.4%
• Acc-2 (binary): 85.6%
• F1: 85.6%
• Corr: 0.800
• MAE: 0.715

Ablation studies confirm that the text LRIB and fusion module critically support overall performance; removing either causes a substantial degradation (Huang et al., 3 Nov 2025).

5. Data Access, Best Practices, and Known Limitations

Data Access:

• Distribution as a zip archive with all modalities and segment metadata; subject to an academic license for non-commercial research, requiring attribution and adherence to YouTube’s original CC terms where applicable.

Best Practices:

• Always use speaker-independent splits.
• Utilize time-alignments for detailed cross-modal modeling.
• Report both MAE and correlation for regression, and accuracy/confusion matrices for classification tasks.
• Explore advanced representation learning: word embeddings, sequence models, spatio-temporal deep learning (Zadeh et al., 2016).

Limitations:

• English-language, US/UK, adult vlog bias restricts generalization to scripted, formal, or demographically different content.
• Single-speaking-face videos preclude hand gestures and off-screen multimodal action.
• Subjectivity annotated only at segment level—intrasegment valence changes are not captured.
• Visual data may be missing or occluded in some frames (Zadeh et al., 2016).

6. Significance for Multimodal Sentiment Research

CMU-MOSI addresses a critical gap for multimodal opinion mining: rigorous, segment-level synchronization of audio-visual-text streams with pragmatic subjective and sentiment labels. The multimodal-dictionary concept—explicitly modeling word–gesture co-occurrences—outperforms simple early fusion, but still leaves a gap to human-level sentiment inference.

Advanced methods such as DIB leverage low-rank Rényi’s entropy to construct compact, noise-robust unimodal and multimodal embeddings, augmented by attention bottleneck fusion to prevent modal cross-contamination. This dual bottleneck paradigm yields state-of-the-art accuracy and robustness under noise by compressing modality-specific information and maximizing task-relevant cross-modal integration. Nevertheless, challenges persist, including label supervision granularity, vision-language grounding for abstract sentiments, and domain transferability (Huang et al., 3 Nov 2025).

7. Continuing Impact and Future Directions

CMU-MOSI remains a foundational resource for benchmarking multimodal sentiment analysis systems. Its comprehensive annotations and tri-modal alignment facilitate research at the intersection of NLP, audio signal processing, and computer vision. Extensions to the MOSI framework—incorporating greater domain heterogeneity, multi-speaker scenarios, or dynamic temporal annotation—represent promising directions. Moreover, recent advances suggest that adaptive supervision, improved vision-language alignment, and meta-labeling strategies could further close the gap to human-level multimodal opinion understanding (Huang et al., 3 Nov 2025).