SentiMMD: Multimodal Sentiment Analysis Benchmark

• SentiMMD is a balanced, large-scale multimodal dataset designed for fine-grained sentiment analysis in social media using both text and visual data.
• It employs a rigorous annotation process with 40 crowd-sourced labels, GPT-based verification, and expert review to ensure high-quality sentiment labeling.
• The dataset supports advanced research in multimodal fusion, retrieval-augmented sentiment inference, and real-time social media emotion tracking.

SentiMMD is a large-scale, balanced multimodal dataset explicitly designed for fine-grained sentiment analysis research in social media contexts. Each instance includes both textual and visual information, enabling research into complex emotion recognition across heterogeneous modalities. SentiMMD’s unique construction and annotation protocols target longstanding challenges surrounding cross-modal data fusion, class imbalance, and annotation quality in the field of multimodal sentiment analysis (Xu et al., 25 Aug 2025).

1. Dataset Construction and Annotation Methodology

SentiMMD’s data originate from a systematic crawl of public social media content spanning microblogs, image sharing platforms, and short-video services. Each dataset entry consists of at least one textual field (post content, video title, or description) and one visual field (either a static image or a sequence of keyframes extracted from a video). For video posts, 3–7 keyframes are algorithmically sampled to represent the visual content. The corpus comprises 3,500 multimodal instances.

Sentiment annotation employs seven discrete categories. Each instance is labeled independently by 40 crowd-workers. After manual labeling, human majority labels are compared with a GPT-based classifier’s prediction. In cases of disagreement, expert reviewers adjudicate or remove ambiguous samples. Aggregated labels are determined by simple majority vote, with expert intervention limited to human-versus-GPT conflicts. Although inter-annotator agreement metrics (e.g., Fleiss’ κ or Cohen’s κ) are not reported, the protocol’s multi-round verification suggests robust label consistency.

2. Sentiment Category Design and Distribution

SentiMMD operationalizes fine-grained sentiment recognition via the label set

$$\mathcal{C} = \{\text{Like},\,\text{Happiness},\,\text{Anger},\,\text{Disgust},\,\text{Fear},\,\text{Sadness},\,\text{Surprise}\}.$$

These categories encompass:

• Like: Approval, support, or admiration
• Happiness: Joy, excitement, or positive affect
• Anger: Hostility, irritation, or rage
• Disgust: Revulsion or strong disapproval
• Fear: Apprehension or anxiety
• Sadness: Sorrow, grief, or unhappiness
• Surprise: Shock or astonishment

The dataset is uniformly balanced at 500 samples per class (total $N=3,500$), yielding a per-class prior $P(c) = 0.143$ (14.3\%), thus ensuring no class imbalance across the sentiment spectrum.

3. Data Splits and Modality Characteristics

SentiMMD is partitioned into 3,150 training samples (90%) and 350 test samples (10%). No distinct validation set is reported; model development and hyperparameter tuning utilize the training split, with the test set held out for final evaluation.

Modality-specific statistics:

• Text: Average length of 28.7 tokens per sample.
• Image/Keyframes: All visual content standardized to $224 \times 224$ pixels.
• Video: Mean duration of 12.3 seconds per video, with 3–7 keyframes extracted post-processing.

4. Comparison with Prior Multimodal Datasets

SentiMMD distinguishes itself from existing benchmarks along four major axes:

• Scale and Balance: Contains 3,500 instances distributed evenly across seven sentiment categories, contrasting with prior multimodal datasets such as XED and MMS, which typically exhibit binary/ternary polarity or long-tailed distributions.
• Modalities: Integrates both text and image/video modalities per instance, unlike EmotiC/HECO (image-only) or traditional Twitter corpora (text-only).
• Granularity: Expands classification beyond the usual 2–3 class schemes by incorporating seven emotions.
• Annotation Quality: Employs 40 annotators per sample, GPT-based cross-verification, and expert review—a significantly more rigorous protocol than typical 3–5 annotator paradigms.

5. Evaluation Metrics and Benchmark Performance

Performance on SentiMMD is measured using macro-averaged accuracy, precision, recall, and F1-score across the seven categories, with the following definitions:

• $$\mathrm{Acc} = \frac{\sum_{i=1}^{N} \mathbf{1}\{\hat{y}_i = y_i\}}{N}$$
• Macro-Precision: $$\text{Macro-P} = \frac{1}{7}\sum_{j=1}^7 \mathrm{P}_j$$, with $$\mathrm{P}_j = \frac{\mathrm{TP}_j}{\mathrm{TP}_j + \mathrm{FP}_j}$$
• Macro-Recall: $$\text{Macro-R} = \frac{1}{7}\sum_{j=1}^7 \mathrm{R}_j$$, with $$\mathrm{R}_j = \frac{\mathrm{TP}_j}{\mathrm{TP}_j + \mathrm{FN}_j}$$
• Macro-F1: $$\text{Macro-F1} = \frac{1}{7}\sum_{j=1}^7 \mathrm{F1}_j$$

Benchmarking Results:

Key models and results (test set):

Model	Acc. (%)	Macro‐P (%)	Macro‐R (%)	Macro‐F1 (%)
GLM-4V-9B	68.2	67.5	66.8	67.1
mPLUG-7B	65.4	64.7	63.9	64.3
Qwen2.5-VL-3B	59.7	57.1	58.2	57.6
Qwen2.5-VL-7B	75.1	74.5	73.9	74.2
GPT-4o-20250513	77.8	77.2	76.7	77.0
SentiMM (Qwen2.5-VL-7B backbone)	82.1	81.5	81.0	81.2
SentiMM (GPT-4o backbone)	89.3	88.7	88.1	88.4

SentiMM with GPT-4o backbone improves accuracy by 11.5 percentage points over the single-pass GPT-4o baseline and achieves 88.4% macro‐F1. Ablation studies indicate that removing components such as the KB Assistant (no retrieval), the Fusion Inspector (no refinement), or either modality (text or image/video) results in accuracy drops to 84.7%, 85.2%, 78.9%, and 74.3%, respectively. Absence of classifier aggregation (direct fusion only) yields 83.5% accuracy.

6. Applications and Research Trajectories

SentiMMD enables multiple research directions:

• Multimodal Fusion Architectures: Facilitates the design and evaluation of architectures targeting fine-grained emotion recognition with multimodal fusion mechanisms.
• Retrieval-Augmented Sentiment Inference: Supports the study of external knowledge retrieval’s effect on multimodal sentiment inference.
• Benchmarking Multi-Agent/Ensemble Approaches: Allows systematic comparison of multi-agent or ensemble models for sentiment and affect analysis.
• Domain Adaptation and Modality Expansion: May be extended to domain-specific tasks such as product reviews or crisis response and further modalities (e.g., audio, speech).
• Real-Time Social Media Analytics: Provides ground truth for tracking public emotion trends during unfolding events.
• Multimodal LLM Fine-Tuning: Serves as training data for large multimodal LLMs focused on emotionally intelligent dialog systems.

7. Significance and Distinguishing Features

SentiMMD’s balanced class distribution, rigorous multi-annotator and GPT-verified labeling, and multimodal (text + image/video) coverage position it as a pivotal benchmark for next-generation research in affective computing and multimodal sentiment analysis. Its protocol resolves multiple chronic deficiencies of prior datasets, specifically in label granularity, annotation reliability, and modality integration. This makes SentiMMD a methodological reference point for researchers pursuing robust evaluation and model development in social media sentiment inference (Xu et al., 25 Aug 2025).