SentiMM: Multi-Agent Sentiment Analysis

• SentiMM is a multimodal, multi-agent framework that processes heterogeneous text and visual data through a structured five-stage pipeline for sentiment analysis.
• The framework employs specialized agents—including text and image analysts, a fusion inspector, a KB assistant, and a classifier aggregator—to enhance performance over state-of-the-art baselines.
• The accompanying SentiMMD benchmark, with seven sentiment categories and detailed ablation studies, validates significant accuracy improvements and highlights future extension possibilities.

SentiMM is a multimodal, multi-agent framework for sentiment analysis in social media that addresses the challenges presented by heterogeneous and multi-label emotional content. The system employs a structured pipeline of specialized agents to systematically process text and visual data, fuse multimodal signals, retrieve relevant contextual knowledge, and deliver fine-grained sentiment classification. The accompanying SentiMMD dataset provides a benchmark with seven annotated sentiment categories, supporting rigorous comparison against state-of-the-art baselines (Xu et al., 25 Aug 2025).

1. Multi-Agent Architecture and Data Flow

SentiMM is organized as a five-stage pipeline, where each functional agent specializes in a distinct aspect of multimodal sentiment analysis. The sequential flow of data and agent responsibilities is as follows:

$$S_\text{text} = \sum_{i=1}^n w_i s_\mathrm{score}(s_i)$$2

1. Text Analyst: Processes input text $T$, including preprocessing, segmentation into units $\{s_i\}_{i=1}^n$, and per-segment sentiment analysis via an LLM $M_\text{text}$ yielding $s_\mathrm{score}(s_i)\in[-1,1]$ and feature vector $f_\text{text}(s_i)\in\mathbb{R}^d$. Global text sentiment is aggregated as $$S_\text{text} = \sum_{i=1}^n w_i s_\mathrm{score}(s_i)$$, with $O_\text{text}$ retaining scores and features.
2. Image Analyst: Accepts an image $I$ or video $V$, decomposes video into frames $\{I_{f_k}\}_{k=1}^m$, extracts visual feature vectors $\{s_i\}_{i=1}^n$0 using CNN-based $\{s_i\}_{i=1}^n$1, and per-frame sentiment via $\{s_i\}_{i=1}^n$2. Video sentiment is aggregated and outputs $\{s_i\}_{i=1}^n$3.
3. Fusion Inspector: Fuses $\{s_i\}_{i=1}^n$4 and $\{s_i\}_{i=1}^n$5 by learned mapping $\{s_i\}_{i=1}^n$6, computes a joint embedding $\{s_i\}_{i=1}^n$7, and applies $\{s_i\}_{i=1}^n$8 for multimodal scoring ($\{s_i\}_{i=1}^n$9, $M_\text{text}$0). Consistency checks are performed (e.g., $M_\text{text}$1), and inconsistencies prompt auxiliary hypotheses via $M_\text{text}$2.
4. KB Assistant: Selects keys via $M_\text{text}$3, retrieves top-$M_\text{text}$4 matching entries via similarity search against vector database $M_\text{text}$5, and generates a contextual report $M_\text{text}$6 using $M_\text{text}$7. Output $M_\text{text}$8 includes retrieved content.
5. Classifier Aggregator: Calculates average score from similar examples $M_\text{text}$9, fuses it with $s_\mathrm{score}(s_i)\in[-1,1]$0 as $s_\mathrm{score}(s_i)\in[-1,1]$1, and predicts label $s_\mathrm{score}(s_i)\in[-1,1]$2 via $s_\mathrm{score}(s_i)\in[-1,1]$3. The final output is the predicted sentiment label $s_\mathrm{score}(s_i)\in[-1,1]$4 and explanation $s_\mathrm{score}(s_i)\in[-1,1]$5.

2. Mathematical and Computational Foundations

SentiMM formalizes its modules and inter-agent communication through specific mathematical constructs:

• Text Encoding: Tokenized inputs $s_\mathrm{score}(s_i)\in[-1,1]$6 are embedded as $s_\mathrm{score}(s_i)\in[-1,1]$7, then transformed by a stack of Transformers to yield $s_\mathrm{score}(s_i)\in[-1,1]$8.
• Visual Encoding: Image patches or CNN-derived features are mapped to $s_\mathrm{score}(s_i)\in[-1,1]$9.
• Cross-Modal Fusion: Joint embedding leverages cross-attention:

$f_\text{text}(s_i)\in\mathbb{R}^d$0

where $f_\text{text}(s_i)\in\mathbb{R}^d$1 and $f_\text{text}(s_i)\in\mathbb{R}^d$2. Output $f_\text{text}(s_i)\in\mathbb{R}^d$3.

• Knowledge Retrieval: For a fusion-derived key $f_\text{text}(s_i)\in\mathbb{R}^d$4, top-$f_\text{text}(s_i)\in\mathbb{R}^d$5 similar vectors $f_\text{text}(s_i)\in\mathbb{R}^d$6 are identified via cosine similarity $f_\text{text}(s_i)\in\mathbb{R}^d$7.
• Classification: The aggregator employs multi-class cross-entropy loss:

$f_\text{text}(s_i)\in\mathbb{R}^d$8

with $f_\text{text}(s_i)\in\mathbb{R}^d$9 as one-hot labels, $$S_\text{text} = \sum_{i=1}^n w_i s_\mathrm{score}(s_i)$$0 output probabilities, and $$S_\text{text} = \sum_{i=1}^n w_i s_\mathrm{score}(s_i)$$1 weight decay.

3. The SentiMMD Benchmark Dataset

SentiMMD is a comprehensive multimodal sentiment analysis benchmark curated to support SentiMM and related research.

• Label Taxonomy: Seven sentiment categories—Like (positive endorsement), Happiness (joy), Anger (hostility), Disgust (revulsion), Fear (anxiety), Sadness (sorrow), and Surprise (unexpectedness).
• Statistics: 3,500 samples (500 per label); 90%/10% train/test split (3,150/350); includes text + image (224×224) or text + video (3–7 frames, ≈12.3s), with median text length of 28.7 tokens.
• Annotation Procedure: Each instance is labeled by 40 human annotators, with multi-stage verification against GPT-based predictions. Discrepancies are adjudicated by expert review or removed.

Label	Definition	Samples
Like	Positive endorsement	500
Happiness	Overt joy or pleasure	500
Anger	Hostility or irritation	500
Disgust	Revulsion or contempt	500
Fear	Anxiety or dread	500
Sadness	Sorrow or regret	500
Surprise	Unexpectedness	500

4. Empirical Evaluation and Ablation

SentiMM demonstrates state-of-the-art performance on SentiMMD, compared to leading multimodal LLMs. Models are evaluated using accuracy (Acc), macro precision (MP), macro recall (MR), and macro F1 (MF1).

Model	Acc (%)	MP (%)	MR (%)	MF1 (%)
GLM-4V-9B	68.2	67.5	66.8	67.1
mPLUG-7B	65.4	64.7	63.9	64.3
Qwen2.5-VL-3B	59.7	57.1	58.2	57.6
Qwen2.5-VL-7B	75.1	74.5	73.9	74.2
GPT-4o	77.8	77.2	76.7	77.0
SentiMM (GPT-4o)	89.3	88.7	88.1	88.4
SentiMM (Qwen2.5-VL-7B)	82.1	81.5	81.0	81.2

Ablation studies using SentiMM (GPT-4o backbone) quantify the contribution of each agent:

Variant	Acc (%)	MF1 (%)
Full SentiMM	89.3	88.4
– without KB Assistant	84.7	83.9
– without Fusion Inspector	85.2	83.7
– without Image Analyst (text only)	78.9	77.8
– without Text Analyst (image only)	74.3	73.5
– without Classifier Aggregator	83.5	81.8

All components, especially the Fusion Inspector and KB Assistant, provide substantial performance gains. Removal of any agent leads to consistent reduction in accuracy and MF1, with end-to-end multi-agent organization yielding improvements up to +11.5 accuracy points over the base GPT-4o baseline.

5. Analysis of Failure Modes and Limitations

Observed limitations include:

• Multimodal Conflict: Cases where text and visual modalities encode opposing emotions (e.g., sarcastic captions over positive images) lead to inconsistency and misclassification, even with inconsistency-checking logic.
• External Knowledge Coverage: Sentiment cues heavily dependent on rare, evolving, or culturally specific references can cause failures in knowledge retrieval, producing mismatches or off-topic supplementary reports.

A plausible implication is that enriching the external KB with up-to-date cultural content and improving multimodal disambiguation mechanisms may further boost robustness.

6. Prospects and Future Extensions

Extensions highlighted for SentiMM include:

• Incorporation of new modalities such as audio and user metadata to capture broader affective cues.
• Enabling online or semi-automated updates of the external KB to adapt to trending memes and slang, addressing the challenge of semantic drift in social content.
• Migration from pipeline prompting to joint end-to-end fine-tuning of all agents, with the goal of improved integration and efficiency across modality and retrieval boundaries.

Ongoing work aims to reinforce SentiMM’s modular interpretability while closing remaining gaps in coverage and adaptability, especially for complex, context-dependent sentiment attributions (Xu et al., 25 Aug 2025).