---
title: 'SentiMM: Multi-Agent Sentiment Analysis'
url: https://www.emergentmind.com/topics/sentimm
type: topic
---

# SentiMM: Multi-Agent Sentiment Analysis

SentiMM is a multimodal, multi-agent framework for sentiment analysis in social media that addresses the challenges presented by heterogeneous and multi-label emotional content. The system employs a structured pipeline of specialized agents to systematically process text and visual data, fuse multimodal signals, retrieve relevant contextual knowledge, and deliver fine-grained sentiment classification. The accompanying SentiMMD dataset provides a benchmark with seven annotated sentiment categories, supporting rigorous comparison against state-of-the-art baselines [2508.18108].

## 1. Multi-Agent Architecture and Data Flow

SentiMM is organized as a five-stage pipeline, where each functional agent specializes in a distinct aspect of multimodal sentiment analysis. The sequential flow of data and agent responsibilities is as follows:

```
[Text Analyst] ──►                         │
   [Image Analyst] ─┼─► [Fusion Inspector] ──► [KB Assistant] ──► [Classifier Aggregator] ─► Sentiment Label
                         │
   [Raw T, I/V ] ───┘
```

1. **Text Analyst**: Processes input text $T$, including preprocessing, segmentation into units $\{s_i\}_{i=1}^n$, and per-segment sentiment analysis via an LLM $M_\text{text}$ yielding $s_\mathrm{score}(s_i)\in[-1,1]$ and feature vector $f_\text{text}(s_i)\in\mathbb{R}^d$. Global text sentiment is aggregated as $S_\text{text} = \sum_{i=1}^n w_i s_\mathrm{score}(s_i)$, with $O_\text{text}$ retaining scores and features.

2. **Image Analyst**: Accepts an image $I$ or video $V$, decomposes video into frames $\{I_{f_k}\}_{k=1}^m$, extracts visual feature vectors $F_\mathrm{vis}(I_j)$ using CNN-based $M_\mathrm{image}$, and per-frame sentiment via $M_\mathrm{senti\text{-}image}$. Video sentiment is aggregated and outputs $O_\mathrm{image}$.

3. **Fusion Inspector**: Fuses $F_\mathrm{text}$ and $F_\mathrm{vis}$ by learned mapping $\phi$, computes a joint embedding $F_\mathrm{combined}$, and applies $M_\mathrm{fusion}$ for multimodal scoring ($S_\mathrm{multimodal}$, $R_\mathrm{fusion}$). Consistency checks are performed (e.g., $\Delta S_\text{text} = |S_\mathrm{multimodal} - S_\text{text}|$), and inconsistencies prompt auxiliary hypotheses via $M_\mathrm{aux}$.

4. **KB Assistant**: Selects keys via $\psi(F_\mathrm{combined}) \to F_\mathrm{key}$, retrieves top-$k$ matching entries via similarity search against vector database $D_\mathrm{rag}$, and generates a contextual report $R_\mathrm{rag}$ using $M_\mathrm{rag}$. Output $O_\mathrm{rag}$ includes retrieved content.

5. **Classifier Aggregator**: Calculates average score from similar examples $S_\mathrm{similar}$, fuses it with $S_\mathrm{multimodal}$ as $S_\mathrm{combined} = \alpha S_\mathrm{multimodal} + \beta S_\mathrm{similar}$, and predicts label $L$ via $M_\mathrm{classify}$. The final output is the predicted sentiment label $L$ and explanation $R_\mathrm{final}$.

## 2. Mathematical and Computational Foundations

SentiMM formalizes its modules and inter-agent communication through specific mathematical constructs:

- **Text Encoding**: Tokenized inputs $x_1,\dots,x_\ell$ are embedded as $\mathbf{e}_t = \mathrm{Embed}(x_t)\in\mathbb{R}^d$, then transformed by a stack of Transformers to yield $\mathbf{h}_t$.

- **Visual Encoding**: Image patches or CNN-derived features are mapped to $\mathbf{v}_j\in\mathbb{R}^p$.

- **Cross-Modal Fusion**: Joint embedding leverages cross-attention:
  $$
  A_{ij} = \frac{\exp((Q_i K_j^\top)/\sqrt{d})}{\sum_{j'} \exp((Q_i K_{j'}^\top)/\sqrt{d})}
  $$
  where $Q = W_Q [F_\mathrm{text}]$ and $K = W_K [F_\mathrm{vis}]$. Output $\phi(F_\mathrm{text}, F_\mathrm{vis}) = \mathrm{Concat}(F_\mathrm{text}, AV)$.

- **Knowledge Retrieval**: For a fusion-derived key $F_\mathrm{key}$, top-$k$ similar vectors $\{d_{i_1}, \dots, d_{i_k}\}$ are identified via cosine similarity $\mathrm{sim}(d_i, F_\mathrm{key})$.

- **Classification**: The aggregator employs multi-class cross-entropy loss:
  $$
  \mathcal{L}_\mathrm{cls} = -\sum_{c=1}^7 y_c\, \log(p_c(L)) + \lambda\|\theta\|_2^2
  $$
  with $y_c$ as one-hot labels, $p_c$ output probabilities, and $\lambda$ weight decay.

## 3. The SentiMMD Benchmark Dataset

SentiMMD is a comprehensive multimodal sentiment analysis benchmark curated to support SentiMM and related research.

- **Label Taxonomy**: Seven sentiment categories—Like (positive endorsement), Happiness (joy), Anger (hostility), Disgust (revulsion), Fear (anxiety), Sadness (sorrow), and Surprise (unexpectedness).
- **Statistics**: 3,500 samples (500 per label); 90%/10% train/test split (3,150/350); includes text + image (224×224) or text + video (3–7 frames, ≈12.3s), with median text length of 28.7 tokens.
- **Annotation Procedure**: Each instance is labeled by 40 human annotators, with multi-stage verification against GPT-based predictions. Discrepancies are adjudicated by expert review or removed.

| Label      | Definition                         | Samples |
|------------|------------------------------------|---------|
| Like       | Positive endorsement               | 500     |
| Happiness  | Overt joy or pleasure              | 500     |
| Anger      | Hostility or irritation            | 500     |
| Disgust    | Revulsion or contempt              | 500     |
| Fear       | Anxiety or dread                   | 500     |
| Sadness    | Sorrow or regret                   | 500     |
| Surprise   | Unexpectedness                     | 500     |

## 4. Empirical Evaluation and Ablation

SentiMM demonstrates state-of-the-art performance on SentiMMD, compared to leading multimodal LLMs. Models are evaluated using accuracy (Acc), macro precision (MP), macro recall (MR), and macro F1 (MF1).

| Model                   | Acc (%) | MP (%) | MR (%) | MF1 (%) |
|-------------------------|---------|--------|--------|---------|
| GLM-4V-9B               |   68.2  |  67.5  |  66.8  |   67.1  |
| mPLUG-7B                |   65.4  |  64.7  |  63.9  |   64.3  |
| Qwen2.5-VL-3B           |   59.7  |  57.1  |  58.2  |   57.6  |
| Qwen2.5-VL-7B           |   75.1  |  74.5  |  73.9  |   74.2  |
| GPT-4o                  |   77.8  |  77.2  |  76.7  |   77.0  |
| SentiMM (GPT-4o)        |   89.3  |  88.7  |  88.1  |   88.4  |
| SentiMM (Qwen2.5-VL-7B) |   82.1  |  81.5  |  81.0  |   81.2  |

Ablation studies using SentiMM (GPT-4o backbone) quantify the contribution of each agent:

| Variant                              | Acc (%) | MF1 (%) |
|---------------------------------------|---------|---------|
| Full SentiMM                         |    89.3 |   88.4  |
| – without KB Assistant                |    84.7 |   83.9  |
| – without Fusion Inspector            |    85.2 |   83.7  |
| – without Image Analyst (text only)   |    78.9 |   77.8  |
| – without Text Analyst (image only)   |    74.3 |   73.5  |
| – without Classifier Aggregator       |    83.5 |   81.8  |

All components, especially the Fusion Inspector and KB Assistant, provide substantial performance gains. Removal of any agent leads to consistent reduction in accuracy and MF1, with end-to-end multi-agent organization yielding improvements up to +11.5 accuracy points over the base GPT-4o baseline.

## 5. Analysis of Failure Modes and Limitations

Observed limitations include:

- **Multimodal Conflict**: Cases where text and visual modalities encode opposing emotions (e.g., sarcastic captions over positive images) lead to inconsistency and misclassification, even with inconsistency-checking logic.
- **External Knowledge Coverage**: Sentiment cues heavily dependent on rare, evolving, or culturally specific references can cause failures in knowledge retrieval, producing mismatches or off-topic supplementary reports.
  
A plausible implication is that enriching the external KB with up-to-date cultural content and improving multimodal disambiguation mechanisms may further boost robustness.

## 6. Prospects and Future Extensions

Extensions highlighted for SentiMM include:

- Incorporation of new modalities such as audio and user metadata to capture broader affective cues.
- Enabling online or semi-automated updates of the external KB to adapt to trending memes and slang, addressing the challenge of semantic drift in social content.
- Migration from pipeline prompting to joint end-to-end fine-tuning of all agents, with the goal of improved integration and efficiency across modality and retrieval boundaries.

Ongoing work aims to reinforce SentiMM’s modular interpretability while closing remaining gaps in coverage and adaptability, especially for complex, context-dependent sentiment attributions [2508.18108].

Source: https://www.emergentmind.com/topics/sentimm