---
title: Multimodal Sentiment Perception Network (MSPN)
url: https://www.emergentmind.com/topics/multimodal-sentiment-perception-network-mspn
type: topic
---

# Multimodal Sentiment Perception Network (MSPN)

Multimodal Sentiment Perception Network (MSPN) denotes a multimodal sentiment-analysis module that infers affective state from more than one signal source and converts that inference into a usable sentiment cue for downstream computation. In the most explicit usage available here, MSPN is the emotion perception core of AIVA, where it addresses a stated limitation of LLM-based conversational agents: they only process text, so they miss emotional cues expressed through non-verbal modalities such as facial expression and visual context. In that system, MSPN operates on image-text pairs, performs cross-modal fusion, and supplies sentiment predictions to response generation [2509.03212]. In a broader and looser sense, the term also overlaps with multimodal sentiment pipelines that separately encode modalities and then fuse them for sentiment inference, although not every such pipeline is presented as a strict MSPN [2501.08085].

## 1. Conceptual scope and problem setting

MSPN is introduced to solve a specific perception problem in emotion-aware human-computer interaction: text alone is insufficient when affect depends on non-verbal evidence. The motivating example is that a phrase such as *“It’s okay”* can indicate different emotions depending on the accompanying image or face. The architectural consequence is that sentiment perception is moved upstream of language generation, so that a downstream conversational model can be conditioned on an inferred affective state rather than on lexical content alone [2509.03212].

Within the available literature, the name “MSPN” is not fully uniform. In AIVA, the term refers to a concrete module with **Vision Transformer (ViT)** and **BERT** encoders, **Cross Attention Fusion (CAF)**, a **Cross-Modal Fusion Transformer (CMFT)**, and **learnable sentiment prototypes** [2509.03212]. By contrast, the CMU-MOSEI study titled “Dynamic Multimodal Sentiment Analysis: Leveraging Cross-Modal Attention for Enabled Classification” is described as best understood as a **transformer-based multimodal sentiment classification pipeline** rather than a full classic “Multimodal Sentiment Perception Network (MSPN)” in the strict sense, even though it shares the MSPN spirit of separate modality encoding followed by fusion [2501.08085].

A common misconception is that an MSPN necessarily includes text, audio, and video. The AIVA MSPN does **not**: it uses **text + image/visual** modalities only. Another misconception is that any multimodal fusion transformer is equivalent to MSPN. The surrounding literature suggests a narrower interpretation: what distinguishes the AIVA formulation is not merely multimodal fusion, but the combination of cross-modal attention, prototype-based sentiment aggregation, and bidirectional supervised contrastive alignment between multimodal embeddings and sentiment prototypes [2509.03212].

## 2. Core architecture in AIVA

AIVA’s MSPN takes a batch of **image-text pairs**,
$$
B = \{ (x_{1}, t_{1}), (x_{2}, t_{2}), \dots, (x_{N}, t_{N}) \},
$$
where \(N\) is the batch size [2509.03212]. The visual modality is encoded by a **Vision Transformer (ViT)**, producing visual tokens
$$
V_i = \{ v_1, v_2, \dots, v_M \}, \quad v \in \mathbb{R}^{768},
$$
where \(M\) is the number of visual tokens. The textual modality is encoded by a **text encoder such as BERT**, producing text tokens
$$
T_i = \{ t_1, t_2, \dots, t_L \}, \quad t \in \mathbb{R}^{768},
$$
where \(L\) is the number of text tokens [2509.03212].

The first fusion stage is **Cross Attention Fusion (CAF)**. Each modality queries the other:
$$
\left\{
\begin{aligned}
\hat{T}_i &= \text{softmax}\left(\frac{W_Q V_i (W_K T_i))^T}{\sqrt{d_k}}\right) W_V T_i \\
\hat{V}_i &= \text{softmax}\left(\frac{W_Q T_i (W_K V_i)^T}{\sqrt{d_k}}\right) W_V V_i
\end{aligned}
\right.
$$
This yields text-aware visual features and image-aware text features. The enhanced features are then concatenated into a unified multimodal representation,
$$
Z_i^0 = \left[ \hat{T}_i ; \hat{V}_i \right],
$$
with
$$
Z_i^0 \in \mathbb{R}^{(L+M) \times 768}.
$$
The result is a joint multimodal token sequence rather than a late decision-level combination [2509.03212].

The second fusion stage is the **Cross-Modal Fusion Transformer (CMFT)**. Its defining feature is the use of **learnable sentiment prototypes**,
$$
S^0 = \{s_1, s_2, \dots, s_c\},\quad S \in \mathbb{R}^{C \times 768},
$$
where \(C\) is the number of sentiment categories. These prototypes act as class-level emotion anchors and are updated through attention over the fused multimodal tokens:
$$
{S}^{j} = \text{softmax}\left(\frac{W_Q S^{j-1} (W_K Z_{i}^{j})^T}{\sqrt{d_k}}\right) W_V Z_{i}^{j} + S^{j-1}.
$$
The paper interprets this as sentiment prototypes acting as queries while fused multimodal tokens act as keys and values, so that each prototype pulls relevant multimodal evidence toward a class-specific emotional representation. After the final transformer layer, the prototype representations are classified:
$$
\hat{p}_i = \text{softmax}(MLP(S^N)).
$$
A plausible implication is that MSPN’s classifier is not only token-fusion-based but also explicitly class-structured through prototype refinement [2509.03212].

## 3. Representation learning and optimization

The emotional representation in MSPN is defined at two levels. First, there are **token-level multimodal embeddings** produced by ViT and BERT and refined by CAF. Second, there are **sentiment prototypes** \(S^0\), one for each sentiment category. The paper states that these prototypes capture **sentiment-specific information** from fused multimodal features [2509.03212].

Training combines a classification objective with **supervised contrastive learning** in both directions: multimodal representations \(\rightarrow\) sentiment prototypes, and sentiment prototypes \(\rightarrow\) multimodal representations. The paper gives a classification loss intended to be standard cross-entropy between the predicted probability \(\hat{p}_i\) and the one-hot ground-truth label \(y_i\); it also notes that the printed equation appears malformed in the paper text, but the intended meaning is clear [2509.03212].

The forward supervised contrastive term aligns each multimodal sample with its positive sentiment prototype(s), while the reverse term treats each sentiment prototype as anchor and aligns it with the corresponding multimodal embeddings. The total objective is described as classification loss plus a weighted average of the two contrastive losses, with \(\lambda\) controlling the trade-off between classification and contrastive learning. The ablation over \(\lambda\) reports the best setting at \(\lambda = 1.0\) [2509.03212].

This training design separates MSPN from simpler fusion-and-classification systems. A plausible implication is that the contrastive component is intended to improve class separability not merely at the decision boundary but in the shared representation space linking fused multimodal samples and class prototypes. That reading is consistent with the reported t-SNE discussion, which states that learned sentiment prototypes form meaningful clusters, with emotions like Happy, Sad, and Angry separated clearly, while Fear and Sad overlap somewhat [2509.03212].

## 4. Role in the AIVA interaction pipeline

MSPN is the perceptual front end of AIVA rather than an isolated classifier. Its outputs are the predicted sentiment distribution,
$$
\hat{p}_i = \text{softmax}(MLP(S^N)),
$$
and the final sentiment labels or sentiment cues extracted from that prediction [2509.03212].

These outputs are passed into **Emotion-aware Prompt Engineering (EPE)**. The sentiment classification result is used as a **prefix in the prompt**, so that the downstream LLM can generate a response that is emotionally aligned, context-sensitive, and empathetic. The prompt template includes **role definition**, **few-shot examples**, **historical context**, and **Chain-of-Thought prompting**. The generation stack is specified as **MSPN → emotion-aware prompt → pretrained LLM (LLaMA2-Chat) → TTS + animated avatar** [2509.03212].

In system terms, MSPN converts multimodal affect perception into a conditioning signal for language generation and expressive output. This suggests that, in AIVA, sentiment perception is not an auxiliary analytic task but a control mechanism for the entire interaction loop. The broader application framing given for the full system includes companion robotics, social care, mental health, and human-centered AI [2509.03212].

## 5. Empirical evaluation

MSPN is evaluated on **MVSA-Single**, **MVSA-Multi**, and **TumEmo**. MVSA-Single is an image + text sentiment dataset with positive, neutral, and negative labels; MVSA-Multi is the multi-label version of MVSA; TumEmo contains over **195,000** image-text-emotion triplets and is used for **pretraining**. The reported protocol is: TumEmo pretraining for **1 epoch**, MVSA fine-tuning for **10 epochs**, **Adam** optimizer, learning rate \(2 \times 10^{-5}\), batch size **16** for MVSA and **24** for TumEmo. Reported metrics are **Accuracy**, **Precision**, **Recall**, and **F1-score** [2509.03212].

| Dataset | MSPN result | Strongest listed baseline |
|---|---:|---:|
| MVSA-Single | 74.25% accuracy / 72.84% F1 | MGNNS: 73.77% accuracy / 72.70% F1 |
| MVSA-Multi | 73.48% accuracy / 70.01% F1 | MGNNS: 72.49% accuracy / 69.34% F1 |
| TumEmo | 81.81% accuracy / 82.48% precision / 81.81% recall / 81.89% F1 | — |

The listed baselines include text-only models (**CNN**, **BiLSTM**, **BERT**), image-only models (**ResNet-50**, **OSDA**), and multimodal models (**MultiSentNet**, **HSAN**, **Co-MN-Hop6**, **MGNNS**) [2509.03212]. The reported gains over MGNNS are modest but consistent on both MVSA datasets.

The ablation study removes **CAF**, **CMFT**, and **SCL**. On **MVSA-Single**, the results are **72.94% acc / 71.17% F1** without CAF, **71.68% acc / 71.19% F1** without CMFT, **73.01% acc / 71.78% F1** without SCL, and **74.25% acc / 72.84% F1** for the full MSPN. On **MVSA-Multi**, the reported table gives **18.70% acc / 67.41% F1** without CAF, **70.76% acc / 68.76% F1** without CMFT, **69.41% acc / 68.25% F1** without SCL, and **73.48% acc / 70.01% F1** for the full MSPN; the source notes that the **18.70%** number appears numerically inconsistent in the paper and is likely a typo, but is reported as written [2509.03212].

## 6. Relation to adjacent multimodal sentiment architectures

MSPN belongs to a broader family of multimodal sentiment systems, but nearby designs differ materially in modality set, fusion timing, and representation strategy. In the CMU-MOSEI study “Dynamic Multimodal Sentiment Analysis: Leveraging Cross-Modal Attention for Enabled Classification,” each modality is encoded by a **ModalityTransformer** with linear projection, Transformer encoder with positional encoding, and classification layer. The study compares **A0: late fusion** via majority vote, **A1: early fusion** by concatenating final hidden states, and **A2: multi-headed attention** to dynamically weigh modality features before classification. On CMU-MOSEI, the reported accuracies are **66.23%** for late fusion, **71.87%** for early fusion, and **72.39%** for multi-headed attention; the paper’s key finding is that early fusion and attention-based fusion both outperform late fusion, but the gain from attention over early fusion is small [2501.08085].

That model shares the MSPN spirit of multimodal perception followed by sentiment inference, but it is explicitly described as simpler than many classic MSPN formulations because it does not define an explicit recurrent perception network, cross-stage interaction module, or elaborate temporal memory [2501.08085]. Relative to AIVA’s MSPN, the main difference is that AIVA uses prototype-based sentiment aggregation and supervised contrastive alignment, whereas the CMU-MOSEI system centers on a comparison among fusion strategies.

A different neighboring line is “Multimodal Sentiment Analysis based on Multi-channel and Symmetric Mutual Promotion Feature Fusion,” which enriches each modality before fusion and then performs pairwise **Symmetric Mutual Promotion (SMP)** fusion across visual-audio, visual-text, and audio-text pairs. Its visual branch uses **ResNet18** pretrained on **AffectNet** together with **OpenFace2.0** Action Units; its audio branch uses **Wav2Vec2.0** fine-tuned on **MSP-Podcast** together with **MFCC**; its text branch uses **Tencent word embeddings** and **BERT**. The final classifier concatenates intra-modal features \(F_v, F_a, F_t\) and inter-modal features \(F_{va}, F_{vt}, F_{at}\) for 3-class sentiment classification. On CMU-MOSI and CH-SIMS, the method is reported to be especially strong in **Acc-2** and **Corr** [2601.02415].

Taken together, these systems indicate that “MSPN” is best understood as a design region rather than a single canonical blueprint. The main architectural axes are: whether modality fusion is late or early; whether class-level structures such as sentiment prototypes are introduced; whether branch-specific features are retained alongside fused features; and whether temporal alignment and bidirectional inter-modal exchange are modeled explicitly.

## 7. Limitations, ambiguities, and future directions

The AIVA paper does **not** provide a dedicated limitations section for MSPN, so explicit failure analysis is limited. The reported material nevertheless suggests several constraints. **Modality coverage is limited**: MSPN only uses text and image, and does not model audio, gesture, or temporal interaction history directly. **Emotion ambiguity remains**: the t-SNE discussion notes overlap between some classes, especially **Fear** and **Sad**. **System-level evaluation is mostly qualitative** for the full AIVA interaction pipeline: the paper shows examples of empathetic responses, but does not report standardized human evaluation of interaction quality [2509.03212].

A separate ambiguity concerns nomenclature. Because some transformer-based multimodal sentiment systems are MSPN-like without being labeled MSPN, the term can blur the distinction between a named architecture and a broader class of multimodal sentiment perception modules. The CMU-MOSEI transformer study is explicit on this point: it is better understood as a fusion-comparison pipeline rather than a full classic MSPN in the strict sense [2501.08085].

Future work implied across the cited papers is convergent. AIVA points toward richer multimodal emotion perception, stronger integration with LLMs, and broader real-world applications in companion robots, social care, and mental-health-oriented interfaces [2509.03212]. The CMU-MOSEI study proposes refining feature fusion techniques, incorporating temporal data, and exploring dynamic feature weighting [2501.08085]. The SMP-based work implicitly highlights additional open issues: no explicit handling of missing modalities is described, the model assumes aligned sequences with uniform length, and pairwise fusion may become complex as modality count or sequence length grows [2601.02415].

In that broader context, MSPN can be viewed as one concrete answer to a general research question: how to convert heterogeneous perceptual evidence into a sentiment representation that is both discriminative for classification and actionable for downstream interaction. The AIVA formulation answers this with **ViT + BERT**, **CAF**, **CMFT**, **learnable sentiment prototypes**, and **bidirectional supervised contrastive learning**; adjacent work suggests that temporal modeling, broader modality coverage, and more explicit robustness mechanisms remain central directions for subsequent multimodal sentiment perception research.

Source: https://www.emergentmind.com/topics/multimodal-sentiment-perception-network-mspn