Multimodal Sentiment Perception Network (MSPN)

• MSPN is a multimodal sentiment analysis module that fuses text and image signals using transformer-based methods and learnable sentiment prototypes.
• It employs cross-attention fusion (CAF) and a cross-modal fusion transformer (CMFT) to integrate modality-specific features into unified representations.
• Evaluated on datasets like MVSA and TumEmo, MSPN demonstrates improved accuracy and F1-scores, underscoring its potential in affective computing.

Multimodal Sentiment Perception Network (MSPN) denotes a multimodal sentiment-analysis module that infers affective state from more than one signal source and converts that inference into a usable sentiment cue for downstream computation. In the most explicit usage available here, MSPN is the emotion perception core of AIVA, where it addresses a stated limitation of LLM-based conversational agents: they only process text, so they miss emotional cues expressed through non-verbal modalities such as facial expression and visual context. In that system, MSPN operates on image-text pairs, performs cross-modal fusion, and supplies sentiment predictions to response generation (Li, 3 Sep 2025). In a broader and looser sense, the term also overlaps with multimodal sentiment pipelines that separately encode modalities and then fuse them for sentiment inference, although not every such pipeline is presented as a strict MSPN (Lee et al., 14 Jan 2025).

1. Conceptual scope and problem setting

MSPN is introduced to solve a specific perception problem in emotion-aware human-computer interaction: text alone is insufficient when affect depends on non-verbal evidence. The motivating example is that a phrase such as “It’s okay” can indicate different emotions depending on the accompanying image or face. The architectural consequence is that sentiment perception is moved upstream of language generation, so that a downstream conversational model can be conditioned on an inferred affective state rather than on lexical content alone (Li, 3 Sep 2025).

Within the available literature, the name “MSPN” is not fully uniform. In AIVA, the term refers to a concrete module with Vision Transformer (ViT) and BERT encoders, Cross Attention Fusion (CAF), a Cross-Modal Fusion Transformer (CMFT), and learnable sentiment prototypes (Li, 3 Sep 2025). By contrast, the CMU-MOSEI study titled “Dynamic Multimodal Sentiment Analysis: Leveraging Cross-Modal Attention for Enabled Classification” is described as best understood as a transformer-based multimodal sentiment classification pipeline rather than a full classic “Multimodal Sentiment Perception Network (MSPN)” in the strict sense, even though it shares the MSPN spirit of separate modality encoding followed by fusion (Lee et al., 14 Jan 2025).

A common misconception is that an MSPN necessarily includes text, audio, and video. The AIVA MSPN does not: it uses text + image/visual modalities only. Another misconception is that any multimodal fusion transformer is equivalent to MSPN. The surrounding literature suggests a narrower interpretation: what distinguishes the AIVA formulation is not merely multimodal fusion, but the combination of cross-modal attention, prototype-based sentiment aggregation, and bidirectional supervised contrastive alignment between multimodal embeddings and sentiment prototypes (Li, 3 Sep 2025).

2. Core architecture in AIVA

AIVA’s MSPN takes a batch of image-text pairs,

$$B = \{ (x_{1}, t_{1}), (x_{2}, t_{2}), \dots, (x_{N}, t_{N}) \},$$

where $N$ is the batch size (Li, 3 Sep 2025). The visual modality is encoded by a Vision Transformer (ViT), producing visual tokens

$$V_i = \{ v_1, v_2, \dots, v_M \}, \quad v \in \mathbb{R}^{768},$$

where $M$ is the number of visual tokens. The textual modality is encoded by a text encoder such as BERT, producing text tokens

$$T_i = \{ t_1, t_2, \dots, t_L \}, \quad t \in \mathbb{R}^{768},$$

where $L$ is the number of text tokens (Li, 3 Sep 2025).

The first fusion stage is Cross Attention Fusion (CAF). Each modality queries the other:

$$\left\{ \begin{aligned} \hat{T}_i &= \text{softmax}\left(\frac{W_Q V_i (W_K T_i))^T}{\sqrt{d_k}}\right) W_V T_i \ \hat{V}_i &= \text{softmax}\left(\frac{W_Q T_i (W_K V_i)^T}{\sqrt{d_k}}\right) W_V V_i \end{aligned} \right.$$

This yields text-aware visual features and image-aware text features. The enhanced features are then concatenated into a unified multimodal representation,

$Z_i^0 = \left[ \hat{T}_i ; \hat{V}_i \right],$

with

$Z_i^0 \in \mathbb{R}^{(L+M) \times 768}.$

The result is a joint multimodal token sequence rather than a late decision-level combination (Li, 3 Sep 2025).

The second fusion stage is the Cross-Modal Fusion Transformer (CMFT). Its defining feature is the use of learnable sentiment prototypes,

$$S^0 = \{s_1, s_2, \dots, s_c\},\quad S \in \mathbb{R}^{C \times 768},$$

where $N$0 is the number of sentiment categories. These prototypes act as class-level emotion anchors and are updated through attention over the fused multimodal tokens:

$N$1

The paper interprets this as sentiment prototypes acting as queries while fused multimodal tokens act as keys and values, so that each prototype pulls relevant multimodal evidence toward a class-specific emotional representation. After the final transformer layer, the prototype representations are classified:

$N$2

A plausible implication is that MSPN’s classifier is not only token-fusion-based but also explicitly class-structured through prototype refinement (Li, 3 Sep 2025).

3. Representation learning and optimization

The emotional representation in MSPN is defined at two levels. First, there are token-level multimodal embeddings produced by ViT and BERT and refined by CAF. Second, there are sentiment prototypes $N$3, one for each sentiment category. The paper states that these prototypes capture sentiment-specific information from fused multimodal features (Li, 3 Sep 2025).

Training combines a classification objective with supervised contrastive learning in both directions: multimodal representations $N$4 sentiment prototypes, and sentiment prototypes $N$5 multimodal representations. The paper gives a classification loss intended to be standard cross-entropy between the predicted probability $N$6 and the one-hot ground-truth label $N$7; it also notes that the printed equation appears malformed in the paper text, but the intended meaning is clear (Li, 3 Sep 2025).

The forward supervised contrastive term aligns each multimodal sample with its positive sentiment prototype(s), while the reverse term treats each sentiment prototype as anchor and aligns it with the corresponding multimodal embeddings. The total objective is described as classification loss plus a weighted average of the two contrastive losses, with $N$8 controlling the trade-off between classification and contrastive learning. The ablation over $N$9 reports the best setting at $$V_i = \{ v_1, v_2, \dots, v_M \}, \quad v \in \mathbb{R}^{768},$$0 (Li, 3 Sep 2025).

This training design separates MSPN from simpler fusion-and-classification systems. A plausible implication is that the contrastive component is intended to improve class separability not merely at the decision boundary but in the shared representation space linking fused multimodal samples and class prototypes. That reading is consistent with the reported t-SNE discussion, which states that learned sentiment prototypes form meaningful clusters, with emotions like Happy, Sad, and Angry separated clearly, while Fear and Sad overlap somewhat (Li, 3 Sep 2025).

4. Role in the AIVA interaction pipeline

MSPN is the perceptual front end of AIVA rather than an isolated classifier. Its outputs are the predicted sentiment distribution,

$$V_i = \{ v_1, v_2, \dots, v_M \}, \quad v \in \mathbb{R}^{768},$$1

and the final sentiment labels or sentiment cues extracted from that prediction (Li, 3 Sep 2025).

These outputs are passed into Emotion-aware Prompt Engineering (EPE). The sentiment classification result is used as a prefix in the prompt, so that the downstream LLM can generate a response that is emotionally aligned, context-sensitive, and empathetic. The prompt template includes role definition, few-shot examples, historical context, and Chain-of-Thought prompting. The generation stack is specified as MSPN → emotion-aware prompt → pretrained LLM (LLaMA2-Chat) → TTS + animated avatar (Li, 3 Sep 2025).

In system terms, MSPN converts multimodal affect perception into a conditioning signal for language generation and expressive output. This suggests that, in AIVA, sentiment perception is not an auxiliary analytic task but a control mechanism for the entire interaction loop. The broader application framing given for the full system includes companion robotics, social care, mental health, and human-centered AI (Li, 3 Sep 2025).

5. Empirical evaluation

MSPN is evaluated on MVSA-Single, MVSA-Multi, and TumEmo. MVSA-Single is an image + text sentiment dataset with positive, neutral, and negative labels; MVSA-Multi is the multi-label version of MVSA; TumEmo contains over 195,000 image-text-emotion triplets and is used for pretraining. The reported protocol is: TumEmo pretraining for 1 epoch, MVSA fine-tuning for 10 epochs, Adam optimizer, learning rate $$V_i = \{ v_1, v_2, \dots, v_M \}, \quad v \in \mathbb{R}^{768},$$2, batch size 16 for MVSA and 24 for TumEmo. Reported metrics are Accuracy, Precision, Recall, and F1-score (Li, 3 Sep 2025).

Dataset	MSPN result	Strongest listed baseline
MVSA-Single	74.25% accuracy / 72.84% F1	MGNNS: 73.77% accuracy / 72.70% F1
MVSA-Multi	73.48% accuracy / 70.01% F1	MGNNS: 72.49% accuracy / 69.34% F1
TumEmo	81.81% accuracy / 82.48% precision / 81.81% recall / 81.89% F1	—

The listed baselines include text-only models (CNN, BiLSTM, BERT), image-only models (ResNet-50, OSDA), and multimodal models (MultiSentNet, HSAN, Co-MN-Hop6, MGNNS) (Li, 3 Sep 2025). The reported gains over MGNNS are modest but consistent on both MVSA datasets.

The ablation study removes CAF, CMFT, and SCL. On MVSA-Single, the results are 72.94% acc / 71.17% F1 without CAF, 71.68% acc / 71.19% F1 without CMFT, 73.01% acc / 71.78% F1 without SCL, and 74.25% acc / 72.84% F1 for the full MSPN. On MVSA-Multi, the reported table gives 18.70% acc / 67.41% F1 without CAF, 70.76% acc / 68.76% F1 without CMFT, 69.41% acc / 68.25% F1 without SCL, and 73.48% acc / 70.01% F1 for the full MSPN; the source notes that the 18.70% number appears numerically inconsistent in the paper and is likely a typo, but is reported as written (Li, 3 Sep 2025).

6. Relation to adjacent multimodal sentiment architectures

MSPN belongs to a broader family of multimodal sentiment systems, but nearby designs differ materially in modality set, fusion timing, and representation strategy. In the CMU-MOSEI study “Dynamic Multimodal Sentiment Analysis: Leveraging Cross-Modal Attention for Enabled Classification,” each modality is encoded by a ModalityTransformer with linear projection, Transformer encoder with positional encoding, and classification layer. The study compares A0: late fusion via majority vote, A1: early fusion by concatenating final hidden states, and A2: multi-headed attention to dynamically weigh modality features before classification. On CMU-MOSEI, the reported accuracies are 66.23% for late fusion, 71.87% for early fusion, and 72.39% for multi-headed attention; the paper’s key finding is that early fusion and attention-based fusion both outperform late fusion, but the gain from attention over early fusion is small (Lee et al., 14 Jan 2025).

That model shares the MSPN spirit of multimodal perception followed by sentiment inference, but it is explicitly described as simpler than many classic MSPN formulations because it does not define an explicit recurrent perception network, cross-stage interaction module, or elaborate temporal memory (Lee et al., 14 Jan 2025). Relative to AIVA’s MSPN, the main difference is that AIVA uses prototype-based sentiment aggregation and supervised contrastive alignment, whereas the CMU-MOSEI system centers on a comparison among fusion strategies.

A different neighboring line is “Multimodal Sentiment Analysis based on Multi-channel and Symmetric Mutual Promotion Feature Fusion,” which enriches each modality before fusion and then performs pairwise Symmetric Mutual Promotion (SMP) fusion across visual-audio, visual-text, and audio-text pairs. Its visual branch uses ResNet18 pretrained on AffectNet together with OpenFace2.0 Action Units; its audio branch uses Wav2Vec2.0 fine-tuned on MSP-Podcast together with MFCC; its text branch uses Tencent word embeddings and BERT. The final classifier concatenates intra-modal features $$V_i = \{ v_1, v_2, \dots, v_M \}, \quad v \in \mathbb{R}^{768},$$3 and inter-modal features $$V_i = \{ v_1, v_2, \dots, v_M \}, \quad v \in \mathbb{R}^{768},$$4 for 3-class sentiment classification. On CMU-MOSI and CH-SIMS, the method is reported to be especially strong in Acc-2 and Corr (Zhu et al., 3 Jan 2026).

Taken together, these systems indicate that “MSPN” is best understood as a design region rather than a single canonical blueprint. The main architectural axes are: whether modality fusion is late or early; whether class-level structures such as sentiment prototypes are introduced; whether branch-specific features are retained alongside fused features; and whether temporal alignment and bidirectional inter-modal exchange are modeled explicitly.

7. Limitations, ambiguities, and future directions

The AIVA paper does not provide a dedicated limitations section for MSPN, so explicit failure analysis is limited. The reported material nevertheless suggests several constraints. Modality coverage is limited: MSPN only uses text and image, and does not model audio, gesture, or temporal interaction history directly. Emotion ambiguity remains: the t-SNE discussion notes overlap between some classes, especially Fear and Sad. System-level evaluation is mostly qualitative for the full AIVA interaction pipeline: the paper shows examples of empathetic responses, but does not report standardized human evaluation of interaction quality (Li, 3 Sep 2025).

A separate ambiguity concerns nomenclature. Because some transformer-based multimodal sentiment systems are MSPN-like without being labeled MSPN, the term can blur the distinction between a named architecture and a broader class of multimodal sentiment perception modules. The CMU-MOSEI transformer study is explicit on this point: it is better understood as a fusion-comparison pipeline rather than a full classic MSPN in the strict sense (Lee et al., 14 Jan 2025).

Future work implied across the cited papers is convergent. AIVA points toward richer multimodal emotion perception, stronger integration with LLMs, and broader real-world applications in companion robots, social care, and mental-health-oriented interfaces (Li, 3 Sep 2025). The CMU-MOSEI study proposes refining feature fusion techniques, incorporating temporal data, and exploring dynamic feature weighting (Lee et al., 14 Jan 2025). The SMP-based work implicitly highlights additional open issues: no explicit handling of missing modalities is described, the model assumes aligned sequences with uniform length, and pairwise fusion may become complex as modality count or sequence length grows (Zhu et al., 3 Jan 2026).

In that broader context, MSPN can be viewed as one concrete answer to a general research question: how to convert heterogeneous perceptual evidence into a sentiment representation that is both discriminative for classification and actionable for downstream interaction. The AIVA formulation answers this with ViT + BERT, CAF, CMFT, learnable sentiment prototypes, and bidirectional supervised contrastive learning; adjacent work suggests that temporal modeling, broader modality coverage, and more explicit robustness mechanisms remain central directions for subsequent multimodal sentiment perception research.