MLLMsent: Visual Sentiment Inference

Updated 9 July 2026

MLLMsent is a framework that investigates visual sentiment by reasoning over complex, scene-level semantics in images.
It decomposes sentiment inference into three tasks: direct classification, description-mediated analysis, and fine-tuned text classification.
The framework outperforms Lexicon-, CNN-, and Transformer-based baselines by 30.9% to 64.8%, proving effective in cross-dataset evaluations.

MLLMsent is a framework for investigating whether Multimodal LLMs (MLLMs) can reason about sentiment from images, rather than only recognize objects or generate captions. It was introduced to study visual sentiment through three complementary perspectives: direct sentiment classification from images by MLLMs, sentiment analysis over MLLM-generated image descriptions using pre-trained text LLMs, and fine-tuning text LLMs on sentiment-labeled image descriptions. On the reported benchmark setting, the fine-tuned variant attains state-of-the-art performance, outperforming Lexicon-, CNN-, and Transformer-based baselines by up to 30.9%, 64.8%, and 42.4%, respectively, across different levels of evaluators' agreement and sentiment polarity categories; in a cross-dataset test, it also exceeds the best runner-up by up to 8.26% without training on the target data (Silva et al., 23 Aug 2025).

1. Definition and research problem

MLLMsent was proposed for a setting in which sentiment perception is closely tied to complex, scene-level semantics. The central question is not merely whether an MLLM can describe visual content, but whether it can infer affective polarity from that content in a way that aligns with human annotation. The framework therefore operationalizes sentiment reasoning as a multimodal inference problem with controlled comparisons across direct visual prompting, text-mediated sentiment inference, and supervised adaptation of the text-side classifier (Silva et al., 23 Aug 2025).

The motivating distinction is methodological. Direct MLLM prompting tests whether a vision-LLM can map images to sentiment labels in a single step. Description-mediated inference decomposes the problem into image-to-text and text-to-sentiment stages. Fine-tuning then asks whether the second stage can be specialized to the linguistic style of MLLM-generated descriptions. This decomposition suggests that visual sentiment may be easier to recover after semantic verbalization than by direct image classification alone, an implication that is borne out empirically in the reported experiments.

2. Framework structure and task decomposition

MLLMsent is organized around three tasks. In the first, an MLLM receives an image and a prompt such as “Analyze this image, and classify it as {sentiment labels}, do not describe the image, and select only one class.” In the second, the MLLM is first prompted to “Describe this image in detail,” and the resulting description is then classified for sentiment by a pre-trained text-only LLM. In the third, the same text classifiers are fine-tuned on sentiment-labeled image descriptions (Silva et al., 23 Aug 2025).

Perspective	Procedure	Models reported
Direct classification	Image + prompt $\rightarrow$ sentiment label	MiniGPT-4, GPT-4o mini, DeepSeek-VL2-Tiny
Description-mediated inference	Image $\rightarrow$ MLLM description $\rightarrow$ text LLM sentiment classification	BART-Large-MNLI, ModernBERT, LLAMA-3
Fine-tuned text classification	Sentiment-labeled image descriptions used to adapt the text classifier	BART, ModernBERT, LLAMA-3 with qLORA

The framework uses two label-control dimensions. The first is annotator agreement threshold: $\sigma_3$ requires at least $3/5$ annotators to agree, while $\sigma_5$ requires unanimity among all five annotators. The second is sentiment granularity: $P_5$ denotes five sentiment classes, $P_3$ merges the “slightly” categories into positive and negative, and $P_2$ denotes binary sentiment. The dominant-label formulation is given as

$c_{\sigma_l}^* (s) = \{ \arg\max_c\, s = (s_1,...,s_C)\;|\;\max (s) \ge l \text{ in } \sigma_l\}.$

The fine-tuning configuration is asymmetric across classifiers. All parameters of BART and ModernBERT are fine-tuned with a classification head mapping to sentiment classes, whereas LLAMA-3 is adapted with qLORA. Stratified 5-fold cross-validation and class-balanced weighted loss are used.

3. Datasets, implementation, and evaluation protocol

The primary benchmark is PerceptSent, which contains 5,000 Instagram, Flickr, and NYC311 images, each annotated by five annotators across five sentiment classes. Cross-dataset generalization is assessed on DeepSent, a binary visual sentiment dataset of 1,269 Twitter images, also with five annotators. The framework is implemented in Python using PyTorch and HuggingFace Transformers, and training is reported on 2 NVIDIA GPUs (Silva et al., 23 Aug 2025).

Evaluation is principally based on F1-score, with confidence intervals reported for all runs. Accuracy is additionally used in the cross-dataset DeepSent test for compatibility with prior work. The paper explicitly studies all combinations of agreement thresholds and sentiment label sets, including $\rightarrow$ 0, $\rightarrow$ 1, $\rightarrow$ 2, $\rightarrow$ 3, and the binary $\rightarrow$ 4, $\rightarrow$ 5 settings.

This protocol makes two aspects of the problem visible. First, higher annotator agreement yields a cleaner supervisory signal. Second, reducing the number of sentiment categories lowers label granularity. The reported experiments show that both factors materially affect performance, which is significant because visual sentiment benchmarks often mix ambiguity in perception with ambiguity in category design.

4. Empirical results and state of the art

The direct-classification setting establishes a baseline for native MLLM sentiment reasoning. GPT-4o mini outperformed the open-source and open-weight alternatives: for $\rightarrow$ 6 it achieved 44.5% F1, while DeepSeek reached 26.6% F1; under the easier $\rightarrow$ 7 condition, direct prompting reached up to 87.7% F1. MiniGPT-4 was reported to struggle with reliable outputs, often producing ambiguous or multi-label responses despite prompt engineering (Silva et al., 23 Aug 2025).

The description-mediated pipeline improved over direct image classification, especially when ModernBERT was used as the text classifier. For $\rightarrow$ 8, GPT-4o mini descriptions combined with ModernBERT reached 47.9% F1; for $\rightarrow$ 9, the same configuration reached 90.6% F1. The strongest results came from fine-tuning the text classifier on sentiment-labeled descriptions. In that setting, GPT-4o mini descriptions combined with fine-tuned ModernBERT achieved 95.8% F1 $\rightarrow$ 0 on $\rightarrow$ 1. Reported relative F1 gains from fine-tuning over the pre-trained text classifiers were +11.6% to 50.9% for BART, +5.7% to 32.2% for ModernBERT, and +4.1% to 102.9% for LLAMA.

The comparative margins against earlier baseline families are also explicit. On $\rightarrow$ 2, VADER obtained 59.2% F1, while MLLMsent reached 77.5% F1, corresponding to the reported +30.9% improvement over the Lexicon-based baseline. On $\rightarrow$ 3, ResNet obtained 51.2% F1 and MLLMsent 84.4% F1, corresponding to +64.8% over the CNN-based baseline. On $\rightarrow$ 4, Swin obtained 41.0% F1 and MLLMsent 58.4% F1, corresponding to +42.4% over the Transformer-based baseline. Post-hoc testing was reported to establish that MLLMsent clearly outperforms all baselines with $\rightarrow$ 5.

5. Generalization, interpretability, and common misconceptions

A common misconception is that direct prompting of an MLLM on an image should be the strongest way to infer sentiment. The reported results contradict that expectation. The two-stage visual-to-text-to-sentiment pipeline outperformed direct image-based classification, and the fine-tuned text-side classifier outperformed both. This indicates that textual semantic compression of the scene can be a more effective substrate for affective inference than direct vision-language sentiment prompting in the tested setting (Silva et al., 23 Aug 2025).

Another misconception is that visual sentiment is simply a by-product of object recognition. The paper frames the task as dependent on complex, scene-level semantics, and the error analysis supports that characterization: some failures are traceable to limitations in the image description stage, not merely to label confusion at the sentiment classifier. Qualitative examples reported in the study show that MLLMs can surface contextual cues such as a person “asking for help” or squirrels scavenging in a trash can, which can align with human sentiment judgments. This makes the pipeline not only predictive but also interpretable, because it produces textual evidence alongside the sentiment label.

The cross-dataset evaluation strengthens the generalization claim. MLLMsent was fine-tuned only on PerceptSent and then tested on DeepSent, yet it still outperformed models trained directly on DeepSent. Specifically, on $\rightarrow$ 6 it achieved 88.0% accuracy versus 81.5% for the best previous model, ResNet50; on $\rightarrow$ 7 it achieved 95.6% accuracy versus 88.3% for DenseNet. The reported absolute advantage over the best runner-up reached 8.26%.

MLLMsent occupies a distinct position within sentiment-analysis research. In text-only sentiment analysis, multi-LLM cooperation has been explored through explicit negotiation, where a reasoning-infused generator and an explanation-deriving discriminator iteratively refine a decision until consensus is reached (Sun et al., 2023). In finance, FinSentLLM combines FinBERT, RoBERTa-sentiment, and structured semantic financial signals via a compact meta-classifier, with consistent 3–6% gains over strong baselines on Financial PhraseBank and econometric links between sentiment and market dynamics (Zhang et al., 16 Sep 2025). MLLMsent differs from both lines by centering image sentiment and by making MLLM-generated descriptions the bridge between vision and affective text classification.

Within multimodal affect modeling, MMoLRE addresses multimodal sentiment analysis and multimodal emotion recognition through shared and task-specific low-rank experts, achieving state-of-the-art performance on the MSA task and competitive MER results on CMU-MOSI and CMU-MOSEI (Zhang et al., 20 May 2025). LLaVA-MoLE, by contrast, is concerned with instruction-finetuning conflicts in MLLMs and uses a sparse Mixture of LoRA Experts to mitigate data conflicts when mixing distinct instruction datasets (Chen et al., 2024). Relative to these systems, MLLMsent is less a new backbone architecture than a model-agnostic evaluation and training scheme for probing sentiment reasoning in MLLMs.

A plausible implication is that MLLMsent connects two research agendas that are often treated separately: multimodal reasoning and affective computing. Its empirical profile suggests that high-performing visual sentiment systems may emerge not from end-to-end visual classifiers alone, but from carefully staged pipelines in which MLLMs provide semantically rich descriptions and text LLMs provide sentiment-specialized decision boundaries. The reported release of generated image descriptions and fine-tuned models further positions the framework as a benchmark-setting reference point for subsequent work on visual sentiment analysis (Silva et al., 23 Aug 2025).