Beyond Acoustic Emotion Recognition: Multimodal Pathos Analysis in Political Speech Using LLM-Based and Acoustic Emotion Models

Abstract: We investigate whether acoustic emotion recognition models can serve as proxies for the Pathos dimension in political speech analysis, as operationalised by the TRUST multi-agent LLM pipeline. Using a Bundestag plenary speech by Felix Banaszak (51 segments, 245 s) as a case study, we compare three analysis modalities: (1) emotion2vec_plus_large, an acoustic speech emotion recognition (SER) model whose continuous Arousal and Valence values are derived via post-hoc Russell Circumplex projection; (2) Gemini 2.5 Flash, an LLM analysing the full speech audio together with its transcript in an open-ended, context-aware fashion; and (3) TRUST-Pathos scores from a three-advocate LLM supervisor ensemble. Spearman rank correlations reveal that Gemini Valence correlates strongly with TRUST-Pathos (rho = +0.664, p < 0.001), whereas emotion2vec Valence does not (rho = +0.097, p = 0.499). We further demonstrate, via a systematic quality evaluation of the Berlin Database of Emotional Speech (EMO-DB) using Gemini in an open-ended annotation paradigm, that standard SER benchmark corpora suffer from acted speech, cultural bias, and category incompatibility. Our results suggest that LLM-based multimodal analysis captures semantically defined political emotion substantially better than acoustic models alone, while acoustic features remain informative for low-level Arousal estimation. Future work will extend this approach to video-based analysis incorporating facial expression and gaze.

• The paper demonstrates that LLM-based evaluations exhibit a strong correlation with TRUST-Pathos measures, unlike the negligible association observed with acoustic SER models.
• The study reveals that LLMs excel in semantic and pragmatic interpretation, effectively disambiguating rhetorical intent, irony, and nuanced emotional cues.
• Results highlight the limitations of traditional acted emotion benchmarks like EMO-DB and advocate for multimodal fusion to improve the analysis of political affect.

Multimodal Pathos Analysis in Political Speech: LLM-Based Versus Acoustic Emotion Models

Overview

This paper systematically evaluates the effectiveness of LLMs and acoustic emotion models as proxies for the Pathos dimension in political speech, particularly within the TRUST framework for computational political communication. Using as a case study a Bundestag plenary speech by Felix Banaszak—segmented into 51 utterances and analyzed across three modalities: a state-of-the-art acoustic speech emotion recognition (SER) model (emotion2vec), Gemini 2.5 Flash (a multimodal LLM), and the TRUST-Pathos scoring pipeline—the work interrogates the correspondence between acoustic and semantic-pragmatic measures of political affect. Additionally, the study assesses the limitations of standard SER benchmarks, especially the EMO-DB corpus, through LLM-based annotation.

Methodology

The core evaluation pipeline entails three components: (1) extraction of Arousal and Valence via emotion2vec with post-hoc Russell Circumplex projection; (2) segment-level Arousal, Valence, and open-ended emotion annotation by Gemini 2.5 Flash, with access to both transcript and speech audio; (3) TRUST-Pathos scoring via an ensemble of LLM advocates aggregated for consensus. Correlations among these measures were determined using Spearman rank, addressing both within- and cross-modality relationships. The analysis also incorporates a granular review of LLM-based annotation on the EMO-DB acted German speech corpus to contextualize the capabilities and biases present in both acoustic and LLM emotion models.

Empirical Results

A principal result is the strong and statistically significant correlation between LLM-based Valence and TRUST-Pathos ($\rho = +0.664$, $p < 0.001$), whereas acoustic emotion2vec-derived Valence is not significantly associated ($\rho = +0.097$, $p = 0.499$). Similarly, Gemini Arousal exhibits a moderate negative correlation with TRUST-Pathos ($\rho = -0.535$, $p < 0.001$), highlighting the multifaceted relationship between prosodic activation and rhetorical polarity in opposition discourse. In all cases, cross-modal agreement between acoustic and LLM estimates is low, underscoring the distinct dimensions captured by the two approaches.

LLM-based open-ended evaluation of EMO-DB exposed notable deficiencies: zero ability to resolve the Disgust category, systematic misclassification of Boredom as Neutral, and overall low semantic match despite high confidence—with only 30.1% aggregate accuracy. These findings reinforce concerns regarding the ecological validity of acted emotion corpora when translated to spontaneous political discourse and highlight a profound taxonomy mismatch between forced-choice classification and open-set LLM annotation.

Theoretical Implications

Critically, the paper demonstrates that acoustic SER models and LLMs target fundamentally different constructs. Acoustic models such as emotion2vec respond largely to energetic properties and spectral cues, estimating Valence as a function of prosodic coloration, often conflating high-energy negative content with positive affect due to limitations in tagging sarcasm, irony, or rhetorical indirection. LLMs, by contrast, leverage full semantic context, pragmatic structure, and world knowledge, enabling fine-grained disambiguation between literal and ironic language, as observed in cases where identical acoustic cues may signal divergent rhetorical intent.

The post-hoc projection of discrete SER class probabilities onto a continuous affective circumplex—while theoretically motivated by established valence-arousal models—remains empirically unvalidated in the context of spontaneous political speech, especially regarding language transferability and mapping errors inherited from legacy annotation schemes.

Practical Significance and Future Directions

For computational political communication, these results substantiate the necessity of multimodal, semantic-pragmatic affect analysis: LLM-based annotations serve as a more robust and functionally valid proxy for societal impact-oriented Pathos scoring than acoustic-only features. Nevertheless, low-level arousal estimation still benefits from acoustic input, motivating a complementary multimodal fusion rather than exclusive reliance on a single approach.

The framework outlined sets the stage for several future directions:

• Curating and evaluating SER models trained on high-quality, naturalistic, and cross-culturally representative datasets to replace or augment acted benchmarks such as EMO-DB.
• Rigorous comparison across multiple LLM providers to assess model-specific versus generalizable affect annotation performance.
• Extension to video-based emotion recognition, incorporating facial Action Units, gaze direction, and gesture for comprehensive multimodal Pathos inference.
• Empirical testing of operationalizations of rhetorical Pathos across diverse political systems and speech genres to ensure broad applicability of the TRUST framework.

Conclusion

LLM-based multimodal analysis presents a clear advancement over acoustic SER for the extraction of rhetorical Pathos in political speech, driven by its capacity for semantic and pragmatic comprehension. However, optimal affect modeling for computational social science and real-world political discourse monitoring will require integrated approaches that leverage both deep acoustic representation and the sophisticated contextual intelligence now attainable via state-of-the-art LLMs. This foundation has significant implications for the design and deployment of automated political communication analysis pipelines and highlights critical limitations in widely used SER benchmarks.