CASE: Acoustic-Semantic Emotion Conflict

• CASE is a phenomenon where acoustic prosody and semantic meaning diverge, creating mismatches in perceived emotion.
• Benchmark datasets for CASE use controlled synthetic speech and expert validation to test emotion recognition under conflicting cues.
• Specialized models like FAS, CARE, and MCAN employ dual-stream architectures and dynamic fusion to reconcile conflicting acoustic and semantic signals.

Conflict in Acoustic-Semantic Emotion (CASE) denotes instances in speech where the affective information conveyed by prosody (acoustic features) diverges from that indicated by the semantic meaning of uttered words. This phenomenon challenges existing speech emotion recognition (SER) and audio LLMs (ALMs), which often presume congruence between “how something is said” and “what is said.” Formally, CASE occurs when the predicted labels of separate acoustic and semantic classifiers disagree on the underlying emotion; i.e., $C(a,s) = 1$ when $\hat y_{\rm a} \neq \hat y_{\rm s}$, where $\hat y_{\rm a}$ is the acoustic and $\hat y_{\rm s}$ the semantic prediction (Huang et al., 8 Jan 2026). CASE is critical for robust SER, as unmodeled cue conflict is prevalent in real-world tasks such as sarcasm detection, emotionally incongruent dialogue, and affective human–machine interaction.

1. Formalism and Problem Definition

CASE is mathematically characterized by a conflict indicator:

$$C(a,s)=\mathbf{1}[\hat y_{\rm a} \neq \hat y_{\rm s}]$$

where $a \in \mathbb{R}^{d_a}$ and $s \in \mathbb{R}^{d_s}$ are the acoustic and semantic representation vectors, respectively, and

$$\hat y_{\rm a} = \arg\max_{y} p_{\rm a}(y|a), \quad \hat y_{\rm s} = \arg\max_{y} p_{\rm s}(y|s)$$

for $y\in\{1,\dots,K\}$ emotion classes (Huang et al., 8 Jan 2026).

In incongruent speech, the emotional content signaled by prosody is intentionally designed to contradict the sentiment that would be inferred from transcript alone. This definition underpins the construction of dedicated CASE benchmarks, ensuring that all included samples satisfy $C(a,s)=1$.

2. Benchmark Datasets and Data Construction

Several recent benchmarks explicitly instantiate CASE:

• CASE Benchmark (Huang et al., 8 Jan 2026):
  • 378 utterances, 7 emotions, all samples are prosody–semantics conflict cases.
  • Generated by combining expert-crafted texts (anchored to a semantic emotion), synthetic vocal rendering with controlled prosodic emotion (via TTS), and rigorous expert panel verification (Fleiss' $\kappa > 0.85$ across 12 raters).
  • Covers English, Mandarin, and dialectal variations, leveraging 21 synthetic speaker timbres.
• EMIS (Emotionally Incongruent Synthetic Speech) (Corrêa et al., 29 Oct 2025):
  • 1248 samples generated by re-synthesizing 104 “emotion-rich” English utterances with four reference prosodic emotions (angry, happy, sad, neutral) using three SOTA TTS engines.
  • Explicit labeling of both proxy (semantic) and target (acoustic) emotion for every instance.
• LISTEN Benchmark (Chen et al., 12 Oct 2025):
  • Four conditions: Neutral-Text, Emotion-Matched, Emotion-Mismatched (explicit CASE), and Paralinguistic.
  • Corpus derived from MUSTARD++ [Ray et al. 2022] for sarcasm (systematic text–audio emotion mismatch); annotations for explicit (transcript) and implicit (prosody) emotion labels with manual verification of disagreement.
  • 10 possible emotion categories; in mismatched condition, the text-only label is lexical, and audio(-plus)-text is acoustic.
• CMU-MOSI/MOSEI Conflict Splits:
  • Utilized for evaluating multi-modal sentiment analysis models with explicit alignment/conflict decompositions (Gao et al., 13 Feb 2025).

Table: Key properties of major CASE-oriented datasets.

Dataset	Emotions	Modality	Conflict Construction
CASE (Huang et al., 8 Jan 2026)	7	Synthetic speech	Scripted text, conflicting TTS
EMIS (Corrêa et al., 29 Oct 2025)	4	Synthetic speech	TTS prosody vs. text semantics
LISTEN (Chen et al., 12 Oct 2025)	up to 10	Natural+synthetic	Sarcasm, paralinguistics

These resources underpin rigorous evaluation and drive the development of robust acoustic-semantic fusion techniques.

3. Limitations of Current Speech Emotion Models under CASE

Empirical findings indicate a pronounced semantic (lexical) dominance in state-of-the-art LALMs, SLMs, and conventional SER approaches:

• LISTEN Benchmark: Text-only models achieve ≈100% accuracy in Neutral-Text, while audio-only and Text+Audio performance under Emotion-Mismatched (conflict) shrinks to 37–43%, near marginal baselines, with models defaulting to few categories such as “frustration”/“ridicule.” In paralinguistic (non-lexical) scenarios, accuracies are only 15–22%, barely above chance for 10-class tasks (Chen et al., 12 Oct 2025).
• EMIS: SLMs, when prompted to ignore word meaning, still predict semantic labels >80% of the time; acoustic accuracy collapses to ≈25% (random) on incongruent trials. Modality-specific SER baselines reach 50% accuracy for prosody but fail for semantic inference (Corrêa et al., 29 Oct 2025).
• CASE Benchmark: Standard baselines (e.g., Whisper) attain 47.26%/44.97% ACC/F1, while the FAS model achieves 59.38%/55.08% on conflict-only samples (Huang et al., 8 Jan 2026).

These results are consistent: modern multimodal or audio LLMs predominantly “transcribe” rather than “listen,” leveraging transcript content as the primary anchor, and exhibiting minimal robustness to emotion label divergence across modalities.

4. Specialized Architectures and Modeling Strategies

Explicit modeling of CASE requires architectural mechanisms to disentangle and reconcile prosodic and semantic cues. Approaches documented in the literature include:

• Fusion Acoustic–Semantic (FAS) Framework (Huang et al., 8 Jan 2026):
  • Parallel extraction of acoustic (MingTok-Audio) and semantic (Whisper-large) embeddings.
  • Top-$k$ token distillation with nonparametric saliency, learnable query-based cross-attention for stream fusion, and final multi-layer perceptron emotion classification.
  • Attention queries learn to upweight the stream most informative for a given example; FAS outperforms prior art by 12.12% ACC on CASE.
• CARE (Content and Acoustic Representations of Emotions) (Dutta et al., 2024):
  • Dual-arm architecture: an “acoustic arm” trained on frame-wise low-level features, and a “semantic arm” distilled from utterance-level text representations.
  • Late fusion via learned nonnegative convex combination $\{\alpha_l\}$ over all arm outputs, allowing dynamic weighting: for ambiguous or conflicting emotion cues, fusion naturally shifts toward the dominant signal for the specific downstream task context.
  • Results: CARE surpasses all tested LLM-based and self-supervised models on weighted F1 across eight benchmarks (avg. 66.02%).
• Multi-Level Conflict-Aware Network (MCAN) (Gao et al., 13 Feb 2025):
  • SVD-based decomposition of fused bimodal representations into aligned and conflict subspaces at both micro (text–audio/visual) and macro (bimodal) levels.
  • Dedicated conflict modeling branch imposes orthogonality and output-level discrepancy constraints, ensuring that conflictive constituents remain mutually inconsistent and thereby detectable.
  • SOTA results on CMU-MOSI (Acc2 84.5%, Acc7 43.1%) and CMU-MOSEI (Acc2 85.8%, Acc7 51.6%).

A common theme is modularization: separate or explicitly disentangled streams for prosodic and lexical information, followed by adaptive or task-driven fusion, are critical for overcoming semantic bias in the presence of CASE.

5. Evaluation Metrics and Diagnostic Analysis

Rigorous evaluation of CASE models requires metrics that expose cue dominance and conflict resolution. Standard metrics include:

• Overall accuracy (percent correctly classified, single-label, multi-class). In e.g., LISTEN, micro-F1 equals accuracy (Chen et al., 12 Oct 2025).
• Prediction-marginal baseline:

$\mathbb{E}[\mathrm{Acc}] = \sum_{i=1}^K p_i q_i$

with $p_i$ (prediction distribution), $q_i$ (ground truth frequency), quantifying accuracy due to class imbalance.

• Cue Dominance Index:

$$\Delta_{\text{text-audio}} = \mathrm{Acc}_{\text{text-only}} - \mathrm{Acc}_{\text{audio-only}}$$

capturing the extent of semantic preference.

• Association statistics (EMIS):

Cramér’s $V$ and $\chi^2$ tests on predicted/true (target vs. proxy) labels; high $V$ for proxy (semantic), low for target (acoustic) labels.

• Confusion matrices and qualitative inspection of attention/fusion weights to ascertain which modality is driving decisions in conflict.

The lead benchmarks include “neutral default bias” analyses, where models revert to “neutral” in ambiguous or non-semantic conditions, and pairwise comparison between matched and mismatched conditions to quantify resilience to CASE.

6. Implications and Future Directions

CASE-centric evaluation exposes a critical failure mode in contemporary SER and multimodal LLMs: semantic anchoring leads to insensitivity toward prosodic emotion, causing dramatic performance drops under cue conflict. Effective mitigation will require:

• Inclusion of conflict-rich samples in pre-training or fine-tuning to prevent shortcut learning.
• Joint or cross-modal alignment objectives (e.g., contrastive losses) to balance representation strengths.
• Architectural innovations for disentangled pathway and dynamic, query- or attention-driven fusion (as in FAS/MCAN).
• Expansion of evaluation to multimodal, paralinguistic, and sarcasm-rich domains.

A plausible implication is that robust CASE resolution is foundational for next-generation human–machine interaction, paralinguistic reasoning, and affect-sensitive AI systems (Huang et al., 8 Jan 2026, Chen et al., 12 Oct 2025, Corrêa et al., 29 Oct 2025, Dutta et al., 2024, Gao et al., 13 Feb 2025).

7. Relation to Broader Multimodal Sentiment Analysis

CASE is one instance of a larger class of multimodal conflicts—inference settings where acoustic, semantic, and possibly visual cues diverge. Methodologies such as MCAN’s SVD-based alignment–conflict disentanglement (Gao et al., 13 Feb 2025) offer a principled framework for generalized multimodal conflict modeling. The expansion of CASE-style benchmarks and architectures to trinary (text–audio–visual) scenarios is a likely next step, necessitating further modularity, dynamic weighting, and fine-grained labeling protocols.

CASE research demonstrates that optimal multimodal emotion recognition demands explicit separation, reconciliation, and sometimes arbitration of competing affective signals—a departure from naive fusion and toward conflict-aware intelligent systems.