EmoSURA: Audio-Grounded Evaluation

• EmoSURA is a framework for evaluating emotional speech captions by decomposing texts into atomic perceptual units (APUs) for detailed verification.
• It integrates audio-grounded verification using specialized LLMs to assess the factual precision and semantic coverage of captions against raw audio.
• EmoSURA’s design mitigates limitations of text-only metrics on caption length and hallucinated details, correlating positively with human judgments.

Searching arXiv for the EmoSURA paper and closely related work on emotional speech caption evaluation. EmoSURA is an evaluation framework for detailed, long-context emotional speech captions that replaces holistic caption scoring with atomic, audio-grounded verification. Rather than assigning a single score to a full caption, it decomposes the caption into Atomic Perceptual Units (APUs), verifies each unit against the raw speech signal, and then measures both factual precision and semantic coverage relative to a reference caption. The framework is paired with SURABench, a balanced benchmark for emotional speech caption evaluation. In reported experiments, EmoSURA exhibits positive correlation with human judgments, whereas standard text-overlap metrics are negatively correlated because of their sensitivity to caption length and their lack of grounding in audio (Jing et al., 10 Mar 2026).

1. Problem formulation and the failure of holistic caption metrics

EmoSURA addresses the evaluation of emotional speech captioning, where a model produces rich, long-form natural-language descriptions of a speech clip’s emotion, prosody, speaking style, paralinguistics, demographic cues, and vocal events. The evaluated content may include valence, arousal, dominance, pitch, tempo, loudness, variation, vocal quality, likely gender, and events such as sighs, sobs, or laughs. The central problem is not caption generation itself, but the reliable assessment of whether a caption is appropriate for a given audio clip.

The framework is motivated by specific failure modes of conventional caption metrics. BLEU-4, ROUGE-L, METEOR, CIDEr, and SPIDER operate through surface matching of n-grams or syntactic units. For long emotional speech captions, they are reported to fail in four systematic ways. First, they are lexically rigid: semantically correct paraphrases receive low scores when wording diverges from the reference. Second, they are highly sensitive to length. In SURABench, reference captions have controlled length with mean approximately 459 characters and standard deviation approximately 60, while modern captioning models such as Qwen-Omni produce much longer outputs with mean approximately 684, standard deviation approximately 280, and maximum 1318. Extra accurate tokens are treated as insertion errors. Third, text-only metrics have no access to the waveform and therefore cannot penalize hallucinated attributes that contradict the audio. Fourth, they collapse many heterogeneous attributes into a single lexical similarity score and do not localize what is correct or incorrect.

These limitations are reflected in the reported correlations with human mean opinion scores. BLEU-4 has Pearson correlation −0.64, ROUGE-L −0.70, METEOR −0.58, CIDEr −0.66, and SPIDER −0.67. The negative signs are significant: better captions, as judged by humans, tend to receive lower scores when they are longer and more descriptive.

EmoSURA also targets weaknesses in LLM-as-a-judge evaluation. Holistic LLM scoring of long captions is described as vulnerable to reasoning inconsistency, truth-bias, context collapse, and unstable open-ended grading. Even when an audio-capable model is used, evaluation that remains unconstrained at the whole-caption level does not systematically enforce audio grounding. EmoSURA’s response is to reduce the task from global judgment to a sequence of local binary decisions.

2. Atomic Perceptual Units

The basic semantic unit in EmoSURA is the Atomic Perceptual Unit (APU); one subsection of the paper also uses the term Atomic Primitive Unit, but the intended concept is the same. An APU is a standalone declarative statement about a single vocal, emotional, or paralinguistic attribute that can be judged true or false with respect to the audio.

Each APU must satisfy three constraints. It must be self-contained, so that it has a truth value without relying on neighboring units. It must encode one atomic attribute or fact, rather than bundling several independent claims into a compound sentence. It must be perceptually grounded, referring to phenomena that a listener could in principle infer from the waveform.

The notation used by the framework is:

• Generated caption: $\mathcal{C}_{gen}$
• Reference caption: $\mathcal{C}_{ref}$
• APUs from the generated caption: $\mathcal{P} = \{p_1, \dots, p_N\}$
• APUs from the reference caption: $\mathcal{O} = \{o_1, \dots, o_M\}$

The evaluation objective is described conceptually as

$$S(\mathcal{C}_{gen}, \mathcal{C}_{ref}, \mathcal{A}) = f(\mathcal{P}, \mathcal{O}, \mathcal{A}),$$

where $\mathcal{A}$ denotes the audio and the function $f$ verifies candidate APUs against the waveform and measures their semantic overlap with reference APUs.

APU extraction is performed with Qwen2.5-7B-Instruct. The model receives the full caption and is prompted to decompose it into atomic statements, each phrased as a full sentence suitable for yes/no evaluation. The decomposition is applied symmetrically to generated captions and reference captions. This step is structurally important because it converts a single high-entropy long-form judgment into a set of small localized truth assessments.

3. Audio-grounded verification and scoring

After decomposition, EmoSURA performs audio-grounded verification. For each generated APU $p_i \in \mathcal{P}$, the system feeds the pair $(\mathcal{A}, p_i)$ to Qwen2-Audio-7B-Instruct and asks for a constrained binary entailment decision:

$$V(p_i \mid \mathcal{A}) = \text{ALM}(\mathcal{A}, p_i) \in \{\text{Yes},\ \text{No}\}.$$

The prompt instructs the model to answer only “Yes” or “No” depending on whether the audio supports the statement. This constrained response format is intended to reduce hallucinated judge rationales and to avoid numerical scoring instability.

The verified subset of generated APUs is

$\mathcal{C}_{ref}$0

From this, EmoSURA defines a precision-oriented score

$\mathcal{C}_{ref}$1

This term directly measures hallucination-free factuality with respect to the waveform.

Verification alone does not capture whether the candidate covers the salient content of the reference. EmoSURA therefore adds a semantic matching stage. Using Qwen2.5-7B-Instruct, the system checks, for each reference APU $\mathcal{C}_{ref}$2, whether there exists a generated APU that semantically entails or matches it. The set of matched reference APUs is denoted $\mathcal{C}_{ref}$3.

The recall-oriented term is then defined as

$\mathcal{C}_{ref}$4

This formulation is unusual in a deliberate way: verified extra details are not penalized. If the candidate contains additional accurate information beyond the reference, those APUs appear symmetrically in both numerator and denominator.

The core F1 score is

$\mathcal{C}_{ref}$5

The framework also computes a descriptive F1 $\mathcal{C}_{ref}$6 restricted to descriptive APUs. The paper notes this restriction but does not detail the exact filtering rule. The final reported metric is

$\mathcal{C}_{ref}$7

Because verification is claim-based rather than label-based, EmoSURA can evaluate a broad range of attributes, including emotion category, valence/arousal style, intensity, prosody, vocal quality, demographic cues, vocal events, and fine-grained paralinguistics. The reported format failure rate of the judge is 5.61%, referring to cases where the audio-LLM fails to produce a valid yes/no output.

4. SURABench

EmoSURA is evaluated on SURABench, a benchmark designed for standardized assessment of long, detailed emotional speech captions. The source corpus is MSP-Podcast v1.11 Test1 split, which already provides valence–arousal–dominance ratings.

SURABench is built through a three-stage curation process. The first stage is acoustic suitability: utterances shorter than 3 seconds or longer than 8 seconds are excluded. The second is label reliability: only utterances with standard deviation of both valence and arousal ratings less than or equal to 1.5 are retained. The third is distributional balance: valence and arousal are normalized to a 1–7 scale, the resulting 2D space is discretized into a 10×10 grid, and up to 15 samples are selected from each bin, prioritizing utterances with the highest consensus, i.e., lowest rating variance.

The resulting benchmark contains 1,018 utterances with relatively uniform coverage of the valence–arousal plane, explicitly avoiding overrepresentation of neutral speech. Dominance ratings are also retained and visualized in the paper.

Reference captions in SURABench are produced through a hybrid human–LLM annotation pipeline. First, ParaCLAP extracts acoustic features such as pitch, pitch variation, loudness, jitter, shimmer, and speech tempo. Second, expert annotators produce gold-standard captions for a representative subset, establishing the desired descriptive style and granularity. Third, GPT-4.1 is prompted with those few-shot examples plus the extracted acoustic features to generate captions for the full benchmark. SURABench therefore combines audio clips, long-form reference captions, affective ratings, and balanced affective coverage.

5. Human correlation, length sensitivity, and perturbation robustness

Human evaluation is based on a 5-point Likert MOS study over 320 audio–caption pairs from SURABench, stratified across valence–arousal space and speaker gender. The raters comprise 14 participants, including 6 audio experts. The evaluated captions come from four categories: Ground Truth, Sabotaged Captions, Unconstrained Qwen-Omni, and Refined Qwen-Omni (Jing et al., 10 Mar 2026).

The reported metric correlations are:

Metric	PCC	Kendall $\mathcal{C}_{ref}$8
BLEU-4	-0.6419	-0.4494
ROUGE-L	-0.7017	-0.4606
METEOR	-0.5813	-0.4541
CIDEr	-0.6640	-0.3732
SPIDER	-0.6679	-0.4481
SPICE	-0.5728	-0.3874
MACE	0.4283	0.2619
EmoSURA	0.4391	0.3277

The corresponding sample-wise $\mathcal{C}_{ref}$9 values are −0.6916 for BLEU-4, −0.6916 for ROUGE-L, −0.6559 for METEOR, −0.6175 for CIDEr, −0.7019 for SPIDER, −0.6240 for SPICE, 0.3709 for MACE, and 0.4480 for EmoSURA. All reported p-values are < 0.001. The principal empirical claim is therefore not merely that EmoSURA is positively correlated with human judgment, but that it better preserves ranking fidelity than alternative metrics, especially under long-caption conditions.

The paper also reports a perturbation study designed to test hallucination detection. Ground-truth captions are modified by changing exactly one semantic attribute per sample. The perturbation classes are acoustic feature swaps, emotion flips, and vocal event fabrication. EmoSURA detects 93.33% of acoustic feature perturbations overall, including 97.50% for gender swaps and 91.25% for pitch/tempo/volume changes. Detection rate for emotion perturbations is 82.50%. Performance drops for vocal event fabrication, where the reported detection rate is 60.00%.

These results indicate a clear asymmetry in robustness. EmoSURA is strongest when the claim concerns relatively stable low-level acoustic cues or broad affective direction, and weaker when the claim concerns fine-grained transient events such as short sighs or subtle sobbing. The paper attributes this weakness to the limitations of current audio-LLMs and to the difficulty of inferring the absence or presence of brief paralinguistic events.

6. Practical use, limitations, and broader significance

Operationally, the framework consists of four stages. First, both candidate and reference captions are decomposed into APUs with Qwen2.5-7B-Instruct. Second, each generated APU is verified against the waveform with Qwen2-Audio-7B-Instruct. Third, semantic matching is performed between generated and reference APUs. Fourth, the precision, recall-like term, F1, descriptive F1, and final EmoSURA score are computed. The pipeline is therefore modular: the decomposition model, the audio-language verifier, or the semantic matcher can in principle be replaced while preserving the scoring structure.

The computational profile follows directly from this design. Decomposition requires one LLM call per caption. Verification is the dominant cost because it requires one audio-language-model call per APU. Matching adds one or more text-LLM calls depending on implementation. The paper notes that APUs can be batched and that evaluation across APUs or samples is straightforwardly parallelizable, although explicit runtime or memory benchmarks are not reported.

The framework’s main limitation is that it still captures only part of human judgment. The reported Pearson correlation, approximately 0.44, leaves substantial unexplained variance. The binary yes/no formulation improves consistency, but it also constrains nuance. Another limitation is the weakness on complex vocal-event hallucinations. A further practical limitation is dependence on the capabilities of the underlying audio-LLM: if the verifier is poor at a given class of perceptual attributes, the metric inherits that weakness.

Within those constraints, EmoSURA is significant because it reformulates multimodal evaluation around atomic, source-grounded claims rather than whole-output similarity. The paper explicitly suggests that the same paradigm can extend beyond emotional speech captioning to general audio captioning, music captioning, video description, and image captioning. It also proposes using EmoSURA as a reward for reinforcement learning or direct preference optimization in emotional speech captioning systems. Additional future directions include moving beyond binary verification, improving robustness to vocal events, and extending SURABench to more languages, channel conditions, and spontaneous conversational settings.

In that sense, EmoSURA is best understood not only as a metric for one captioning task, but as a decompositional evaluation architecture for multimodal generation: decompose long outputs into verifiable units, ground each unit in the source modality, and aggregate claim-level precision and coverage into a score that is less vulnerable to verbosity bias and semantic drift.