SURABench: Balanced Emotional Speech Captions Dataset

• SURABench is a balanced and stratified benchmark dataset designed for evaluating long, emotionally detailed speech captions.
• It employs a three-stage curation process—duration filtering, consensus filtering, and stratified grid sampling—to ensure acoustic suitability and label reliability.
• The dataset enables reference-based evaluation and controlled perturbation testing for hallucination detection in the EmoSURA evaluation framework.

SURABench is a balanced and stratified benchmark dataset for evaluating emotional speech captions, introduced as part of the EmoSURA framework in “EmoSURA: Towards Accurate Evaluation of Detailed and Long-Context Emotional Speech Captions” (Jing et al., 10 Mar 2026). It is presented as a standardized resource for reproducible reference-based evaluation, motivated by the claim that existing resources were insufficient for long, detailed, emotionally grounded speech captions. Within that framework, SURABench functions as the benchmark substrate: it provides the audio, the associated long-form captions, the emotional stratification, and the controlled perturbation setting used to validate caption-evaluation methods (Jing et al., 10 Mar 2026).

1. Origin, purpose, and problem setting

SURABench was created in response to three problems identified in emotional speech caption evaluation. First, long-form captions are difficult to evaluate with traditional metrics: N-gram metrics such as BLEU, ROUGE, METEOR, and CIDEr are described as poorly reflecting semantic correctness for rich emotional descriptions and as especially sensitive to caption length and lexical variation. Second, LLM judges are described as unreliable on long, information-dense captions because they can collapse context, lose details, or reason inconsistently. Third, there was a lack of a standardized benchmark. The benchmark is therefore intended to provide a controlled, balanced, and stratified test set suitable for reference-based evaluation, with sufficient emotional diversity and quality to support evaluation of detailed captions and hallucination detection (Jing et al., 10 Mar 2026).

In this sense, SURABench is not merely a held-out dataset. It is the benchmark foundation on which EmoSURA is evaluated and validated. The paper distinguishes the roles clearly: SURABench is the benchmark dataset, whereas EmoSURA is the reference-based evaluation framework applied to that dataset (Jing et al., 10 Mar 2026).

2. Corpus derivation and three-stage curation

SURABench is derived from the MSP-Podcast v1.11 Test1 split. Its construction follows a three-stage curation process intended to ensure acoustic suitability, label reliability, and distributional balance (Jing et al., 10 Mar 2026).

The first stage is duration filtering. Utterances were excluded if they were less than 3 seconds or more than 8 seconds. The stated rationale is that short segments may not convey complete emotional semantics, whereas long segments may hinder stable caption generation (Jing et al., 10 Mar 2026).

The second stage is consensus filtering. Only utterances whose valence and arousal annotations had sufficiently high agreement were retained. The stated criterion is that the standard deviations of both valence and arousal ratings must be ≤ 1.5. This stage is explicitly intended to remove ambiguous samples with low inter-annotator agreement (Jing et al., 10 Mar 2026).

The third stage is stratified grid sampling. To address class imbalance, the authors discretized the Valence-Arousal space into a 10 × 10 grid on a 1–7 scale. From each bin, they selected up to 15 samples, prioritizing samples with the highest annotation consensus. The stated goals are to improve uniform coverage across emotional space, ensure representation across all four semantic quadrants, and reduce over-representation of neutral speech (Jing et al., 10 Mar 2026).

The final benchmark contains 1,018 utterances. The paper reports qualitative distributional properties rather than a full class breakdown: broad emotional coverage, uniformity across the valence-arousal space, and reduced dominance of neutral samples. In the benchmark figure, point colors represent dominance, and marginal histograms show the dataset’s uniformity (Jing et al., 10 Mar 2026).

3. Caption creation pipeline and benchmark contents

SURABench includes high-fidelity captions produced by a hybrid annotation pipeline that combines acoustic feature extraction, human guidance, and LLM generation (Jing et al., 10 Mar 2026).

The first step is the extraction of paralinguistic features. Following ParaCLAP, the authors extracted objective acoustic evidence including pitch, pitch variation, loudness, jitter, shimmer, and speech tempo. These features are used as evidence to support detailed captioning (Jing et al., 10 Mar 2026).

The second step is the production of “gold-standard” descriptions by expert annotators for a representative subset of the data. These human-written captions serve two purposes stated in the paper: they establish the desired level of descriptive granularity and define a preferred syntactic/style template (Jing et al., 10 Mar 2026).

The third step is LLM-based caption generation for the full benchmark using GPT-4.1. GPT-4.1 was prompted with the gold-standard descriptions as few-shot examples, which means the benchmark captions were generated in a style aligned with the human-written reference subset (Jing et al., 10 Mar 2026).

As a result, SURABench contains more than audio clips. It contains spontaneous speech samples from MSP-Podcast, emotion-labeled utterances, valence/arousal annotations, dominance as visualized in the benchmark distribution plot, long, detailed emotional speech captions, and acoustic/paralinguistic descriptors tied to the speech signal. The captions are intended to describe vocal characteristics, emotional state, prosodic style, and fine-grained paralinguistic details (Jing et al., 10 Mar 2026).

This construction suggests that SURABench was designed not only for caption-quality scoring in the narrow lexical sense, but also for assessment of semantic adequacy and factual grounding against the speech signal.

4. Function within the EmoSURA evaluation framework

Within EmoSURA, SURABench is the standardized dataset against which candidate captions are evaluated. The paper states that SURABench is used in at least three ways: reference-based caption evaluation, a human evaluation study, and a controlled perturbation set for hallucination testing (Jing et al., 10 Mar 2026).

EmoSURA itself shifts from holistic scoring to atomic verification by decomposing captions into Atomic Perceptual Units and validating each unit against the raw speech signal. For a generated atomic perceptual unit $p_i$, verification is defined as

$$V(p_i \mid \mathcal{A}) = \text{ALM}(\mathcal{A}, p_i) \in \{\text{Yes}, \text{No}\}$$

and the set of supported units is

$$\mathcal{P}_{true} = \{ p_i \in \mathcal{P} \mid V(p_i \mid \mathcal{A}) = \text{Yes} \}.$$

The precision-oriented score is

$s_p = \frac{|\mathcal{P}_{true}|}{|\mathcal{P}|}.$

For semantic matching and recall, letting $\mathcal{O}$ denote reference APUs and $\mathcal{Q} \subseteq \mathcal{O}$ the matched reference units, the recall-oriented score is

$$s_r = \frac{|\mathcal{Q}| + |\mathcal{P}_{true} \setminus \mathcal{Q}|}{|\mathcal{O}| + |\mathcal{P}_{true} \setminus \mathcal{Q}|}.$$

The paper emphasizes that this formulation rewards matched reference content while also giving credit for additional verified content in the generated caption. The combined F1 score is

$s_f = 2 \cdot \frac{s_p \cdot s_r}{s_p + s_r},$

with a descriptive F1 score $s_f'$ computed similarly but restricted to descriptive APUs only. The final EmoSURA score is

$$\mathcal{F} = \frac{1}{2} \left( s_f + s_f' \right).$$

These equations define the scoring logic used when EmoSURA is applied to SURABench (Jing et al., 10 Mar 2026).

The operational relationship is therefore direct: SURABench provides the references and the stratified audio substrate, while EmoSURA provides APU decomposition, audio-grounded verification, reference matching, and final scoring.

5. Subjective evaluation and perturbation-based robustness testing

SURABench is also the basis for the paper’s subjective evaluation. The MOS study used 320 audio-caption pairs stratified across the Valence-Arousal circumplex and speaker gender. It involved 14 participants, comprising 6 males and 8 females, including 6 audio experts, and used a 5-point Likert scale (Jing et al., 10 Mar 2026).

The captions in that study were grouped into four categories: Ground Truth (GT), Sabotaged Captions, Unconstrained Qwen-Omni, and Refined Qwen-Omni. The “Sabotaged Captions” category consisted of captions in which factual details were intentionally corrupted. The resulting human judgments were used to assess automated metrics, including EmoSURA (Jing et al., 10 Mar 2026).

The benchmark also supports a dedicated controlled perturbation evaluation set. In that setting, an acoustic hallucination is defined as any caption attribute contradicting the annotated acoustic evidence, including speaker gender, emotional state, vocal events, and low-level acoustic features. The perturbation categories are Emotion Flip, Vocal Event Fabrication, and Acoustic Feature Swap. The perturbation process changes one semantic attribute per sample while maintaining consistency elsewhere; examples given in the paper include swapping male ↔ female and correspondingly changing pitch and vocal texture descriptions (Jing et al., 10 Mar 2026).

This perturbation protocol is significant because it operationalizes hallucination detection as contradiction with acoustic evidence rather than mere deviation from a reference wording. A plausible implication is that SURABench supports evaluation at two distinct levels: agreement with human preference and sensitivity to injected factual inconsistency.

6. Empirical findings and benchmark-specific effects

SURABench is central to the empirical validation of EmoSURA. On the benchmark, traditional rule-based metrics are reported to have negative correlations with human ratings, whereas MACE is positive but moderate, and EmoSURA performs best among the listed metrics. The reported correlations are as follows (Jing et al., 10 Mar 2026):

Metric	PCC	Kendall $$V(p_i \mid \mathcal{A}) = \text{ALM}(\mathcal{A}, p_i) \in \{\text{Yes}, \text{No}\}$$0
BLEU-4	-0.6419	-0.4494
ROUGE-L	-0.7017	-0.4606
METEOR	-0.5813	-0.4541
CIDEr	-0.6640	-0.3732
SPIDER	-0.6679	-0.4481
SPICE	-0.5728	-0.3874
MACE	0.4283	0.2619
EmoSURA	0.4391	0.3277

The paper also reports sample-wise $$V(p_i \mid \mathcal{A}) = \text{ALM}(\mathcal{A}, p_i) \in \{\text{Yes}, \text{No}\}$$1, with EmoSURA at 0.4480, and states that all p-values are < 0.001 (Jing et al., 10 Mar 2026).

A benchmark-specific effect highlighted in the paper is length sensitivity. SURABench reveals a strong distributional mismatch between reference captions, averaging 459 characters with $$V(p_i \mid \mathcal{A}) = \text{ALM}(\mathcal{A}, p_i) \in \{\text{Yes}, \text{No}\}$$2, and model-generated captions, averaging 684 characters with $$V(p_i \mid \mathcal{A}) = \text{ALM}(\mathcal{A}, p_i) \in \{\text{Yes}, \text{No}\}$$3 and extremes up to 1,318 characters. The paper argues that this matters because the benchmark contains long, detailed captions and thereby exposes the failure mode of standard metrics on verbose outputs (Jing et al., 10 Mar 2026).

On the SURABench-based perturbation set, EmoSURA detects hallucinations with category-dependent rates. For Acoustic Features, the reported rate is 93.33%; within that category, Gender reaches 97.50%, while Pitch/Tempo/Volume/etc. reaches 91.25%. For Emotion, the rate is 82.50%, and for Vocal Event, 60.00% (Jing et al., 10 Mar 2026). The paper interprets this as showing high sensitivity to low-level acoustic factual errors and emotional flips, with lower reliability on fabricated vocal events.

Taken together, these results position SURABench as a benchmark that stresses long-context description, semantic fidelity, and hallucination robustness rather than lexical overlap alone.

7. Scope, interpretation, and disambiguation

Within the cited literature, SURABench denotes the emotional speech caption benchmark introduced alongside EmoSURA (Jing et al., 10 Mar 2026). It should not be conflated with several unrelated benchmark systems that appear in adjacent domains. SQuASH is a surrogate-assisted benchmark for QAS in quantum architecture search (Martyniuk et al., 7 Jun 2025). Bencher is a benchmarking framework for black-box optimization built around isolated virtual environments and a gRPC interface (Papenmeier et al., 27 May 2025). scarab is software for creating scalable benchmarks from quantum circuits and algorithms using process fidelity estimation (Siekierski et al., 3 Nov 2025). SolBench is a benchmark for functional correctness in Solidity code completion and repair (Chen et al., 3 Mar 2025).

That distinction matters because SURABench, in the sense established by EmoSURA, is not a generic benchmarking framework, a quantum benchmarking suite, or a code-generation benchmark. It is specifically a balanced and stratified benchmark dataset for emotional speech captions, derived from MSP-Podcast, annotated through a hybrid human/LLM pipeline, and used for both reference-based evaluation and perturbation-driven hallucination testing (Jing et al., 10 Mar 2026).

A common misconception would be to read SURABench primarily as an evaluation metric. The paper instead assigns that role to EmoSURA. SURABench is the testing ground; EmoSURA is the method evaluated on that ground.