SpeechFeedback: Speech Preference Data

• SpeechFeedback is a synthetic speech preference dataset designed to evaluate S2S LLMs with detailed semantic and acoustic annotations.
• It provides pairwise judgments with rationales, capturing aspects such as truthfulness, instruction following, and diverse speech styles.
• Utilized in SageLM’s two-stage training, it achieves high human agreement and accuracy, outperforming cascaded ASR and text-based evaluations.

SpeechFeedback is a synthetic preference dataset introduced for evaluating Speech-to-Speech (S2S) LLMs in settings where both semantic content and acoustic realization matter. It was proposed together with SageLM, an end-to-end, multi-aspect, and explainable speech judge, to address the scarcity of large-scale, diverse, and multi-aspect annotated speech preference data. In SpeechFeedback, each instance contains a speech query, two contrasting speech responses, detailed preference annotations across semantic and acoustic dimensions, and rationales explaining the preference decision. The dataset is designed for training and analyzing systems that judge spoken responses directly rather than through a cascaded automatic speech recognition pipeline (Ge et al., 28 Aug 2025).

1. Definition and scope

SpeechFeedback was introduced to support comprehensive evaluation of S2S LLMs, a setting in which textual adequacy alone is insufficient because prosody, speaking style, and acoustic instruction-following are also part of the model output. In the reported formulation, the dataset contains 324,774 instances. Each instance includes a query, two candidate speech responses, pairwise preference annotations, and accompanying rationales (Ge et al., 28 Aug 2025).

The central design choice is multi-aspect supervision. The semantic side covers truthfulness, honesty, helpfulness, and instruction following. The acoustic side covers speech instruction following. This makes SpeechFeedback materially different from scalar reward datasets: the target is not a single undifferentiated preference label, but a structured judgement that can be decomposed by aspect and accompanied by an explanation (Ge et al., 28 Aug 2025).

A recurring source of ambiguity in the literature is the word “feedback.” In spoken dialogue research, feedback can denote short listener responses such as backchannels; in language learning, it can denote corrective pronunciation guidance; in S2S evaluation, SpeechFeedback denotes a preference dataset for judging generated spoken responses. These usages are related by their concern with spoken interaction, but they refer to different objects and tasks. Short feedback responses are modeled in spoken dialogue as contextually appropriate listener behavior (Qian et al., 2024), while pronunciation systems such as Flowchase and PTeacher provide personalized corrective feedback to learners (Tits et al., 2023, Bu et al., 2021). SpeechFeedback, by contrast, is an evaluation resource for speech judges (Ge et al., 28 Aug 2025).

2. Internal structure and annotation scheme

The dataset is organized around pairwise preference comparison. Given a query and two speech responses, the target output is an evaluation result comprising aspect-wise comparison labels and aspect-wise rationales. In the described formulation, each aspect label belongs to the set $\{\text{win}, \text{lose}, \text{tie}\}$, and each label is paired with an explanation. This structure is used to train models that not only choose between two spoken outputs but also explain why one is preferred (Ge et al., 28 Aug 2025).

SpeechFeedback explicitly couples judgement with explanation. The rationales are not ancillary metadata; they are part of the supervision signal. The corresponding training setup in SageLM uses supervised fine-tuning on “label + rationale” sequences rather than on labels alone. The intended effect is twofold: improve alignment with human evaluation outcomes and increase explainability of the resulting judge (Ge et al., 28 Aug 2025).

This rationale-centric design distinguishes SpeechFeedback from datasets aimed solely at response ranking or reward estimation. A plausible implication is that the dataset is intended not just to optimize comparative accuracy, but also to regularize the internal decision process of the judge toward human-readable criteria. That interpretation is consistent with the reported analysis that rationale-based supervision improves agreement and reduces contradictions between explanations and verdicts (Ge et al., 28 Aug 2025).

3. Dataset construction

SpeechFeedback is constructed in two main stages, one semantic and one acoustic. For the semantic dimension, the source is UltraFeedback. Textual responses rated by GPT-4 with scores and rationales are synthesized into speech using seven TTS models. The pipeline then filters the data, including removal of math/code/multilingual prompts and low-quality TTS outputs, and transforms scalar-style evaluations into pairwise win–lose preferences with rewritten comparative rationales (Ge et al., 28 Aug 2025).

For the acoustic dimension, the construction process targets explicit and implicit speaking-style control. The reported acoustic categories include emotion, gender, and cartoon voice, generated through synthetic prompts, TTS manipulations, and explicit evaluation rationales. This stage is what allows SpeechFeedback to supervise criteria that are invisible to text-only judges and largely inaccessible to cascaded ASR-plus-text evaluation (Ge et al., 28 Aug 2025).

The dataset is therefore synthetic in a specific sense: not merely text-to-speech conversion of pre-existing preference data, but a composite resource in which semantic preference supervision is ported into the speech domain and acoustic preference supervision is created to cover style-sensitive phenomena. This construction choice addresses the stated problem that speech preference data are scarce. It also explains why the dataset is useful for end-to-end speech judgement rather than only for response selection in text space (Ge et al., 28 Aug 2025).

4. Role in SageLM and the two-stage training paradigm

SpeechFeedback underpins the training of SageLM, an end-to-end, multi-aspect, and explainable judge for S2S LLM evaluation. SageLM directly processes speech responses and jointly assesses semantic and acoustic dimensions. The reported evaluation interface takes a semantic query and two speech responses and produces aspect-wise comparisons and rationales. The query is text, while the responses are audio, which the authors describe as a way to ensure normalized, fair evaluation across diverse S2S systems (Ge et al., 28 Aug 2025).

Training proceeds through a two-stage training paradigm. In Stage 1, the model learns semantic preference learning from the semantic portion of SpeechFeedback, including truthfulness, honesty, helpfulness, and instruction-following. In Stage 2, it is fine-tuned on acoustic preference learning, covering emotion, speaking style, and other speech instruction-following phenomena (Ge et al., 28 Aug 2025).

The paper’s analysis presents this decomposition as important rather than incidental. Stage 1 improves semantic scores, but acoustic performance degrades; Stage 2 adds acoustic supervision, restores high semantic performance, and enables strong acoustic judgement. This suggests that semantic and acoustic preferences are not trivially co-learned, and that staged supervision helps prevent one capability from dominating the other (Ge et al., 28 Aug 2025).

Rationale-based supervision is a further core component. The authors report that rationale-augmented supervised fine-tuning outperforms both label-only supervised fine-tuning and reinforcement-learning approaches such as GRPO in agreement and judgement consistency. They also note that rule-based reinforcement learning is more vulnerable to reward hacking and inconsistent rationales. In that sense, SpeechFeedback is not only a data source but also a vehicle for a particular training philosophy: comparative speech judgement should be explainable by construction (Ge et al., 28 Aug 2025).

5. Empirical performance

The principal empirical claim associated with SpeechFeedback is that, when used to train SageLM, it supports strong alignment with human evaluators on both semantic and acoustic judgement tasks. On human-annotated semantic test sets, SageLM (Qwen2.5-omni-7B-SFT) achieves 82.79% agreement with human evaluators, outperforming cascaded ASR+GPT-4o by 7.42% and S2T LLM baselines by at least 26.20% (Ge et al., 28 Aug 2025).

On human-annotated acoustic test sets, the same model reaches up to 99.66% agreement on explicit emotion/gender/voice control tasks and 91.36% on mixed acoustic aspects, with 95–99% accuracy across categories. By contrast, reported S2T baselines remain around 30–50% in agreement and accuracy, while cascaded baselines are not applicable because they disregard acoustic cues (Ge et al., 28 Aug 2025).

The paper also reports strong generalization and explanation-related results. On an unseen AlpacaEval test set, SageLM attains 87.97% agreement and 81.01% accuracy, surpassing Whisper+GPT-4o by 18.67% agreement and 27.21% accuracy. Human assessment finds 90.89% rationale consistency with judgements. Position consistency is reported as 85% on semantic and 96% on acoustic tests (Ge et al., 28 Aug 2025).

Setting	Reported result	Context
Semantic agreement	82.79%	SageLM vs. human evaluators
Acoustic agreement	up to 99.66%	Explicit emotion/gender/voice control
OOD agreement	87.97%	Unseen AlpacaEval test set

These results are presented as evidence that speech judgement benefits from direct access to both the acoustic signal and rationale supervision. A plausible implication is that SpeechFeedback functions as both a coverage expansion mechanism—because it supplies missing speech preference data—and a supervision-shaping mechanism—because it encodes the expected structure of judgement itself (Ge et al., 28 Aug 2025).

6. Relation to broader research on spoken feedback

SpeechFeedback occupies a specific niche within a broader research landscape in which “feedback” has multiple technical meanings. In dialogue systems, supporting backchannels are treated as socially meaningful listener responses. Earlier work on backchannel prediction used acoustic features such as power, pitch, and fundamental frequency variation, with LSTM-based causal prediction reaching $F_1 = 0.390$ when word2vec linguistic context was added (Ruede et al., 2017). More recent work on spoken dialogue embeds contexts and feedback responses in a shared contrastive space and shows that the resulting ranking model can outperform humans on feedback-response ranking tasks (Qian et al., 2024). Complementary work on perceived prosodic similarity shows that self-supervised speech embeddings and spectral representations align with human judgements of feedback prosody better than extracted pitch features, and that contrastive learning can further align them to perception (Qian et al., 19 May 2025).

In pronunciation training and CAPT, “feedback” usually denotes corrective guidance. Flowchase provides personalized and instant feedback to English learners through a mobile application that performs joint forced-alignment and phonetic recognition and analyzes both segmental and suprasegmental pronunciation aspects (Tits et al., 2023). PTeacher provides personalized exaggerated audio-visual corrective feedback and adapts exaggeration level to learner proficiency using accumulated diagnosis and three explicit metrics: Distinguishability, Understandability, and Perceptibility (Bu et al., 2021). Related work on exaggerated visual-speech synthesis combines Tacotron-based emphatic speech generation with ADC Viseme Blending—Amplitude, Duration, and Color Contrast—to improve pronunciation identification and improvement (Bu et al., 2020).

These neighboring lines of work clarify what SpeechFeedback is not. It is not a backchannel corpus, not a pronunciation tutoring system, and not a generic collection of spoken comments. It is a structured preference dataset for training judges of spoken responses in S2S systems (Ge et al., 28 Aug 2025).

7. Significance, limitations, and interpretive issues

The principal significance of SpeechFeedback lies in its attempt to make speech evaluation simultaneously end-to-end, multi-aspect, and explainable. The dataset addresses a concrete bottleneck identified in S2S evaluation: existing reward systems often rely on scalar scores and fail to capture diverse semantic and acoustic dimensions. By supplying pairwise labels plus rationales across both kinds of dimensions, SpeechFeedback enables training regimes that treat explanation as part of the target rather than as a post hoc add-on (Ge et al., 28 Aug 2025).

At the same time, the dataset is explicitly synthetic. Its semantic component is derived from text preference data synthesized into speech, and its acoustic component is generated through synthetic prompts and TTS manipulations. This does not invalidate the dataset, but it does define its epistemic status: the resource is engineered to fill a data gap, not presented as a direct corpus of naturally occurring human comparative speech judgements (Ge et al., 28 Aug 2025).

A common misconception would be to treat SpeechFeedback as a universal account of speech feedback. The broader literature shows that speech feedback can refer to listener acknowledgements in dialogue (Ruede et al., 2017), prosodically differentiated conversational responses (Qian et al., 19 May 2025), or instructional feedback in pronunciation training (Tits et al., 2023). SpeechFeedback is narrower and more technical: it is a speech preference dataset designed to support explainable evaluation of S2S LLMs (Ge et al., 28 Aug 2025).

A plausible longer-term implication is that datasets of this kind could become infrastructure for model selection, reward modeling, and comparative auditing in spoken AI systems. That interpretation is consistent with the reported gains of SageLM over cascaded and S2T baselines, but the present contribution is the dataset-and-judge framework itself rather than a general theory of spoken interaction (Ge et al., 28 Aug 2025).