SageLM: Explainable Speech Evaluation Model

• SageLM is an end-to-end, multi-aspect speech evaluation model that assesses both semantic content and acoustic delivery directly from speech.
• It employs a two-stage, rationale-based training paradigm on a synthetic preference dataset to deliver detailed pairwise judgments.
• It outperforms cascaded ASR-based methods with an 82.79% human agreement rate and robust performance across semantic and acoustic dimensions.

Searching arXiv for papers on “SageLM” and closely related SAGE usages to ground the article. SageLM is an end-to-end, multi-aspect, and explainable speech LLM for evaluating speech-to-speech LLMs. It is designed to judge both semantic and acoustic dimensions directly from speech, rather than through an automatic-speech-recognition-to-text cascade, and it is trained with rationale-based supervision on the synthetic preference dataset SpeechFeedback through a two-stage post-training paradigm. In the reported evaluation, SageLM achieves an 82.79\% agreement rate with human evaluators, outperforming cascaded and SLM-based baselines by at least 7.42\% and 26.20\%, respectively (Ge et al., 28 Aug 2025).

1. Problem setting and motivation

SageLM addresses a central difficulty in speech-to-speech evaluation: spoken dialogue must be judged on both what is said and how it is said. The paper frames this as the joint assessment of semantic quality—helpfulness, honesty, truthfulness, and instruction following—and acoustic quality, especially whether speech follows requested or implied vocal style. The same content may be delivered with different emotion, tone, prosody, speaking style, gendered voice, or character voice, so a speech judge that only inspects transcribed text is structurally incomplete (Ge et al., 28 Aug 2025).

The work argues that the standard cascaded pipeline—first transcribe speech with ASR, then evaluate the transcript with a text LLM—is insufficient for two stated reasons. First, it incurs error propagation, because ASR mistakes distort the input seen by the judge. Second, it causes loss of acoustic information, since the transcript erases tone, emotion, voice identity, speaking rate, and prosody. The paper further states that human evaluation, while a gold standard, is expensive, slow, not scalable for model iteration, and usually not explainable in a structured way. It also treats semantic-only SLM-based judges as inadequate because they are not trained for pairwise judgment, lack speech preference supervision, may fail to align final labels with coherent explanations, and do not reliably cover the acoustic axis (Ge et al., 28 Aug 2025).

2. Formal task definition and judgment protocol

The model is formulated as a pairwise judge over a textual query and two speech responses. Given a textual instruction or query $Q$ and two speech responses $R_1, R_2$, SageLM outputs an evaluation result

$ER = J(Q, R_1, R_2).$

For each aspect $a_j$, where

$$a_j \in \{\text{truthfulness, honesty, helpfulness, instruction following, speech instruction following}\},$$

the model predicts a comparison label

$C_{a_j} \in \{\text{win, lose, tie}\}$

together with a textual explanation $E_{a_j}$ (Ge et al., 28 Aug 2025).

A notable design choice is that the query is textual rather than spoken. The paper states that this standardizes evaluation, avoids extra variability from different query-audio synthesis systems, and avoids distribution mismatch from different S2S training pipelines. SageLM is therefore not a single-score evaluator. It is explicitly multi-aspect: stage 1 covers four semantic aspects, and stage 2 adds the acoustic aspect of speech instruction following (Ge et al., 28 Aug 2025).

The training prompt format reflects this pairwise design. Inputs contain the instruction, Response 1 as <audio>, Response 2 as <audio>, and an aspect tag; outputs contain a comparison among 1, 2, or Tie, with a rationale. The judging protocol is therefore comparative rather than absolute, and explanation generation is part of the core interface rather than an auxiliary post hoc module (Ge et al., 28 Aug 2025).

3. SpeechFeedback and the two-stage training paradigm

The data substrate for SageLM is SpeechFeedback, a synthetic preference dataset with 324,774 instances in total. The semantic portion is derived from UltraFeedback, which contains 64k instructions, each with four LLM-generated responses annotated by GPT-4 on instruction following, truthfulness, honesty, and helpfulness, together with scores and rationales. To adapt these data to speech evaluation, the pipeline removes math and code examples, removes multilingual prompts, sanitizes special characters and formatting that hurt prosody, synthesizes responses with seven TTS systems—CosyVoice, CosyVoice2, SparkTTS, ChatTTS, F5-TTS, Index-TTS, and gpt-4o-mini-tts—discards utterances with high Whisper ASR WER or duration below 0.2 seconds, converts absolute preference scores into pairwise win/lose labels, rewrites rationales into comparative form using Qwen2.5-32B-Instruct, and caps combined text-speech sequence length at 4096 tokens (Ge et al., 28 Aug 2025).

The acoustic portion evaluates emotion, gender, and cartoon / character voice across three task types: explicit TTS, explicit dialogue, and implicit dialogue. In the explicit TTS task, the paper samples 1,000 instructions per style category from UltraFeedback, randomly chooses a text response to synthesize, generates prompt templates with GPT-4o, and constructs paired responses with correct and incorrect style labels using sampling ratios 8:1:1 for correct-correct, correct-incorrect, and incorrect-incorrect pairs. The explicit dialogue task follows a similar construction, including 180 mixed samples jointly controlling emotion and gender. In the implicit dialogue task, both instructions and responses are generated by GPT-4o, seed prompts are curated from Kimi-GenTest emotion data via Self-Instruct, 500 evaluation samples are created, and rationales first explain the implied emotion of the query and then the emotional tone of each candidate response (Ge et al., 28 Aug 2025).

Training follows a two-stage post-training paradigm. Stage 1 performs semantic preference learning over helpfulness, honesty, truthfulness, and instruction following. Stage 2 then adds acoustic preference learning for speech instruction following. Both stages use rationale-augmented supervised fine-tuning. The main SageLM instantiation is Qwen2.5-omni-7B-SFT, while additional finetuned variants include Qwen2-Audio-Instruct-7B-SFT and Qwen2.5-omni-3B-SFT. The reported split comprises 316,544 semantic and 4,270 acoustic training instances, with 728 semantic and 410 acoustic test instances, and training is conducted on 8 NVIDIA A100-SXM4-80GB GPUs (Ge et al., 28 Aug 2025).

4. Rationale-based supervision and optimization

The paper’s central methodological claim is that rationale-based supervision is more suitable for this judgment task than rule-based reinforcement learning. Under supervised fine-tuning, the target can be either label-only or label + rationale, with the instruction-tuning objective given as

$$\mathcal{L}_{\text{SFT}} = - \mathbb{E}_{(x, y) \sim \mathcal{D}} \left[ \sum_{t=1}^{T} \log P_\theta(y_t \mid y_{<t}, x) \right].$$

The rationale-augmented variant is intended to make the model reason about why one response is better, connect evidence to the final judgment, improve alignment with human decisions, and increase interpretability (Ge et al., 28 Aug 2025).

The paper contrasts this with GRPO, which uses a rule-based reward

$$r_i = \alpha \cdot R_a(\hat{s}, s) + \gamma \cdot R_f,$$

with $\alpha = 1.0$ and $R_1, R_2$0, and an accuracy reward

$R_1, R_2$1

The stated critique is that rule rewards are brittle for subtle dialogue reasoning and may optimize the final answer without ensuring that the explanation is consistent. In a manual analysis, the paper reports that in 39\% of 100 sampled cases, the GRPO explanation contradicted the final score. By contrast, a separate human evaluation finds that SageLM rationales align with final results at 90.89\% (Ge et al., 28 Aug 2025).

This comparison is framed as a generation-verification gap problem: checking a judgment is easy, but generating a good judgment is hard. In that framing, label-plus-rationale supervision is meant to directly constrain both the decision and the explanation, whereas rule-based reinforcement learning constrains the output only indirectly (Ge et al., 28 Aug 2025).

5. Empirical performance, robustness, and qualitative findings

The reported headline result is 82.79\% agreement with human judgments. On semantic evaluation, the main SageLM model, Qwen2.5-omni-7B-SFT, reports agreement scores of 83.61 for helpfulness, 81.78 for honesty, 81.68 for instruction following, and 84.07 for truthfulness. On acoustic evaluation, the reported agreement scores are 97.89 for emotion, 99.31 for gender, 99.66 for voice, 98.65 for implicit evaluation, and 91.36 for mixed evaluation (Ge et al., 28 Aug 2025).

The paper also reports strong out-of-distribution performance on unseen AlpacaEval-derived real S2S outputs. In that setting, Whisper + GPT-4o achieves 69.30\% agreement and 53.80\% accuracy, whereas SageLM achieves 87.97\% agreement and 81.01\% accuracy. This is presented as evidence that end-to-end speech judging generalizes better than cascaded ASR-plus-text evaluation on real output distributions (Ge et al., 28 Aug 2025).

Robustness analyses include position consistency, where SageLM reports 84.95\% semantic consistency and 96.10\% acoustic consistency under response-order reversal. The paper also finds that agreement decreases slightly as total response length increases, but remains high, around 80\% in supplementary analysis. For long outputs, SageLM uses a 60s truncation window; performance is described as stable when pair length is at most 60 seconds, and as declining for longer pairs (Ge et al., 28 Aug 2025).

A qualitative case study is used to show a failure mode of cascaded evaluation. Whisper is reported to mis-transcribe Kimi-Audio under hoarse timbre and prosody, producing noisy and distorted transcriptions; GPT-4o, operating on those transcripts, is then biased toward the wrong model, whereas SageLM and human judges recognize the semantic content correctly. The paper uses this example to argue that ASR errors and missing acoustic information jointly distort downstream evaluation (Ge et al., 28 Aug 2025).

6. Limitations and nomenclatural ambiguity

The paper is explicit about several limitations. Agreement drops slightly as total response length increases, especially beyond 60s total pair length. Acoustic evaluation data are much scarcer than semantic data in the two-stage setup. SpeechFeedback is largely synthetic, so real-world variability may still expose gaps. The 60s cutoff is usually sufficient but not always for longer generations. Position bias is reduced but not eliminated. These caveats delimit SageLM’s reported performance and suggest that evaluation quality remains sensitive to data coverage and utterance duration (Ge et al., 28 Aug 2025).

A separate source of ambiguity is terminological rather than methodological. In a 2026 reasoning paper, “SageLM” does not denote the speech judge; instead, it refers to the model family obtained by training a base LLM with SAGE — Self-evolving Agents for Generalized reasoning Evolution. In that usage, “SageLM” is the resulting LLM after a closed-loop, multi-agent RL process involving Challenger, Planner, Solver, and Critic, and the domain is verifiable math and code reasoning rather than speech evaluation (Peng et al., 16 Mar 2026). This suggests that the term is paper-local and must be disambiguated by task domain: in speech evaluation, SageLM refers to the multi-aspect explainable judge of S2S outputs; in the self-evolution literature, it refers to the backbone after SAGE training.

Within the speech-evaluation context, however, the term is technically specific: an end-to-end judge that evaluates paired speech responses across semantic and acoustic dimensions, uses rationale-augmented supervision, and is intended for benchmarking and developing end-to-end spoken dialogue systems (Ge et al., 28 Aug 2025).