---
title: 'SageLM: Explainable Speech Evaluation Model'
url: https://www.emergentmind.com/topics/sagelm
type: topic
---

# SageLM: Explainable Speech Evaluation Model

Searching arXiv for recent papers on “SageLM” and closely related SAGE usages to ground the article.
SageLM is an end-to-end, multi-aspect, and explainable speech large language model for evaluating speech-to-speech large language models. It is designed to judge both semantic and acoustic dimensions directly from speech, rather than through an automatic-speech-recognition-to-text cascade, and it is trained with rationale-based supervision on the synthetic preference dataset SpeechFeedback through a two-stage post-training paradigm. In the reported evaluation, SageLM achieves an **82.79\% agreement** rate with human evaluators, outperforming cascaded and SLM-based baselines by at least **7.42\%** and **26.20\%**, respectively [2508.20916].

## 1. Problem setting and motivation

SageLM addresses a central difficulty in speech-to-speech evaluation: spoken dialogue must be judged on both **what is said** and **how it is said**. The paper frames this as the joint assessment of semantic quality—helpfulness, honesty, truthfulness, and instruction following—and acoustic quality, especially whether speech follows requested or implied vocal style. The same content may be delivered with different emotion, tone, prosody, speaking style, gendered voice, or character voice, so a speech judge that only inspects transcribed text is structurally incomplete [2508.20916].

The work argues that the standard cascaded pipeline—first transcribe speech with ASR, then evaluate the transcript with a text LLM—is insufficient for two stated reasons. First, it incurs **error propagation**, because ASR mistakes distort the input seen by the judge. Second, it causes **loss of acoustic information**, since the transcript erases tone, emotion, voice identity, speaking rate, and prosody. The paper further states that human evaluation, while a gold standard, is expensive, slow, not scalable for model iteration, and usually not explainable in a structured way. It also treats semantic-only SLM-based judges as inadequate because they are not trained for pairwise judgment, lack speech preference supervision, may fail to align final labels with coherent explanations, and do not reliably cover the acoustic axis [2508.20916].

## 2. Formal task definition and judgment protocol

The model is formulated as a pairwise judge over a textual query and two speech responses. Given a textual instruction or query \(Q\) and two speech responses \(R_1, R_2\), SageLM outputs an evaluation result
$$
ER = J(Q, R_1, R_2).
$$
For each aspect \(a_j\), where
$$
a_j \in \{\text{truthfulness, honesty, helpfulness, instruction following, speech instruction following}\},
$$
the model predicts a comparison label
$$
C_{a_j} \in \{\text{win, lose, tie}\}
$$
together with a textual explanation \(E_{a_j}\) [2508.20916].

A notable design choice is that the query is textual rather than spoken. The paper states that this standardizes evaluation, avoids extra variability from different query-audio synthesis systems, and avoids distribution mismatch from different S2S training pipelines. SageLM is therefore not a single-score evaluator. It is explicitly **multi-aspect**: stage 1 covers four semantic aspects, and stage 2 adds the acoustic aspect of **speech instruction following** [2508.20916].

The training prompt format reflects this pairwise design. Inputs contain the instruction, `Response 1` as `<audio>`, `Response 2` as `<audio>`, and an aspect tag; outputs contain a comparison among **1**, **2**, or **Tie**, with a rationale. The judging protocol is therefore comparative rather than absolute, and explanation generation is part of the core interface rather than an auxiliary post hoc module [2508.20916].

## 3. SpeechFeedback and the two-stage training paradigm

The data substrate for SageLM is **SpeechFeedback**, a synthetic preference dataset with **324,774 instances** in total. The semantic portion is derived from **UltraFeedback**, which contains **64k instructions**, each with four LLM-generated responses annotated by GPT-4 on instruction following, truthfulness, honesty, and helpfulness, together with scores and rationales. To adapt these data to speech evaluation, the pipeline removes math and code examples, removes multilingual prompts, sanitizes special characters and formatting that hurt prosody, synthesizes responses with **seven TTS systems**—CosyVoice, CosyVoice2, SparkTTS, ChatTTS, F5-TTS, Index-TTS, and gpt-4o-mini-tts—discards utterances with high Whisper ASR WER or duration below **0.2 seconds**, converts absolute preference scores into pairwise win/lose labels, rewrites rationales into comparative form using **Qwen2.5-32B-Instruct**, and caps combined text-speech sequence length at **4096 tokens** [2508.20916].

The acoustic portion evaluates **emotion**, **gender**, and **cartoon / character voice** across three task types: **explicit TTS**, **explicit dialogue**, and **implicit dialogue**. In the explicit TTS task, the paper samples **1,000 instructions per style category** from UltraFeedback, randomly chooses a text response to synthesize, generates prompt templates with GPT-4o, and constructs paired responses with correct and incorrect style labels using sampling ratios **8:1:1** for correct-correct, correct-incorrect, and incorrect-incorrect pairs. The explicit dialogue task follows a similar construction, including **180 mixed samples** jointly controlling emotion and gender. In the implicit dialogue task, both instructions and responses are generated by GPT-4o, seed prompts are curated from **Kimi-GenTest** emotion data via Self-Instruct, **500 evaluation samples** are created, and rationales first explain the implied emotion of the query and then the emotional tone of each candidate response [2508.20916].

Training follows a two-stage post-training paradigm. **Stage 1** performs semantic preference learning over helpfulness, honesty, truthfulness, and instruction following. **Stage 2** then adds acoustic preference learning for speech instruction following. Both stages use rationale-augmented supervised fine-tuning. The main SageLM instantiation is **Qwen2.5-omni-7B-SFT**, while additional finetuned variants include **Qwen2-Audio-Instruct-7B-SFT** and **Qwen2.5-omni-3B-SFT**. The reported split comprises **316,544 semantic** and **4,270 acoustic** training instances, with **728 semantic** and **410 acoustic** test instances, and training is conducted on **8 NVIDIA A100-SXM4-80GB GPUs** [2508.20916].

## 4. Rationale-based supervision and optimization

The paper’s central methodological claim is that **rationale-based supervision** is more suitable for this judgment task than **rule-based reinforcement learning**. Under supervised fine-tuning, the target can be either **label-only** or **label + rationale**, with the instruction-tuning objective given as
$$
\mathcal{L}_{\text{SFT}} = - \mathbb{E}_{(x, y) \sim \mathcal{D}} \left[ \sum_{t=1}^{T} \log P_\theta(y_t \mid y_{<t}, x) \right].
$$
The rationale-augmented variant is intended to make the model reason about why one response is better, connect evidence to the final judgment, improve alignment with human decisions, and increase interpretability [2508.20916].

The paper contrasts this with **GRPO**, which uses a rule-based reward
$$
r_i = \alpha \cdot R_a(\hat{s}, s) + \gamma \cdot R_f,
$$
with \(\alpha = 1.0\) and \(\gamma = 0.5\), and an accuracy reward
$$
R_a(\hat{s}, s) =
\begin{cases}
\exp\left( - \frac{(\hat{s} - s)^2}{2\sigma^2} \right) & 0 \le |\hat{s} - s| \le 4 \\
0 & |\hat{s} - s| > 4.
\end{cases}
$$
The stated critique is that rule rewards are brittle for subtle dialogue reasoning and may optimize the final answer without ensuring that the explanation is consistent. In a manual analysis, the paper reports that in **39\% of 100 sampled cases**, the GRPO explanation contradicted the final score. By contrast, a separate human evaluation finds that SageLM rationales align with final results at **90.89\%** [2508.20916].

This comparison is framed as a **generation-verification gap** problem: checking a judgment is easy, but generating a good judgment is hard. In that framing, label-plus-rationale supervision is meant to directly constrain both the decision and the explanation, whereas rule-based reinforcement learning constrains the output only indirectly [2508.20916].

## 5. Empirical performance, robustness, and qualitative findings

The reported headline result is **82.79\% agreement** with human judgments. On semantic evaluation, the main SageLM model, **Qwen2.5-omni-7B-SFT**, reports agreement scores of **83.61** for helpfulness, **81.78** for honesty, **81.68** for instruction following, and **84.07** for truthfulness. On acoustic evaluation, the reported agreement scores are **97.89** for emotion, **99.31** for gender, **99.66** for voice, **98.65** for implicit evaluation, and **91.36** for mixed evaluation [2508.20916].

The paper also reports strong out-of-distribution performance on unseen AlpacaEval-derived real S2S outputs. In that setting, **Whisper + GPT-4o** achieves **69.30\%** agreement and **53.80\%** accuracy, whereas **SageLM** achieves **87.97\%** agreement and **81.01\%** accuracy. This is presented as evidence that end-to-end speech judging generalizes better than cascaded ASR-plus-text evaluation on real output distributions [2508.20916].

Robustness analyses include **position consistency**, where SageLM reports **84.95\%** semantic consistency and **96.10\%** acoustic consistency under response-order reversal. The paper also finds that agreement decreases slightly as total response length increases, but remains high, around **80\%** in supplementary analysis. For long outputs, SageLM uses a **60s truncation window**; performance is described as stable when pair length is at most **60 seconds**, and as declining for longer pairs [2508.20916].

A qualitative case study is used to show a failure mode of cascaded evaluation. Whisper is reported to mis-transcribe Kimi-Audio under hoarse timbre and prosody, producing noisy and distorted transcriptions; GPT-4o, operating on those transcripts, is then biased toward the wrong model, whereas SageLM and human judges recognize the semantic content correctly. The paper uses this example to argue that ASR errors and missing acoustic information jointly distort downstream evaluation [2508.20916].

## 6. Limitations and nomenclatural ambiguity

The paper is explicit about several limitations. Agreement drops slightly as total response length increases, especially beyond **60s** total pair length. Acoustic evaluation data are much scarcer than semantic data in the two-stage setup. SpeechFeedback is largely synthetic, so real-world variability may still expose gaps. The **60s** cutoff is usually sufficient but not always for longer generations. Position bias is reduced but not eliminated. These caveats delimit SageLM’s reported performance and suggest that evaluation quality remains sensitive to data coverage and utterance duration [2508.20916].

A separate source of ambiguity is terminological rather than methodological. In a 2026 reasoning paper, **“SageLM”** does not denote the speech judge; instead, it refers to the model family obtained by training a base LLM with **SAGE — Self-evolving Agents for Generalized reasoning Evolution**. In that usage, “SageLM” is the resulting LLM after a closed-loop, multi-agent RL process involving Challenger, Planner, Solver, and Critic, and the domain is verifiable math and code reasoning rather than speech evaluation [2603.15255]. This suggests that the term is paper-local and must be disambiguated by task domain: in speech evaluation, SageLM refers to the multi-aspect explainable judge of S2S outputs; in the self-evolution literature, it refers to the backbone after SAGE training.

Within the speech-evaluation context, however, the term is technically specific: an end-to-end judge that evaluates paired speech responses across semantic and acoustic dimensions, uses rationale-augmented supervision, and is intended for benchmarking and developing end-to-end spoken dialogue systems [2508.20916].

Source: https://www.emergentmind.com/topics/sagelm