SpeechJudge: Human-Level Judgment for Speech Naturalness

• SpeechJudge is a comprehensive suite for speech naturalness evaluation, combining a large-scale human feedback dataset, a pairwise benchmark, and a generative reward model.
• The framework addresses the bottleneck in aligning TTS models with human perception by achieving up to 79.4% accuracy with inference-time scaling.
• It supports diverse languages and expressive speech styles, offering practical insights for improving human-aligned speech synthesis in downstream applications.

Searching arXiv for the primary paper and closely related work on speech preference evaluation and benchmark usage. arXiv_search(query="SpeechJudge Towards Human-Level Judgment for Speech Naturalness", max_results=5, sort_by="relevance") SpeechJudge is a suite for speech naturalness judgment that comprises a dataset, a benchmark, and a reward model centered on naturalness, described as one of the most fundamental subjective metrics for speech synthesis. It was introduced to address a specific bottleneck in aligning large generative models with human feedback in speech synthesis: the lack of a large-scale human preference dataset that reflects human perception. The suite contains SpeechJudge-Data, a large-scale human feedback corpus; SpeechJudge-Eval, a benchmark for pairwise naturalness judgment; and SpeechJudge-GRM, a generative reward model based on Qwen2.5-Omni-7B. On SpeechJudge-Eval, the reported gap between human judgment and existing metrics remains substantial: the leading evaluated AudioLLM, Gemini-2.5-Flash, achieves less than 70% agreement with human judgment, whereas SpeechJudge-GRM reaches 77.2% accuracy and 79.4% after inference-time scaling at 10 samples (Zhang et al., 11 Nov 2025).

1. Problem formulation and conceptual scope

SpeechJudge is framed around pairwise human preference prediction for speech naturalness rather than scalar quality scoring. In the benchmark task, the input is a triplet $(t, a_1, a_2)$ consisting of target text and two synthesized speech candidates, and the objective is to predict which audio is more natural. The benchmark uses accuracy as the primary metric:

$$\mathrm{Accuracy} = \frac1{|\mathcal{D}|} \sum_{d\in\mathcal{D}} \mathbb{I}\bigl(y_{\text{model}(d)}=y_{\text{human}(d)}\bigr).$$

This formulation is motivated by the observation that naturalness is a subjective attribute for which direct human preference signals are especially valuable. SpeechJudge therefore treats human pairwise comparison as the central supervision source and constructs both the dataset and the reward model around that signal (Zhang et al., 11 Nov 2025).

A common misconception in speech evaluation is that objective metrics or generic multimodal models are already adequate proxies for human naturalness judgments. The reported benchmark results contradict that assumption. WER, FAD, SIM, most MOS predictors, and deepfake detectors operate near random or only modestly above it, and even the strongest evaluated AudioLLMs remain below 71% accuracy under Chain-of-Thought prompting. This establishes naturalness judgment as a distinct evaluation problem rather than a straightforward reuse of intelligibility, similarity, or anti-spoofing metrics (Zhang et al., 11 Nov 2025).

2. SpeechJudge-Data

SpeechJudge-Data is a human feedback corpus of 99 K synthesized speech pairs $(t, a_1, a_2)$, with each pair annotated on average by 2.49 human raters and an estimated market cost of $\sim 70$ K USD. The corpus is constructed from six zero-shot text-to-speech systems spanning three architectures: autoregressive systems (ARS, CosyVoice2, CosyVoice2-INTP, Ints-INTP), a flow-matching system (F5-TTS), and a masked generative codec system (MaskGCT). The speech reference styles include regular samples from Emilia-Large and expressive samples including emotional speech from ParaSpeechCaps, accented speech from L2-Arctic and KeSpeech, whisper speech, and “character voices” from Genshin Impact (Zhang et al., 11 Nov 2025).

The dataset covers Chinese, English, and code-switched speech in both monolingual and cross-lingual synthesis settings, including examples such as en$\rightarrow$en, zh$\rightarrow$en, en$\rightarrow$zh, and mixed-language conditions. This breadth is important because the paper explicitly positions SpeechJudge as a resource spanning diverse speech styles and multiple languages rather than a narrow single-domain evaluation set (Zhang et al., 11 Nov 2025).

The annotation protocol separates intelligibility from naturalness. Intelligibility is labeled pointwise per audio clip as binary OK/fail according to whether the read-aloud covers the text exactly, with no omissions, insertions, or substitutions. Naturalness is labeled pairwise on a five-point CMOS scale: A +2, A +1, Tie, B +1, B +2. The raters consist of 69 professional annotators, with native Chinese annotators for Chinese data and annotators with at least CET-6 English proficiency for English and mixed data. Quality control includes detailed pilot-tested guidelines and examples, dual annotation with a third rater in cases of disagreement, and removal of any rater whose overall agreement with peers falls below approximately 30%. Final inter-annotator agreement is reported as approximately 60–70% per person (Zhang et al., 11 Nov 2025).

The naturalness labels also expose the structure of subjective disagreement. Full agreement among raters accounts for 51.5% of the data, weak agreement for 17.2%, weak disagreement for 19.7%, and full disagreement for 11.6%. This distribution is consequential for later benchmark construction and model training. It indicates that the corpus contains both high-consensus cases suitable for evaluation and lower-consensus or adversarial cases suitable for difficult post-training (Zhang et al., 11 Nov 2025).

3. SpeechJudge-Eval

SpeechJudge-Eval is a benchmark for pairwise naturalness judgment derived from SpeechJudge-Data. Its construction begins with the 99 K raw pairs, filters out “Tie” pairs, retains only full-agreement examples, and then draws a stratified sample of 1,000 pairs. The evaluation set is divided into a regular subset of 400 pairs, consisting of 200 English and 200 Chinese examples, and an expressive subset of 600 pairs, consisting of 200 English, 200 Chinese, and 200 mixed-language examples (Zhang et al., 11 Nov 2025).

The benchmark compares four categories of baselines: objective metrics, MOS predictors, deepfake detectors, and AudioLLMs. The objective metrics are WER, SIM, and FAD. The MOS predictors are DNSMOS, UTMOS, Content Enjoyment, Content Usefulness, Production Complexity, and Production Quality. The deepfake detectors are AASIST and ADV. The evaluated AudioLLMs include open-source systems such as Phi-4-Multimodal, Qwen2.5-Omni-7B, Kimi-Audio-7B, Gemma-3n, Voxtral-Mini-3B, MiDashengLM, and MiMo-Audio-7B, as well as closed-source systems accessed through APIs, including Gemini-2.5-Flash, Gemini-2.5-Pro, GPT-4o mini, and GPT-4o Audio. Prompting is tested in both plain and Chain-of-Thought variants (Zhang et al., 11 Nov 2025).

The quantitative pattern is consistent across baseline families. WER, FAD, SIM, most MOS predictors, and the deepfake detectors lie in the 45–60% accuracy range, often near random. Among AudioLLMs with plain prompting, Gemini-2.5-Flash reaches 69.1%, GPT-4o Audio 67.4%, and Kimi-Audio 67.0%. Even with Chain-of-Thought prompting, top models remain under 71%, with Gemini-2.5-Flash CoT at 70.5%. The paper further reports that CoT prompting alone yields mixed effects on open-source AudioLLMs: it helps Gemini but can hurt some others. This result is significant because it separates “reasoning-style prompting” from actual alignment with human auditory preference; prompting alone does not close the gap (Zhang et al., 11 Nov 2025).

4. SpeechJudge-GRM

SpeechJudge-GRM is a generative reward model whose base “Thinker” is Qwen2.5-Omni-7B. Its input format consists of a textual prompt describing the task together with a representation of $a_1$, $a_2$, and the target text $t$. Its output has two parts: a Chain-of-Thought rationale and a final preference decision, either A or B (Zhang et al., 11 Nov 2025).

Training proceeds in two stages. The first stage is supervised fine-tuning with Chain-of-Thought rationales. Gemini-2.5-Flash serves as the teacher and generates rationales on the training portion of SpeechJudge-Data. Only those samples where the teacher’s choice matches the human label are retained, yielding approximately 25 K examples out of 42 K. The objective is standard next-token cross-entropy over the teacher-generated rationale and answer:

$$\mathrm{Accuracy} = \frac1{|\mathcal{D}|} \sum_{d\in\mathcal{D}} \mathbb{I}\bigl(y_{\text{model}(d)}=y_{\text{human}(d)}\bigr).$$0

The reported optimization settings are LoRA rank 128, AdamW with learning rate $$\mathrm{Accuracy} = \frac1{|\mathcal{D}|} \sum_{d\in\mathcal{D}} \mathbb{I}\bigl(y_{\text{model}(d)}=y_{\text{human}(d)}\bigr).$$1, and batch tokens 4,000 (Zhang et al., 11 Nov 2025).

The second stage is reinforcement learning with Generative Reward Policy Optimization, implemented as DAPO, an enhanced GRPO. The RL dataset is the remaining 17 K “challenging” samples where Gemini disagreed with the human label. The policy is the SFT-initialized Qwen2.5-Omni model $$\mathrm{Accuracy} = \frac1{|\mathcal{D}|} \sum_{d\in\mathcal{D}} \mathbb{I}\bigl(y_{\text{model}(d)}=y_{\text{human}(d)}\bigr).$$2. For each prompt, the model samples $$\mathrm{Accuracy} = \frac1{|\mathcal{D}|} \sum_{d\in\mathcal{D}} \mathbb{I}\bigl(y_{\text{model}(d)}=y_{\text{human}(d)}\bigr).$$3 rollouts, each yielding a decision $$\mathrm{Accuracy} = \frac1{|\mathcal{D}|} \sum_{d\in\mathcal{D}} \mathbb{I}\bigl(y_{\text{model}(d)}=y_{\text{human}(d)}\bigr).$$4, and assigns reward $$\mathrm{Accuracy} = \frac1{|\mathcal{D}|} \sum_{d\in\mathcal{D}} \mathbb{I}\bigl(y_{\text{model}(d)}=y_{\text{human}(d)}\bigr).$$5 if $$\mathrm{Accuracy} = \frac1{|\mathcal{D}|} \sum_{d\in\mathcal{D}} \mathbb{I}\bigl(y_{\text{model}(d)}=y_{\text{human}(d)}\bigr).$$6 and $$\mathrm{Accuracy} = \frac1{|\mathcal{D}|} \sum_{d\in\mathcal{D}} \mathbb{I}\bigl(y_{\text{model}(d)}=y_{\text{human}(d)}\bigr).$$7 otherwise. The optimization target is

$$\mathrm{Accuracy} = \frac1{|\mathcal{D}|} \sum_{d\in\mathcal{D}} \mathbb{I}\bigl(y_{\text{model}(d)}=y_{\text{human}(d)}\bigr).$$8

Implementation is reported through ms-swift with LoRA rank 64, learning rate $$\mathrm{Accuracy} = \frac1{|\mathcal{D}|} \sum_{d\in\mathcal{D}} \mathbb{I}\bigl(y_{\text{model}(d)}=y_{\text{human}(d)}\bigr).$$9, and batch size 32 (Zhang et al., 11 Nov 2025).

The benchmark also includes a classic Bradley–Terry reward model, SpeechJudge-BTRM, as a baseline. It assigns each audio a scalar reward $(t, a_1, a_2)$0 and models pairwise preference probability as

$(t, a_1, a_2)$1

Its loss is

$(t, a_1, a_2)$2

SpeechJudge-GRM differs from this formulation by being generative and rationale-producing rather than purely scalar and discriminative. The paper also introduces inference-time scaling: at test time, it samples $(t, a_1, a_2)$3 independent outputs and selects the majority vote. A scaling factor of $(t, a_1, a_2)$4 yields a reported gain of 2 percentage points, from 77.2% to 79.4% (Zhang et al., 11 Nov 2025).

5. Empirical performance and downstream use

On SpeechJudge-Eval, the reported overall accuracies are as follows (Zhang et al., 11 Nov 2025):

System	Accuracy
Qwen2.5-Omni-7B (zero-shot)	60.6%
Gemini-2.5-Flash (plain)	69.1%
SpeechJudge-BTRM	72.7%
SpeechJudge-GRM (SFT only)	75.3%
SpeechJudge-GRM (SFT only, Voting@10)	77.6%
SpeechJudge-GRM (SFT + RL)	77.2%
SpeechJudge-GRM (SFT + RL, Voting@10)	79.4%

These results support two specific ablation conclusions reported in the paper. First, the RL stage adds approximately 2 percentage points relative to SFT alone. Second, Voting@$(t, a_1, a_2)$5 consistently adds approximately 2 percentage points when moving from $(t, a_1, a_2)$6 to $(t, a_1, a_2)$7. The combination of generative post-training and inference-time scaling therefore improves agreement with human naturalness judgments beyond both zero-shot AudioLLMs and the Bradley–Terry baseline (Zhang et al., 11 Nov 2025).

SpeechJudge is also used as a reward function in two downstream settings. In high-naturalness sample selection, a zero-shot TTS model, Qwen2.5-Omni-0.5B-TTS, generates 100 samples per prompt, after which SpeechJudge-BTRM or SpeechJudge-GRM selects the “best” sample. Human A/B evaluation against random selection shows that both methods outperform random, with GRM yielding higher win rates. In post-training for speech generation, the base model is Qwen2.5-0.5B-TTS (AR+Diffusion), and four methods are compared: DPO on INTP intelligibility data, DPO on SpeechJudge-Data directly, DPO on INTP relabeled offline by SpeechJudge-GRM, and online DPO using SpeechJudge-GRM as reward function. The reported subjective results show that all SpeechJudge-GRM variants improve naturalness CMOS more than INTP, with the online case highest at +0.25 CMOS. Speaker similarity remains largely tied, which the paper identifies as evidence that multi-objective alignment remains unresolved (Zhang et al., 11 Nov 2025).

6. Subsequent use, related benchmarks, and open problems

SpeechJudge has already been incorporated into subsequent evaluation work. In SwanBench-Speech, a benchmark for long-form speech generation, Prosodic Coherence is implemented using a fine-tuned Qwen2.5-Omni model identified as “SpeechJudge,” prompted to rate coherence on a 1–5 scale with the full utterance and target text, averaged over 10 independent runs. This places SpeechJudge within a broader evaluation stack for long-form acoustics, semantics, and expressiveness rather than limiting it to short-form pairwise naturalness comparison (Pan et al., 27 May 2026).

SpeechJudge also appears as a benchmark dataset in preference-prediction research beyond its original paper. PrefSQA evaluates on a SpeechJudge split of 42,097/1,000/1,000 train/validation/test pairs and reports 68.2% accuracy on SpeechJudge as a human-preference set. That paper studies MOS-free pairwise preference prediction and emphasizes the role of high-quality preference datasets, which is consistent with the design rationale of SpeechJudge itself (Fan et al., 17 Jun 2026).

The open challenges identified for SpeechJudge define a broader research agenda. These include extending reward modeling to other subjective dimensions such as speaker similarity, expressiveness, and emotion; improving handling of highly expressive or cross-lingual data, where human agreement is lower; jointly optimizing multi-objective preferences without trade-offs such as naturalness versus speaker similarity; and developing more efficient inference scaling strategies to reduce latency. A plausible implication is that SpeechJudge marks a shift from using automatic speech metrics as imperfect surrogates toward training explicit human-aligned evaluators, while leaving unresolved the more general problem of multi-dimensional preference alignment in speech synthesis (Zhang et al., 11 Nov 2025).