---
title: 'SpeechJudge-Eval: Speech Naturalness Benchmark'
url: https://www.emergentmind.com/topics/speechjudge-eval
type: topic
---

# SpeechJudge-Eval: Speech Naturalness Benchmark

SpeechJudge-Eval is the benchmark component of SpeechJudge, designed for **pairwise speech naturalness judgment**: given a target text and two synthesized speech outputs, the evaluator must decide which output is more natural or human-like. It was introduced to address a gap in speech synthesis evaluation: existing speech quality reporting is fragmented across papers, and proxy measures such as **WER**, **FAD**, **MOS predictors**, and **deepfake detectors** do not necessarily align with human preference for naturalness. The benchmark is built from **high-agreement human labels** derived from a larger human-feedback corpus, and its initial evaluation shows that existing automatic metrics and AudioLLMs remain substantially below the benchmark’s intended target, with **Gemini-2.5-Flash** as the leading baseline at **69.1%** agreement and still below 70% [2511.07931].

## 1. Scope and motivation

SpeechJudge-Eval is centered on **naturalness**, which the underlying paper treats as one of the most fundamental subjective criteria in text-to-speech (TTS). Its specific task is not MOS prediction, deepfake detection, or transcript fidelity estimation; rather, it is a **comparative preference judgment** over two speech samples conditioned on the same target text. This framing places the benchmark close to preference-modeling paradigms used in alignment, but adapted to speech rather than text [2511.07931].

The benchmark was created because the speech field lacked a large-scale human preference corpus focused specifically on naturalness. The paper argues that automatic evaluators and strong AudioLLMs still do not reliably match human judgments of which speech sounds more natural. SpeechJudge-Eval is therefore intended as a controlled evaluation substrate for testing whether a model can reproduce human comparative judgments on speech naturalness, rather than merely optimize surface proxies such as intelligibility or lexical correctness [2511.07931].

A central design choice is the use of **high-agreement human labels**. The benchmark is intentionally filtered to retain only confident naturalness comparisons, so that disagreement in automated systems can be more plausibly attributed to evaluator limitations than to annotation ambiguity. This makes SpeechJudge-Eval not only a benchmark for model comparison, but also a reference point for studying the gap between human and automated judgment in speech evaluation [2511.07931].

## 2. SpeechJudge-Data as the benchmark substrate

SpeechJudge-Eval is derived from **SpeechJudge-Data**, a human-feedback corpus of **99K annotated speech pairs** with target texts, synthesized from multiple strong zero-shot TTS systems and annotated for both **intelligibility** and **naturalness**. The corpus uses **six TTS models** spanning three architecture families: **AR-based** (**ARS, CosyVoice2, CosyVoice2-INTP, Ints-INTP**), **FM-based** (**F5-TTS**), and **MGM-based** (**MaskGCT**). This construction is explicitly intended to create diverse speech quality profiles and reduce overfitting to a single generator family [2511.07931].

Reference speech includes both **regular speech**, sampled from **Emilia-Large**, and **expressive speech**, sourced from **ParaSpeechCaps**, **L2-Arctic**, **KeSpeech**, in-house **whisper** samples, and **Genshin Impact** character voices. Target texts cover **Chinese (zh)**, **English (en)**, and **Chinese-English code-switching (mixed)**, with monolingual settings (**en2en**, **zh2zh**) and cross-lingual settings (**zh2en**, **en2zh**, **zh2mixed**, **en2mixed**). Pair formation includes both **intra-model pairs** and **inter-model pairs**, broadening the space of naturalness contrasts represented in the data [2511.07931].

Each sample \((t, a_1, a_2)\) receives two annotation types. **Intelligibility** is a **binary** judgment about whether each audio correctly reads the target text without insertion, omission, or mispronunciation. **Naturalness** uses a **five-level Comparative Mean Opinion Score (CMOS)** pairwise preference scale: **A +2**, **A +1**, **Tie**, **B +1**, and **B +2**. The annotation guidelines specify that naturalness should reflect **human-likeness, prosody, pacing, clarity, and stress**, and that minor isolated pronunciation errors should not dominate the naturalness decision unless they seriously disrupt listening [2511.07931].

The annotation process involved **69 annotators** over **2 months**, yielding **99K raw pairs**, an average of **2.49 annotations per pair**, and a reported annotation value of over **500K RMB**. For agreement analysis, the five-level CMOS labels are collapsed to **A / B / Tie**, and four agreement levels are defined: **FA** (full agreement), **WA** (weak agreement), **WD** (weak disagreement), and **FD** (full disagreement). The paper reports that about **70%** of the data is in **FA or WA**, and that the expressive subset is harder, with lower agreement than the regular subset [2511.07931].

## 3. Benchmark construction and task protocol

SpeechJudge-Eval formalizes speech naturalness judgment as a **binary pairwise comparison**: given a target text \(t\) and two audio samples \((a_1, a_2)\), the model predicts which audio is more natural. The paper defines benchmark accuracy as the fraction of examples for which model and human preferences match. This metric directly operationalizes alignment with comparative human judgment, rather than agreement with a scalar quality proxy [2511.07931].

The benchmark is constructed from **SpeechJudge-Data (hq)** through a strict filtering pipeline. First, all **Tie** samples are removed. Second, only **Full Agreement (FA)** samples are retained. Third, the resulting pool is stratified over **regular vs expressive** subsets and over target languages **zh / en / mixed**. The final benchmark contains **1,000 pairs**, designed to preserve both label confidence and coverage across speech styles and language conditions [2511.07931].

| Benchmark component | Composition | Size |
|---|---|---:|
| Regular / Emilia-Large | **200 en + 200 zh** | 400 |
| Expressive / ParaSpeechCaps, L2-Arctic, KeSpeech, Genshin, etc. | **200 en + 200 zh + 200 mixed** | 600 |
| Total | FA-only, no ties, stratified sampling | 1,000 |

This construction has two methodological consequences. First, it turns SpeechJudge-Eval into a high-confidence benchmark for **relative** naturalness rather than absolute scoring. Second, it makes the benchmark sensitive to phenomena that are often underrepresented in transcript-centric evaluation, especially expressive variation and multilingual or code-switched synthesis. The paper’s repeated contrast between regular and expressive speech indicates that benchmark difficulty is not uniform across subsets [2511.07931].

## 4. Baselines and empirical behavior

The paper evaluates four broad classes of baselines on SpeechJudge-Eval: **objective metrics**, **MOS predictors**, **deepfake detectors**, and **AudioLLMs**. The overall pattern is that none of these families reaches human-level agreement, and several are only near chance for pairwise naturalness judgment [2511.07931].

Among **objective metrics**, the reported total accuracies are **WER: 57.9%**, **SIM: 44.5%**, and **FAD: 48.6%**. These results are close to chance or only modestly above it, supporting the paper’s claim that naturalness is not reducible to transcript correctness, speaker similarity, or distributional audio distance. Among **MOS predictors**, the paper reports **DNSMOS: 57.9%**, **UTMOS: 53.7%**, and **Audiobox-aesthetics** variants with **CE: 60.8%**, **CU: 57.3%**, **PC: 44.9%**, and **PQ: 57.1%**; the best result in this group is **CE at 60.8%**, still well below the stronger pairwise evaluators [2511.07931].

**Deepfake detectors** perform poorly: **AASIST** reaches **46.7%**, and **ADV** reaches **38.3%**. This is consistent with the paper’s observation that real-vs-fake discrimination is not the same problem as ranking two synthetic outputs by naturalness. The detector objective does not encode the comparative and fine-grained perceptual criteria that human annotators use in SpeechJudge-Eval [2511.07931].

AudioLLMs fare better but still fall short of the benchmark’s apparent difficulty. Reported open-source results include **Phi-4-Multimodal: 57.0%**, **Qwen2.5-Omni-7B: 60.6%**, **Kimi-Audio-7B-Instruct: 67.0%**, **Gemma-3n-E4B-it: 48.2%**, **Voxtral-Mini-3B-2507: 56.0%**, **MiDashengLM: 61.6%**, and **MiMo-Audio-7B-Instruct: 54.1%**. Closed-source systems include **Gemini-2.5-Flash: 69.1%**, **Gemini-2.5-Pro: 66.5%**, **GPT-4o mini Audio: 50.5%**, and **GPT-4o Audio: 67.4%**. The headline result is that **Gemini-2.5-Flash** is the strongest baseline, but still remains below 70% agreement [2511.07931].

The paper also reports prompt-sensitivity for some judge models. **Chain-of-Thought prompting** improves **Gemini-2.5-Flash** from **69.1%** to **70.5%**, while **Kimi-Audio-7B-Instruct** declines from **67.0%** to **66.5%**. This indicates that reasoning-style prompting does not uniformly help, and that performance depends on whether the underlying multimodal model already has strong instruction-following and evaluative capacity. The analysis further notes that **expressive speech is harder than regular speech**, reinforcing that benchmark difficulty is not explained by a single global score [2511.07931].

## 5. SpeechJudge-GRM and benchmark-driven alignment

SpeechJudge-Eval is not only an evaluation set; within the SpeechJudge framework it also serves as the target benchmark for a dedicated **generative reward model**, **SpeechJudge-GRM**. This model is built on **Qwen2.5-Omni-7B (Thinker)**. The paper argues for a generative reward model rather than a classic scalar **Bradley-Terry reward model** because a GRM can produce **Chain-of-Thought rationales** and supports **test-time scaling / majority voting** [2511.07931].

Training proceeds in two stages. In **Stage 1**, the model is given a **cold start** via **Supervised Fine-Tuning (SFT)** on Chain-of-Thought rationales generated by **Gemini-2.5-Flash**. For each training sample \(d=(t,a_1,a_2,y_{\mathcal{H}})\), the teacher produces a rationale and a preference label, and only samples where the teacher label matches the human label are retained. This stage uses **25K** teacher-agreeing samples, **LoRA rank 128**, and learning rate \(5 \times 10^{-5}\). In **Stage 2**, the model is optimized with **GRPO** on the hard cases where the teacher disagrees with humans; the appendix states that **DAPO**, an enhanced GRPO variant, is actually used. This stage uses **17K** teacher-disagreeing samples, **LoRA rank 64**, **8 rollouts per prompt**, batch size **32**, and learning rate \(5 \times 10^{-6}\). The reward is verifiable against human preference:
```latex
r = 1 \;\text{if}\; y_{\mathcal{M}} = y_{\mathcal{H}}, \quad r=-1 \;\text{otherwise}
```  
[2511.07931]

The benchmark results show a staged improvement:

| Model | Total accuracy |
|---|---:|
| Qwen2.5-Omni-7B | 60.6% |
| Gemini-2.5-Flash | 69.1% |
| SpeechJudge-BTRM | 72.7% |
| SpeechJudge-GRM (SFT) | 75.3% |
| SpeechJudge-GRM (SFT + RL) | 77.2% |
| SpeechJudge-GRM (SFT) w/ Voting@10 | 77.6% |
| SpeechJudge-GRM (SFT + RL) w/ Voting@10 | 79.4% |

These numbers establish two points. First, the dedicated benchmark supports meaningful separation between generic AudioLLMs and reward models trained specifically for speech naturalness. Second, the gain from **SFT** (**60.6% → 75.3%**) followed by **RL** (**75.3% → 77.2%**) suggests that benchmark-aligned post-training contributes substantially beyond the base multimodal model [2511.07931].

The paper further reports that SpeechJudge-GRM can be used as a reward function during post-training of speech generation models. Two practical uses are described: **Best-of-N sample selection** and **post-training TTS models** with **DPO/online alignment**. According to the paper, these uses improve both intelligibility and naturalness, while naturalness gains do not materially harm speaker similarity, though similarity alignment remains imperfect and is identified as future work [2511.07931].

## 6. Relation to adjacent speech and judge-evaluation research

Within speech evaluation, SpeechJudge-Eval occupies a narrower but more specialized role than benchmarks such as **SD-Eval**. **SD-Eval** evaluates **spoken dialogue understanding beyond words** in a **single-turn speech-to-text dialogue** setting, with four subsets—**emotion**, **accent**, **age**, and **background sound / environment**—and a dataset of **7,303 utterances** totaling **8.76 hours**. Its central question is whether a model can generate an appropriate response conditioned on paralinguistic and environmental cues, whereas SpeechJudge-Eval asks which of two speech outputs is more natural. SD-Eval also reports that **GPT-4o**-based LLM metrics correlate more strongly with human evaluation than BLEU/ROUGE-style metrics, which reinforces the broader claim that open-ended speech evaluation often requires richer judgment mechanisms than lexical overlap [2406.13340].

More generally, recent work on judge models indicates that evaluator quality is multidimensional rather than reducible to a single agreement number. Research on hidden shortcuts shows that LLM judges can change verdicts under irrelevant metadata perturbations while failing to acknowledge those cues in their rationales, yielding an **explanation gap** [2602.07996]. Work on **evaluative fingerprints** reports low inter-judge agreement but high judge-specific stability, arguing that judges are stable yet non-interchangeable measurement devices [2601.05114]. A psychometric **Judge Datasheet** protocol further decomposes judge behavior into **dark current**, **stable cross-sensitivity**, **positional false preference**, **target sensitivity**, and **criterion shifts**, explicitly treating judges as instruments rather than scalar classifiers [2606.15610].

Related literature also highlights the role of pairwise robustness and judge assignment. **J4R** argues that reasoning-heavy pairwise evaluation is strongly affected by **positional bias**, and introduces training over equivalent response-order states to improve judge consistency [2505.13346]. **CyclicJudge** shows that judge identity can contribute substantial variance in LLM-as-judge pipelines and that round-robin judge assignment can cancel judge bias without increasing per-item cost [2603.01865]. Capability-oriented benchmarking work such as **M-JudgeBench** argues that reliable judge evaluation should separately probe pairwise comparison, length bias avoidance, and process error detection [2603.00546].

This broader literature suggests that SpeechJudge-Eval should be understood as measuring a **specific evaluative construct**—speech naturalness under pairwise human preference—rather than generic judge competence. Its contribution is therefore twofold: it supplies a high-confidence benchmark for one foundational speech criterion, and it provides a substrate on which broader judge-reliability questions in multimodal evaluation can be posed with unusually direct human supervision [2511.07931].

Source: https://www.emergentmind.com/topics/speechjudge-eval