---
title: 'SpeechJudge-Data: TTS Naturalness Evaluation'
url: https://www.emergentmind.com/topics/speechjudge-data
type: topic
---

# SpeechJudge-Data: TTS Naturalness Evaluation

Searching arXiv for the specified papers and closely related work on speech judgment and process-evaluation datasets.
SpeechJudge-Data is a large-scale human feedback corpus for synthesized speech that is designed for intelligibility judgment and, centrally, naturalness judgment. Its base unit is the triplet $\mathcal{D}=\{(t,a_1,a_2)\}$, where $t$ is the target text and $a_1,a_2$ are two synthesized waveforms for the same text; annotators supply pointwise intelligibility labels for each waveform and a pairwise naturalness preference between them on a five-level Comparative MOS (CMOS) scale. Introduced as part of the broader SpeechJudge suite, the dataset is intended to support reward modeling, DPO-style preference alignment, RLHF for TTS, and the construction of a dedicated naturalness benchmark and evaluator models [2511.07931].

## 1. Definition, motivation, and scope

SpeechJudge-Data was built to address a specific deficiency in speech synthesis evaluation: the absence of a large-scale human preference dataset focused on holistic naturalness for modern zero-shot TTS systems. The work positions naturalness as “one of the most fundamental subjective metrics for speech synthesis,” and distinguishes its objective from prior resources that either emphasize pointwise MOS or focus on narrower attributes such as low-level acoustic quality or intelligibility [2511.07931].

The dataset is explicitly structured around pairwise comparison rather than scalar scoring alone. This design matches its intended downstream uses: reward modeling, pairwise preference learning, and alignment of speech generation models with human judgments. In related speech quality assessment research, pairwise preference prediction is motivated as less sensitive to rater variability, scale-use differences, and cross-study protocol effects than MOS-based supervision, which provides a broader methodological context for SpeechJudge-Data’s preference-centric construction [2606.19597].

SpeechJudge-Data is not a generic speech quality corpus. It is specifically centered on synthesized speech naturalness under modern zero-shot TTS conditions, with multilingual and multi-style coverage, and with simultaneous intelligibility annotation so that naturalness can later be disentangled from gross text-accuracy failures [2511.07931].

## 2. Corpus construction and coverage

SpeechJudge-Data (raw) contains **99K** triplets $(t,a_1,a_2)$. Each pair receives **2–3 independent annotators**, with an average of **2.49 annotations** per pair. The paper estimates the market value of annotation at **500K RMB (~70K USD)** [2511.07931].

The waveform pairs are generated from **six zero-shot TTS models** spanning three architectural families:

- **AR-based**: ARS, CosyVoice2, CosyVoice2-INTP, Ints-INTP  
- **FM-based**: F5-TTS  
- **MGM-based**: MaskGCT  

The target texts cover **Chinese (zh)**, **English (en)**, and **mixed code-switching (zh–en)**. Reference speech is divided into two broad style regimes. **Regular speech** references are sampled from **Emilia-Large**. **Expressive speech** references come from corpora and sources including **ParaSpeechCaps**, **L2-Arctic**, **KeSpeech**, a **Whisper-style in-house corpus**, and character voices from **Genshin Impact** [2511.07931].

For each sample, a prompt $(a_{\mathrm{ref}}, t)$ is formed, where the reference waveform $a_{\mathrm{ref}}$ defines speaker, style, and prosody, and the target text $t$ defines content. A TTS model $\mathcal{M}_{\mathrm{tts}}$ then generates audio according to
\[
a_i \sim \mathcal{M}_{\mathrm{tts}}(a_{\mathrm{ref}}, t).
\]
Pairs may be **intra-model**, where both audios come from the same TTS system and differ due to sampling or decoding variability, or **inter-model**, where the two audios are generated by different systems [2511.07931].

The language configuration is deliberately broader than simple monolingual synthesis. The corpus includes **en2en** and **zh2zh** settings, as well as cross-lingual and code-switching conditions such as **zh2en**, **en2zh**, **zh2mixed**, and **en2mixed**. For expressive references, DeepSeek-V3 is used to generate new scripts in multiple writing styles, returning monolingual continuations, a random continuation, a translation, and a code-switching version; for regular references, DeepSeek-V3 is used to clean ASR-like transcriptions by fixing typos and punctuation [2511.07931].

This composition makes the corpus a study of naturalness under high-quality, often subtle, TTS comparisons rather than an evaluation of overtly degraded speech alone. A plausible implication is that the corpus is designed to stress judgment in regimes where differences are perceptually meaningful but not trivially captured by lexical correctness or simple acoustic heuristics.

## 3. Annotation protocol and label semantics

The annotation process uses a professional data annotation company in China. For Chinese samples, annotators are native Chinese speakers. For English and code-switching samples, the requirement is **CET-6 or higher**. The protocol includes standardized training based on a detailed manual, and a pilot study among researchers was used to refine the guidelines [2511.07931].

Each sample is initially labeled by **two annotators** independently; if they disagree, a **third annotator** is introduced. Reliability is quantified at the annotator level using
\[
r_{ij}=\frac{1}{M-1}\sum_{x\neq i}\mathbb{I}[y_{xj}=y_{ij}], \qquad
R_i=\frac{1}{N_i}\sum_{j=1}^{N_i} r_{ij}.
\]
The agreement distribution peaks at **60–70%**, and one annotator with **<30%** agreement was removed [2511.07931].

Annotators perform two distinct tasks.

First, they conduct **pronunciation error detection (intelligibility)** for each audio separately. Given the target text $t$ and one waveform, they decide whether the audio accurately reads the text. The guidelines define intelligibility errors as **omission**, **insertion**, and **substitution**, including misread names, numbers, and word-order errors. This is recorded as a binary text-accuracy decision for each of $a_1$ and $a_2$ [2511.07931].

Second, they perform **naturalness comparison** between the two audios using a **5-point CMOS scale**:

- **A +2**: A is significantly more natural than B  
- **A +1**: A is slightly more natural than B  
- **Tie**: similar naturalness, too close to judge  
- **B +1**: B is slightly more natural than A  
- **B +2**: B is significantly more natural than A  

The annotation criteria for naturalness include the absence of robotic artifacts and disturbing noise, natural and expressive intonation, reasonable speaking rhythm, and appropriate stress and prosody. The instructions explicitly state that small intelligibility errors should not dominate naturalness judgment, although severe intelligibility errors can reduce naturalness [2511.07931].

For agreement analysis, the five-level CMOS labels are collapsed into ternary outcomes: **A better**, **B better**, and **Tie**. The paper defines four sample-level agreement regimes:

1. **Full Agreement (FA)**: all annotators choose the same polarity  
2. **Weak Agreement (WA)**: two annotators share a polarity and the third chooses Tie  
3. **Weak Disagreement (WD)**: two annotators share a polarity and the third chooses the opposite polarity  
4. **Full Disagreement (FD)**: one annotator chooses each of A, B, and Tie  

For naturalness, about **51.5%** of samples are FA and about **17.2%** are WA, so roughly **70%** are FA or WA. The expressive subset shows lower agreement than the regular subset, indicating that expressive naturalness is harder to judge consistently [2511.07931].

## 4. Derived subsets and benchmark formation

SpeechJudge-Data is released conceptually as a raw corpus plus several processed subsets created for different analytical and modeling purposes [2511.07931].

| Subset | Construction | Size |
|---|---|---:|
| SpeechJudge-Data (raw) | 5-scale CMOS, multi-annotator corpus | 99K |
| SpeechJudge-Data (pref) | Majority vote; Tie pairs discarded | 79K |
| SpeechJudge-Data (hq) | `pref` filtered by absolute WER gap < 12% | 44K |
| SpeechJudge-Data (train) | Remaining FA/WA/WD preference pairs after benchmark/dev sampling | ~42K |
| SpeechJudge-Data (dev) | Stratified sample from `hq` | 1,000 |
| SpeechJudge-Eval | Stratified FA-only benchmark from `hq` | 1,000 |

The transformation from raw labels to these subsets proceeds in stages. First, multi-annotator CMOS labels are aggregated by majority vote into a ternary preference; discarding ties yields **SpeechJudge-Data (pref)** with **79K** A-vs-B pairs. Next, the authors filter pairs using ASR-derived WER, retaining only cases with **absolute WER difference < 12%** so that intelligibility asymmetries do not swamp naturalness judgments; this yields **SpeechJudge-Data (hq)** with **44K** pairs [2511.07931].

From the high-quality set, the authors perform stratified sampling over **reference style** and **target text language** to create **SpeechJudge-Eval**, a **1,000-pair** naturalness benchmark. SpeechJudge-Eval uses only **no-Tie**, **Full Agreement** samples. Its composition includes **regular** and **expressive** conditions, with target languages spanning **en**, **zh**, and **mixed** [2511.07931].

The benchmark task is binary naturalness judgment: given $(t,a_1,a_2)$, a model must predict which audio is more natural. Accuracy is defined by comparison to the human label:
\[
\mathrm{Accuracy}=\frac{1}{|\mathcal{D}|}\sum_{d=0}^{|\mathcal{D}|}\mathbb{I}(y_{\mathcal{M}}=y_{\mathcal{H}}).
\]
Because SpeechJudge-Eval is derived from high-agreement, WER-filtered pairs generated by advanced zero-shot TTS systems, it is meant to isolate naturalness rather than intelligibility or gross synthesis failure [2511.07931].

## 5. Empirical role in naturalness evaluation and reward modeling

SpeechJudge-Data functions both as a data source for benchmarking and as training material for learned evaluators. On SpeechJudge-Eval, conventional metrics and existing AudioLLMs perform substantially below full human agreement. Reported total accuracies are **57.9%** for WER, **44.5%** for SIM, **48.6%** for FAD, **57.9%** for DNSMOS, **53.7%** for UTMOS, **60.8%** for the best Meta Audiobox aesthetics predictor (CE), **46.7%** for AASIST, and **38.3%** for ADV. Among open-source AudioLLMs, **Kimi-Audio-7B-Instruct** reaches **67.0%** and **Qwen2.5-Omni-7B** reaches **60.6%**. Among closed-source models, **Gemini-2.5-Flash** is the best baseline at **69.1%**, with **GPT-4o Audio** at **67.4%** and **Gemini-2.5-Pro** at **66.5%** [2511.07931].

These benchmark results motivate model training on SpeechJudge-Data itself. The paper derives **SpeechJudge-Data (train)** with **42K** preference pairs for reward-model learning. Two models are trained on this set: a classical **SpeechJudge-BTRM** using a Bradley–Terry objective over scalar rewards, and **SpeechJudge-GRM**, a generative reward model built on **Qwen2.5-Omni-7B (Thinker)** [2511.07931].

SpeechJudge-GRM uses a **two-stage post-training** pipeline. In the **SFT** stage, the authors prompt **Gemini-2.5-Flash** with a chain-of-thought naturalness-evaluation instruction. If Gemini’s predicted preference matches the human label, the resulting reasoning and output are retained as an SFT example. Out of the 42K training samples, **25K** such cases are used for SFT. The remaining **17K** challenging cases, where Gemini disagrees with human judgment, are reserved for **RL**. In the RL stage, the policy generates multiple rollouts per prompt and is updated with **GRPO/DAPO** using a verifiable reward of **$+1$** for matching the human preference and **$-1$** otherwise [2511.07931].

The resulting gains on SpeechJudge-Eval are substantial. **SpeechJudge-BTRM** achieves **72.7%** total accuracy, exceeding the strongest zero-shot baseline. **SpeechJudge-GRM (SFT)** reaches **75.3%**; **SpeechJudge-GRM (SFT+RL)** reaches **77.2%**; and **Voting@10** further increases this to **79.4%**. The same paper also uses SpeechJudge-Data and SpeechJudge-GRM in TTS post-training with **Qwen2.5-0.5B-TTS**, reporting that methods leveraging SpeechJudge-GRM yield larger gains in **naturalness CMOS (N-CMOS)**, with **w/ SpeechJudge-GRM (online)** achieving the best **N-CMOS (+0.25 vs base)** while also improving WER and intelligibility relative to the base model [2511.07931].

## 6. Position within related research, limitations, and open directions

SpeechJudge-Data sits at the intersection of two broader research trajectories. The first is the movement from scalar MOS supervision to **pairwise human preference** in speech judgment. PrefSQA argues that forced-choice preferences reduce rater-scale variability and cross-test inconsistency, and shows that high-quality preference labels are critical for exposing real model differences; this provides a general statistical rationale for SpeechJudge-Data’s use of pairwise labels rather than relying only on pointwise MOS [2606.19597].

The second is the emergence of specialized “judge” datasets. In a different domain, ProJudgeBench and ProJudge-173k formalize a judge as a model that evaluates another model’s intermediate reasoning steps, outputting per-step correctness, error type, and explanation. SpeechJudge-Data differs fundamentally in target and label structure: it evaluates **pairwise perceptual naturalness** of synthesized speech, rather than **step-level reasoning validity** over multimodal scientific solutions. The contrast is useful because it highlights two distinct judge paradigms now appearing in foundation-model research: process evaluation on reasoning traces and preference evaluation on perceptual outputs [2503.06553].

Several limitations are explicit. Language coverage is restricted to **Chinese**, **English**, and **zh–en code-switching**. The corpus does not explicitly cover broader multilingual settings, real-world noisy conditions, conversational multi-speaker interactions, or long-form dialogue. Annotators come from a single region, and agreement, while often high, is not perfect; expressive speech is notably harder to judge. The authors also report that WER correlates with intelligibility in regular speech but that this correlation breaks down for expressive speech at high WER, while SIM and FAD align poorly with naturalness judgments. Finally, SpeechJudge-GRM may inherit biases from both the training data and the Gemini-2.5-Flash chain-of-thought teacher used during SFT [2511.07931].

The stated future directions are to extend the SpeechJudge framework beyond naturalness to other subjective dimensions such as **speaker similarity** and **emotional expressiveness**, to broaden language and condition coverage, and to improve AudioLLM-based judges and training strategies. This suggests a wider research program in which SpeechJudge-Data is not merely a standalone corpus but an initial substrate for multi-aspect speech preference modeling and preference-aligned speech generation [2511.07931].

Source: https://www.emergentmind.com/topics/speechjudge-data