SpeechJudge-Data: TTS Naturalness Evaluation

Updated 4 July 2026

The paper introduces SpeechJudge-Data, a large-scale corpus that uses pairwise naturalness judgments alongside pointwise intelligibility labels to assess modern TTS systems.
It comprises 99K triplets generated from six zero-shot TTS models across multilingual and multi-style conditions, with an average of 2.49 annotations per pair.
The dataset supports reward modeling, preference alignment, and benchmark evaluation, achieving accuracy gains up to 79.4% on naturalness prediction tasks.

Searching arXiv for the specified papers and closely related work on speech judgment and process-evaluation datasets. SpeechJudge-Data is a large-scale human feedback corpus for synthesized speech that is designed for intelligibility judgment and, centrally, naturalness judgment. Its base unit is the triplet $\mathcal{D}=\{(t,a_1,a_2)\}$ , where $t$ is the target text and $a_1,a_2$ are two synthesized waveforms for the same text; annotators supply pointwise intelligibility labels for each waveform and a pairwise naturalness preference between them on a five-level Comparative MOS (CMOS) scale. Introduced as part of the broader SpeechJudge suite, the dataset is intended to support reward modeling, DPO-style preference alignment, RLHF for TTS, and the construction of a dedicated naturalness benchmark and evaluator models (Zhang et al., 11 Nov 2025).

1. Definition, motivation, and scope

SpeechJudge-Data was built to address a specific deficiency in speech synthesis evaluation: the absence of a large-scale human preference dataset focused on holistic naturalness for modern zero-shot TTS systems. The work positions naturalness as “one of the most fundamental subjective metrics for speech synthesis,” and distinguishes its objective from prior resources that either emphasize pointwise MOS or focus on narrower attributes such as low-level acoustic quality or intelligibility (Zhang et al., 11 Nov 2025).

The dataset is explicitly structured around pairwise comparison rather than scalar scoring alone. This design matches its intended downstream uses: reward modeling, pairwise preference learning, and alignment of speech generation models with human judgments. In related speech quality assessment research, pairwise preference prediction is motivated as less sensitive to rater variability, scale-use differences, and cross-study protocol effects than MOS-based supervision, which provides a broader methodological context for SpeechJudge-Data’s preference-centric construction (Fan et al., 17 Jun 2026).

SpeechJudge-Data is not a generic speech quality corpus. It is specifically centered on synthesized speech naturalness under modern zero-shot TTS conditions, with multilingual and multi-style coverage, and with simultaneous intelligibility annotation so that naturalness can later be disentangled from gross text-accuracy failures (Zhang et al., 11 Nov 2025).

2. Corpus construction and coverage

SpeechJudge-Data (raw) contains 99K triplets $(t,a_1,a_2)$ . Each pair receives 2–3 independent annotators, with an average of 2.49 annotations per pair. The paper estimates the market value of annotation at 500K RMB (~70K USD) (Zhang et al., 11 Nov 2025).

The waveform pairs are generated from six zero-shot TTS models spanning three architectural families:

AR-based: ARS, CosyVoice2, CosyVoice2-INTP, Ints-INTP
FM-based: F5-TTS
MGM-based: MaskGCT

The target texts cover Chinese (zh), English (en), and mixed code-switching (zh–en). Reference speech is divided into two broad style regimes. Regular speech references are sampled from Emilia-Large. Expressive speech references come from corpora and sources including ParaSpeechCaps, L2-Arctic, KeSpeech, a Whisper-style in-house corpus, and character voices from Genshin Impact (Zhang et al., 11 Nov 2025).

For each sample, a prompt $(a_{\mathrm{ref}}, t)$ is formed, where the reference waveform $a_{\mathrm{ref}}$ defines speaker, style, and prosody, and the target text $t$ defines content. A TTS model $\mathcal{M}_{\mathrm{tts}}$ then generates audio according to

$a_i \sim \mathcal{M}_{\mathrm{tts}}(a_{\mathrm{ref}}, t).$

Pairs may be intra-model, where both audios come from the same TTS system and differ due to sampling or decoding variability, or inter-model, where the two audios are generated by different systems (Zhang et al., 11 Nov 2025).

The language configuration is deliberately broader than simple monolingual synthesis. The corpus includes en2en and zh2zh settings, as well as cross-lingual and code-switching conditions such as zh2en, en2zh, zh2mixed, and en2mixed. For expressive references, DeepSeek-V3 is used to generate new scripts in multiple writing styles, returning monolingual continuations, a random continuation, a translation, and a code-switching version; for regular references, DeepSeek-V3 is used to clean ASR-like transcriptions by fixing typos and punctuation (Zhang et al., 11 Nov 2025).

This composition makes the corpus a study of naturalness under high-quality, often subtle, TTS comparisons rather than an evaluation of overtly degraded speech alone. A plausible implication is that the corpus is designed to stress judgment in regimes where differences are perceptually meaningful but not trivially captured by lexical correctness or simple acoustic heuristics.

3. Annotation protocol and label semantics

The annotation process uses a professional data annotation company in China. For Chinese samples, annotators are native Chinese speakers. For English and code-switching samples, the requirement is CET-6 or higher. The protocol includes standardized training based on a detailed manual, and a pilot study among researchers was used to refine the guidelines (Zhang et al., 11 Nov 2025).

Each sample is initially labeled by two annotators independently; if they disagree, a third annotator is introduced. Reliability is quantified at the annotator level using

$r_{ij}=\frac{1}{M-1}\sum_{x\neq i}\mathbb{I}[y_{xj}=y_{ij}], \qquad R_i=\frac{1}{N_i}\sum_{j=1}^{N_i} r_{ij}.$

The agreement distribution peaks at 60–70%, and one annotator with <30% agreement was removed (Zhang et al., 11 Nov 2025).

Annotators perform two distinct tasks.

First, they conduct pronunciation error detection (intelligibility) for each audio separately. Given the target text $t$ 0 and one waveform, they decide whether the audio accurately reads the text. The guidelines define intelligibility errors as omission, insertion, and substitution, including misread names, numbers, and word-order errors. This is recorded as a binary text-accuracy decision for each of $t$ 1 and $t$ 2 (Zhang et al., 11 Nov 2025).

Second, they perform naturalness comparison between the two audios using a 5-point CMOS scale:

A +2: A is significantly more natural than B
A +1: A is slightly more natural than B
Tie: similar naturalness, too close to judge
B +1: B is slightly more natural than A
B +2: B is significantly more natural than A

The annotation criteria for naturalness include the absence of robotic artifacts and disturbing noise, natural and expressive intonation, reasonable speaking rhythm, and appropriate stress and prosody. The instructions explicitly state that small intelligibility errors should not dominate naturalness judgment, although severe intelligibility errors can reduce naturalness (Zhang et al., 11 Nov 2025).

For agreement analysis, the five-level CMOS labels are collapsed into ternary outcomes: A better, B better, and Tie. The paper defines four sample-level agreement regimes:

Full Agreement (FA): all annotators choose the same polarity
Weak Agreement (WA): two annotators share a polarity and the third chooses Tie
Weak Disagreement (WD): two annotators share a polarity and the third chooses the opposite polarity
Full Disagreement (FD): one annotator chooses each of A, B, and Tie

For naturalness, about 51.5% of samples are FA and about 17.2% are WA, so roughly 70% are FA or WA. The expressive subset shows lower agreement than the regular subset, indicating that expressive naturalness is harder to judge consistently (Zhang et al., 11 Nov 2025).

4. Derived subsets and benchmark formation

SpeechJudge-Data is released conceptually as a raw corpus plus several processed subsets created for different analytical and modeling purposes (Zhang et al., 11 Nov 2025).

Subset	Construction	Size
SpeechJudge-Data (raw)	5-scale CMOS, multi-annotator corpus	99K
SpeechJudge-Data (pref)	Majority vote; Tie pairs discarded	79K
SpeechJudge-Data (hq)	`pref` filtered by absolute WER gap < 12%	44K
SpeechJudge-Data (train)	Remaining FA/WA/WD preference pairs after benchmark/dev sampling	~42K
SpeechJudge-Data (dev)	Stratified sample from `hq`	1,000
SpeechJudge-Eval	Stratified FA-only benchmark from `hq`	1,000

The transformation from raw labels to these subsets proceeds in stages. First, multi-annotator CMOS labels are aggregated by majority vote into a ternary preference; discarding ties yields SpeechJudge-Data (pref) with 79K A-vs-B pairs. Next, the authors filter pairs using ASR-derived WER, retaining only cases with absolute WER difference < 12% so that intelligibility asymmetries do not swamp naturalness judgments; this yields SpeechJudge-Data (hq) with 44K pairs (Zhang et al., 11 Nov 2025).

From the high-quality set, the authors perform stratified sampling over reference style and target text language to create SpeechJudge-Eval, a 1,000-pair naturalness benchmark. SpeechJudge-Eval uses only no-Tie, Full Agreement samples. Its composition includes regular and expressive conditions, with target languages spanning en, zh, and mixed (Zhang et al., 11 Nov 2025).

The benchmark task is binary naturalness judgment: given $t$ 3, a model must predict which audio is more natural. Accuracy is defined by comparison to the human label: $t$ 4 Because SpeechJudge-Eval is derived from high-agreement, WER-filtered pairs generated by advanced zero-shot TTS systems, it is meant to isolate naturalness rather than intelligibility or gross synthesis failure (Zhang et al., 11 Nov 2025).

5. Empirical role in naturalness evaluation and reward modeling

SpeechJudge-Data functions both as a data source for benchmarking and as training material for learned evaluators. On SpeechJudge-Eval, conventional metrics and existing AudioLLMs perform substantially below full human agreement. Reported total accuracies are 57.9% for WER, 44.5% for SIM, 48.6% for FAD, 57.9% for DNSMOS, 53.7% for UTMOS, 60.8% for the best Meta Audiobox aesthetics predictor (CE), 46.7% for AASIST, and 38.3% for ADV. Among open-source AudioLLMs, Kimi-Audio-7B-Instruct reaches 67.0% and Qwen2.5-Omni-7B reaches 60.6%. Among closed-source models, Gemini-2.5-Flash is the best baseline at 69.1%, with GPT-4o Audio at 67.4% and Gemini-2.5-Pro at 66.5% (Zhang et al., 11 Nov 2025).

These benchmark results motivate model training on SpeechJudge-Data itself. The paper derives SpeechJudge-Data (train) with 42K preference pairs for reward-model learning. Two models are trained on this set: a classical SpeechJudge-BTRM using a Bradley–Terry objective over scalar rewards, and SpeechJudge-GRM, a generative reward model built on Qwen2.5-Omni-7B (Thinker) (Zhang et al., 11 Nov 2025).

SpeechJudge-GRM uses a two-stage post-training pipeline. In the SFT stage, the authors prompt Gemini-2.5-Flash with a chain-of-thought naturalness-evaluation instruction. If Gemini’s predicted preference matches the human label, the resulting reasoning and output are retained as an SFT example. Out of the 42K training samples, 25K such cases are used for SFT. The remaining 17K challenging cases, where Gemini disagrees with human judgment, are reserved for RL. In the RL stage, the policy generates multiple rollouts per prompt and is updated with GRPO/DAPO using a verifiable reward of $t$ 5 for matching the human preference and $t$ 6 otherwise (Zhang et al., 11 Nov 2025).

The resulting gains on SpeechJudge-Eval are substantial. SpeechJudge-BTRM achieves 72.7% total accuracy, exceeding the strongest zero-shot baseline. SpeechJudge-GRM (SFT) reaches 75.3%; SpeechJudge-GRM (SFT+RL) reaches 77.2%; and Voting@10 further increases this to 79.4%. The same paper also uses SpeechJudge-Data and SpeechJudge-GRM in TTS post-training with Qwen2.5-0.5B-TTS, reporting that methods leveraging SpeechJudge-GRM yield larger gains in naturalness CMOS (N-CMOS), with w/ SpeechJudge-GRM (online) achieving the best N-CMOS (+0.25 vs base) while also improving WER and intelligibility relative to the base model (Zhang et al., 11 Nov 2025).

SpeechJudge-Data sits at the intersection of two broader research trajectories. The first is the movement from scalar MOS supervision to pairwise human preference in speech judgment. PrefSQA argues that forced-choice preferences reduce rater-scale variability and cross-test inconsistency, and shows that high-quality preference labels are critical for exposing real model differences; this provides a general statistical rationale for SpeechJudge-Data’s use of pairwise labels rather than relying only on pointwise MOS (Fan et al., 17 Jun 2026).

The second is the emergence of specialized “judge” datasets. In a different domain, ProJudgeBench and ProJudge-173k formalize a judge as a model that evaluates another model’s intermediate reasoning steps, outputting per-step correctness, error type, and explanation. SpeechJudge-Data differs fundamentally in target and label structure: it evaluates pairwise perceptual naturalness of synthesized speech, rather than step-level reasoning validity over multimodal scientific solutions. The contrast is useful because it highlights two distinct judge paradigms now appearing in foundation-model research: process evaluation on reasoning traces and preference evaluation on perceptual outputs (Ai et al., 9 Mar 2025).

Several limitations are explicit. Language coverage is restricted to Chinese, English, and zh–en code-switching. The corpus does not explicitly cover broader multilingual settings, real-world noisy conditions, conversational multi-speaker interactions, or long-form dialogue. Annotators come from a single region, and agreement, while often high, is not perfect; expressive speech is notably harder to judge. The authors also report that WER correlates with intelligibility in regular speech but that this correlation breaks down for expressive speech at high WER, while SIM and FAD align poorly with naturalness judgments. Finally, SpeechJudge-GRM may inherit biases from both the training data and the Gemini-2.5-Flash chain-of-thought teacher used during SFT (Zhang et al., 11 Nov 2025).

The stated future directions are to extend the SpeechJudge framework beyond naturalness to other subjective dimensions such as speaker similarity and emotional expressiveness, to broaden language and condition coverage, and to improve AudioLLM-based judges and training strategies. This suggests a wider research program in which SpeechJudge-Data is not merely a standalone corpus but an initial substrate for multi-aspect speech preference modeling and preference-aligned speech generation (Zhang et al., 11 Nov 2025).