TTSDS: Objective TTS Quality Metric

• TTSDS is a multi-factor, reference-free metric that quantifies distributional similarity between synthetic and actual speech using optimal transport distances.
• It decomposes evaluation into prosody, speaker identity, and intelligibility by extracting empirical feature distributions and aggregating normalized scores into a single interpretable value.
• Empirical validation across 35 TTS systems shows TTSDS achieves robust, era-agnostic correlation with human judgments, outperforming traditional MOS-based metrics.

TTSDS is an objective, reference-free evaluation metric for modern Text-to-Speech (TTS) systems. Its principal design is to quantify the distributional similarity between synthetic and real speech across multiple orthogonal perceptual factors—primarily prosody, speaker identity, and intelligibility—using optimal transport distances. TTSDS directly addresses the growing limitations of single-dimension metrics and MOS-prediction approaches when synthetic speech quality approaches that of human speech. Its aggregation scheme and empirical validation demonstrate robust, era-agnostic correlation with human perceptual judgments (Minixhofer et al., 2024).

1. Motivation and Theoretical Foundation

Traditional evaluation of TTS systems, such as Mean Opinion Score (MOS) or MOS-predictor neural networks, has become increasingly unreliable as modern TTS approaches produce audio nearly indistinguishable from real speech. MOS and similar metrics are difficult to compare across studies or over historical time, and single-factor metrics (e.g., WER, MCD, pitch distance) fail to measure overall perceptual similarity.

TTSDS was developed to address these deficiencies by reframing synthetic speech quality assessment as a multi-factor, distributional distance problem. The goal is to compute, for each major perceptual factor, how closely the statistical distribution of features in synthetic speech matches that from real speech, and to aggregate these factor-wise distances into a single interpretable score. Comparison to noise (nuisance) feature distributions anchors the score and provides a diagnostic interpretation (Minixhofer et al., 2024).

2. Factorization and Feature Extraction

TTSDS decomposes TTS evaluation into three principal factors:

• Prosody: Encompassing rhythm, pitch contours, speaking rate, and expressive timing. Features extracted include frame-level pitch (from WORLD vocoder F₀), prosodic SSL representations (such as Masked Prosody Model, MPM), and utterance duration proxies (Hubert-token length).
• Speaker Identity: Captures how closely the synthetic speaker matches real human timbre; features include d-vector embeddings [Wan et al., 2018] and WeSpeaker embeddings [Wang et al., 2023].
• Intelligibility: Operationalized as ASR-derived Word Error Rate (WER), using systems such as wav2vec 2.0 and Whisper.

For each of these factors, empirical feature distributions $\hat P(X \mid D)$ are estimated from synthetic data ($D_\mathrm{syn}$) and multiple real speech datasets ($D_\mathrm{real}$), using 80–100 utterances per system for robust estimation (Minixhofer et al., 2024).

3. Distributional Distance Computation

The core technical step involves quantifying the distance between empirical feature distributions corresponding to a synthetic system and the closest available real and noise datasets. The principal distance is the 2-Wasserstein (optimal transport) distance:

• For 1D features (e.g., F₀, WER):

$$W_2\left(\hat P_1, \hat P_2\right) = \sqrt{\frac{1}{n} \sum_{i=1}^n (x_i - y_i)^2}$$

• For high-dimensional embeddings (e.g., d-vectors):

$$W_2\left(\hat P_1, \hat P_2\right) = \sqrt{ \|\mu_1-\mu_2\|^2 + \mathrm{Tr}\left( \Sigma_1 + \Sigma_2 - 2\left(\Sigma_2^{1/2}\Sigma_1\Sigma_2^{1/2}\right)^{1/2} \right) }$$

For each feature $X$:

1. $$W_\mathrm{real}(X) = \min_{D_\mathrm{real}} W_2(\hat P(X \mid D_\mathrm{syn}), \hat P(X \mid D_\mathrm{real}))$$
2. $$W_\mathrm{noise}(X) = \min_{D_\mathrm{noise}} W_2(\hat P(X \mid D_\mathrm{syn}), \hat P(X \mid D_\mathrm{noise}))$$

Feature-level scores are normalized:

$$S_X = 100 \times \frac{W_\mathrm{real}(X)}{W_\mathrm{real}(X) + W_\mathrm{noise}(X)}$$

Each factor score is calculated as the unweighted average over its constituent feature scores (Minixhofer et al., 2024).

4. Aggregation and Interpretation

The final TTSDS is computed as the unweighted mean of the three main factor scores (prosody, speaker identity, intelligibility):

$$\mathrm{TTSDS} = \frac{1}{3}(S_\mathrm{prosody} + S_\mathrm{identity} + S_\mathrm{intelligibility})$$

Equal weighting is motivated by the orthogonality and independent perceptual saliency of the three factors. This aggregation strategy produces a scalar objective score in $D_\mathrm{syn}$0, directly interpretable as the proximity of synthetic speech to real speech across orthogonal perceptual axes. Factor-wise scores also serve as diagnostic indicators for targeted system improvement (Minixhofer et al., 2024).

5. Empirical Validation and Correlation with Human Judgement

TTSDS was benchmarked on 35 TTS systems across three historical eras: Blizzard’08 audiobook systems, the “Back-to-the-Future” set (hybrid/HMM vs. neural), and the “TTS Arena” of LLM-based recent methods. Scores were correlated with human MOS and Elo ratings. Results include:

Time Period	# Systems	Human Metric	TTSDS ρ	WVMOS ρ	UTMOS ρ
Blizzard '08	22	MOS	0.60	0.68	0.32
BTTF (~10)	MOS	0.72	0.05	0.85
TTS Arena	9	Elo ratings	0.83	0.10	0.23

TTSDS achieves consistent positive Spearman ρ (0.60–0.83) across all settings, indicating strong alignment with subjective preferences, outperforming contemporary MOS-predictor networks in era-robustness and reliability (Minixhofer et al., 2024).

6. Properties, Limitations, and Implications

TTSDS possesses several key properties:

• Reference-free: Requires no human-annotated references or manual listening.
• Diagnostics: Factor-level and feature-level results enable detailed analysis and targeted improvements.
• Non-parametric: Applies directly to new architectures and datasets without training.
• Era-agnostic generalization: Maintains correlation with human scores across multiple architectures and eras.

Limitations include the dependence on the quality and representativeness of real and noise reference datasets, and the inability of any single feature or factor to robustly track subjective quality—a combination is necessary. TTSDS does not directly account for transcript-fidelity, lexical accuracy, or long-form/discourse-level aspects, presenting open questions for metric extension. Its objective, reproducible design facilitates the diagnosis and development of high-quality TTS systems in the regime approaching human parity (Minixhofer et al., 2024).