TTSDS2: Evaluating Synthetic Speech Quality

• TTSDS2 is a distribution-based metric that evaluates synthetic speech by comparing key perceptual factors such as prosody and speaker identity.
• It employs the 2-Wasserstein distance to quantify similarity between synthetic and real speech features across various languages and conditions.
• Validated against over 11,000 MOS ratings, TTSDS2 offers reliable benchmarking for TTS system ranking, model tuning, and deepfake detection.

Text to Speech Distribution Score 2 (TTSDS2) is a distribution-based objective metric and benchmarking framework for evaluating the quality of synthetic speech produced by modern text-to-speech (TTS) systems. TTSDS2 extends upon earlier efforts (notably TTSDS) to provide robust, perceptually meaningful, and language-agnostic quantification of how closely synthetic speech matches human speech across key factors such as prosody, speaker identity, intelligibility, and general naturalness. TTSDS2 emerged in response to limitations of both subjective evaluation (e.g., MOS) and traditional objective metrics, providing a reliable method that correlates strongly with human perception even as synthetic audio increasingly becomes indistinguishable from real recordings (2506.19441).

1. Conceptual Overview and Motivation

Traditional assessments of TTS systems relied heavily on subjective listening tests—such as Mean Opinion Score (MOS)—and objective metrics like PESQ and STOI. However, these metrics are challenged by the remarkable progress of recent neural TTS architectures, whose outputs often equal or surpass the quality of ground-truth speech, making stable and reproducible cross-system comparisons difficult (2407.12707, 2506.19441).

TTSDS2 was designed to address these challenges by:

• Framing TTS evaluation as a problem of distributional similarity between synthesized and real speech.
• Evaluating multiple perceptually critical factors (e.g., prosody, speaker identity, intelligibility) separately and aggregating them into an overall score.
• Ensuring domain and language robustness, vital for benchmarking systems across clean, noisy, spontaneous, and child speech, and for scaling to multilingual TTS.

2. Underlying Methodology

TTSDS2 operates by extracting feature representations associated with several distinct perceptual factors. For each factor, features are computed for both synthetic and reference (real) datasets. The empirical distributions of these features are then compared using the 2-Wasserstein (Earth Mover’s) distance, yielding a score that reflects how closely the synthetic speech matches “real” speech, as opposed to noise or distractor data (2506.19441, 2407.12707).

Scoring Formula

For each feature $X$, TTSDS2 computes:

• $W_{\text{real}}(X)$: The minimum 2-Wasserstein distance between synthetic and real speech.
• $W_{\text{noise}}(X)$: The minimum 2-Wasserstein distance between synthetic and noise/distractor data.

The per-feature score is given by:

$$S(X) = 100 \times \frac{W_{\text{noise}}(X)}{W_{\text{real}}(X) + W_{\text{noise}}(X)}$$

A score $S(X) > 50$ means the synthetic speech distribution is closer to real speech than to noise for that feature.

Scores for each factor are computed by averaging feature-level scores, and the overall TTSDS2 score is the (typically unweighted) average across factors (2506.19441, 2407.12707).

Perceptual Factors

TTSDS2 evaluates the following major factors:

• Generic properties: SSL-based representations (e.g., wav2vec 2.0, Hubert) capturing holistic characteristics.
• Speaker: Embeddings (e.g., d-vectors, WeSpeaker) reflecting speaker identity similarity.
• Prosody: Features such as F0 (pitch), token durations, and SSL-based prosody vectors.
• Intelligibility: WERs using multiple ASR backends (wav2vec 2.0, Whisper).
• Environment: Noise/distortion measures, e.g., PESQ (post-denoising), SNR estimates.

Each factor is captured via multiple features, leading to a fine-grained multi-dimensional assessment.

3. Benchmarking and Correlation with Human Judgement

TTSDS2 has been validated against a comprehensive set of subjective opinion scores (over 11,000 MOS/CMOS/SMOS ratings) collected from listening tests across clean, noisy, spontaneous (“wild”), and children’s speech domains, as well as across 14 languages (2506.19441).

Among 16 evaluated objective and “MOS prediction” metrics, TTSDS2 was reported as the only one to achieve Spearman correlations exceeding 0.50 across all evaluated domains and subjective metrics, with an average correlation of approximately 0.67 (2506.19441). Benchmarking experiments covering 20 TTS systems confirm that TTSDS2 reliably tracks system ranking as perceived by human listeners.

Notably, as synthetic speech approaches or surpasses ground-truth audio in subjective ratings, traditional MOS predictors become increasingly unstable, while TTSDS2 continues to provide fine-grained distinctions due to its distributional foundation and multi-factor structure (2407.12707, 2506.19441).

4. Practical Construction: Datasets and Pipelines

TTSDS2 is supported by public resources to ensure reproducibility and scalability:

• A curated dataset with over 11,000 subjective ratings, spanning multiple languages, domains, and system architectures.
• A pipeline for automatically creating and refreshing a multilingual test set (via data scraping, topic filtering, speaker diarization, and language identification), reducing the risk of data leakage or stale benchmarking.
• Open benchmarks including more than 20 TTS systems, facilitating system comparison and longitudinal trend analysis (2506.19441).

This infrastructure enables continual, comparable evaluation as new TTS technologies or languages emerge.

5. Applications and Impact

TTSDS2 is especially suited for:

• Cross-system benchmarking: Enabling objective comparisons between TTS systems in both monolingual and multilingual settings.
• Model selection and ablation: Providing feedback on incremental model changes during research and development pipelines.
• Security/risk assessment: Quantifying the degree to which synthetic speech matches real human speech, relevant for anti-spoofing and deepfake detection.
• Low-resource and spontaneous domains: Maintaining robustness in challenging, noisy, or spontaneous spoken data scenarios (2506.19441).

Additionally, because TTSDS2 captures the one-to-many nature of speech (e.g., capturing variability in prosodic or speaker features rather than assessing only means), it is less prone to mask oversmoothing or overfitting in system outputs.

6. Limitations and Future Directions

While TTSDS2 signifies an advance in TTS evaluation, several areas for further development are recognized:

• Computational cost: The need for extensive feature extraction and metric computation may be substantial, especially at scale (2506.19441).
• Coverage of long-form/conversational speech: Current implementation typically assesses utterances of 3–30 seconds; future extensions may address longer or more interactive samples.
• Enhanced feature sets: Integration of novel representations (e.g., more advanced self-supervised or neural prosody models) may further strengthen sensitivity.
• Failure case detection: Augmenting TTSDS2 to guard against adversarial or degenerate systems (e.g., those maximizing text faithfulness without natural prosody) is a prospective area of research.
• Refined weighting: Empirical investigation into non-uniform (factor-weighted) aggregation could help reflect evolving priorities as new systems reach parity in traditional quality dimensions.

A plausible implication is that as TTS systems continue to advance, distributional metrics such as TTSDS2—backed by transparent, continually refreshed benchmarks—will serve as a principal means for both fair evaluation and the guiding of next-generation speech synthesis research (2506.19441, 2407.12707).

7. Summary Table: Key Features of TTSDS2

Aspect	Description	Reference
Score Type	Distribution-based, multi-factor, 2-Wasserstein distance to real/noise data	(2506.19441)
Factors	Generic, Speaker, Prosody, Intelligibility, Environment	(2407.12707)
Validation	Robust Spearman correlation (>0.50) with subjective ratings in all conditions	(2506.19441)
Resources	11K+ ratings, multilingual benchmark, continuous dataset and pipeline	(2506.19441)
Applications	System ranking, cross-language evaluation, security/risk assessment, model tuning	(2506.19441)

TTSDS2 thus constitutes a rigorously validated, transparent, and extensible metric for the comparative assessment of human-quality TTS. Its multi-factor, distributional approach underpins its utility across languages, domains, and the advancing state-of-the-art in speech synthesis.