Grounded SAM: A Robust TTS Evaluation Metric

• Grounded SAM is a composite metric that evaluates synthetic speech by comparing distributions of features like prosody, speaker identity, and intelligibility.
• It employs optimal transport distances such as the 2-Wasserstein distance to quantify similarity between synthetic, real, and noise-based speech feature distributions.
• Demonstrating robust correlation with human judgments across TTS systems, Grounded SAM provides diagnostic insights for targeted system improvements.

TTSDS (Text-to-Speech Distribution Score) is an objective, multi-factor evaluation metric for synthetic speech quality designed to address the shortcomings of traditional subjective and single-dimension metrics. Developed in response to advances in TTS (Text-to-Speech) systems that produce audio perceptually close to real speech, TTSDS employs distributional comparisons over various perceptual dimensions—primarily prosody, speaker identity, and intelligibility—using optimal transport distances between feature distributions extracted from both synthetic and real speech, as well as from noise baselines. It serves as a reference-free, non-parametric, and diagnostic score that demonstrates robust correlation with human judgments across multiple system generations and evaluation periods (Minixhofer et al., 2024).

1. Motivation and Limitations of Existing Metrics

Existing TTS evaluation methodologies, such as Mean Opinion Score (MOS) and MOS-prediction networks, show diminishing reliability as synthetic and real speech converge in quality. Specifically, MOS values become hard to interpret or compare across studies and time periods, and their predictive models lack generalizability. Single-dimension, task-specific metrics (e.g., word error rate (WER), mel cepstral distortion (MCD), pitch distance) only capture isolated aspects of speech quality and miss holistic perceptual alignment. TTSDS was developed to provide a composite, distributional approach that evaluates how closely synthetic speech mirrors the statistical and perceptual properties of real speech across critical factors (Minixhofer et al., 2024).

2. Factorization and Feature Extraction

TTSDS decomposes speech quality into a set of orthogonal perceptual factors, each characterized by specific feature types:

• Prosody: Encompasses rhythm, pitch, speaking rate, and expressive timing.
  • Features: Frame-level pitch (obtained via WORLD vocoder), SSL-based prosody representations (e.g., Masked Prosody Model), and proxies for duration (Hubert-token length).
• Speaker Identity: Measures the correspondence of voice timbre to those of real speakers.
  • Features: d-vector embeddings, WeSpeaker embeddings.
• Intelligibility: Quantifies the ease of recognizing the verbal content.
  • Features: Word error rate (WER) as computed from ASR systems (e.g., wav2vec 2.0, Whisper).

For each system and factor, features $X$ are extracted from datasets $D$ (synthetic or real), yielding empirical distributions $\hat{P}(X\mid D)$ (Minixhofer et al., 2024).

3. Distributional Distance Computation

TTSDS evaluates the proximity of synthetic-speech feature distributions to those of both real speech and noise. The core methodology is as follows:

• For each extracted feature $X$ and synthetic dataset $D_{\text{syn}}$, calculate the 2-Wasserstein distance to each real-speech dataset $D_{\text{real}}$ and to noise baselines $D_{\text{noise}}$:

$$W_{\text{real}}(X) = \min_{D_\text{real}} W_2\bigl(\hat P(X|D_\text{syn}), \hat P(X|D_\text{real})\bigr)$$

$$W_{\text{noise}}(X) = \min_{D_\text{noise}} W_2\bigl(\hat P(X|D_\text{syn}), \hat P(X|D_\text{noise})\bigr)$$

• For 1D features, 2-Wasserstein distance is

$$W_2(\hat P_1, \hat P_2) = \sqrt{\frac{1}{n}\sum_{i=1}^n (x_i - y_i)^2}$$

• For higher-dimensional (Gaussian-assumed) features:

$D$0

• Feature similarity is normalized as:

$D$1

This process ensures rigorous, distribution-based comparison between systems (Minixhofer et al., 2024).

4. Aggregation and Final Scoring

Each factor's score is computed as the average of its constituent feature scores ($D$2), and the final TTSDS is the unweighted mean of the three core factors:

$D$3

This aggregation strategy enforces orthogonality between dimensions and avoids over-weighting any single perceptual aspect. The resulting score ranges between 0 (indistinguishable from noise) and 100 (empirically as close to real speech as possible) (Minixhofer et al., 2024).

5. Experimental Validation and Comparative Performance

TTSDS was evaluated on 35 TTS systems spanning three major eras: Blizzard'08, Back-to-the-Future (BTTF) hybrids, and TTS Arena (LLM-based systems). Human ratings were benchmarked via MOS or Elo scores, and TTSDS was compared against leading MOS-predictor networks (WVMOS, UTMOS). Spearman's rank correlation $D$4 with human scores was observed as follows:

Period	#Systems	Human Metric	TTSDS $D$5	WVMOS $D$6	UTMOS $D$7
Blizzard'08	22	MOS	0.60	0.68	0.32
BTTF	~10	MOS	0.72	0.05	0.85
TTS Arena	9	Elo rating	0.83	0.10	0.23

Unlike MOS predictors, TTSDS maintains consistency across eras and generalizes to novel architectures. Combining factor scores is essential for robust cross-system comparison, as individual factors are not sufficient (Minixhofer et al., 2024).

6. Diagnostic Properties, Implications, and Legacy

TTSDS is reference-free and non-parametric, requiring only a collection of real and noise reference corpora. It facilitates factor-specific analysis, enabling developers to isolate deficiencies (e.g., poor prosody, low intelligibility). Because it operates without human annotation, it significantly reduces resource constraints for rapid system prototyping while still providing actionable diagnostic output.

Key findings are:

• Strong alignment with human evaluation across multiple TTS system generations.
• Outperformance of SOTA MOS prediction on modern, highly natural TTS.
• The factor-wise breakdown offers targeted guidance for system improvement.
• The metric can be immediately applied to new architectures and datasets without retraining or tuning for specific systems (Minixhofer et al., 2024).

A plausible implication is that, as synthetic speech continues to reach human parity, distributional, factorized scores such as TTSDS will become essential for distinguishing subtle qualitative differences and for ensuring evaluation reproducibility across research groups.

7. Extensions, Successors, and Future Directions

TTSDS has established a methodological foundation for objective TTS evaluation. It has been extended by TTSDS2, which incorporates modifications for robustness, domain adaptation (e.g., "wild" YouTube speech, children’s speech), a broadened feature pool (e.g., WavLM, mHuBERT-147, XLSR-53), and the replacement of WER by "non-matching" ASR activations to improve intelligibility scoring. TTSDS2 is empirically validated on 20+ open-source systems across 14 languages, achieving higher and more stable correlations with subjective MOS, CMOS, and SMOS ratings in diverse domains (Minixhofer et al., 24 Jun 2025).

Future design recommendations include integration of lexical fidelity checking, contextual/long-form evaluation, and continual multilingual expansion facilitated by automatic data pipelines. Though it does not fully replace listening tests, TTSDS and its successors provide a reproducible, scientific standard for the next generation of TTS evaluation.