PMOS: Evaluating Prosody in TTS

• PMOS is a subjective metric that evaluates the perceptual diversity and quality of prosody in synthesized speech using human pairwise ratings.
• It involves controlled comparisons of speech variants focusing on pitch, rhythm, stress, and expressive timing to benchmark TTS systems.
• PMOS serves as a validation benchmark for objective measures like DS-WED, aiding in the assessment of prosody in zero-shot and cross-lingual contexts.

Prosody Mean Opinion Score (PMOS) is a subjective evaluation metric designed to quantify the perceptual diversity and quality of prosodic patterns in synthesized speech. It serves both as a standalone metric for capturing the human judgment of prosody—such as pitch, intonation, stress, rhythm, emotional expressiveness—and as a validation benchmark for newly developed objective prosody diversity and quality metrics. PMOS has become increasingly important with the progression of natural-sounding text-to-speech (TTS) technology, particularly in zero-shot and cross-lingual settings, where achieving varied, context-appropriate prosody is a key goal.

1. Definition and Rationale

PMOS—Prosody Mean Opinion Score—is a human-annotated, subjective measure formulated to assess the diversity and perceptual distinctiveness of prosody in synthetic speech (Yang et al., 24 Sep 2025). Unlike canonical MOS, which aggregates listeners’ ratings on overall naturalness or intelligibility, PMOS explicitly focuses on prosodic features: pitch contours, rhythm, stress, intonation, and expressive variability. PMOS is collected by presenting listeners with synthetic utterances (often as sets of samples generated from a TTS system for the same text and speaker), and requesting pairwise ratings of prosodic difference using a fixed-point Likert scale. A score of 1 reflects nearly identical prosody between samples; a score of 5 denotes clearly distinguishable prosodic variation. The principal motivation for PMOS is to bridge the well-documented gap between what acoustic metrics capture (typically, local or single-dimensional features) and what listeners actually perceive.

2. Collection Methodology and Benchmark Datasets

The ProsodyEval dataset (Yang et al., 24 Sep 2025) is the primary open-source resource designed around PMOS. It comprises 1,000 speech samples produced by seven major TTS systems, spanning diverse generative paradigms (autoregressive, non-autoregressive flow-matching, masked generative modeling), with 2,000 human-rated PMOS annotations. Twenty expert listeners performed exhaustive pairwise comparisons within sets of five utterances, rating each pair on prosodic distinctiveness. This dataset enables direct benchmarking of system-generated prosodic diversity and provides the ground-truth for correlation analysis with objective evaluation metrics.

PMOS can be collected using any protocol where multiple samples for the same text/speaker are synthesized; raters must be instructed to ignore intelligibility, text, and global audio quality, focusing only on perceived differences in manner of speaking (e.g., pitch, rhythm, stress, and expressive timing).

3. Relationship to Objective Metrics

PMOS is positioned as the principal ground-truth for validating objective measures of prosodic diversity and quality. Conventional metrics such as log F₀ RMSE and Mel Cepstral Distortion (MCD) are reference-dependent and assess isolated attributes (typically, frequency or spectral distortion), often yielding incomplete or weak correlation with human perception. For instance, alignment by dynamic time warping (DTW) can obscure expressive variation and correlate poorly with listener judgments (Yang et al., 24 Sep 2025).

To address these shortcomings, the Discretized Speech Weighted Edit Distance (DS-WED) metric was introduced (Yang et al., 24 Sep 2025). DS-WED computes a weighted Levenshtein distance between tokenized representations of speech derived from self-supervised models. The formula is: $$\text{DS-WED}(c_1, c_2) = \min_{\pi \in \mathcal{A}(c_1, c_2)} \sum_{(i, j, o) \in \pi} w_o \cdot c_o(c_{1,i}, c_{2,j})$$ where $\mathcal{A}(c_1, c_2)$ is the set of alignments, $o$ is the edit operation (substitution, insertion, deletion), $c_o$ is the operation-specific cost, and $w_o$ is the operation weight (with substitution typically set to 1.2 for higher perceptual impact, insertions/deletions at 1).

Extensive experiments show that DS-WED demonstrates much higher Pearson correlation with PMOS than previous acoustic metrics, suggesting that it more closely reflects human-perceived prosodic diversity.

4. Applications in TTS Evaluation and System Diagnostics

By leveraging PMOS and ProsodyEval, researchers systematically benchmark TTS systems on their ability to generate prosodic diversity. Comparative analyses reveal that autoregressive TTS models generally exhibit more expressive prosody than flow-matching non-autoregressive models, though certain masked generative models can match or surpass AR systems (Yang et al., 24 Sep 2025). Inclusion of PMOS as a metric makes it possible to:

• Quantify the diversity of expressive speaking styles (crucial for applications like audiobook narration, virtual assistants, or dialog systems).
• Diagnose over-regularization introduced by reinforcement learning approaches such as Direct Preference Optimization (DPO), which can systematically reduce prosodic variability in pursuit of higher intelligibility (Yang et al., 24 Sep 2025).

PMOS also serves as a target metric for both supervised and RL-based fine-tuning in speech synthesis, especially in frameworks designed for style transfer, emotion conversion, or expressive speech generation.

5. Methodological Considerations: Statistical Handling and Bias

Given that PMOS (like conventional MOS) is an average of subjective listener ratings, statistical issues—such as non-independence, rater bias, and over-interpretation of small numerical differences—are prominent. Recent work (Naderi et al., 2020) demonstrates that direct application of rank-based statistical methods to averaged opinion scores (whether MOS or PMOS) can yield misleading conclusions due to the noise and small differences intrinsic to subjective ratings. A transformation method is recommended: treat two MOS/PMOS values as statistically tied if at least one lies within the other's 95% confidence interval, with grouping and rounding to avoid spurious discrimination. Open-source implementations are available to automate this process for robust statistical inference (Naderi et al., 2020).

6. Limitations and Future Directions

Current PMOS-based evaluation protocols are limited by linguistic coverage (e.g., ProsodyEval covers English only (Yang et al., 24 Sep 2025)), scalability (due to human annotation requirements), and inherent subjectivity, which may introduce rater variance and dataset-specific effects. Moreover, cross-lingual PMOS validity and transferability to tonal languages are open research challenges (Yang et al., 24 Sep 2025).

Advancing objective prosody evaluation remains an active area. Metrics such as DS-WED show promise but require further adaptation for robustness across languages and TTS paradigms. Large audio LLMs are not yet sufficiently sensitive to prosodic nuances to serve as effective standalone evaluators (Yang et al., 24 Sep 2025). Combining objective and subjective PMOS-driven evaluations should foster more sensitive, ecologically valid, and scalable metrics for future expressive TTS research.

7. Summary Table: Core Properties of PMOS vs. Acoustic Metrics

Metric	Captures Human Perceptual Judgments?	Reference-Free	Measures Multi-dimensional Prosody?
PMOS	Yes	Yes	Yes
F₀ RMSE	No	No	No (Pitch Only)
MCD	No	No	No (Spectral Only)
DS-WED	Indirectly (Validated on PMOS)	Yes	Yes (via token sequences)

PMOS establishes a critical benchmark for prosodic diversity and quality, validating and challenging objective evaluation methods, guiding the evolution of expressive, human-like TTS generation.