Investigating Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbations

Abstract: Mean opinion score (MOS) prediction models are widely used as proxy metrics in text-to-speech (TTS) research, yet their ability to capture quality differences beyond acoustic fidelity remains unclear. We investigate this via controlled perturbations on speech: acoustic degradation, prosodic errors, and manipulation of speaker-specific characteristics such as pitch and speaking rate. We obtained MOS predictions for these speech samples from both human listeners and the model, and analyzed the differences in their perceptual characteristics. Results show that most models track acoustic degradation well, while all are insensitive to prosodic errors despite large subjective score drops. For speaker characteristics, models exhibit a double dissociation: strong mean fundamental frequency (F0) biases absent in human ratings, yet insensitivity to speaking rate and F0 variability that humans notice. These findings highlight limitations of scalar MOS prediction beyond acoustic fidelity.

• The paper shows that MOS models effectively assess acoustic degradations but fail to capture prosodic nuances critical for natural speech.
• Evaluation experiments demonstrate strong alignment between models and human ratings for signal-level errors while exposing insensitivity to prosodic and speaker characteristics.
• Results imply that enhancing MOS models requires specialized prosodic and speaker-centric supervision to better mirror native perceptual judgments.

Analysis of Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbations

Motivations and Problem Statement

Automatic speech quality assessment, primarily via mean opinion score (MOS) prediction models, is integral to modern TTS research. While current models provide efficient proxy metrics for large-scale evaluation, their capacity to capture subjective quality dimensions—especially beyond coarse acoustic fidelity—remains uncertain. Nuanced aspects such as prosodic naturalness, pitch-accent accuracy, and speaker-specific characteristics increasingly dominate perceptual quality differences in high-quality TTS output. The central question addressed is whether MOS prediction models align with the perceptual dimensions that guide human judgments, specifically under controlled perturbations in acoustics and prosody.

Experimental Framework and Methodology

The paper implements a systematic comparison between human listeners and state-of-the-art MOS prediction models—including SSL-MOS variants (SHEET-MB, SHEET-BV), UTMOS, UTMOSv2, NISQA, and DNSMOS—using the VERSA evaluation toolkit. Evaluation samples are divided into three groups:

• Group A: Controlled signal-level degradations (clipping, noise, compression) applied to natural Japanese speech;
• Group B: TTS-synthesized speech with explicit pitch-accent errors (flipped accent patterns);
• Group C: Speech samples manipulated along speaker dimensions (mean $F_0$, speaking rate, and $F_0$ variability), including both natural variation and artificial shifts.

Fifteen native Japanese listeners provided reference MOS ratings for all perturbations, while models were assessed for both utterance- and system-level ranking fidelity.

Model Sensitivity to Acoustic Degradation

Under signal-level degradations (Group A), most models demonstrated high system-level SRCC with human MOS, validating their robustness for acoustic fidelity assessment. Notably, architectures trained solely on TTS/VC data (BVCC) provided superior sensitivity to codec and noise artifacts compared to multi-domain models, underscoring the importance of training data composition. At the utterance level, models preserved intra-condition ordering but did not always match human differentiation across conditions.

Prosodic Error Indifference in Models

For prosodic accent errors (Group B), human listeners exhibited pronounced sensitivity—MOS declining by 1.84 points with increased accent swapping. Every MOS prediction model, irrespective of SSL backbone and training corpus, remained insensitive to prosodic manipulation, with negligible score variation across conditions. Altering training data composition alone was insufficient; explicit supervision targeting prosodic quality is absent in current MOS datasets. Language mismatch (lack of Japanese speech in training data) may contribute to insensitivity but does not explain the uniformity, as these models are commonly applied cross-linguistically.

Misaligned Sensitivity to Speaker Characteristics

Analysis of speaker-dependent characteristics (Group C) revealed a double dissociation:

• Mean $F_0$: Models demonstrated strong negative correlation (higher scores for lower-$F_0$ speakers), while human listeners showed no substantial association;
• Speaking Rate & $F_0$ Variability: Humans rated higher variability as more natural, with slower speaking rates penalized, yet models failed to encode these perceptual cues, displaying near-zero correlation.

SHEET-MB, influenced by inclusion of singing data in training (SingMOS), was a consistent outlier, exhibiting positive $F_0$ correlation and moderate speaking rate sensitivity.

Figure 1: Scatter plots of MOS against speaker characteristics (mean $\log F_0$, speaking rate) for Groups C-1--C-3.

Implications for MOS Modeling and Evaluation

The research establishes two major limitations in the prevailing scalar MOS modeling paradigm:

1. Incomplete Quality Representation: Models reliably quantify signal degradations but systematically fail at capturing prosodic appropriateness and speaker-specific perceptual cues critical for native listener judgments in pitch-accent languages.
2. Training Dataset Insufficiency: Merely extending training datasets does not resolve insensitivity to prosodic errors or idiosyncratic speaker attributes; specialized prosodic and speaker-centric supervision is required.

These findings challenge the routine use of MOS models as language-independent evaluation metrics in TTS research and suggest the need to shift towards multidimensional or compositional perceptual modeling frameworks.

Theoretical and Practical Outlook

From a theoretical perspective, scalar MOS fails to encode the multidimensionality inherent in human perceptual quality assessment. This is aggravated in languages with complex prosodic structure (e.g., Japanese), where accentual errors hold lexical consequences. Practically, objective metrics used for benchmarking TTS, VC, and speech enhancement systems may obscure real-world perceptual deficits, particularly when prosodic appropriateness is critical.

Future developments should focus on:

• Incorporation of explicit prosodic and speaker-centric supervision in MOS datasets;
• Development of multidimensional perceptual quality models that account for acoustic, prosodic, and idiosyncratic speaker factors;
• Benchmarking frameworks that utilize frame-level or multidimensional targets to better mimic human evaluation paradigms.

Conclusion

Controlled experiments reveal that current MOS prediction models are highly effective for signal-level acoustic quality assessment but fundamentally deficient in capturing prosodic and speaker-related quality dimensions. Their divergence from human judgments underscores a gap between model sensitivity and multidimensional perceptual evaluation. Addressing these limitations will require rethinking both dataset composition and modeling strategies in automated speech quality assessment to achieve closer alignment with human perceptual structure.