PASQA: Pitch-Accent-Focused Speech Quality Assessment Model Trained on Synthetic Speech with Accent Errors

Abstract: Existing mean opinion score (MOS) prediction models typically predict utterance-level naturalness MOS and can be insensitive to localized pitch-accent errors. We propose Pitch-Accent-focused Speech Quality Assessment (PASQA), which explicitly targets pitch-accent correctness. To train our model, we construct a controlled Japanese accent-error dataset by changing accent patterns using an accent-controllable text-to-speech system, and compute a pseudo accent-quality score from the accent-error rate. PASQA builds on self-supervised representations and employs mora-conditioned fusion, ranking loss, an auxiliary accent-error localization task, and speaker-invariant training. Experiments show that conventional models fail to preserve the ordering by accent-error severity, whereas PASQA achieves high ordering accuracy on both seen and unseen speakers. Further, PASQA shows stronger agreement with human accent-correctness judgments. The code is available at https://github.com/lycorp-jp/PASQA.

• The paper introduces PASQA, a novel model that integrates self-supervised learning with mora-conditioned fusion and ranking loss to detect localized pitch-accent errors.
• It details a multi-component architecture including an auxiliary frame-level error head and speaker-invariant adversarial training to enhance error detection explainability and robustness.
• Experimental results demonstrate that PASQA outperforms standard MOS predictors with order accuracy up to 0.785 and strong correlations with human judgments in accent-sensitive evaluations.

PASQA: Pitch-Accent-Focused Speech Quality Assessment Model for Japanese Synthetic Speech

Problem Context and Motivation

Standard utterance-level MOS prediction models frequently fail to detect localized prosodic degradations, particularly pitch-accent errors crucial for intelligibility and lexical disambiguation in pitch-accent languages such as Japanese. In modern DNN-based TTS pipelines, accent-related intermediate representations are often inaccessible or absent, necessitating direct, non-intrusive assessment from the speech signal. Previous frame-level explainability approaches do not explicitly address prosodic correctness and are limited in applicability to accent-sensitive languages.

Data Construction for Accent Error Assessment

The cornerstone of this work is a controlled Japanese accent-error dataset generated using an accent-controllable TTS system. By systematically manipulating accent nuclei within accent phrases (drawing from automatically derived prosodic labels based on morphological analysis and DNN-based prediction), sentences are synthesized at multiple error rates, providing fine-grained supervision for learning pitch-accent correctness. Each sample is assigned a pseudo accent-quality score, constructed as a monotonic function of the mora-level accent error proportion, enabling supervised training without labor-intensive manual labeling.

PASQA Model Architecture

PASQA builds upon the SSL-MOS backbone with wav2vec2-based self-supervised acoustic representations. The architecture is augmented with several components designed to enhance sensitivity to pitch-accent correctness:

1. Mora-conditioned fusion: Explicit linguistic information is derived from input text, tokenized into mora sequences, embedded, and contextualized via Transformer encoding, then fused into acoustic features via cross-attention. This enables precise modeling of accent placement in the speech signal.
2. Ranking loss (Bradley–Terry pairwise logistic loss): Training emphasizes ordinal relations among accent-quality scores, penalizing incorrect ordering on severity-controlled triplets.
3. Auxiliary frame-level error head: Provides localized supervision by predicting which frames correspond to phrases with manipulated nuclei, improving both error detection explainability and utterance-level score accuracy.
4. Speaker-invariant training via scheduled GRL: Adversarial training mitigates speaker identity confounds, focusing model representations toward prosodic, not speaker-specific, cues.

   Figure 1: The PASQA model architecture integrates mora-conditioned fusion, ranking loss, frame-level error detection, and speaker-independent adversarial learning.

Experimental Results

The objective evaluation on both seen and unseen speakers demonstrates that conventional non-intrusive MOS predictors (DNSMOS P.835/P.808, NISQA, SHEET SSL-MOS, UTMOS/UTMOSv2) exhibit near-chance order accuracy and negligible correlation between their predicted scores and accent error severity. In stark contrast, PASQA achieves substantially higher order accuracy and correlation across evaluation metrics. Specifically, PASQA reaches order accuracy of up to 0.785 (unseen speakers), LCC of 0.879, SRCC of 0.751, and KTAU of 0.559, outperforming ACC-WORLD-MOS and baseline SSL-MOS even when trained on the same dataset.

Ablation experiments indicate that removing mora-conditioned fusion, ranking loss, the frame-level error head, or GRL each results in reduced accuracy and correlation, confirming the complementary contribution of each architectural enhancement.

In subjective listening tests (15 native speakers), PASQA achieves SRCC of 0.828 and KTAU of 0.614 with human judgments of accent-correctness, outperforming alternative predictors. The model remains robust and discriminative in OOD evaluation with GPT-4o-mini-TTS outputs, exhibiting statistically significant pairwise accuracy (0.780, $p < 0.001$) where conventional predictors fail.

Implications and Theoretical Significance

PASQA establishes that explicit modeling of accent errors is crucial for speech quality assessment in pitch-accent languages. The integration of mora-conditioned linguistic context and frame-level error detection enables accurate, explainable, and robust assessment of localized prosodic degradations. The application of speaker-invariant adversarial training prevents confounding by speaker characteristics, enhancing generalization to unseen speakers and OOD TTS models.

Practically, PASQA facilitates scalable, non-intrusive quality evaluation of Japanese synthetic speech for both research benchmarking and commercial deployments, without reliance on manual prosodic annotation or intermediate accent labels in proprietary TTS systems. The methodology is amenable to extension toward other prosody-sensitive languages and can be adapted for assessment of additional localized speech errors.

Future Prospects and Directions

Potential developments include:

• Extending PASQA’s framework to multilingual prosodic assessment, including tone languages and systems with complex prosodic structures.
• Improving robustness and calibration for absolute score prediction under diverse TTS models and speaker populations.
• Integrating PASQA for real-time quality monitoring, voice cloning validation, and generative evaluation in large-scale production TTS applications.
• Augmenting PASQA training with both synthetic and real-world spontaneous speech, improving domain adaptability and coverage.

Conclusion

PASQA delivers a technically advanced, pitch-accent-sensitive speech quality assessment pipeline, combining self-supervised representations, linguistically informed fusion, ranking-based training, frame-level error localization, and speaker-adversarial techniques. The model consistently outperforms conventional predictors in both objective and subjective evaluation, establishing the necessity of prosody-focused assessment for Japanese synthetic speech and providing a rigorous template for future quality evaluation in prosodically complex languages.