CodecMOS-Accent: A MOS Benchmark of Resynthesized and TTS Speech from Neural Codecs Across English Accents

Abstract: We present the CodecMOS-Accent dataset, a mean opinion score (MOS) benchmark designed to evaluate neural audio codec (NAC) models and the LLM-based text-to-speech (TTS) models trained upon them, especially across non-standard speech like accented speech. The dataset comprises 4,000 codec resynthesis and TTS samples from 24 systems, featuring 32 speakers spanning ten accents. A large-scale subjective test was conducted to collect 19,600 annotations from 25 listeners across three dimensions: naturalness, speaker similarity, and accent similarity. This dataset does not only represent an up-to-date study of recent speech synthesis system performance but reveals insights including a tight relationship between speaker and accent similarity, the predictive power of objective metrics, and a perceptual bias when listeners share the same accent with the speaker. This dataset is expected to foster research on more human-centric evaluation for NAC and accented TTS.

• The paper introduces CodecMOS-Accent, a benchmark that evaluates naturalness, speaker similarity, and accent preservation in resynthesized and TTS speech.
• It employs a diverse dataset of 4,000 samples from 24 systems across 10 English accents, carefully balancing accent, gender, and duration.
• The study reveals discrepancies between objective metrics and human judgments, highlighting listener biases and hierarchical trait preservation in modern TTS models.

Authoritative Summary of "CodecMOS-Accent: A MOS Benchmark of Resynthesized and TTS Speech from Neural Codecs Across English Accents" (2603.14328)

Motivation and Dataset Construction

The paper addresses limitations in existing evaluation frameworks for neural audio codecs (NACs) and TTS systems, particularly the lack of comprehensive subjective assessment across accented speech. Prior benchmarks (e.g., DASB, Codec-SUPERB) have primarily focused on reconstruction quality and objective metrics, neglecting nuanced human perceptual evaluations—especially for speech exhibiting non-standard attributes such as accent. With advancements in LLM-based TTS models and in-context learning (ICL) capabilities enabling voice and accent cloning, systematic evaluation of these emergent properties remains underexplored.

The CodecMOS-Accent dataset was curated to fill this gap, comprising 4,000 samples from 24 open-source codec and TTS systems, using VCTK as a diverse source of 32 speakers spanning ten English accents. Five utterances per speaker were chosen and processed, with careful balancing of accent, gender, and duration. The dataset includes both codec resynthesis (compression/reconstruction) and voice cloning (text-driven generation imitating reference speech) samples.

Experimental Design and Evaluation Metrics

A large-scale mean opinion score (MOS) listening test was conducted, collecting 19,600 annotations from 25 listeners across three perceptual dimensions:

• Naturalness (S-NAT): Assessment of audio fidelity covering articulation, prosody, and noise.
• Speaker Similarity (S-SPK-SIM): Degree to which the system retains speaker identity relative to a reference.
• Accent Similarity (S-ACC-SIM): Ability of the system to replicate reference accent, irrespective of speaker identity.

Objective metrics were computed for cross-validation, including:

• O-WER: Word error rate via Whisper transcription.
• O-SPK-SIM: ECAPA-TDNN speaker embedding cosine similarity.
• O-ACC-SIM: ECAPA-TDNN accent embedding cosine similarity from CommonAccent-pretrained models.
• O-UTMOS: Predicted MOS via UTMOS, a neural speech quality predictor.

System-Level Score Analysis

In system-level aggregated scores, the ground truth samples ranked below multiple state-of-the-art TTS systems in naturalness, confirming known limitations of VCTK’s inherent acoustic artifacts. However, ground truth maintained the top rank in speaker and accent similarity.

A salient finding is the discrepancy between naturalness and identity/trait preservation: even with notable acoustic degradation (e.g., low-layer SpeechTokenizer), speaker and accent traits were preserved, directly challenging assumptions that low-level NAC layers encode only linguistic content. This reveals a hierarchical learning structure in modern speech models, wherein global identity attributes are captured earlier than high-fidelity synthesis.

Resynthesis samples exhibited performance upper-bounded by ground truth, consistent with expectations—while TTS models often surpassed ground truth in naturalness by selectively omitting environmental noise, focusing on essential speaker/accent information.

Objective Metric Predictiveness

Analysis of Pearson correlations demonstrated:

• Strong correlation between O-SPK-SIM and S-SPK-SIM (0.86) and between O-SPK-SIM and S-ACC-SIM (0.90), indicating speaker identification models are unexpectedly predictive of accent similarity.
• O-WER showed weak correlation with subjective scores, reinforcing the inadequacy of relying solely on transcription accuracy for speech quality assessment.
• High correlation (0.96) between O-UTMOS and S-NAT, despite UTMOS being trained on obsolete architectures, revealing persistent bottlenecks in fundamental speech quality despite algorithmic innovation.

The above findings establish the sufficiency of certain objective metrics for high-level trait evaluation but underscore the necessity of expanding training data (including accented samples) for next-generation SQA model robustness.

Listener Bias and Accent Effects

Statistical analysis revealed a "same-accent bias": listeners sharing the speaker’s accent consistently rate speaker and accent similarity higher, both for ground truth and synthesized speech ($p < 0.05$). This bias did not manifest for naturalness in ground truth, but emerged for synthesized samples, which the paper hypothesizes is attributable to training data bias favoring US English. Spearman rank correlations demonstrated moderate to strong inter-listener agreement across test accents, regardless of region.

These results reinforce the imperative for culturally and demographically balanced training and evaluation in speech synthesis, particularly as TTS models become increasingly deployed globally.

Implications and Future Directions

The insights offered by CodecMOS-Accent highlight several research directions:

• Refinement of SQA metrics: Incorporation of accent-diverse datasets into SQA predictor training (e.g., UTMOS) is necessary to improve alignment with human judgment.
• Direct training for accent identification: Objective evaluation models should leverage human labels to surpass current proxy-based approaches for accent similarity.
• Model evaluation protocols: Systematic perceptual benchmarking across accent, speaker traits, and environmental factors should become standard for NAC and TTS research.

The dataset will serve as a foundation for further research in accent-robust TTS, speaker and trait disentanglement, and the development of perceptually guided evaluation methods that reflect global deployment realities.

Conclusion

CodecMOS-Accent establishes a comprehensive MOS benchmark for evaluating resynthesized and TTS speech from NACs across multiple English accents, offering rigorous perceptual and objective analysis. Its findings elucidate modeling hierarchies in trait preservation, confirm substantial listener biases linked to accent familiarity, and critically assess the validity of existing objective speech quality metrics. The dataset is positioned to drive the development of accent-aware, human-centric SQA evaluation and training paradigms in speech synthesis research.