---
title: 'CodecMOS-Accent: MOS Benchmark for Accented Speech'
url: https://www.emergentmind.com/topics/codecmos-accent-dataset
type: topic
---

# CodecMOS-Accent: MOS Benchmark for Accented Speech

Searching arXiv for the specified paper and adjacent work to ensure citations are current and accurate.
CodecMOS-Accent is a mean opinion score benchmark for evaluating neural audio codec resynthesis systems and codec-based text-to-speech systems on accented English speech. It is designed around a gap identified in codec evaluation: prior benchmarks had largely emphasized reconstruction fidelity and objective downstream metrics, while providing limited perceptual evidence about system behavior on non-standard speech, especially accented speech [2603.14328]. The benchmark therefore centers human judgment rather than purely automatic measures, and operationalizes evaluation along three dimensions—naturalness, speaker similarity, and accent similarity—over a collection of resynthesized and synthesized speech produced from multiple contemporary systems [2603.14328]. In the available literature, CodecMOS-Accent is distinct from CodecBench, which does not explicitly include or mention a dataset by that name and does not present an accent-focused MOS subset [2508.20660].

## 1. Definition and scope

CodecMOS-Accent is explicitly framed as a MOS benchmark for neural audio codecs and LLM-based or codec-based voice-cloning systems, with particular emphasis on accented speech [2603.14328]. The benchmark covers two task families. The first is codec resynthesis, in which an original utterance is encoded and decoded by a neural audio codec. The second is voice cloning or TTS, in which a system is given target text and a same-speaker reference utterance and is required to synthesize the target while preserving speaker identity and accent [2603.14328].

The dataset comprises 4,000 evaluated audio samples from 24 systems, with 32 speakers spanning ten English accents [2603.14328]. The paper further specifies that the listener study collected 19,600 annotations from 25 listeners, and that judgments were obtained for naturalness, speaker similarity, and accent similarity on five-point scales [2603.14328]. This places the dataset at the intersection of codec evaluation, accented-speech assessment, and subjective speech quality analysis.

A central conceptual feature of the benchmark is that accent similarity is treated as a first-class perceptual dimension rather than being folded into speaker similarity or generic naturalness. This suggests a broader evaluation target than conventional voice-cloning work, which had often foregrounded speaker similarity more than accent preservation. The paper argues that accent similarity has become an important emergent capability of in-context TTS and therefore requires systematic benchmarking [2603.14328].

## 2. Data source and corpus construction

CodecMOS-Accent is built from VCTK, which the authors selected over alternatives such as CommonVoice or AccentDB on the basis of quality, quantity, and diversity [2603.14328]. All VCTK audio was downsampled to 16 kHz, and leading and trailing silences were trimmed using open-source forced-alignment labels from a public repository [2603.14328].

To obtain a relatively balanced accent distribution, the benchmark selects 32 speakers across 10 accents using VCTK’s accent labels [2603.14328]. The ten accents identified in the analysis are South African, Australian, English, Northern Irish, American, Canadian, Scottish, New Zealand, Irish, and Welsh [2603.14328]. The speaker set consists of 20 female and 12 male speakers [2603.14328].

Each selected speaker contributes 5 target utterances, yielding 160 ground-truth target utterances in total [2603.14328]. All target utterances are constrained to be between 3 and 7 seconds long [2603.14328]. For the voice-cloning condition, each target utterance is paired with a different reference utterance from the same speaker, chosen randomly [2603.14328]. With 25 evaluated conditions when ground truth is counted as a condition, the total number of evaluated samples is consistent with
$$160 \times 25 = 4{,}000.$$

The paper does not describe train/dev/test splits for CodecMOS-Accent, and it does not present a formal metadata schema, directory layout, or licensing statement in the provided text [2603.14328]. It also does not provide a release URL, stating only that the dataset “will be made public in the near future” [2603.14328]. Accordingly, the benchmark is well specified at the protocol level, but its packaging and access mechanics are not documented in the excerpted material.

## 3. Systems and evaluated conditions

The benchmark evaluates 9 codec resynthesis systems and 15 voice-cloning or TTS systems, in addition to ground truth [2603.14328]. Only open-source models were included; commercial or black-box systems were excluded [2603.14328].

| Category | Systems |
|---|---|
| Resynthesis | Encodec; DAC; SpeechTokenizer; FACodec; Mimi; SNAC; WavTokenizer; NanoCodec; NeuCodec |
| Voice cloning / TTS | VALL-E-X; TorToiSe; XTTS; FireRedTTS; MaskGCT; OpenAudio s1 mini; VevoTTS; CosyVoice 2; Llasa-1B; MetaVoice; Orpheus-TTS; VoiceStar; IndexTTS2; Chatterbox; NeuTTS Air |

For the resynthesis setting, each selected VCTK utterance is passed through a given neural audio codec encoder and decoder to produce a waveform reconstruction [2603.14328]. The paper notes that most modern NACs use residual vector quantization and that some low-bitrate configurations were intentionally included to create a broad quality range [2603.14328]. Table 1 in the paper identifies specific tested variants for some systems, including Mimi at 4.4 kbps, WavTokenizer at 40 Hz, DAC at 6 kbps, SpeechTokenizer with 2 layers, and Encodec at 1.5 kbps [2603.14328].

For the TTS setting, each model is given the target text plus a same-speaker reference utterance, and the task is defined as generating speech that correctly speaks the target text while preserving both speaker identity and accent from the reference [2603.14328]. The paper does not provide implementation-level generation parameters such as decoding temperatures, prompt truncation, or model-specific control settings [2603.14328]. This means the benchmark protocol is clear at the task level, but not exhaustively specified for bitwise reproduction.

## 4. Subjective annotation protocol

The listening test was conducted through Intergroup, a crowdsourcing company [2603.14328]. The unit of annotation is explicitly defined as
$$\langle \text{test sample}, \text{reference sample}, \text{scores} \rangle$$
for a specific listener [2603.14328]. Each annotation contains three subjective scores:

- **S-NAT**: naturalness of the test sample, considering pronunciation, prosody, noise, and related factors.
- **S-SPK-SIM**: whether the test sample and reference sample are spoken by the same speaker.
- **S-ACC-SIM**: whether the test sample and reference sample have the same accent, regardless of whether they sound like the same person.

All three dimensions use a five-point scale, although the paper does not provide verbal anchors for the individual points [2603.14328]. The study included 25 listeners, each of whom rated 784 samples, for a total of
$$25 \times 784 = 19{,}600$$
annotations [2603.14328]. This corresponds to approximately 4.9 annotations per sample on average [2603.14328].

No personally identifying information was collected except age, headphone information, and self-reported accent [2603.14328]. Listener accent backgrounds were 19 US annotators, 2 Canadian annotators, 3 English annotators, and 1 Scottish annotator [2603.14328]. The paper notes that it follows VCTK in treating English and Scottish as different accents [2603.14328].

One post-collection quality-control step is described. By inspecting listener comments, the authors rejected 55 problematic samples, together with their corresponding 275 annotations, because those samples were reported to be either silent or of such low quality that fair evaluation was difficult [2603.14328]. The paper does not describe qualification tests, hidden anchors, duplicate items, session blocking, or detailed randomization procedures [2603.14328].

## 5. Benchmark dimensions, statistical analysis, and objective correlates

CodecMOS-Accent uses human ratings as the primary benchmark signal, rather than relying exclusively on automatic proxies [2603.14328]. System-level results are reported as means with \(95\%\) confidence intervals, Pearson correlation coefficients are used to compare objective and subjective metrics at utterance and system levels, Welch’s \(t\)-test is used for same-accent versus different-accent bias analysis, and Spearman’s rank correlation \(\rho\) is used to measure agreement across listener-accent groups [2603.14328]. Only listener-group pairs with statistically significant correlation \((p < 0.05)\) are retained in the listener-agreement table [2603.14328].

The benchmark also includes four objective metrics:

- **O-WER**: word error rate using Whisper large-v3.
- **O-SPK-SIM**: cosine similarity between speaker embeddings from an open-source ECAPA-TDNN speaker verification model.
- **O-ACC-SIM**: cosine similarity between accent embeddings from an ECAPA-TDNN model trained on CommonAccent.
- **O-UTMOS**: predicted speech quality from UTMOS.

At the utterance level, the paper reports modest correlations between objective and subjective measures except for UTMOS on naturalness [2603.14328]. At the system level, correlations are substantially stronger [2603.14328]. Two findings are emphasized. First, O-UTMOS is highly predictive of subjective naturalness at the system level, with \(r(\text{S-NAT}, \text{O-UTMOS}) = 0.96\) [2603.14328]. Second, O-SPK-SIM predicts subjective accent similarity better than O-ACC-SIM at the system level, with \(0.90\) versus \(0.81\) correlation respectively [2603.14328]. The authors interpret this as evidence that automatic speaker discrimination implicitly captures accent variation strongly [2603.14328].

The paper explicitly argues against using O-WER as a stand-alone overall metric [2603.14328]. This is consequential for codec evaluation, because it distinguishes intelligibility-oriented proxies from broader perceptual constructs such as accent similarity and speaker preservation.

## 6. Empirical findings

The benchmark reports several findings about system behavior on accented speech [2603.14328]. One is that some modern TTS systems are judged more natural than the VCTK ground truth itself. The authors attribute this to recording artifacts in VCTK and the possibility that modern TTS systems generate cleaner speech [2603.14328]. By contrast, resynthesis systems are conceptually upper-bounded by source quality, since they reconstruct an existing recording rather than synthesizing a cleaned rendition [2603.14328].

Another major finding is a very strong relationship between speaker similarity and accent similarity. At the utterance level, the Pearson correlation between subjective speaker similarity and accent similarity is \(0.75\), and at the system level it rises to \(0.97\) [2603.14328]. The authors interpret this as evidence that, for current systems, the ability to capture speaker identity is intrinsically linked to the ability to synthesize the corresponding accent [2603.14328].

The benchmark also finds that even low-quality codec reconstructions can preserve speaker and accent cues better than their naturalness scores might suggest [2603.14328]. Systems such as Encodec 1.5 kbps, SpeechTokenizer 2 layers, and DAC 6 kbps receive very low naturalness scores—\(1.364 \pm 0.056\), \(1.386 \pm 0.053\), and \(1.418 \pm 0.055\), respectively—yet still retain nontrivial speaker- and accent-similarity ratings [2603.14328]. The authors describe this as evidence for a learning hierarchy in which broad global traits such as speaker identity and accent may be captured before high-fidelity, artifact-free generation is achieved [2603.14328].

Among all systems, the best TTS naturalness score is reported for CosyVoice 2 at
$$S\text{-}NAT = 4.430 \pm 0.052,$$
followed by OpenAudio s1 mini \((4.291 \pm 0.053)\), Llasa-1B \((4.240 \pm 0.062)\), and Chatterbox \((4.230 \pm 0.058)\) [2603.14328]. Ground truth scores \(4.045 \pm 0.063\) on naturalness [2603.14328]. For resynthesis, NanoCodec \((4.073 \pm 0.062)\), FACodec \((4.070 \pm 0.061)\), and NeuCodec \((4.048 \pm 0.064)\) are reported as the strongest systems on naturalness [2603.14328].

For speaker similarity, ground truth is highest at \(4.756 \pm 0.042\) [2603.14328]. FACodec is the strongest resynthesis system at \(4.671 \pm 0.048\), with NanoCodec at \(4.558 \pm 0.055\) [2603.14328]. Among TTS systems, MaskGCT \((4.586 \pm 0.056)\), IndexTTS2 \((4.530 \pm 0.058)\), VevoTTS \((4.484 \pm 0.062)\), and VoiceStar \((4.397 \pm 0.066)\) are highlighted [2603.14328]. For accent similarity, ground truth reaches \(4.678 \pm 0.050\); FACodec reaches \(4.609 \pm 0.053\); NanoCodec reaches \(4.493 \pm 0.062\); and the leading TTS systems include MaskGCT \((4.483 \pm 0.061)\), VevoTTS \((4.348 \pm 0.070)\), VoiceStar \((4.293 \pm 0.074)\), and IndexTTS2 \((4.248 \pm 0.081)\) [2603.14328].

## 7. Bias, limitations, and relation to adjacent benchmarks

A notable contribution of CodecMOS-Accent is its analysis of listener-accent bias [2603.14328]. Using Welch’s \(t\)-test, the authors compare ratings from listeners who share the speaker’s accent with ratings from listeners with different accents [2603.14328]. For ground-truth speech, same-accent listeners give significantly higher speaker-similarity and accent-similarity scores, but not significantly higher naturalness scores [2603.14328]. For the full dataset, same-accent listeners give higher scores on all three dimensions:

- **S-NAT**: same accent 3.616 vs different accent 3.489, \(p < 0.001\)
- **S-SPK-SIM**: same accent 4.124 vs different accent 3.997, \(p < 0.001\)
- **S-ACC-SIM**: same accent 4.236 vs different accent 3.955, \(p < 0.001\)

The authors hypothesize that the naturalness bias in the full dataset may arise because most listeners are US-based and many models are trained primarily on US English data, causing synthesized outputs to align more closely with US listeners’ perceptual expectations [2603.14328]. This suggests that subjective evaluation on accented speech is not only a measurement problem but also a sampling problem in listener demographics.

Agreement across listener accent groups is reported as generally moderate to strong, although some accents such as Welsh show weaker agreement in the table summarized in the paper [2603.14328]. The authors state that the data do not provide strong enough evidence to conclude that highly specific regional accents necessarily create greater disagreement [2603.14328]. Thus, accent-wise difficulty is suggested but not decisively resolved.

The benchmark’s limitations are also explicit. It is limited to English accents, specifically the 10 accents represented in the selected VCTK speakers [2603.14328]. The source corpus itself contains recording artifacts and may not represent all varieties of accented speech equally [2603.14328]. The listener pool is relatively small and heavily skewed toward US annotators [2603.14328]. The paper also does not provide exhaustive reproduction details for all TTS decoding settings [2603.14328].

Relative to adjacent work, CodecMOS-Accent fills a role not covered by CodecBench. CodecBench evaluates codecs across acoustic and semantic dimensions and includes accent-adjacent resources such as KeSpeech, which contains standard Mandarin and eight subdialects, but it does not explicitly include a dataset named CodecMOS-Accent and does not present accent-focused MOS evaluation or human MOS listening tests [2508.20660]. PASQA, by contrast, addresses accent-focused quality modeling in Japanese through a synthetic accent-error dataset and pseudo accent-quality scores, but it is not a codec-MOS benchmark and does not use CodecMOS-Accent [2606.20137]. Taken together, these works place CodecMOS-Accent in a distinct niche: human evaluation of codec resynthesis and codec-based TTS specifically for accented English speech, with accent similarity separated from both naturalness and speaker similarity as an explicit perceptual target [2603.14328].

Source: https://www.emergentmind.com/topics/codecmos-accent-dataset