CodecMOS-Accent: MOS Benchmark for Accented Speech

• CodecMOS-Accent is a MOS benchmark designed to evaluate neural audio codec and TTS systems on accented English speech, emphasizing perceptual accent similarity.
• It collects human ratings on naturalness, speaker similarity, and accent similarity from 4,000 samples across 32 speakers representing 10 diverse English accents.
• The dataset reveals a strong link between speaker identity and accent preservation, offering valuable insights for enhancing both codec resynthesis and voice cloning techniques.

Searching arXiv for the specified paper and adjacent work to ensure citations are current and accurate. CodecMOS-Accent is a mean opinion score benchmark for evaluating neural audio codec resynthesis systems and codec-based text-to-speech systems on accented English speech. It is designed around a gap identified in codec evaluation: prior benchmarks had largely emphasized reconstruction fidelity and objective downstream metrics, while providing limited perceptual evidence about system behavior on non-standard speech, especially accented speech (Huang et al., 15 Mar 2026). The benchmark therefore centers human judgment rather than purely automatic measures, and operationalizes evaluation along three dimensions—naturalness, speaker similarity, and accent similarity—over a collection of resynthesized and synthesized speech produced from multiple contemporary systems (Huang et al., 15 Mar 2026). In the available literature, CodecMOS-Accent is distinct from CodecBench, which does not explicitly include or mention a dataset by that name and does not present an accent-focused MOS subset (Deng et al., 28 Aug 2025).

1. Definition and scope

CodecMOS-Accent is explicitly framed as a MOS benchmark for neural audio codecs and LLM-based or codec-based voice-cloning systems, with particular emphasis on accented speech (Huang et al., 15 Mar 2026). The benchmark covers two task families. The first is codec resynthesis, in which an original utterance is encoded and decoded by a neural audio codec. The second is voice cloning or TTS, in which a system is given target text and a same-speaker reference utterance and is required to synthesize the target while preserving speaker identity and accent (Huang et al., 15 Mar 2026).

The dataset comprises 4,000 evaluated audio samples from 24 systems, with 32 speakers spanning ten English accents (Huang et al., 15 Mar 2026). The paper further specifies that the listener study collected 19,600 annotations from 25 listeners, and that judgments were obtained for naturalness, speaker similarity, and accent similarity on five-point scales (Huang et al., 15 Mar 2026). This places the dataset at the intersection of codec evaluation, accented-speech assessment, and subjective speech quality analysis.

A central conceptual feature of the benchmark is that accent similarity is treated as a first-class perceptual dimension rather than being folded into speaker similarity or generic naturalness. This suggests a broader evaluation target than conventional voice-cloning work, which had often foregrounded speaker similarity more than accent preservation. The paper argues that accent similarity has become an important emergent capability of in-context TTS and therefore requires systematic benchmarking (Huang et al., 15 Mar 2026).

2. Data source and corpus construction

CodecMOS-Accent is built from VCTK, which the authors selected over alternatives such as CommonVoice or AccentDB on the basis of quality, quantity, and diversity (Huang et al., 15 Mar 2026). All VCTK audio was downsampled to 16 kHz, and leading and trailing silences were trimmed using open-source forced-alignment labels from a public repository (Huang et al., 15 Mar 2026).

To obtain a relatively balanced accent distribution, the benchmark selects 32 speakers across 10 accents using VCTK’s accent labels (Huang et al., 15 Mar 2026). The ten accents identified in the analysis are South African, Australian, English, Northern Irish, American, Canadian, Scottish, New Zealand, Irish, and Welsh (Huang et al., 15 Mar 2026). The speaker set consists of 20 female and 12 male speakers (Huang et al., 15 Mar 2026).

Each selected speaker contributes 5 target utterances, yielding 160 ground-truth target utterances in total (Huang et al., 15 Mar 2026). All target utterances are constrained to be between 3 and 7 seconds long (Huang et al., 15 Mar 2026). For the voice-cloning condition, each target utterance is paired with a different reference utterance from the same speaker, chosen randomly (Huang et al., 15 Mar 2026). With 25 evaluated conditions when ground truth is counted as a condition, the total number of evaluated samples is consistent with

$160 \times 25 = 4{,}000.$

The paper does not describe train/dev/test splits for CodecMOS-Accent, and it does not present a formal metadata schema, directory layout, or licensing statement in the provided text (Huang et al., 15 Mar 2026). It also does not provide a release URL, stating only that the dataset “will be made public in the near future” (Huang et al., 15 Mar 2026). Accordingly, the benchmark is well specified at the protocol level, but its packaging and access mechanics are not documented in the excerpted material.

3. Systems and evaluated conditions

The benchmark evaluates 9 codec resynthesis systems and 15 voice-cloning or TTS systems, in addition to ground truth (Huang et al., 15 Mar 2026). Only open-source models were included; commercial or black-box systems were excluded (Huang et al., 15 Mar 2026).

Category	Systems
Resynthesis	Encodec; DAC; SpeechTokenizer; FACodec; Mimi; SNAC; WavTokenizer; NanoCodec; NeuCodec
Voice cloning / TTS	VALL-E-X; TorToiSe; XTTS; FireRedTTS; MaskGCT; OpenAudio s1 mini; VevoTTS; CosyVoice 2; Llasa-1B; MetaVoice; Orpheus-TTS; VoiceStar; IndexTTS2; Chatterbox; NeuTTS Air

For the resynthesis setting, each selected VCTK utterance is passed through a given neural audio codec encoder and decoder to produce a waveform reconstruction (Huang et al., 15 Mar 2026). The paper notes that most modern NACs use residual vector quantization and that some low-bitrate configurations were intentionally included to create a broad quality range (Huang et al., 15 Mar 2026). Table 1 in the paper identifies specific tested variants for some systems, including Mimi at 4.4 kbps, WavTokenizer at 40 Hz, DAC at 6 kbps, SpeechTokenizer with 2 layers, and Encodec at 1.5 kbps (Huang et al., 15 Mar 2026).

For the TTS setting, each model is given the target text plus a same-speaker reference utterance, and the task is defined as generating speech that correctly speaks the target text while preserving both speaker identity and accent from the reference (Huang et al., 15 Mar 2026). The paper does not provide implementation-level generation parameters such as decoding temperatures, prompt truncation, or model-specific control settings (Huang et al., 15 Mar 2026). This means the benchmark protocol is clear at the task level, but not exhaustively specified for bitwise reproduction.

4. Subjective annotation protocol

The listening test was conducted through Intergroup, a crowdsourcing company (Huang et al., 15 Mar 2026). The unit of annotation is explicitly defined as

$$\langle \text{test sample}, \text{reference sample}, \text{scores} \rangle$$

for a specific listener (Huang et al., 15 Mar 2026). Each annotation contains three subjective scores:

• S-NAT: naturalness of the test sample, considering pronunciation, prosody, noise, and related factors.
• S-SPK-SIM: whether the test sample and reference sample are spoken by the same speaker.
• S-ACC-SIM: whether the test sample and reference sample have the same accent, regardless of whether they sound like the same person.

All three dimensions use a five-point scale, although the paper does not provide verbal anchors for the individual points (Huang et al., 15 Mar 2026). The study included 25 listeners, each of whom rated 784 samples, for a total of

$25 \times 784 = 19{,}600$

annotations (Huang et al., 15 Mar 2026). This corresponds to approximately 4.9 annotations per sample on average (Huang et al., 15 Mar 2026).

No personally identifying information was collected except age, headphone information, and self-reported accent (Huang et al., 15 Mar 2026). Listener accent backgrounds were 19 US annotators, 2 Canadian annotators, 3 English annotators, and 1 Scottish annotator (Huang et al., 15 Mar 2026). The paper notes that it follows VCTK in treating English and Scottish as different accents (Huang et al., 15 Mar 2026).

One post-collection quality-control step is described. By inspecting listener comments, the authors rejected 55 problematic samples, together with their corresponding 275 annotations, because those samples were reported to be either silent or of such low quality that fair evaluation was difficult (Huang et al., 15 Mar 2026). The paper does not describe qualification tests, hidden anchors, duplicate items, session blocking, or detailed randomization procedures (Huang et al., 15 Mar 2026).

5. Benchmark dimensions, statistical analysis, and objective correlates

CodecMOS-Accent uses human ratings as the primary benchmark signal, rather than relying exclusively on automatic proxies (Huang et al., 15 Mar 2026). System-level results are reported as means with $95\%$ confidence intervals, Pearson correlation coefficients are used to compare objective and subjective metrics at utterance and system levels, Welch’s $t$-test is used for same-accent versus different-accent bias analysis, and Spearman’s rank correlation $\rho$ is used to measure agreement across listener-accent groups (Huang et al., 15 Mar 2026). Only listener-group pairs with statistically significant correlation $(p < 0.05)$ are retained in the listener-agreement table (Huang et al., 15 Mar 2026).

The benchmark also includes four objective metrics:

• O-WER: word error rate using Whisper large-v3.
• O-SPK-SIM: cosine similarity between speaker embeddings from an open-source ECAPA-TDNN speaker verification model.
• O-ACC-SIM: cosine similarity between accent embeddings from an ECAPA-TDNN model trained on CommonAccent.
• O-UTMOS: predicted speech quality from UTMOS.

At the utterance level, the paper reports modest correlations between objective and subjective measures except for UTMOS on naturalness (Huang et al., 15 Mar 2026). At the system level, correlations are substantially stronger (Huang et al., 15 Mar 2026). Two findings are emphasized. First, O-UTMOS is highly predictive of subjective naturalness at the system level, with $r(\text{S-NAT}, \text{O-UTMOS}) = 0.96$ (Huang et al., 15 Mar 2026). Second, O-SPK-SIM predicts subjective accent similarity better than O-ACC-SIM at the system level, with $0.90$ versus $0.81$ correlation respectively (Huang et al., 15 Mar 2026). The authors interpret this as evidence that automatic speaker discrimination implicitly captures accent variation strongly (Huang et al., 15 Mar 2026).

The paper explicitly argues against using O-WER as a stand-alone overall metric (Huang et al., 15 Mar 2026). This is consequential for codec evaluation, because it distinguishes intelligibility-oriented proxies from broader perceptual constructs such as accent similarity and speaker preservation.

6. Empirical findings

The benchmark reports several findings about system behavior on accented speech (Huang et al., 15 Mar 2026). One is that some modern TTS systems are judged more natural than the VCTK ground truth itself. The authors attribute this to recording artifacts in VCTK and the possibility that modern TTS systems generate cleaner speech (Huang et al., 15 Mar 2026). By contrast, resynthesis systems are conceptually upper-bounded by source quality, since they reconstruct an existing recording rather than synthesizing a cleaned rendition (Huang et al., 15 Mar 2026).

Another major finding is a very strong relationship between speaker similarity and accent similarity. At the utterance level, the Pearson correlation between subjective speaker similarity and accent similarity is $$\langle \text{test sample}, \text{reference sample}, \text{scores} \rangle$$0, and at the system level it rises to $$\langle \text{test sample}, \text{reference sample}, \text{scores} \rangle$$1 (Huang et al., 15 Mar 2026). The authors interpret this as evidence that, for current systems, the ability to capture speaker identity is intrinsically linked to the ability to synthesize the corresponding accent (Huang et al., 15 Mar 2026).

The benchmark also finds that even low-quality codec reconstructions can preserve speaker and accent cues better than their naturalness scores might suggest (Huang et al., 15 Mar 2026). Systems such as Encodec 1.5 kbps, SpeechTokenizer 2 layers, and DAC 6 kbps receive very low naturalness scores—$$\langle \text{test sample}, \text{reference sample}, \text{scores} \rangle$$2, $$\langle \text{test sample}, \text{reference sample}, \text{scores} \rangle$$3, and $$\langle \text{test sample}, \text{reference sample}, \text{scores} \rangle$$4, respectively—yet still retain nontrivial speaker- and accent-similarity ratings (Huang et al., 15 Mar 2026). The authors describe this as evidence for a learning hierarchy in which broad global traits such as speaker identity and accent may be captured before high-fidelity, artifact-free generation is achieved (Huang et al., 15 Mar 2026).

Among all systems, the best TTS naturalness score is reported for CosyVoice 2 at

$$\langle \text{test sample}, \text{reference sample}, \text{scores} \rangle$$5

followed by OpenAudio s1 mini $$\langle \text{test sample}, \text{reference sample}, \text{scores} \rangle$$6, Llasa-1B $$\langle \text{test sample}, \text{reference sample}, \text{scores} \rangle$$7, and Chatterbox $$\langle \text{test sample}, \text{reference sample}, \text{scores} \rangle$$8 (Huang et al., 15 Mar 2026). Ground truth scores $$\langle \text{test sample}, \text{reference sample}, \text{scores} \rangle$$9 on naturalness (Huang et al., 15 Mar 2026). For resynthesis, NanoCodec $25 \times 784 = 19{,}600$0, FACodec $25 \times 784 = 19{,}600$1, and NeuCodec $25 \times 784 = 19{,}600$2 are reported as the strongest systems on naturalness (Huang et al., 15 Mar 2026).

For speaker similarity, ground truth is highest at $25 \times 784 = 19{,}600$3 (Huang et al., 15 Mar 2026). FACodec is the strongest resynthesis system at $25 \times 784 = 19{,}600$4, with NanoCodec at $25 \times 784 = 19{,}600$5 (Huang et al., 15 Mar 2026). Among TTS systems, MaskGCT $25 \times 784 = 19{,}600$6, IndexTTS2 $25 \times 784 = 19{,}600$7, VevoTTS $25 \times 784 = 19{,}600$8, and VoiceStar $25 \times 784 = 19{,}600$9 are highlighted (Huang et al., 15 Mar 2026). For accent similarity, ground truth reaches $95\%$0; FACodec reaches $95\%$1; NanoCodec reaches $95\%$2; and the leading TTS systems include MaskGCT $95\%$3, VevoTTS $95\%$4, VoiceStar $95\%$5, and IndexTTS2 $95\%$6 (Huang et al., 15 Mar 2026).

7. Bias, limitations, and relation to adjacent benchmarks

A notable contribution of CodecMOS-Accent is its analysis of listener-accent bias (Huang et al., 15 Mar 2026). Using Welch’s $95\%$7-test, the authors compare ratings from listeners who share the speaker’s accent with ratings from listeners with different accents (Huang et al., 15 Mar 2026). For ground-truth speech, same-accent listeners give significantly higher speaker-similarity and accent-similarity scores, but not significantly higher naturalness scores (Huang et al., 15 Mar 2026). For the full dataset, same-accent listeners give higher scores on all three dimensions:

• S-NAT: same accent 3.616 vs different accent 3.489, $95\%$8
• S-SPK-SIM: same accent 4.124 vs different accent 3.997, $95\%$9
• S-ACC-SIM: same accent 4.236 vs different accent 3.955, $t$0

The authors hypothesize that the naturalness bias in the full dataset may arise because most listeners are US-based and many models are trained primarily on US English data, causing synthesized outputs to align more closely with US listeners’ perceptual expectations (Huang et al., 15 Mar 2026). This suggests that subjective evaluation on accented speech is not only a measurement problem but also a sampling problem in listener demographics.

Agreement across listener accent groups is reported as generally moderate to strong, although some accents such as Welsh show weaker agreement in the table summarized in the paper (Huang et al., 15 Mar 2026). The authors state that the data do not provide strong enough evidence to conclude that highly specific regional accents necessarily create greater disagreement (Huang et al., 15 Mar 2026). Thus, accent-wise difficulty is suggested but not decisively resolved.

The benchmark’s limitations are also explicit. It is limited to English accents, specifically the 10 accents represented in the selected VCTK speakers (Huang et al., 15 Mar 2026). The source corpus itself contains recording artifacts and may not represent all varieties of accented speech equally (Huang et al., 15 Mar 2026). The listener pool is relatively small and heavily skewed toward US annotators (Huang et al., 15 Mar 2026). The paper also does not provide exhaustive reproduction details for all TTS decoding settings (Huang et al., 15 Mar 2026).

Relative to adjacent work, CodecMOS-Accent fills a role not covered by CodecBench. CodecBench evaluates codecs across acoustic and semantic dimensions and includes accent-adjacent resources such as KeSpeech, which contains standard Mandarin and eight subdialects, but it does not explicitly include a dataset named CodecMOS-Accent and does not present accent-focused MOS evaluation or human MOS listening tests (Deng et al., 28 Aug 2025). PASQA, by contrast, addresses accent-focused quality modeling in Japanese through a synthetic accent-error dataset and pseudo accent-quality scores, but it is not a codec-MOS benchmark and does not use CodecMOS-Accent (Kawamura et al., 18 Jun 2026). Taken together, these works place CodecMOS-Accent in a distinct niche: human evaluation of codec resynthesis and codec-based TTS specifically for accented English speech, with accent similarity separated from both naturalness and speaker similarity as an explicit perceptual target (Huang et al., 15 Mar 2026).