TokAN: Accent Normalization Using Self-Supervised Speech Tokens

Abstract: Accent normalization (AN) seeks to convert non-native (L2) accented speech into standard (L1) speech while preserving speaker identity. The current techniques either require naturally recorded parallel L1-L2 speech for training, or suffer from quality degradation when supervised by synthesized targets. In this paper, we present TokAN, a token-based accent normalization framework that operates on self-supervised discrete speech tokens extracted from a L1-L2 jointly trained vector-quantization (VQ) tokenizer, without the need of synthetic supervisory speech. An autoregressive encoder-decoder model performs token-to-token conversion, translating L2-accented token sequences into the tokens of standard voice. We also introduce reinforcement learning (RL) post-training based on Group Relative Policy Optimization (GRPO), using word error rate and accent classifier confidence as complementary rewards. A non-autoregressive flow-matching synthesizer recovers the Mel-spectrogram from the converted tokens, conditioned on the source speaker embedding. We also develop a flow-matching duration predictor that supports total-duration-aware synthesis, making TokAN applicable to duration-critical tasks such as voice dubbing and live casting. Experiments on seven English accents demonstrate that TokAN reduced the word error rate from 12.40% to 9.89% after supervised fine-tuning, and further to 9.23% after RL post-training, consistently outperforming frame-to-frame, direct flow-matching, and prompt-based token-conversion baselines in terms of accent reduction and intelligibility.

• The paper introduces a modular token-based framework leveraging self-supervised learning to normalize L2-accented speech while preserving speaker identity.
• It employs a joint-trained VQ tokenizer, autoregressive token conversion, and GRPO-based reinforcement learning to optimize WER and accent classifier rewards.
• Experimental results show TokAN outperforms existing systems with a WER of 9.23% and near-native accent conversion across multiple English accents.

TokAN: Accent Normalization Using Self-Supervised Speech Tokens

Introduction and Motivation

Accent normalization (AN) addresses the phonetic transformation of L2-accented speech into native-like (L1) speech while maintaining speaker identity, critical for applications such as computer-assisted language learning, dubbing, and personalized TTS. Conventional systems either require naturally paired L1-L2 corpora or depend on TTS- or ASR-generated supervision, which introduces perceptual and prosodic artifacts, adversely affecting system quality and model generalization. Further, reference- and prompt-based accent conversion architectures are impractical for large-scale deployment.

TokAN proposes a modular, token-based framework that leverages self-supervised learning (SSL) and discrete speech tokens via a jointly optimized vector quantization (VQ) codebook. The solution eliminates synthetic target supervision in conversion, mitigates voice synthesis artifacts, and enables the system to operate without paired L1-L2 speech. In addition to supervised token-level fine-tuning, reinforcement learning (RL) with group relative policy optimization (GRPO) is introduced for task-anchored post-training, directly optimizing content preservation and accentedness via word error rate (WER) and accent classifier rewards.

System Architecture

TokAN’s architecture consists of three primary components, each designed for modular training and deployment:

1. VQ Tokenizer with Joint Training: Extracts discrete, phonetically salient tokens from a frozen WavLM-Large SSL encoder. The VQ codebook is co-trained with a flow-matching speech synthesizer (using Mel-spectrogram reconstruction) and a CTC-based ASR head to encourage tokens that retain both reconstructive and recognition information pertinent to AN tasks.
2. Autoregressive Token Conversion Model: Employs a Transformer encoder-decoder architecture with rotary positional embeddings (RoPE). Distinctively, it uses a self-attention-only decoder, in which encoder output is prefixed to the decoder inputs; this eliminates explicit cross-attention and the need for a source-accent embedding, yielding an accent-universal mapping. Pre-training is conducted with BART-style token corruption to improve generalization to diverse input distributions, followed by supervised fine-tuning on semi-synthetic token pairs.
3. Flow-Matching Synthesizer and Duration Predictor: Non-autoregressive synthesis reconstructs Mel-spectrograms, conditioned on the converted tokens and speaker embedding. A flow-matching duration predictor enables total-duration-aware synthesis, essential for applications like dubbing. Source and total-duration conditioning modes allow flexible but controlled prosody and temporal alignment.

Training Pipeline and Datasets

The training pipeline is strictly staged to maximize data efficiency and system modularity:

• Tokenizer/Synthesizer Joint Training: Initially, the codebook is tuned for both phonetic discrimination and speech reconstruction using high-quality native data (LibriTTS-R, then Emilia-EN for scale and diversity).
• BART-Style Pre-Training: Conversion model learns general sequence transformation priors on unlabeled native speech.
• Supervised Token-Level Fine-Tuning: Non-parallel semi-synthetic token pairs are constructed from synthesized L2 utterances/targets for SFT, avoiding cross-modal artifacts.
• GRPO-Based RL Post-Training: Performed with real unpaired multi-accented speech from GLOBE; RL rewards are derived from a native ASR (WER) and accent classifier confidence, obviating the need for reference targets.

Experimental Evaluation

Tokenizer Analysis

Joint optimization of the VQ tokenizer with ASR and synthesis objectives (using WavLM-Large, layer 22, codebook size 1024) substantially reduces WER in reconstructive evaluation, outperforming standard K-Means- and baseline VQ-based approaches. The codebook size and SSL backbone are shown to significantly impact native recognition and accent-removal in ablation.

Accent Normalization Performance

TokAN achieves a WER of 9.23% (duration-free mode after RL), outperforming CosyAccent (12.40%), FramAN (17.55%), and VEVO (28.94%). L1 accent probability as measured by the classifier is 99.09%, with the lowest $\Delta$PPG reflecting near-native phonetic realization post-conversion. On subjective metrics, TokAN secures the highest naturalness and lowest accentedness, though there is a measurable trade-off observed between accent normalization and speaker similarity in token-based architectures.

Duration Control

TokAN’s flow-matching duration predictor, conditioned on total duration, offers robust prosodic control. Source-length-preserving mode (TokAN-2) can enforce strict duration constraints with negligible intelligibility or accent reduction cost, exhibiting only a minute $\Delta$PPG increase compared to unconstrained mode yet drastically improving match to reference duration (deviation reduced to 1.64%).

Ablation and Analysis

Each component—pre-training, SFT, RL post-training, and CTC phoneme supervision—meaningfully contributes to final performance. RL post-training yields notable further reductions in WER and accentedness, with minimal degradation to speaker similarity. Phoneme-level analysis, specifically on Mandarin-L2 speakers, illustrates that systematic L1-transfer phenomena (e.g., dental fricative realizations, affricate devoicing, nasal coda weakening) are effectively corrected, with token distribution statistics converging toward native targets after conversion.

Implications and Future Directions

From a practical perspective, TokAN’s modular, token-centric architecture maximizes data efficiency, enabling training on disparate datasets and adaptation to new accents or languages with token-level supervision. The use of RL post-training without reference data is particularly compelling for extending AN to domains where parallel L1-L2 data is sparse or unavailable.

Theoretically, this work reifies the advantage of self-supervised discrete representations in speech: such tokens provide a robust intermediate space for prosodic and phonetic manipulation that decouples speaker and accent attributes, facilitating efficient transfer and normalization. The demonstrated effectiveness of GRPO in speech RL suggests further cross-pollination with methods developed for LLMs.

Future research should focus on replacing speaker embedding-based synthesis with prompt-based mechanisms to enhance speaker similarity, incorporating additional reward signals (e.g., speaker verification) in the RL stage, and expanding TokAN’s coverage to other languages and dialects by modularly adapting tokenizer and synthesizer components for non-English data. Exploration of discrete diffusion models for sequence modeling and tighter integration with large-scale multimodal systems presents additional directions.

Conclusion

TokAN represents a comprehensive framework for accent normalization leveraging self-supervised tokenization, autoregressive sequence modeling, and task-level reinforcement learning for direct accent reduction and intelligibility improvement (2607.03928). The architecture is highly modular, does not require real paired L1-L2 data, and achieves state-of-the-art performance on seven English accents. The results and analyses presented provide a compelling case for the broad adoption of discrete token approaches in accent normalization and related speech-to-speech transformation tasks.