Joycent: Diffusion-based Accent TTS without Accented Phone Prediction

Abstract: Accent text-to-speech (TTS) aims to synthesize speech with target accents. Existing accent TTS systems typically rely on a two-stage pipeline that first converts standard phone sequences into accented phone sequences and then synthesizes accented speech. However, such approaches suffer from error accumulation and require paired standard-accented phone sequence data, which is often limited in practice. Moreover, text-based accented phone representations are insufficient to model acoustic accent characteristics such as prosody and rhythm. In this work, we propose Joycent, a diffusion-based accent TTS model that synthesizes accented speech directly from standard phone sequences and speech references without accented phone prediction. Joycent integrates accent and speaker representations through conditional layer normalization (CLN) in the text encoder. We introduce WhisAID, a Mandarin accent identification model trained on accented Mandarin speech to extract accent representations. Experimental results show that Joycent improves accentedness while preserving speaker identity compared with baseline systems. We release our code and demos at: https://github.com/oshindow/Joycent-code.

• The paper proposes Joycent, a diffusion-based TTS framework that eliminates the need for accented phone prediction.
• It leverages disentangled accent and speaker embeddings via conditional layer normalization to enhance accent fidelity and efficiency.
• Experimental results on multi-accent Mandarin datasets show improved MOS scores, accent accuracy, and inference speed compared to baselines.

Diffusion-Based Accent TTS Without Accented Phone Prediction: The Joycent Framework

Overview

The paper "Joycent: Diffusion-based Accent TTS without Accented Phone Prediction" (2606.16417) introduces a novel approach for Mandarin accent text-to-speech (TTS) synthesis that forgoes the conventional two-stage pipeline requiring accented phone sequence prediction. By leveraging disentangled acoustic accent and speaker embeddings integrated via conditional layer normalization (CLN) in a diffusion-based neural TTS architecture, Joycent directly synthesizes accented speech from standard phone sequences and reference utterances. This paradigm shift addresses limitations in error propagation, the scarcity of paired labeled data, and the inability of text-based accent representations to capture core prosodic and rhythmic accent features. The work is evaluated on multi-accent Mandarin datasets and compared systematically to both phone-sequence–driven baselines and commercial speech models.

Model Architecture and Methodology

The Joycent system utilizes Grad-TTS as the core backbone, incorporating a text encoder built on stacked Conformer blocks, a monotonic alignment model, and a diffusion-based decoder for mel-spectrogram generation.

Figure 1: Overview of the Joycent TTS framework integrating WhisAID for accent embedding extraction and FACodec for speaker embedding extraction, with CLN-based fusion into the Conformer text encoder.

Disentangled Accent and Speaker Embeddings

A critical design aspect is the disentanglement of accent and speaker information, which are inherently confounded in natural speech corpora. Accent embeddings are extracted with WhisAID, a fine-tuned variant of Whisper, augmented with an accent head and a speaker head. A gradient reversal layer (GRL) is applied prior to the speaker classifier to adversarially suppress speaker identity information within the accent embedding, as assessed by the Silhouette Coefficient for Speaker Clusters (SCSC). Speaker embeddings are derived from FACodec, supporting zero-shot adaptation and robust speaker identity preservation.

Conditional Layer Normalization Integration

Joycent moves beyond simple embedding concatenation by leveraging CLN. Specifically, accent embeddings modulate the layer normalization in the initial Conformer block, while speaker embeddings modulate the normalization in the terminal block. This hierarchical injection empirically outperforms injecting these factors at the decoder input, substantiating that phone-level accent modulations enhance accent rendering compared to frame-level conditioning.

Diffusion-Based TTS Decoder

The diffusion decoder refines noisy latent variables towards target mel-spectrograms, conditioned on prior means from text encoding. The reverse SDE is solved deterministically, as per Grad-TTS conventions, with amplitude controlled by a temperature parameter. Output mel-spectrograms are vocoded using a Parallel WaveGAN trained on the union of relevant datasets.

Experimental Results

Accent Embedding Analysis

WhisAID outperforms previous accent classifiers such as GenAID, especially in the unseen-speaker setting, achieving accuracy/F1 scores of 0.60/0.57 (Mandarin, Whisper-medium). SCSC consistently decreases with the incorporation of GRL, confirming improved speaker independence of the accent representation.

Accent TTS Benchmarks

Joycent is evaluated on multi-accent Mandarin datasets, focusing on the Singaporean Mandarin accent as a test case. Both subjective (MOS, SMOS) and objective (WhisAID-based identification, cosine similarity metrics) criteria are considered, comparing Joycent to MacST and CosyVoice3. Key findings include:

• Joycent achieves MOS scores of 3.45 (subjective quality) and SMOS up to 3.0 (accent similarity), approaching groundtruth (3.5) while surpassing baselines (2.1–2.85).
• Accent identification accuracy of 0.77 (seen-speaker) and 0.50 (unseen-speaker) signifies substantial improvements over text-based pipelines (e.g., MacST, CosyVoice3: 0.11–0.15).
• Speaker similarity (cosine) remains on par with larger LLM-based models, despite Joycent’s compact size (18.79M parameters) and its use of disentangled representations.
• Inference speed is dramatically superior (RTF 0.069 vs 0.642 for CosyVoice3), highlighting the efficiency advantage.

Ablation analysis demonstrates that early CLN-based conditioning of accent information (block 1 of encoder) yields the highest accent similarity, vindicating the hierarchical design choice.

Theoretical and Practical Implications

Joycent empirically validates the claim that robust accent control in TTS is not achievable through textual accent tokenization or phone substitution alone, particularly for language varieties where prosodic and phonetic subtleties define the accent. The explicit decoupling of accent and speaker variables enables generalization to unseen speakers and more faithful accent rendering—critical for low-resource or multi-accent environments.

Practically, Joycent’s approach is immediately applicable for generating high-variance accent data for downstream tasks such as mispronunciation detection, robust ASR, or speech assessment systems. This is especially valuable in educational and diagnostic contexts where authentic accented data is scarce.

Future Directions

Two main avenues are prominent:

1. Accent-Agnostic Data Augmentation: Systematic synthesis of accented data for supervised models targeting pronunciation assessment or accent robustness.
2. Extensibility to Other Languages and Accents: Adapting the architecture to languages with more complex prosodic systems, or to L2 accents where the distribution of error patterns diverges further from L1 phonology.

Further, the modularity of the WhisAID accent extractor supports plug-and-play integration with other large speech backbones and may facilitate scaling to highly multilingual or code-switched scenarios.

Conclusion

Joycent provides a compelling demonstration that diffusion models, coupled with adversarially-disentangled acoustic conditioning, set a new standard for accent TTS quality and control beyond text-based approaches. Its architecture establishes a template for future controllable TTS research—balancing efficiency, accent fidelity, and speaker generalization. The system is poised to enable practical advancement in both data creation and speech technology aimed at multi-accent, low-resource, or personalized applications.