CrossAccent-TTS: Cross-Lingual Accent-Intensity Controllable Text-to-Speech via Disentangled Speaker and Accent Representations

Abstract: Accent conversion and controllability remain fundamental challenges in cross-lingual text-to-speech (TTS), particularly for low-resource and phonetically diverse Indic languages. While recent LLM-based TTS systems exhibit strong cross-lingual generalization, they provide limited explicit control over accent characteristics and intensity. In this paper, we propose CrossAccentTTS, a framework that enables both accent control and conversion while preserving speaker identity. Specifically, we introduce an Accent Intensity Controller (AIC) that injects weighted language embeddings into the accent subspace, allowing smooth interpolation between accents and fine-grained modulation of accent strength at inference time. Experiments on the Indic Multilingual and L2-arctic datasets shows that CrossAccent-TTS achieves precise control of accent intensity, outperforming strong baselines in accent similarity and controllability by maintaining speaker similarity and naturalness.

• The paper introduces a modular TTS system that enables explicit control of accent and intensity via disentangled speaker and accent embeddings.
• It utilizes adversarial training and weighted language embeddings, significantly improving accent similarity and reducing accent leakage over baselines.
• Empirical evaluations on Indic and L2 datasets confirm robust speaker identity preservation and naturalness in synthesized speech.

CrossAccent-TTS: Accent-Controllable and Intensity-Modulable Cross-Lingual Speech Synthesis

Framework and Architectural Innovations

CrossAccent-TTS presents a modular cross-lingual TTS system with explicit control over accent and accent intensity, leveraging disentangled speaker and accent representations via neural codec tokenization, adversarial disentanglement, and weighted language embeddings. The architecture includes four primary components: Neucodec-based speech tokenization, Perceiver Resampler for speaker/style encoding, Accent Suppression Module (utilizing adversarial training with a GRL), and a Qwen-2.5-based autoregressive decoder conditioned on text, disentangled speaker/style slots, and weighted language embeddings for accent control.

The Accent Intensity Controller (AIC) injects weighted language embeddings into the accent subspace, yielding continuous interpolations and enabling fine-grained accent strength modulation at inference, with no requirement for accent-specific training instances. During training, adversarial suppression ensures that speaker embeddings are accent-invariant, maximizing speaker identity preservation.

Figure 1: Model architecture of the proposed CrossAccent TTS.

Dataset Utilization and Experimental Setup

The experimental evaluation spans two distinct categorical domains:

• Indic Multilingual Dataset: Six languages (Hindi, Telugu, Tamil, Bengali, Marathi, English), combining 986 hours of internal and open data, with explicit language label-based conditioning and adversarial objectives.
• L2 ARCTIC Dataset: 27 hours, 24 speakers, six non-native English accent backgrounds, accompanied by aligned text and accent annotations.

Training proceeds in a two-stage curriculum: large-scale multilingual pretraining for five epochs, followed by L2 ARCTIC fine-tuning. IPA text tokenization and a shared tokenizer are used. Speaker embeddings (32 slots, $d=768$) are extracted, and the Qwen 2.5 decoder autoregressively generates acoustic tokens conditioned on text and speaker-language representation. Accent intensity is modulated via linear combination of learned language/accent embeddings.

Objective and Subjective Evaluation

Performance metrics include Accent Similarity (cosine similarity between generated and target accent embeddings via GenAID), Accent Leakage (source accent suppression), UTMOS for overall speech quality, and Speaker Similarity using Resemblyzer.

Empirical results demonstrate:

• Accent similarity for Indic conversion improved from 0.312 (IndicF5) and 0.284 (XTTS-v2) to 0.371 (CrossAccent-TTS).
• Accent leakage reduced from 0.312 (IndicF5) and 0.284 (XTTS-v2) to 0.203; speaker similarity remains comparable with baselines.
• L2 Arctic: Accent similarity up to 0.686 (CrossAccent-TTS) versus 0.612 (CVAE-L) and 0.670 (GST); UTMOS improved from 3.044 (GST) to 4.001 (CrossAccent-TTS).

Subjective MOS tests with 20 participants confirm perceptually improved accent similarity and control with CrossAccent-TTS compared to all baselines.

Figure 2: Accent Similarity MOS of Indic Accents.

Accent Intensity Control: Analysis and Outcomes

Accent strength is modulated continuously by adjusting the linear weight of language embeddings. As intensity increases from 0 to 1.0, the accent similarity score rises in a predictable, monotonic fashion, validating effective accent interpolation. The system enables not only binary accent transfer but direct control and smooth accent modulation.

Figure 3: Effect of Accent Intensity on Accent Similarity score.

Implications and Future Directions

CrossAccent-TTS demonstrates accent conversion and intensity control for both low-resource Indic languages and L2 English accents, surpassing baseline models in objective and subjective accent similarity, controllability, and speaker identity preservation. The adversarial Accent Suppression Module achieves robust disentanglement between speaker and accent, preventing accent leakage during synthesis. The linear weighted language embedding mechanism supports smooth, continuous accent transitions and intensity control, expanding practical utility for applications including voice dubbing, language education, and personalized conversational agents.

Theoretically, this model advances the explicit conditional control paradigm in speech synthesis and provides a scalable solution for accent modularity—potential to extend to hierarchical or compositional accent/style control. Practically, future developments may include multimodal accent conditioning (prosodic cues, speech rhythm), hierarchical accent intensity models for finer degrees of expressivity, and transfer to unsupervised/sparse accent scenarios as well as further cross-lingual generalization.

Conclusion

CrossAccent-TTS establishes a principled approach to cross-lingual, accent-intensity-controllable TTS via adversarially disentangled representations and explicit weighted accent embeddings. The results demonstrate superior controllability and accent similarity metrics across diverse datasets and languages, with strong speaker identity preservation and naturalness. These findings underline the utility of codec-based tokenization, language-conditioned modular embeddings, and adversarial speaker/accent disentanglement for practical accent synthesis and conversion in multilingual, low-resource, and native/non-native settings.