CrossAccentTTS: Controllable Accent Synthesis

• CrossAccentTTS is a text-to-speech framework that achieves continuous accent control by disentangling speaker identity and accent representations.
• It employs a fixed-length speaker bottleneck, an accent suppression module, and an explicit accent intensity controller to ensure natural synthesis.
• Empirical evaluations demonstrate improved accent similarity and controllability compared to baselines in multilingual, low-resource settings.

CrossAccentTTS denotes a line of text-to-speech and accent-conversion research centered on controlled manipulation of accent while preserving speaker identity; in its most specific recent usage, it refers to "CrossAccent-TTS: Cross-Lingual Accent-Intensity Controllable Text-to-Speech via Disentangled Speaker and Accent Representations" (Annamdevula et al., 24 Jun 2026). The problem setting combines cross-lingual synthesis, accent conversion, and controllability: the system must render a target text in a desired accent, often with adjustable intensity, without collapsing speaker similarity or naturalness. Closely related work frames the same objective through non-parallel accent conversion guided by TTS representations (Chen et al., 2024, Zhou et al., 2022), disentangled latent-variable synthesis (Melechovsky et al., 2024, Melechovsky et al., 2024), zero-shot accent embeddings (Zhong et al., 2024), parameter-space accent vectors (Lertpetchpun et al., 8 Mar 2026), multilingual accent embeddings (Badlani et al., 2023), low-resource accent modeling without accent-specific frontends (Tinchev et al., 2023), and cross-dialect modeling in pitch-accent languages (Yamauchi et al., 2024).

1. Problem setting and scope

Accent conversion aims to convert the accent of a source speech to a target accent, meanwhile preserving the speaker's identity (Chen et al., 2024). In cross-lingual TTS, the same requirement expands to synthesis from text rather than direct waveform conversion, with the additional constraint that accent should remain explicitly controllable at inference time. The 2026 CrossAccent-TTS formulation places this problem in the context of low-resource and phonetically diverse Indic languages, and argues that recent LLM-based TTS systems exhibit strong cross-lingual generalization but provide limited explicit control over accent characteristics and intensity (Annamdevula et al., 24 Jun 2026).

Within this scope, CrossAccentTTS is not a single methodological template. Some variants operate as accent conversion systems trained on non-parallel data, where speech representations are aligned with linguistic representations obtained from a TTS system (Chen et al., 2024, Zhou et al., 2022). Other variants are fully generative TTS systems that separate speaker and accent factors and expose accent as an independent control variable (Melechovsky et al., 2024, Annamdevula et al., 24 Jun 2026). Still others treat accent as a continuous embedding extracted from reference speech (Zhong et al., 2024) or as a task vector in parameter space (Lertpetchpun et al., 8 Mar 2026).

A recurring design objective across these formulations is disentanglement. Speaker identity, accent, language, and prosody are strongly correlated in multilingual corpora, especially when bilingual or multiaccent data are scarce (Badlani et al., 2023). CrossAccentTTS systems therefore tend to impose architectural bottlenecks, adversarial objectives, vector quantization, grouped latent variables, or explicit conditioning paths so that accent can be modified without uncontrolled drift in timbre or content.

2. Core architecture of CrossAccent-TTS

The CrossAccent-TTS framework of 2026 is organized around six major modules: a Neucodec Speech Tokenizer, a Text Encoder operating on IPA, a Speaker Encoder implemented with a Perceiver Resampler, an Accent Encoder coupled with an Accent Intensity Controller (AIC), an autoregressive Acoustic Decoder based on Qwen 2.5, and a Neucodec Vocoder for waveform synthesis (Annamdevula et al., 24 Jun 2026). Text is converted into token embeddings; reference audio is encoded by Neucodec into discrete acoustic tokens; the Perceiver Resampler produces a fixed-length bottleneck intended to represent speaker and style; accent information is injected separately; and the decoder predicts acoustic tokens autoregressively before final waveform reconstruction.

Three design choices are identified as central to disentanglement in this architecture (Annamdevula et al., 24 Jun 2026). The first is a fixed-length Perceiver bottleneck, which forces speaker and style embeddings to discard phonetic content. The second is an Accent Suppression Module using gradient reversal with an accent/language classifier, so residual accent or language cues are adversarially removed from the speaker/style pathway. The third is a separate learnable accent, or language, embedding table, which provides an explicit route for accent injection rather than allowing accent to remain entangled with speaker identity.

This arrangement places CrossAccent-TTS in a hybrid position between classical speaker-conditioned TTS and explicit factorized latent-variable models. It does not rely on a single shared embedding to represent all global variation. Instead, speaker and accent are routed through distinct mechanisms and only recombined downstream, which is the operational basis for accent conversion, accent interpolation, and speaker-preserving synthesis (Annamdevula et al., 24 Jun 2026).

3. Disentangled representations and the Accent Intensity Controller

In CrossAccent-TTS, the speaker embedding is defined as

$$s = f_{spk}(Neucodec(x_{ref})) \in R^{N_s \times d},$$

and the accent embedding as

$a = E_{lang}(label) \in R^{1 \times d}.$

The intended condition is that speaker and accent be approximately independent, expressed as $I(s; a) \approx 0$ (Annamdevula et al., 24 Jun 2026). In practice, this is encouraged through the adversarial accent-classification objective

$$L_{GRL}(\theta_{spk}, \theta_D) = E_{(s,\ell)}[- \log D_{acc}(\ell \mid GRL(s))],$$

where the classifier minimizes $L_{GRL}$ while the speaker encoder, under gradient reversal, maximizes the classification error (Annamdevula et al., 24 Jun 2026).

The Accent Intensity Controller provides the explicit mechanism for continuous accent modulation. For source and target accent embeddings $a_{src}$ and $a_{tgt}$, CrossAccent-TTS defines

$$e_{accent}(\alpha) = \alpha \cdot a_{tgt} + (1-\alpha) \cdot a_{src}, \quad \alpha \in [0,1].$$

This interpolated accent vector is broadcast and added to the speaker embedding:

$s' = s + 1_{N_s} \cdot e_{accent}(\alpha).$

The interpretation given in the paper is direct: $\alpha=0$ corresponds to source-style neutral output, while $a = E_{lang}(label) \in R^{1 \times d}.$0 corresponds to full target accent (Annamdevula et al., 24 Jun 2026).

The significance of the AIC is that accent strength becomes an inference-time control variable rather than a discrete training-time label only. This contrasts with systems that represent accent solely as a categorical token or as a latent inferred once per utterance. Related work explores similar factorization with different mechanisms. DART uses multi-level variational autoencoders and separate vector-quantization bottlenecks for speaker and accent codes (Melechovsky et al., 2024). "Accent Conversion in Text-To-Speech Using Multi-Level VAE and Adversarial Training" uses an adversarial accent classifier to force the speaker latent $a = E_{lang}(label) \in R^{1 \times d}.$1 to be accent-agnostic (Melechovsky et al., 2024). CrossAccent-TTS differs in making accent interpolation itself a first-class operation in the forward path (Annamdevula et al., 24 Jun 2026).

4. Objectives, training pipeline, and inference

CrossAccent-TTS optimizes a two-term objective:

$a = E_{lang}(label) \in R^{1 \times d}.$2

The decoder loss is cross-entropy over acoustic tokens,

$a = E_{lang}(label) \in R^{1 \times d}.$3

and the adversarial suppression term is weighted by $a = E_{lang}(label) \in R^{1 \times d}.$4 in all experiments (Annamdevula et al., 24 Jun 2026).

The reported training setup uses two datasets. Indic Multilingual contains approximately $a = E_{lang}(label) \in R^{1 \times d}.$5 hours covering Hindi, Tamil, Telugu, Bengali, Marathi, and English, while L2-ARCTIC contains $a = E_{lang}(label) \in R^{1 \times d}.$6 hours with six non-native English accents (Annamdevula et al., 24 Jun 2026). Objective metrics are Accent Similarity, defined as cosine similarity between GenAID embeddings of generated and target-accent speech; Accent Leakage, defined as cosine similarity between generated speech and the reference speech in GenAID space; UTMOS for overall quality; and Speaker Similarity, defined using Resemblyzer embeddings (Annamdevula et al., 24 Jun 2026). Subjective evaluation uses MOS on Accent Similarity with $a = E_{lang}(label) \in R^{1 \times d}.$7 listeners; MOS on Naturalness is described as not explicitly tabulated in the paper but implicit in UTMOS trends (Annamdevula et al., 24 Jun 2026).

Inference follows a fixed sequence (Annamdevula et al., 24 Jun 2026). Given text, reference audio, a target accent label, and intensity $a = E_{lang}(label) \in R^{1 \times d}.$8, the system computes the speaker embedding from the reference waveform, looks up source and target accent embeddings, interpolates them through the AIC, forms the modified speaker representation $a = E_{lang}(label) \in R^{1 \times d}.$9, autoregressively samples acoustic tokens from the decoder, and reconstructs the waveform with the Neucodec vocoder.

Earlier CrossAccentTTS-style pipelines adopt a different training decomposition. In the non-parallel TTS-guided accent-conversion method, a target-accent TTS system is first pretrained and frozen; a speech encoder is then optimized so that its output matches the TTS text embedding and can be decoded by the target-accent TTS decoder (Zhou et al., 2022). The speech encoder loss is

$I(s; a) \approx 0$0

with $I(s; a) \approx 0$1 and $I(s; a) \approx 0$2 (Zhou et al., 2022). This establishes a distinct but related paradigm: accent control through representation transfer rather than explicit accent embedding interpolation.

5. Empirical performance and controllability

CrossAccent-TTS reports objective results on both Indic Multilingual and L2-ARCTIC (Annamdevula et al., 24 Jun 2026). The metric order in the table below is UTMOS $I(s; a) \approx 0$3, AccLeak $I(s; a) \approx 0$4, AccSim $I(s; a) \approx 0$5, and SpkSim $I(s; a) \approx 0$6.

Dataset	Baseline snapshot	CrossAccent-TTS
Indic Multilingual	IndicF5: 2.817 / 0.312 / 0.312 / 0.843; XTTS_v2: 3.168 / 0.284 / 0.284 / 0.832	3.181 / 0.203 / 0.371 / 0.842
L2-ARCTIC	CVAE-L: 2.810 / 0.487 / 0.612 / 0.677; CVAE-NL: 2.714 / 0.530 / 0.491 / 0.673; GST: 3.044 / 0.544 / 0.670 / 0.732	4.001 / 0.439 / 0.686 / 0.693

On Indic Multilingual, the proposed model improves accent similarity over both IndicF5 and XTTS_v2, reduces accent leakage relative to both baselines, and maintains speaker similarity at essentially the same level as IndicF5 (Annamdevula et al., 24 Jun 2026). On L2-ARCTIC, the model attains the highest reported UTMOS and accent similarity among the listed systems while reducing accent leakage relative to GST and both CVAE baselines, although GST retains a higher speaker-similarity score in that table (Annamdevula et al., 24 Jun 2026). The abstract summarizes these findings by stating that CrossAccent-TTS outperforms strong baselines in accent similarity and controllability by maintaining speaker similarity and naturalness (Annamdevula et al., 24 Jun 2026).

A central empirical claim is controllability. As $I(s; a) \approx 0$7 varies from $I(s; a) \approx 0$8 to $I(s; a) \approx 0$9, the accent-similarity score grows monotonically, which is presented as confirmation that linear interpolation of language embeddings yields fine-grained, perceptually meaningful control over accent strength without retraining (Annamdevula et al., 24 Jun 2026). Subjectively, the model is reported to outperform IndicF5 and XTTS_v2 in listener-rated accent similarity for Indic accents by approximately $$L_{GRL}(\theta_{spk}, \theta_D) = E_{(s,\ell)}[- \log D_{acc}(\ell \mid GRL(s))],$$0 MOS, and to obtain significant MOS gains over CVAE and GST baselines for foreign accents (Annamdevula et al., 24 Jun 2026).

This pattern aligns with other accent-control studies. Accent Vector likewise reports that stronger accent scaling produces monotonically increasing accent probability but higher WER and slightly lower UTMOS, revealing a trade-off between accent strength and utility (Lertpetchpun et al., 8 Mar 2026). A plausible implication is that continuous accent control is operationally useful only when accompanied by explicit monitoring of intelligibility and naturalness, not accent similarity alone.

6. Related paradigms and alternative formulations

A major precursor to CrossAccentTTS is TTS-guided training for accent conversion without parallel data (Zhou et al., 2022). There, a TTS system pretrained on target-accented speech provides hidden text representations associated only with the target accent, and a speech encoder is trained to map source-accented speech to that representation space. At run-time, the speech encoder is combined with the pretrained TTS decoder to convert the source-accented speech toward the target (Zhou et al., 2022). The method reports WER of $$L_{GRL}(\theta_{spk}, \theta_D) = E_{(s,\ell)}[- \log D_{acc}(\ell \mid GRL(s))],$$1 for TTS-AC, compared with $$L_{GRL}(\theta_{spk}, \theta_D) = E_{(s,\ell)}[- \log D_{acc}(\ell \mid GRL(s))],$$2 for natural-source, $$L_{GRL}(\theta_{spk}, \theta_D) = E_{(s,\ell)}[- \log D_{acc}(\ell \mid GRL(s))],$$3 for natural-target, and $$L_{GRL}(\theta_{spk}, \theta_D) = E_{(s,\ell)}[- \log D_{acc}(\ell \mid GRL(s))],$$4 for Oracle TTS; MUSHRA around $$L_{GRL}(\theta_{spk}, \theta_D) = E_{(s,\ell)}[- \log D_{acc}(\ell \mid GRL(s))],$$5; accentedness around $$L_{GRL}(\theta_{spk}, \theta_D) = E_{(s,\ell)}[- \log D_{acc}(\ell \mid GRL(s))],$$6 on a scale where $$L_{GRL}(\theta_{spk}, \theta_D) = E_{(s,\ell)}[- \log D_{acc}(\ell \mid GRL(s))],$$7 is target and $$L_{GRL}(\theta_{spk}, \theta_D) = E_{(s,\ell)}[- \log D_{acc}(\ell \mid GRL(s))],$$8 is source; and speaker similarity above $$L_{GRL}(\theta_{spk}, \theta_D) = E_{(s,\ell)}[- \log D_{acc}(\ell \mid GRL(s))],$$9 listeners judging the output as the same speaker (Zhou et al., 2022).

Another line emphasizes latent disentanglement. "Accent Conversion in Text-To-Speech Using Multi-Level VAE and Adversarial Training" extends Tacotron 2 with a multi-level VAE and adversarial accent classifier, producing MLVAE-ADV with MCD $L_{GRL}$0, compared with $L_{GRL}$1 for MLVAE and $L_{GRL}$2 for GST, but with higher WER at $L_{GRL}$3 and lower MOS at $L_{GRL}$4 (Melechovsky et al., 2024). DART moves the same problem to a non-autoregressive FastSpeech2-style backbone with multi-level VAE grouping and dual VQ bottlenecks for speaker and accent. Its pre-trained DART512 configuration reports the best speaker cosine similarity at $L_{GRL}$5, MCD $L_{GRL}$6 dB, FFE $L_{GRL}$7, WER $L_{GRL}$8, and MOS $L_{GRL}$9 (Melechovsky et al., 2024).

A third line treats accent as a reusable control signal external to the synthesis backbone. AccentBox first trains a speaker-agnostic accent identification model, GenAID, achieving unseen-speaker macro $a_{src}$0 and accuracy $a_{src}$1 on a 13-way accent identification task, and then conditions a zero-shot TTS model on the resulting continuous accent embeddings (Zhong et al., 2024). Accent Vector, by contrast, defines accent as a LoRA task vector in parameter space:

$a_{src}$2

with scaling and interpolation performed directly on model parameters rather than embedding activations (Lertpetchpun et al., 8 Mar 2026). The paper explicitly contrasts this with CrossAccentTTS, stating that CrossAccentTTS typically uses learned accent embeddings conditioned on text or phonemes, while Accent Vector manipulates model parameters directly (Lertpetchpun et al., 8 Mar 2026).

Multilingual embedding-based TTS offers another related formulation. RADTTS-based multilingual multiaccented multispeaker synthesis injects explicit accent and speaker embeddings into alignment, duration, and mel-decoding modules, and combines this with VicReg-style regularization and cross-correlation penalties to reduce entanglement without bilingual training data (Badlani et al., 2023). Low-resource accent modeling without accent-specific frontends instead uses voice conversion to generate donor-voice target-accent data, then trains multi-speaker multi-accent TTS on real and synthetic data using a single off-the-shelf G2P frontend (Tinchev et al., 2023). In pitch-accent languages, cross-dialect TTS has been formulated with phoneme-level Accent Latent Variables predicted by a multi-dialect phoneme-level BERT (Yamauchi et al., 2024). Taken together, these systems show that CrossAccentTTS is best understood as a design space rather than a single architecture.

7. Evaluation challenges, trade-offs, and open problems

A persistent issue in CrossAccentTTS research is evaluation bias. Accent Vector explicitly notes that VoxProfile, LID, Whisper, and UTMOS are biased toward dominant English varieties, and that cross-lingual accent scores require cautious interpretation (Lertpetchpun et al., 8 Mar 2026). This is especially relevant when the target of control is accent intensity rather than categorical accent transfer, since a model may increase classifier-detected accentedness without improving perceived target-accent appropriateness.

A second issue is the trade-off between accent strength and preservation objectives. MLVAE-ADV improves accent-conversion ability relative to MLVAE and GST, but sacrifices some speaker identity, voice quality, and intelligibility (Melechovsky et al., 2024). DART reports a slight MOS drop when strongly disentangling, framed as a trade-off between accent accuracy and naturalness, and also requires accent labels at training time (Melechovsky et al., 2024). Accent Vector reports that higher scaling increases accent probability but also increases WER and slightly lowers UTMOS (Lertpetchpun et al., 8 Mar 2026). These results make clear that “better accent control” is not a scalar property.

A third issue is generalization beyond seen accents or tightly supervised settings. RADTTS-based multilingual accent control identifies extension to unseen accents as open (Badlani et al., 2023). The low-resource frontend-free method shows that a very small VC-augmented set of $a_{src}$3 utterances can teach a new accent, but still relies on a supporting multi-accent training mixture and a donor-target setup (Tinchev et al., 2023). AccentBox addresses unseen-accent generation by conditioning on continuous accent embeddings extracted from reference speech rather than on discrete accent IDs (Zhong et al., 2024). This suggests that future CrossAccentTTS systems may increasingly move from discrete accent labels toward continuous or compositional accent spaces.

Against this background, CrossAccent-TTS occupies a specific point in the design spectrum: it combines explicit disentanglement, autoregressive acoustic token generation, and a lightweight inference-time interpolation rule for accent intensity (Annamdevula et al., 24 Jun 2026). Its defining contribution is not merely cross-lingual synthesis, but the claim that accent can be modulated continuously through a dedicated accent subspace while speaker identity is preserved through a separate bottlenecked pathway.