Accent Vector: Controllable Speech Variation

• Accent vector is a representation that captures accent-specific speech patterns, distinct from speaker identity and lexical content.
• It is implemented through various forms such as utterance-level embeddings, i-vectors, continuous latent codes, and parameter-space differences.
• It facilitates robust accent control in speech tasks, supporting applications like accent recognition, conversion, and multi-accent text-to-speech systems.

An accent vector is a representation that encodes accent-bearing variation in speech while, ideally, remaining separable from speaker identity and lexical content. Across recent speech recognition, accent identification, accent conversion, and text-to-speech systems, the term does not denote a single canonical object. Instead, it is realized as utterance-level i-vectors and learned embeddings, discrete accent-ID embeddings, continuous latent variables, speaker-agnostic non-timbral embeddings, local phoneme- or mora-conditioned features, and even parameter-space task vectors that can be added to a pretrained TTS model (Siddhant et al., 2017, Jin et al., 2022, Annamdevula et al., 24 Jun 2026, Lertpetchpun et al., 8 Mar 2026).

1. Conceptual scope

In most accent-recognition and accent-controllable TTS work, “accent” refers to regional, national, or L1-influenced variation in pronunciation, rhythm, and prosody. In that setting, an accent vector is a global or sequence-level representation used to classify, transfer, or control those patterns. Voice-preserving accent conversion, for example, treats accent as a factor that should be changed while timbre and pitch-related speaker traits are preserved (Jin et al., 2022).

A different meaning appears in pitch-accent research. In neural pitch-accent detection, the relevant representation is a word-level or local prominence representation derived from acoustic and lexical features rather than a global regional-accent embedding (Stehwien et al., 2018). In PASQA, the target is Japanese pitch-accent correctness, where accent is defined through accent phrases, moras, and accent nucleus position, and the model learns localized accent-error representations together with an utterance-level accent-quality embedding (Kawamura et al., 18 Jun 2026). This distinction matters because the same phrase, “accent vector,” can refer either to a global accent identity or to a local prosodic correctness signal.

2. Major representational forms

Recent work implements accent vectors in several recurrent forms. Some are explicit fixed-dimensional embeddings extracted from speech; some are discrete learnable codes indexed by accent labels; some are latent variables in probabilistic models; and some are parameter differences between a pretrained and adapted model. The common thread is that accent is treated as a manipulable representational factor rather than a by-product of speaker embeddings alone. This suggests a useful editor’s shorthand: an accent vector is any representation dedicated to accent-bearing variability and used operationally for control or discrimination.

Representation family	Example realization	Representative papers
Utterance embedding	300-dim DNN i-vector, refined to a 128-dim Siamese embedding	(Siddhant et al., 2017)
Accent-ID embedding	Trainable embedding $\mathbf{e}_a = E[a]$, repeated frame-wise in a pronunciation encoder	(Jin et al., 2022)
Continuous latent variable	Pronunciation/accent latent $z_{pr}$; group-level accent latent $z_a^g$ with VQ	(Ma et al., 2023, Melechovsky et al., 2024)
Language/accent embedding	Language embedding $e_{\text{lang}} \in \mathbb{R}^{d}$ injected into an accent subspace	(Annamdevula et al., 24 Jun 2026)
Speaker-agnostic accent embedding	WNTA64 non-timbral embedding; 256-dim WhisAID accent embedding	(Bakari et al., 28 Apr 2026, Wang et al., 15 Jun 2026)
Parameter-space vector	Task vector $$\tau_{\text{accent}} = \theta_{\text{ft}} - \theta_{\text{pre}}$$	(Lertpetchpun et al., 8 Mar 2026)

The earliest form in the supplied literature is the utterance-level i-vector. In “Leveraging Native Language Speech for Accent Identification using Deep Siamese Networks” (Siddhant et al., 2017), accent vectors are implemented as 400-dimensional GMM i-vectors or 300-dimensional DNN i-vectors and then mapped by a shared two-layer Siamese network into a 128-dimensional accent-discriminative space. In “Voice-preserving Zero-shot Multiple Accent Conversion” (Jin et al., 2022), the corresponding object is a trainable accent embedding associated with each accent ID and concatenated to every frame of wav2vec 2.0 text predictions inside the pronunciation encoder.

Later TTS systems move toward explicitly disentangled latent variables. Accent-VITS represents accent primarily through the pronunciation latent $z_{pr}$, whose posterior is inferred from ASR bottleneck features and whose prior is predicted from accent-dependent phoneme sequences (Ma et al., 2023). DART instead uses a group-level ML-VAE latent $z_a^g$ followed by a VQ bottleneck, so the effective accent vector at runtime is a discrete codebook entry $\tilde{z}_a^g$ selected from an accent codebook (Melechovsky et al., 2024). CrossAccent-TTS makes the implementation even more explicit: the accent vector is the language embedding $e_{\text{lang}}$, or a linear mixture of such embeddings, injected into an accent subspace with $d=768$ (Annamdevula et al., 24 Jun 2026).

Two additional lines extend the notion further. Robust AID work uses WNTA64 non-timbral embeddings, $z_{pr}$0, as speaker-invariant accent vectors for both classification and TTS conditioning (Bakari et al., 28 Apr 2026). Joycent uses a 256-dimensional WhisAID accent embedding produced by a Whisper-based accent identification model and injected through conditional layer normalization (Wang et al., 15 Jun 2026). Finally, “Accent Vector: Controllable Accent Manipulation for Multilingual TTS Without Accented Data” (Lertpetchpun et al., 8 Mar 2026) defines the accent vector directly in parameter space: $z_{pr}$1 which, under LoRA fine-tuning, is exactly the learned LoRA weight update.

3. Disentanglement from speaker identity

A central technical issue is that accent and speaker identity are strongly entangled in speech. Several systems therefore treat an accent vector as meaningful only if it is forced into a representation space distinct from timbre and speaker-specific acoustics.

A common strategy is adversarial training. In “Voice-preserving Zero-shot Multiple Accent Conversion” (Jin et al., 2022), the acoustic encoder outputs a 256-dimensional global vector $z_{pr}$2 from MFCC and periodicity features, while accent is supplied by a separate accent embedding $z_{pr}$3. A binary accent discriminator, labeling speech as native or foreign, is trained adversarially on $z_{pr}$4 so that $z_{pr}$5 becomes accent-invariant and the accent embedding becomes the only explicit accent-conditioned global code. CrossAccent-TTS makes the same separation more explicitly: the Perceiver Resampler produces $z_{pr}$6 with $z_{pr}$7 and $z_{pr}$8, and an Accent Suppression Module with a GRL and auxiliary accent classifier forces $z_{pr}$9 to be accent-agnostic, with $z_a^g$0 in the total loss (Annamdevula et al., 24 Jun 2026). On evaluation, the proposed model reports speaker similarity of $z_a^g$1 for Indic accents and $z_a^g$2 on L2-ARCTIC, supporting the intended separation of timbre from accent.

Other systems achieve disentanglement structurally rather than only adversarially. Accent-VITS keeps speaker conditioning out of the pronunciation latent $z_a^g$3 and lets speaker identity enter only through the acoustic prior $z_a^g$4, with BN features acting as a speaker-independent constraint on the accent latent (Ma et al., 2023). DART uses ML-VAE grouping assumptions, separate priors for speaker and accent latents, and distinct VQ codebooks for speaker and accent; its appendix t-SNE plots show accent embeddings clustering by accent and speaker embeddings clustering by speaker, and the DART$z_a^g$5 variant shows high cosine similarity, approximately $z_a^g$6 (Melechovsky et al., 2024).

Multi-scale accent modeling applies the same principle at two temporal scales. In the Tacotron-2-based MSADM system, a Global Accent Disentangling Model learns an utterance-level vector $z_a^g$7, a Local Accent Disentangling Model learns phoneme-level vectors $z_a^g$8, and both are trained with accent classification losses and GRL-based speaker losses, while a separate 256-dimensional speaker embedding $z_a^g$9 is injected only in the decoder (Zhou et al., 2024). The resulting global accent embeddings form tight, well-separated clusters by accent, whereas the non-disentangled baseline clusters by both accent and speaker.

Recent AID models also turn disentanglement into a first-class objective. The WNTA64-based AID system regularizes the speaker head toward a uniform distribution with a KL term and reaches a new state-of-the-art F1-score of $e_{\text{lang}} \in \mathbb{R}^{d}$0, compared to the previous score of $e_{\text{lang}} \in \mathbb{R}^{d}$1, on unseen speakers in GenAID (Bakari et al., 28 Apr 2026). Joycent’s WhisAID uses a Whisper encoder, an accent head that produces a 256-dimensional embedding, and a speaker head connected through a GRL with $e_{\text{lang}} \in \mathbb{R}^{d}$2; lower SCSC indicates less speaker clustering in the accent embeddings (Wang et al., 15 Jun 2026). A plausible implication is that an accent vector becomes most useful when it is not merely accent-discriminative but actively speaker-agnostic.

4. Accent vectors in recognition and identification

In recognition tasks, accent vectors are used either as the direct object of classification or as intermediate control variables for feature adaptation. The literature shows a progression from utterance-level acoustic summaries to explicitly structured and speaker-robust accent spaces.

The i-vector/Siamese line is one early example. “Leveraging Native Language Speech for Accent Identification using Deep Siamese Networks” (Siddhant et al., 2017) represents each utterance by a 300-dimensional DNN i-vector and learns a 128-dimensional embedding with contrastive loss over pairs of accented English and native-language speech. The best Siamese test strategy, Siamese-4, achieves $e_{\text{lang}} \in \mathbb{R}^{d}$3 test accuracy on a 10-class accent identification task, outperforming SVM and NNET baselines on the same DNN i-vectors.

Another strand treats accent as a structured adaptation variable inside ASR. “Layer-wise Fast Adaptation for End-to-End Multi-Accent Speech Recognition” (Gong et al., 2022) replaces one-hot accent vectors with a continuous 256-dimensional accent embedding $e_{\text{lang}} \in \mathbb{R}^{d}$4 extracted from a pretrained AID model and converts it into mixture coefficients $e_{\text{lang}} \in \mathbb{R}^{d}$5 over multiple accent bases. The accent-conditioned adapter computes

$e_{\text{lang}} \in \mathbb{R}^{d}$6

and the proposed structure yields $e_{\text{lang}} \in \mathbb{R}^{d}$7 and $e_{\text{lang}} \in \mathbb{R}^{d}$8 relative WER reduction on AESRC2020 and LibriSpeech, respectively. Because $e_{\text{lang}} \in \mathbb{R}^{d}$9 is a soft mixture over bases rather than a one-hot label, the method can handle unseen accents more naturally.

Accent-shift modeling goes further by defining the accent vector as a relative deviation from a text-derived anchor. In LASAS, frame-aligned text embeddings are projected into multiple accent-associated spaces, acoustic embeddings are projected into the same spaces, and a concatenation of similarity scores

$$\tau_{\text{accent}} = \theta_{\text{ft}} - \theta_{\text{pre}}$$0

serves as an accent shift vector (Shao et al., 2022). Combined with a dimension-reduced text reference, this yields a bimodal representation that reaches $$\tau_{\text{accent}} = \theta_{\text{ft}} - \theta_{\text{pre}}$$1 accuracy on the AESRC test set, a $$\tau_{\text{accent}} = \theta_{\text{ft}} - \theta_{\text{pre}}$$2 relative improvement over a competitive challenge system.

Deep discriminative accent recognition explicitly optimizes the utterance-level accent vector geometry. In “Deep Discriminative Feature Learning for Accent Recognition” (Wang et al., 2020), a CRNN front-end and many-to-one BiGRU produce an utterance embedding $$\tau_{\text{accent}} = \theta_{\text{ft}} - \theta_{\text{pre}}$$3. With ASR initialization, CTC auxiliary training, and Circle-Loss, the system reaches $$\tau_{\text{accent}} = \theta_{\text{ft}} - \theta_{\text{pre}}$$4 test accuracy on AESRC2020, and the 3D visualizations show the most compact and well-separated accent clusters among the compared losses.

Speaker-agnostic AID embeddings now feed directly into downstream generation. AccentBox’s GenAID uses an XLSR backbone, a 64-dimensional bottleneck, inverse-frequency sampling, speed and noise augmentation, and an adversarial speaker objective, reducing SCSC from $$\tau_{\text{accent}} = \theta_{\text{ft}} - \theta_{\text{pre}}$$5 in the baseline to $$\tau_{\text{accent}} = \theta_{\text{ft}} - \theta_{\text{pre}}$$6 and improving unseen-speaker F1 from $$\tau_{\text{accent}} = \theta_{\text{ft}} - \theta_{\text{pre}}$$7 to $$\tau_{\text{accent}} = \theta_{\text{ft}} - \theta_{\text{pre}}$$8 (Zhong et al., 2024). Robust Accent Identification via Voice Conversion and Non-Timbral Embeddings reports that WNTA64-based AID reaches F1 $$\tau_{\text{accent}} = \theta_{\text{ft}} - \theta_{\text{pre}}$$9 and accuracy $z_{pr}$0 on unseen speakers, while VC augmentation with kNN-VC lifts a raw-feature baseline from F1 $z_{pr}$1 to $z_{pr}$2 (Bakari et al., 28 Apr 2026). Taken together, these results suggest that modern accent vectors in recognition are increasingly defined by speaker invariance and cross-speaker transfer rather than by raw acoustic separability alone.

5. Accent vectors in synthesis, conversion, and controllability

In speech generation, accent vectors function as conditioning variables that alter phonetic realization, rhythm, and prosody while preserving speaker identity. The main implementation styles are accent-ID embeddings, continuous latent codes, extracted reference embeddings, and parameter-space task vectors.

Early explicit control appears in voice-preserving accent conversion. The 2022 zero-shot multiple accent conversion system uses a trainable accent embedding $z_{pr}$3 injected into a pronunciation encoder, while a separate acoustic vector $z_{pr}$4 carries speaker timbre and F0-related patterns (Jin et al., 2022). At inference, keeping $z_{pr}$5 fixed and switching $z_{pr}$6 enables conversion from any input accent to any of eight target accents for unseen speakers. Accent-VITS replaces discrete accent IDs with a continuous pronunciation latent $z_{pr}$7 inferred from BN features and predicted from accent-dependent phoneme sequences, thereby decomposing text-to-wave generation into text-to-accent and accent-to-wave mappings (Ma et al., 2023).

Multi-speaker multi-accent TTS extends this to multiple temporal scales. In the MSADM system, a global accent vector $z_{pr}$8 captures utterance-level accent characteristics and a local accent sequence $z_{pr}$9 captures phoneme-level accent variation, while a Local Accent Prediction Model predicts $z_a^g$0 from text and a chosen global accent vector so that accented speech can be generated from phonemes alone (Zhou et al., 2024). DART uses a group-level latent $z_a^g$1 and accent VQ codebook alongside a separate speaker VQ codebook, enabling arbitrary speaker–accent combinations at inference (Melechovsky et al., 2024).

Zero-shot accent generation systems externalize accent extraction. AccentBox conditions a YourTTS-derived model on pretrained speaker-agnostic accent embeddings from GenAID, replacing the one-hot language embeddings in the text encoder and stochastic duration predictor (Zhong et al., 2024). Joycent instead extracts a 256-dimensional WhisAID accent embedding from reference speech and injects it via conditional layer normalization in the first Conformer block of a Grad-TTS-style text encoder, while the speaker embedding is injected by CLN only in the last block (Wang et al., 15 Jun 2026). The ablation reported as E5, which conditions only the decoder by concatenating accent and speaker embeddings with the score-network input, collapses accent metrics, indicating that accent vectors are most effective when they shape text-side representations before acoustic generation.

CrossAccent-TTS makes control algebraic at inference time. Each language/accent has a learnable embedding $z_a^g$2, with $z_a^g$3, and the Accent Intensity Controller forms a weighted mixture

$z_a^g$4

This accent vector is repeated over $z_a^g$5 speaker/style slots and added to the accent-neutral representation: $z_a^g$6 The system synthesizes accent intensities $z_a^g$7, $z_a^g$8, $z_a^g$9, and $\tilde{z}_a^g$0, and accent similarity increases monotonically with intensity. On Indic accents it reports AccSim $\tilde{z}_a^g$1 and AccLeak $\tilde{z}_a^g$2, compared with higher leakage and lower similarity for the baselines; on L2-ARCTIC it reports AccSim $\tilde{z}_a^g$3 and AccLeak $\tilde{z}_a^g$4 (Annamdevula et al., 24 Jun 2026).

The most radical formulation is parameter-space control. “Accent Vector: Controllable Accent Manipulation for Multilingual TTS Without Accented Data” (Lertpetchpun et al., 8 Mar 2026) defines accent as a task vector: $\tilde{z}_a^g$5 and applies it at inference through

$\tilde{z}_a^g$6

Because fine-tuning uses LoRA, $\tilde{z}_a^g$7 is exactly the LoRA weights. Scaling $\tilde{z}_a^g$8 gives fine-grained control over accent strength, and multiple accents can be mixed by summing their task vectors. The same work shows that accent control generalizes beyond English: English-accented Spanish, German, and Mandarin can be produced by using English-derived Accent Vectors with non-English base languages.

6. Evaluation criteria, misconceptions, and open issues

Evaluation of accent vectors varies sharply with task, which is one reason the term remains heterogeneous. Accent recognition work emphasizes accuracy, F1, WER reduction, or cluster structure on unseen speakers (Gong et al., 2022, Zhong et al., 2024, Bakari et al., 28 Apr 2026). TTS and conversion work adds accent similarity, accent leakage, speaker similarity, MOS, AMOS, SMOS, UTMOS, or cosine similarity in an accent-embedding space (Annamdevula et al., 24 Jun 2026, Zhou et al., 2024, Wang et al., 15 Jun 2026). PASQA evaluates a different axis entirely: ordering by accent-error severity, agreement with human accent-correctness judgments, and frame-level localization of pitch-accent errors (Kawamura et al., 18 Jun 2026).

A frequent misconception is that an accent vector must be a single utterance-level embedding. The cited work shows otherwise. It may be a global vector, such as WNTA64 or a 64-dimensional GenAID bottleneck (Bakari et al., 28 Apr 2026, Zhong et al., 2024); a sequence of local phoneme-level vectors $\tilde{z}_a^g$9 (Zhou et al., 2024); a latent variable with prior and posterior distributions such as $e_{\text{lang}}$0 (Ma et al., 2023); a group-level VQ code $e_{\text{lang}}$1 (Melechovsky et al., 2024); a 256-dimensional reference embedding extracted by WhisAID (Wang et al., 15 Jun 2026); or a parameter delta in the weights of a multilingual TTS model (Lertpetchpun et al., 8 Mar 2026).

A second misconception is that accent vectors are interchangeable with speaker embeddings. Multiple papers explicitly reject that assumption. One-hot accent vectors require prior knowledge of the target accent and cannot handle unseen accents well (Gong et al., 2022). Speaker embeddings often entangle accent with timbre, which is why GenAID, WNTA64, CrossAccent-TTS, DART, and WhisAID all introduce explicit mechanisms to suppress or bypass speaker information (Zhong et al., 2024, Bakari et al., 28 Apr 2026, Annamdevula et al., 24 Jun 2026, Melechovsky et al., 2024, Wang et al., 15 Jun 2026). This suggests that accent vectors are most stable when they are learned with either adversarial pressure or architectural factorization.

The literature also identifies trade-offs. Stronger accent control can increase WER and slightly reduce UTMOS when parameter-space Accent Vectors are scaled up (Lertpetchpun et al., 8 Mar 2026). DART reports that stronger disentanglement via VQ and pre-training slightly hurts MOS even while improving accent similarity (Melechovsky et al., 2024). Joycent argues that text-based accented phone prediction is insufficient for prosody and rhythm, implying that symbolic accent representations alone may under-model acoustic accent characteristics (Wang et al., 15 Jun 2026). Voice-preserving zero-shot multiple accent conversion explicitly notes that zero-shot generalization is with respect to new speakers, not new accents; new accent IDs would still require training or fine-tuning (Jin et al., 2022).

A final open issue is generalization beyond the training accent taxonomy. Joycent evaluates unseen speakers but not unseen accents (Wang et al., 15 Jun 2026). AccentBox presents unseen-accent generation qualitatively but leaves more systematic evaluation for future work (Zhong et al., 2024). Layer-wise accent adaptation motivates continuous accent spaces partly because one-hot schemes fail on unseen accents (Gong et al., 2022). Taken together, these works suggest that the field is converging on a broader view: an accent vector is not merely a label embedding but a controllable accent-bearing subspace whose usefulness depends on speaker invariance, interpolation behavior, and robustness outside the closed set of training accents.