Few-Shot Accent Synthesis for ASR with LLM-Guided Phoneme Editing

Abstract: Accented automatic speech recognition (ASR) often degrades due to the limited availability of accented training data. Prior work has explored accent modeling in low-resource settings, but existing approaches typically require minutes to hours of labeled speech, which may still be impractical for truly scarce accent scenarios. We propose a pipeline that adapts a text-to-speech (TTS) decoder to a target-accent speaker using fewer than ten reference utterances and employs LLM-based phoneme editing to generate accent-conditioned pronunciations. The resulting synthetic speech is used to fine-tune a self-supervised ASR model. Experiments demonstrate consistent word error rate (WER) reductions on real accented speech, including cross-speaker evaluation and ultra-low data regimes. A matched-rate random phoneme baseline shows that phoneme-space perturbation itself is a strong form of augmentation, while LLM-guided edits provide additional gains through accent-conditioned structure.

• The paper introduces a novel TTS adaptation pipeline that leverages LLM-guided phoneme editing to generate synthetic accented speech from few-shot data.
• It employs phoneme-level prosody extraction and FiLM conditioning to achieve accent-specific acoustic biases, yielding significant improvements in accent similarity.
• ASR experiments demonstrate that synthetic augmentation reduces WER effectively in low-resource regimes and generalizes across different speakers.

Few-Shot Accent Synthesis for ASR with LLM-Guided Phoneme Editing

Methodological Framework

The paper introduces a pipeline for accented ASR adaptation that leverages a few-shot speaker-adapted TTS system and LLM-guided phoneme editing to generate synthetic accented speech for data augmentation. The method is situated in ultra-low-resource regimes, requiring fewer than ten reference utterances from the target accent speaker. The backbone TTS is pretrained on Standard American English datasets and subsequently fine-tuned to the target accent speaker using mel-spectrogram reconstruction losses, eschewing auxiliary prosodic and adversarial losses for stability in limited data conditions.

The approach employs externally extracted phoneme-level prosody (duration, log-$F_0$, energy), which increases prosodic variability and broadens the data distribution. Zero-shot speaker embeddings and style vectors are combined for global conditioning. The adapted decoder is FiLM-conditioned on these vectors, resulting in accent-specific acoustic biases.

LLM-guided phoneme editing operates on aligned phoneme-prosody inputs. The editing process is constrained to insertion, deletion, splitting, and merging to preserve phoneme-prosody alignment—a crucial constraint for acoustic model consistency. In-context examples are constructed using paired source and target-accent phoneme-prosody sequences; the LLM is directed to approximate the observed phoneme substitution rate while maintaining validity. Informative in-context examples are selected based on pitch variability ratios, favoring cases with stronger accent cues.

An overview of the pipeline, including TTS adaptation and LLM-based editing, is shown below.

Figure 1: Overview of the proposed few-shot TTS adaptation and LLM-based pronunciation editing pipeline for synthetic accented speech generation.

Experimental Design and Analysis

The evaluation is performed on Indian and Korean English accents, using L2-ARCTIC for accented speech, matched Standard American speech for source utterances, and human-transcribed accent phonemes for LLM prompts. The adapted decoder generates synthetic audio from edited phoneme-prosody sequences, which are used for ASR fine-tuning with wav2vec2.0 Base pretrained on 960h English speech. Metrics include WER from Whisper as an intelligibility proxy, UTMOS for naturalness, and accent similarity (AccSim) via a wav2vec2-based accent classifier.

The experiments probe (1) acoustic accent realization across system variants, (2) ASR performance scaling with synthetic fine-tuning budgets, (3) cross-speaker ASR generalization, and (4) few-shot data efficiency.

For acoustic evaluation, adaptation increases AccSim sharply relative to the American TTS baseline (Indian: 0.27→0.69; Korean: 0.32→0.61), evidencing adaptation-induced accent biases. LLM edits further improve AccSim (+0.03 for Indian), but for Korean, high substitution density degrades WER and does not increase AccSim due to model limitations under aggressive symbolic edits. Adapt+Random phoneme perturbation shows markedly increased WER (Indian: 47.2%; Korean: 93.4%), highlighting decoder sensitivity to non-accented phonemic inputs and demonstrating that arbitrary perturbation is ineffective compared to accent-structured edits.

ASR Adaptation Efficacy

ASR fine-tuning experiments reveal that synthetic speech delivers substantial WER reductions, particularly in the few-shot regime. Adapt+LLM synthetic data outperforms Adapt+Random, manifesting additional gains attributable to accent-structured edits. However, the random control approximates Adapt+LLM at larger $N$, indicating that phoneme-space perturbation itself is a potent augmentation signal. This challenges the assumption that accent-fidelity is the principal driver—pronunciation variability is a crucial factor (Figure 2).

Figure 2: ASR performance as a function of fine-tuning budget $N$ for Indian (TNI) and Korean (HKK) English, demonstrating the relative contributions of LLM-guided and random phoneme editing.

Oracle phoneme and prosody analysis (Figure 3) shows that ground-truth phonemes with aligned prosody yield modest gains over random edits, with convergence at larger $N$. Ground-truth prosody provides the best WER, yet the incremental benefit is less pronounced than the overall synthetic-vs-real gap.

Figure 3: ASR performance using ground-truth phoneme and prosody annotations, highlighting the diminishing gains of oracle conditions over random perturbations as training budget increases.

Mixed Real+Synth training stabilizes WER under ultra-low-resource conditions (e.g., Indian English $N=3$: WER 19.48%→16.81%; Korean $N=5$: WER 20.40%→15.83%), and reduces variance (standard deviation drops from 3.11→0.49 for Indian). As $N$ increases, the influence of real speech grows—Indian English adapts rapidly and surpasses mixed training at $N ≈ 8$, while Korean English improves more gradually. Synthetic augmentation is most beneficial when real data is very limited; optimal balance shifts as more real speech is available.

Cross-speaker evaluation demonstrates that synthetic data generated from one adaptation speaker transfers effectively, reducing WER for other speakers in the same accent group, confirming that accent-structured edits are not merely speaker-dependent.

Few-Shot Adaptation Dynamics

Data efficiency analysis (Figure 4) demonstrates that joint adaptation stabilizes accent similarity and naturalness at $K≥3$ reference utterances. Decoder fine-tuning is most sensitive, with robust stabilization at $K=3$; in contrast, LLM editing quality is largely invariant to the number of in-context examples, and speaker/style embeddings contribute marginally across $N$0. This establishes that 3–5 utterances suffice for effective accent synthesis in this regime.

Figure 4: Few-shot analysis on Indian English (TNI); accent similarity and naturalness saturate with as few as three reference utterances.

Implications and Future Directions

The pipeline achieves practical ASR improvements for accented speech in ultra-low-resource settings, demonstrating that both phonological variability and accent-focused structural edits are key. The matched-rate random phoneme baseline makes a bold claim: generic phoneme perturbation alone is a strong augmentation signal, challenging established emphasis on speech realism and accent fidelity in synthetic data generation.

Theoretically, this work advances the understanding of how symbolic variability interacts with acoustic adaptation in few-shot TTS-based ASR augmentation. Practically, it enables rapid accent adaptation with minimal data investment, relevant to inclusive speech technologies and personalized ASR systems.

Future directions include explicit prosody modeling for accent-specific suprasegmental variation, disentangled accent-speaker representations for multi-speaker accent generation, optimized real/synthetic data balancing for ASR adaptation, and broader accent coverage. Robust LLM phoneme editing in the low-data regime, and improved decoder resilience to aggressive symbolic edits, are critical for maximizing accent fidelity and intelligibility.

Conclusion

The study presents an effective pipeline for few-shot accent synthesis and ASR adaptation, combining LLM-guided phoneme editing with speaker-adapted TTS, yielding substantive ASR gains in low-resource accent scenarios. It demonstrates that accent-conditioned synthetic data, intelligently generated even with sparse references, can reliably augment training for robust, inclusive ASR, with symbolic perturbation playing a critical role alongside accent fidelity. Future research should focus on prosody modeling and accent-speaker disentanglement to further advance multi-accent synthesis and adaptation.