DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech

Abstract: Autoregressive (AR) text-to-speech (TTS) models generate discrete speech tokens sequentially, which makes inference slow and can degrade robustness by propagating local errors and hallucinations. This limitation stems from their left-to-right AR commitment: each token must be determined before future speech-token context is available. However, such ordering is not an inherent requirement for TTS, as the full input text is available before synthesis. In this paper, we introduce DELTA-TTS, a lightweight LoRA-based adaptation framework that converts a pretrained AR TTS model into a discrete diffusion LLM (dLLM) for confidence-ordered speech-token decoding. To better capture the local structure of speech, DELTA-TTS incorporates a convolution module that injects local acoustic context, together with a $1/t$-weighted training objective and a time-shifted inference schedule that defer low-confidence positions to later steps. Trained on only $585$ hours of LibriTTS, DELTA-TTS achieves a $\textbf{1.75}\%$ WER on Seed-TTS test-en, outperforming its AR backbone while generating tokens $\textbf{3.3}\times$ faster. Further analysis shows that DELTA-TTS produces sharper text--speech alignment, increases overall decoding confidence, and mitigates hallucinations observed in AR generation.

• The paper introduces DELTA-TTS, a method that converts an AR TTS system into a diffusion-based model using bidirectional self-attention and parameter-efficient LoRA adapters to achieve lower WER and faster synthesis.
• It leverages convolution modules for local context modeling and a time-shifted unmasking schedule that defers uncertain token predictions, thereby improving synthesis stability and text-speech alignment.
• Empirical evaluations on Seed-TTS and LibriSpeech demonstrate significant improvements in speed and accuracy without compromising quality, highlighting the practical viability of the approach.

DELTA-TTS: Conversion of AR TTS into Diffusion LLM for Efficient, Robust Speech Synthesis

Introduction

DELTA-TTS introduces a methodology for converting a pretrained autoregressive (AR) text-to-speech (TTS) model into a discrete diffusion LLM (dLLM) using parameter-efficient Low-Rank Adaptation (LoRA) modules paired with a convolutional architecture. The motivation arises from two major AR TTS constraints: linear latency due to left-to-right token generation and restricted unidirectional context, which complicates modeling speech’s strong local dependencies. Unlike typical AR constraints, text-to-speech allows access to the full input sequence from the onset, making the strict autoregressive formulation an artificial restriction. DELTA-TTS leverages this by enabling non-autoregressive, bidirectional, confidence-ordered generation for improved synthesis efficiency and robustness.

Model Architecture and Adaptation Scheme

DELTA-TTS operates by directly augmenting a frozen, pretrained AR TTS backbone (CosyVoice3/Qwen2-0.5B) with minimal intervention. The conversion to dLLM is realized via two core architectural changes: (1) All self-attention is made bidirectional, and (2) each transformer block is decorated with two lightweight adapters—a LoRA module for adapting attention and MLP components, and a residual Conformer-style convolution module for explicit local context modeling.

Figure 1: Overview of DELTA-TTS conversion, highlighting bidirectional attention and block-local adaptation using LoRA and convolutional modules.

By freezing the AR backbone and inserting only $\sim$15% additional parameters, DELTA-TTS circumvents the immense data and compute demands of from-scratch NAR training while enabling robust parallel generation. Unlike prior text-domain AR-to-dLLM conversions, mask-annealing is not necessary due to the architectural isolation of adaptation.

Training Objective and Decoding Strategy

Leveraging the masked discrete diffusion paradigm, DELTA-TTS is trained to predict original speech tokens at randomly masked positions, with explicit $1/t$ weighting in the objective to upweight low-masking-ratio (high-confidence, late-stage) steps. A shift operation ensures the model predicts next-token positions, maintaining backbone alignment.

Decoding employs a time-shifted unmasking schedule: at each iteration, only the most confident predictions are committed, and less-certain positions are deferred to subsequent steps. This produces a confidence-ordered, “easy-first, hard-last” trajectory that structurally avoids committing on under-specified (and AR-hallucination-prone) positions until sufficient local and global context is present.

Figure 3: The shifted schedule unmasking trajectory and associated $1/t$ loss weighting for maximum training efficiency and deferred difficult-token prediction.

Empirical Evaluation & Analysis

DELTA-TTS is empirically tested on Seed-TTS test-en and LibriSpeech-PC B, demonstrating strong results:

• Seed-TTS test-en: WER of 1.75% (vs. 2.02% for AR backbone) at 3.3× speedup (RTF of 0.144)
• LibriSpeech-PC B: Outperforms all NAR baselines in WER and matches the best AR benchmarks
• No quality compromise in speaker similarity (SIM) or UTMOS, with subjective CMOS/SMOS judgments slightly favoring DELTA-TTS over original AR outputs

  Figure 4: Speed–quality Pareto front, with DELTA-TTS achieving the lowest WER and RTF among all systems at a moderate (16-step) decode budget.

Qualitative analysis reveals tighter text–speech alignment and substantially higher model confidence—especially on tokens where AR models exhibit classic failures (typically utterance-initial tokens):

Figure 5: Text–speech attention alignment in DELTA-TTS shows sharper diagonals and improved locality, with bidirectional attention enabling superior upcoming-text integration.

Figure 2: In AR, teacher-forced model confidence is nearly zero at utterance starts; DELTA-TTS consistently commits only at high certainty, averting speculative early errors.

Key ablation findings:

• Both time-shifted schedule and convolution adapters are critical—naive LoRA-only adaptation is insufficient, degrading WER and SIM
• Full backbone finetuning overfits given 585h training data, confirming the necessity of parameter-efficient adaptation in the low-resource regime
• Knowledge distillation from AR gives subpar results compared to direct AR-to-dLLM conversion

Practical and Theoretical Implications

Practically, DELTA-TTS offers an efficient mechanism for deploying zero-shot, high-quality TTS systems with minimal labeled data and compute. This is achieved while using only frozen, publicly available AR models and a small set of trainable adapters. Its superiority in speed–quality tradeoff and demonstrated stability against AR hallucination and error propagation highlight the utility of bidirectional, confidence-ordered, diffusion-based decoding for sequence generation tasks with abundant global context (such as TTS).

Theoretically, the framework extends previous text-domain AR-to-dLLM conversions to modalities with richer local dependencies and variable-length outputs. The role of local convolution in enabling robust speech alignment, and the structural advantage of deferred, context-rich token commitment, provide concrete design criteria for future diffusion LMs in TTS and possibly related structured generation domains.

Future Directions

Immediate extensions include multilingual adaptation, modeling with multi-codebook tokenizers, integration of explicit length predictors or duration modeling (removing rule-based speech length estimation), and reward-based finetuning with end-to-end metrics like WER and speaker similarity. The conversion strategy provides a direct bridge from the rapid advances in LLMs and text diffusion LMs to the domain of speech synthesis, substantially narrowing resource and performance gaps.

Conclusion

DELTA-TTS sets a new standard for the efficient adaptation of AR TTS systems to non-autoregressive, diffusion-based, confidence-ordered decoding with minimal intervention and training cost. Both in terms of synthesis speed and objective/subjective output quality, the framework consistently surpasses or rivals previous state-of-the-art AR and NAR TTS models. The generality of the architectural and schedule design, plus the robustness gains derived from bidirectional, context-aware commitment, demonstrate significant progress towards practical, scalable, and stable text-to-speech synthesis.

Figure 6: Informative-token (phoneme) decoding is progressively deferred, supporting the easy-first, hard-last generation hypothesis enabled by the confident decoding schedule.

Figure 7: DELTA-TTS commits AR-low-confidence positions only at late unmasking steps, illustrating context accumulation and robust deferment.

For further technical detail, ablations, and subjective evaluation methodologies, see (2607.04140).