DELTA-TTS: Diffusion-Style TTS

• DELTA-TTS is a text-to-speech method that converts a pretrained autoregressive model into a discrete diffusion framework using confidence-ordered masked token prediction.
• It integrates lightweight LoRA adapters and Conformer-style convolutions to leverage both bidirectional attention and local acoustic continuity, enhancing text–speech alignment.
• Empirical results show a 3.3× speedup, lower WER, and reduced hallucination, demonstrating significantly improved intelligibility and efficiency in synthesis.

DELTA-TTS is a text-to-speech method that adapts a pretrained autoregressive text-to-speech model into a discrete diffusion LLM for confidence-ordered speech-token decoding. Instead of generating discrete speech tokens strictly left to right, it predicts masked positions iteratively, committing high-confidence tokens first and deferring uncertain positions until more bidirectional context is available. In the reported implementation, DELTA-TTS uses a frozen CosyVoice3 zero-shot synthesis pipeline with a Qwen2-0.5B LLM, 25 Hz semantic speech tokens, and a flow-matching decoder, while training only lightweight adapters on 585 hours of LibriTTS. On Seed-TTS test-en, it reports 1.75% WER and a 3.3× token-generation speedup over its autoregressive backbone, together with sharper text–speech alignment and reduced hallucination (Moon et al., 5 Jul 2026).

1. Conceptual basis and problem setting

DELTA-TTS is motivated by a structural limitation of autoregressive TTS. A standard autoregressive model factorizes speech-token generation as

$$P(A \mid T) = P(a_1, a_2, \dots \mid T) = \prod_i P(a_i \mid a_{<i}, T),$$

which forces a left-to-right commitment order even though the full input text is already available before synthesis (Moon et al., 5 Jul 2026).

The method argues that this ordering is not intrinsic to TTS. Speech tokens exhibit strong local dependencies, and early autoregressive errors can cascade because each token must be determined before future speech-token context is available. DELTA-TTS therefore replaces temporal decoding order with a confidence-ordered iterative masked prediction process. The target speech sequence is treated as a masked sequence; at training time random target positions are replaced by a mask token, and at inference time the entire target speech sequence is initialized as masks. Each masked target token can then exploit the transcript, prompt speech, and already revealed nearby target tokens through bidirectional attention (Moon et al., 5 Jul 2026).

This places DELTA-TTS within a broader shift from causal speech-token generation toward diffusion-style or masked generation. A closely related line is LLaDA-TTS, which also replaces autoregressive decoding with masked diffusion and reports that a fixed number of denoising steps decouples latency from output length (Fan et al., 27 Mar 2026). DELTA-TTS differs in the adaptation strategy emphasized in its paper: it is explicitly presented as a lightweight LoRA-based conversion of a pretrained autoregressive TTS system, augmented with a convolution module to better capture local acoustic structure (Moon et al., 5 Jul 2026).

2. Backbone, input representation, and adaptation modules

The reported DELTA-TTS system uses CosyVoice3 as its autoregressive backbone. The backbone consists of a Qwen2-0.5B LLM that produces 25 Hz semantic speech tokens, followed by a flow-matching decoder that converts tokens to waveform. A central design choice is that the speech tokenizer and the flow-matching decoder are frozen; only the autoregressive backbone is adapted into a discrete diffusion LLM. This preserves the zero-shot synthesis pipeline while changing the token generator (Moon et al., 5 Jul 2026).

The bidirectional input format follows CosyVoice3’s zero-shot configuration: $$[\text{SOS}, \mathbf{t}_\text{inst}, \mathbf{t}_\text{prompt}, \mathbf{t}_\text{target}, \text{TASK}, \mathbf{s}_\text{prompt}, \mathbf{s}_\text{target}, \text{EOS}].$$ Masking is applied only to $\mathbf{s}_\text{target}$. This choice preserves the text prompt, instruction tokens, prompt speech tokens, and special symbols as fully visible conditioning context during both training and inference (Moon et al., 5 Jul 2026).

Architecturally, DELTA-TTS makes two modifications to the pretrained transformer. First, it replaces causal attention with bidirectional self-attention. Second, it inserts lightweight trainable adapters into every transformer block: LoRA modules and a Conformer-style convolution module. The LoRA adapters target the query, key, value, output, gate, up, and down projections, with scaling $\alpha = 128$. The trainable parameter budget is about 94M parameters, broken down as 35M from LoRA and 59M from the convolution modules, or about 15% extra parameters on top of the frozen CosyVoice3 pipeline (Moon et al., 5 Jul 2026).

The convolution branch is a residual Conformer-style module inserted after each transformer block. It consists of depth-wise convolution, GLU gating, and Swish activation. The stated motivation is that 25 Hz speech tokens have very strong local continuity, whereas bidirectional attention supplies global context without strongly biasing the model toward short-range acoustic structure. The convolution module therefore provides a local acoustic inductive bias for neighboring speech-token interactions, prosodic continuity, and phoneme transitions (Moon et al., 5 Jul 2026).

This design choice distinguishes DELTA-TTS from masked-diffusion conversion methods that rely only on attention-mask changes. The paper’s interpretation is that global bidirectional context and local convolutional mixing are complementary: bidirectional attention supports contextual disambiguation, while convolution sharpens local text–speech alignment and reduces hallucination-prone instability (Moon et al., 5 Jul 2026).

3. Training objective and confidence-ordered inference

DELTA-TTS uses a masked discrete diffusion objective. Let $\mathbf{s}_0$ denote the original target speech-token sequence. A masking ratio $t \in (0,1)$ is sampled, and $\mathbf{s}_t$ is formed by replacing each token independently with a mask token $\mathrm{M}$ with probability $t$. The training loss is

$$\mathcal{L}(\theta) \triangleq -\mathbb{E}_{t, \mathbf{s}_0, \mathbf{s}_t}\biggl[\frac{1}{t} \sum_{i=1}^{L} \mathbf{1}[\mathbf{s}_t^i = \mathrm{M}]\, \log p_\theta(\mathbf{s}_0^i \mid \mathbf{c}, \mathbf{s}_t)\biggr],$$

where $$[\text{SOS}, \mathbf{t}_\text{inst}, \mathbf{t}_\text{prompt}, \mathbf{t}_\text{target}, \text{TASK}, \mathbf{s}_\text{prompt}, \mathbf{s}_\text{target}, \text{EOS}].$$0 is the target speech length and $$[\text{SOS}, \mathbf{t}_\text{inst}, \mathbf{t}_\text{prompt}, \mathbf{t}_\text{target}, \text{TASK}, \mathbf{s}_\text{prompt}, \mathbf{s}_\text{target}, \text{EOS}].$$1 denotes the visible context. The $$[\text{SOS}, \mathbf{t}_\text{inst}, \mathbf{t}_\text{prompt}, \mathbf{t}_\text{target}, \text{TASK}, \mathbf{s}_\text{prompt}, \mathbf{s}_\text{target}, \text{EOS}].$$2 factor is described as coming from the ELBO of the masked diffusion process and amplifies the loss at low masking ratios, where the model should predict with high confidence (Moon et al., 5 Jul 2026).

A shift operation is retained from prior autoregressive-to-diffusion conversion methods: the hidden state at position $$[\text{SOS}, \mathbf{t}_\text{inst}, \mathbf{t}_\text{prompt}, \mathbf{t}_\text{target}, \text{TASK}, \mathbf{s}_\text{prompt}, \mathbf{s}_\text{target}, \text{EOS}].$$3 predicts $$[\text{SOS}, \mathbf{t}_\text{inst}, \mathbf{t}_\text{prompt}, \mathbf{t}_\text{target}, \text{TASK}, \mathbf{s}_\text{prompt}, \mathbf{s}_\text{target}, \text{EOS}].$$4 rather than $$[\text{SOS}, \mathbf{t}_\text{inst}, \mathbf{t}_\text{prompt}, \mathbf{t}_\text{target}, \text{TASK}, \mathbf{s}_\text{prompt}, \mathbf{s}_\text{target}, \text{EOS}].$$5. This preserves the backbone’s next-token prediction behavior under the new masked-diffusion objective. The only newly added embedding is the mask token $$[\text{SOS}, \mathbf{t}_\text{inst}, \mathbf{t}_\text{prompt}, \mathbf{t}_\text{target}, \text{TASK}, \mathbf{s}_\text{prompt}, \mathbf{s}_\text{target}, \text{EOS}].$$6, initialized as the mean of all speech-token embeddings and refined during training (Moon et al., 5 Jul 2026).

Inference is iterative and parallel. At each decoding step, the model predicts all currently masked positions, samples candidate tokens using top-$$[\text{SOS}, \mathbf{t}_\text{inst}, \mathbf{t}_\text{prompt}, \mathbf{t}_\text{target}, \text{TASK}, \mathbf{s}_\text{prompt}, \mathbf{s}_\text{target}, \text{EOS}].$$7 nucleus sampling, computes a confidence score for each sampled token, commits the most confident tokens, and leaves the remaining positions masked for later steps. Confidence is defined as the probability of the sampled token (Moon et al., 5 Jul 2026).

The paper introduces a time-shifted schedule rather than a uniform reveal schedule. The cumulative fraction of tokens unmasked after step $$[\text{SOS}, \mathbf{t}_\text{inst}, \mathbf{t}_\text{prompt}, \mathbf{t}_\text{target}, \text{TASK}, \mathbf{s}_\text{prompt}, \mathbf{s}_\text{target}, \text{EOS}].$$8 is

$$[\text{SOS}, \mathbf{t}_\text{inst}, \mathbf{t}_\text{prompt}, \mathbf{t}_\text{target}, \text{TASK}, \mathbf{s}_\text{prompt}, \mathbf{s}_\text{target}, \text{EOS}].$$9

with total decoding steps $\mathbf{s}_\text{target}$0, current step $\mathbf{s}_\text{target}$1, and time-shifting parameter $\mathbf{s}_\text{target}$2. The number of decisions at step $\mathbf{s}_\text{target}$3 is

$\mathbf{s}_\text{target}$4

For $\mathbf{s}_\text{target}$5, few tokens are unmasked early and most decisions are deferred to later stages. The intended effect is a confidence-to-context trajectory: early steps resolve only easy positions, and later steps use those positions as anchors for more difficult predictions (Moon et al., 5 Jul 2026).

This inference logic is closely aligned with the paper’s robustness claim. A plausible implication is that DELTA-TTS treats token order as an optimization variable rather than a temporal inevitability: token commitment is driven by certainty under context, not by absolute position in the acoustic sequence.

4. Reported empirical performance

The DELTA-TTS paper trains on LibriTTS, using 585 hours of English speech, and evaluates zero-shot synthesis on Seed-TTS test-en and LibriSpeech-PC test-clean Subset B. The reported metrics are WER, speaker similarity via cosine similarity from a WavLM-large-based ECAPA-TDNN, UTMOS, real-time factor on a single NVIDIA A100, and subjective CMOS and SMOS (Moon et al., 5 Jul 2026).

On Seed-TTS test-en, DELTA-TTS reports WER 1.75% and SIM 0.688. The CosyVoice3 autoregressive backbone reports WER 2.02% and SIM 0.692. DELTA-TTS is therefore reported as better on intelligibility with slightly lower SIM than the backbone. In subjective evaluation, DELTA-TTS reports CMOS $\mathbf{s}_\text{target}$6 and SMOS 4.30, compared with $\mathbf{s}_\text{target}$7 and 4.03 for CosyVoice3. The paper also notes that SMOS in that setup is above the ground-truth reference’s 3.94 (Moon et al., 5 Jul 2026).

On LibriSpeech-PC Subset B, DELTA-TTS reports WER 2.17%, SIM 0.69, and UTMOS 4.27. The paper characterizes these results as better than every non-autoregressive baseline, second only to the strongest autoregressive system in WER, and competitive in naturalness and similarity (Moon et al., 5 Jul 2026).

Speed is a central result. On Seed-TTS test-en, DELTA-TTS reports RTF 0.144 and a 3.3× speedup over CosyVoice3. The advantage grows with utterance duration because the number of decoding iterations is fixed while autoregressive cost grows linearly with output length. The paper gives length-bucketed speedups of 2.08× for 0–3 s, 3.30× for 3–5 s, and 4.46× for 5–10 s, while WER remains better than CosyVoice3 in every bucket. A decoding-step sweep over $\mathbf{s}_\text{target}$8 indicates that 16 steps provides the best speed–quality balance and is used as the default setting (Moon et al., 5 Jul 2026).

These results place DELTA-TTS alongside recent non-autoregressive or semi-parallel TTS systems that explicitly target the latency bottleneck of speech-token generation. LLaDA-TTS likewise reports that generation can be completed in a fixed number of denoising steps, reaching 0.98% CER on Chinese and 1.96% WER on English with 64 steps while enabling zero-shot speech editing (Fan et al., 27 Mar 2026). DELTA-TTS is narrower in scope—it focuses on converting an existing AR TTS model rather than redesigning the full stack—but its reported 3.3× speedup with only adapter training is the core practical claim (Moon et al., 5 Jul 2026).

5. Ablations, alignment analysis, and hallucination behavior

The ablation study is a central part of the DELTA-TTS evidence. A naive conversion using LoRA, no convolution, and a uniform schedule reports WER 3.01, SIM 0.574, and UTMOS 4.04. Adding the time-shifted schedule improves WER to 2.59 and SIM to 0.586. Adding the convolution module with $\mathbf{s}_\text{target}$9 further reduces WER to 1.61, with SIM 0.584 and UTMOS 4.09. This is one of the largest reported gains and is the main empirical argument for local acoustic bias (Moon et al., 5 Jul 2026).

Prompt conditioning variants indicate that target-length estimation matters but is not dominant. With ground-truth length, the model reports WER 1.63 and SIM 0.686; with rule-based length, WER 1.75 and SIM 0.688. Full fine-tuning of the backbone performs worse than parameter-efficient adaptation, with WER 1.97 and 494M trainable parameters. Knowledge distillation from CosyVoice3 also underperforms direct autoregressive-to-diffusion conversion, reporting WER 2.37 and SIM 0.591 (Moon et al., 5 Jul 2026).

The paper gives several analyses of why the method works. For alignment, DELTA-TTS is reported to produce a much cleaner middle-layer attention diagonal, while later layers can attend to upcoming text in a way that causal autoregressive models cannot. The convolution module increases local attention to neighboring unmasked speech while keeping text attention stable (Moon et al., 5 Jul 2026).

For confidence, the paper reports that the autoregressive model has very low confidence at the beginning of utterances. Teacher-forced confidence on ground-truth tokens is described as near zero at the first five positions, with a median confidence of 0.023. DELTA-TTS instead commits positions at substantially higher confidence, with median confidence 0.193, about 8× higher. This is presented as the central explanation for the improvement in robustness (Moon et al., 5 Jul 2026).

Hallucination reduction is illustrated through failure cases in which CosyVoice3 collapses into hallucinated transcripts while DELTA-TTS synthesizes them correctly. The paper attributes this to deferred commitment: low-confidence positions are left masked until richer context has accumulated. Across multiple random seeds, the autoregressive model repeatedly fails on the same hallucination cases, whereas DELTA-TTS does not (Moon et al., 5 Jul 2026).

This suggests that DELTA-TTS is not merely a faster decoder. Its decoding policy changes the distribution of commitment errors: instead of locking in uncertain early tokens, it delays them until they are conditioned on more evidence. That is a different robustness mechanism from approaches that improve TTS by changing intermediate representations, as in DelightfulTTS 2, which replaces mel-spectrograms with learned codec latents and jointly optimizes acoustic model and waveform decoder (Liu et al., 2022).

6. Relation to adjacent TTS directions, scope, and limitations

DELTA-TTS belongs to a broader family of work that challenges conventional TTS factorization, alignment, or decoding order. Several neighboring directions are relevant but technically distinct.

The first is end-to-end representation learning. DelightfulTTS 2 addresses the mismatch between separately trained acoustic model and vocoder by learning an intermediate frame-level codec representation with a VQ-GAN-style speech codec and jointly optimizing the acoustic model and waveform decoder. Its target is the mel-spectrogram bottleneck and train–inference mismatch in cascaded TTS, rather than left-to-right token decoding (Liu et al., 2022).

The second is TTS-driven data augmentation. “TTS-by-TTS” uses a Tacotron 2-based autoregressive teacher with LP-WaveNet to synthesize a large aligned corpus for training a non-autoregressive student, specifically FastSpeech 2 + Parallel WaveGAN. That work addresses low-resource data scarcity and non-autoregressive quality degradation, not autoregressive commitment order itself (Hwang et al., 2020).

The third is multilingual and disentangled controllable synthesis. The RADTTS-based multilingual multiaccented system introduces explicit accent, language, speaker, $\alpha = 128$0, and energy control, together with embedding regularization to disentangle speaker and accent without bilingual speakers. Its technical emphasis is factor disentanglement under multilingual data imbalance, rather than diffusion-style token infilling (Badlani et al., 2023).

The fourth is diffusion or flow-based alternatives that remove or weaken domain-specific alignment constraints. DiTTo-TTS uses a latent diffusion model over compressed speech latents, frozen pretrained text encoders, and an utterance-level speech length predictor instead of phoneme durations, scaled to 82K hours and up to 790M parameters (Lee et al., 2024). ARDiT models continuous speech latents autoregressively with a diffusion or flow-matching objective and then distills the sampler to one step, emphasizing continuous representations and speech editing (Liu et al., 2024). TADA adopts text-acoustic dual alignment to create one-to-one synchronized acoustic tokens per text token, reducing hallucination and inference cost in LLM-based TTS and spoken language modeling (Dang et al., 26 Feb 2026). LLaDA-TTS converts an autoregressive speech-token decoder into masked diffusion with bidirectional attention and fixed-step parallel denoising, while also supporting zero-shot insertion, deletion, and substitution without additional training (Fan et al., 27 Mar 2026).

Within this landscape, DELTA-TTS is most precisely characterized as a parameter-efficient autoregressive-to-diffusion conversion method for discrete speech-token TTS. Its distinctive ingredients are the LoRA-based adaptation regime, the Conformer-style local convolution branch, the $\alpha = 128$1-weighted masked objective, and the time-shifted confidence-ordered inference schedule (Moon et al., 5 Jul 2026).

The paper is also explicit about its limits. Target length is still rule-based, which it describes as a general challenge for discrete diffusion LLM generation. Experiments are English only. Results are reported in a low-resource adaptation setting with 585 hours of LibriTTS, and multilingual extension remains future work. The paper also notes voice misuse risk, recommending consent and disclosure for synthetic audio (Moon et al., 5 Jul 2026).

In sum, DELTA-TTS advances a specific claim about TTS decoding: when the full text is known, speech-token generation need not be committed left to right. The reported evidence is that converting a pretrained autoregressive model into a bidirectional masked decoder, while preserving most of the pretrained stack and adding only lightweight adapters, yields faster token generation, better intelligibility, sharper alignment, and fewer hallucinations than the autoregressive backbone under the tested conditions (Moon et al., 5 Jul 2026).