Reliable Neural-Codec Text-to-Speech by ASR Self-Verification and Distillation: Near-Zero Catastrophic Failures Across Models and Codecs

Abstract: Open autoregressive neural-codec text-to-speech (TTS) models sound excellent on typical inputs yet suffer stochastic catastrophic failures: on a meaningful fraction of utterances they emit silence, terminate early, or collapse into repetitive or hallucinated content. We show this failure mode is cheap to remove. Under a single format-robust metric (a catastrophic-failure rate via an ASR round-trip), best-of-N ASR self-verification drives failures to near-zero: no observed failures remain by N=2 on a standard corpus (LibriSpeech) and by N=4 on a hard prompt set. This is not an artifact of one model: the reduction replicates across four open codec-TTS systems and three neural codecs (XCodec2, SNAC, Mimi), reaching the near-zero floor by N=2 on three of the four. We then make the fix free at inference time by distilling the self-verified behaviour into the model, which recovers much of the robustness in single-shot decoding, closing ~52-58% of the failure mass on hard inputs at no test-time cost. The distillation gain concentrates where it is needed (hard inputs); on already-reliable prose there is no headroom and no detectable change. A controlled comparison adds a clean negative: offline direct preference optimization (DPO/IPO) does not beat plain supervised distillation, and an online iterative variant is promising but not statistically separable at our evaluation size. We report honestly the one model that resists (a larger Llasa where scale did not obviously help) and a rare-word capability ceiling that no self-distillation method overcomes

• The paper demonstrates that best-of-N sampling with ASR self-verification can statistically reduce catastrophic TTS failures to near-zero across multiple models and codecs.
• It introduces a unified metric based on ASR round-trip transcriptions to reliably identify failure modes like dropouts and content collapse.
• LoRA-based distillation transfers the robustness from multiple candidates to a single-shot model, significantly lowering failure rates without extra inference cost.

Near-Zero Catastrophic Failures in Neural-Codec TTS via ASR Self-Verification and Distillation

Overview

The paper "Reliable Neural-Codec Text-to-Speech by ASR Self-Verification and Distillation: Near-Zero Catastrophic Failures Across Models and Codecs" (2606.18323) investigates the inherent reliability issues in open autoregressive neural-codec TTS models. While these models deliver impressive output on standard prompts, they can unpredictably fail—manifesting as silence, early truncation, repetitive collapse, or hallucinatory content—on a non-trivial subset of inputs. These catastrophic failures impede deployment in production systems that require consistent quality and predictability.

The work makes two primary contributions: (1) It demonstrates that catastrophic failures can be driven to statistically near-zero using a simple best-of-$N$-sampling approach with ASR-based self-verification, and (2) it shows that LoRA-based distillation of these verified samples into the model yields much of these gains in single-shot inference, thus eliminating the inference-time cost of sampling-based reliability mechanisms.

Catastrophic Failure Modes and Unified Metric

The study introduces a unified and format-robust metric for catastrophic failures in codec-TTS: for a given generation, failures are identified by ASR-based round-trip transcriptions, with explicit criteria distinguishing dropouts (speech too short or empty) and content collapses (WER $>0.5$ vs reference). This reframing addresses prior evaluation artifacts—such as digit/word formatting issues in numbers and dates—that make WER ill-suited for certain TTS robustness assessments.

Best-of-$N$ ASR Self-Verification: Eliminating Failures

The central claim is that using best-of-$N$ sampling with ASR-based self-verification is an efficient and highly effective strategy for mitigating catastrophic failures. In this procedure, $N$ candidate utterances are generated for each prompt, and the candidate passing the ASR-based criterion is selected; a prompt only counts as a failure if all $N$ candidates fail.

Across four open codec-TTS models and three distinct neural codecs (XCodec2, SNAC, Mimi), best-of-2 or best-of-4 sampling reduces the catastrophic failure rate to a statistically negligible upper bound ($\leq 0.008$ in 95% CI, rule-of-three) for three of the four systems. The sole exception is Llasa-3B, which, under a stricter generation budget, sees failures plateau at a nonzero floor.

Figure 1: Single-shot (base) vs.\ best-of-2 catastrophic-failure rate across four codec-TTS models and three codecs. best-of-2 reaches no observed failures on three of four models; Llasa-3B is the lone exception. 0.000 cells are 95\% upper-bounded by 0.008.

The effect generalizes across model size, architecture, and codec design, strongly indicating that this is not an idiosyncratic property of any one system. Best-of-$N$ saturates rapidly for easy (prose) inputs and more slowly for adversarial/hard prompts.

Distillation of Robustness: Efficient and Effective

While best-of-$N$ self-verification is highly effective, it multiplies inference cost by $N$, hindering deployment. The paper solves this through a single round of supervised distillation (LoRA fine-tuning) where the targets are the best ASR-verified candidates. The distilled model inherits much of the robustness at zero additional inference cost: on hard prompts, the single-shot catastrophic failure rate drops from 0.199 to 0.083–0.096, closing 52–58% of the failure mass. Importantly, the gain is input-difficulty-dependent: for standard LibriSpeech prose, where the base rate is already low, no measurable improvement is obtained.

Figure 2: Best-of-N failure rate by input difficulty (primary model). Hard inputs saturate at $>0.5$0; easy prose at $>0.5$1; the saturation point acts as a proxy for distillation headroom.

The observed pattern is that the benefit of distillation closely tracks where best-of-$>0.5$2 improves reliability most—the "saturation $>0.5$3" for a prompt effectively predicts the attainable gain. This provides a natural metric for evaluating the future limits of such transfer.

Controlled Preference Optimization Baselines

To assess whether additional preference optimization strategies provide incremental value in learning from the sampled candidate pool, the authors compare standard supervised distillation to DPO, IPO, and targeted pairing strategies. None outperform the supervised baseline in the offline setting; only the online iterative on-policy variant achieves a marginally lower failure rate, but these differences are statistically indistinguishable given the evaluation budget. This finding underscores the sufficiency of plain supervised distillation in transferring self-verified robustness.

Failure Analysis and Residual Issues

Detailed error analysis reveals four recurring patterns: (i) silence/dropout, (ii) repetitive collapse, (iii) content hallucination, and (iv) rare-word mispronunciations. Rare-word capability remains a hard ceiling: for inputs like "otorhinolaryngologist", successful generation is rare even with increased sampling, indicating a representation and generalization bottleneck outside of reliability per se. Number/date handling is dominated by tokenization artifacts, not fixed by these interventions.

Limitations and Future Directions

Several experimental caveats are acknowledged. The "hard-vs-easy" comparison interleaves input difficulty with corpus differences (handwritten vs. natural prose), impeding precise attribution. The rare-word and number/date buckets remain open. The ASR-based failure metric inherits ASR error floors. Extension to more languages, spontaneous speech, and within-corpus difficulty gradients are proposed as the next key steps.

The online iterative variant of preference optimization, though not conclusively superior here, is presented as the most promising future avenue for closing the remaining hard-prompt robustness gap.

Conclusion

This work establishes that catastrophic reliability failures in open neural codec TTS models are not inherent but are predominantly a function of inference procedure and can be robustly mitigated by ASR-based best-of-$>0.5$4 sampling. With a single round of distillation, much of this robustness becomes intrinsic, requiring no added inference cost. Further, these results generalize across architectures and codecs, indicating broad applicability. Major open problems are rare-word and formatting capability ceilings and a deeper understanding of difficulty-dependent transfer within single corpora. This research provides a practical blueprint for making open TTS systems deployment-grade without extensive retraining.