WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modeling

Abstract: Recently, diffusion models operating on VAE latents or mel-spectrograms have become the dominant paradigm for zero-shot TTS. Although these compressed representations improve generation efficiency, they inevitably suffer from information loss and non-end-to-end training. Theoretically, directly modeling raw waveforms circumvents these issues; however, this direction remains underexplored and is often deemed difficult due to the extremely long sequence length of audio signals. To overcome this, we propose WavTTS, the first raw waveform generative TTS model that substantially narrows the gap with latent-space generative models. Built upon the flow matching with Diffusion Transformer (DiT), WavTTS directly models speech waveforms via a simple patchification strategy, while integrating multi-scale mel-spectrogram supervision to provide perceptual guidance during training. Furthermore, we investigate the impact of prediction targets and noise scheduling in waveform diffusion, and develop an effective schedule design to improve generation quality. Evaluations on open-source benchmarks demonstrate that WavTTS closely approaches the performance of current state-of-the-art latent generative zero-shot TTS models, while substantially outperforming previous end-to-end speech generation models. Our findings demonstrate the feasibility of scaling diffusion-based TTS directly in the waveform space, opening a new direction for end-to-end speech generation.

• The paper introduces a 673-million-parameter non-autoregressive flow-matching Transformer that generates 16 kHz speech directly from raw waveforms, avoiding autoencoders, codecs, and vocoders.
• WavTTS aligns speech and noise variance, uses an x-prediction objective with multi-scale mel supervision, and applies PolyShift inference scheduling to improve convergence, intelligibility, and acoustic quality.
• On the Seed-TTS benchmark, WavTTS achieves 1.50% English WER and 3.92 UTMOS, but its speaker similarity of 0.65 remains below leading latent-space systems, highlighting an important direction for future work.

Overview

WavTTS is a non-autoregressive (NAR) zero-shot text-to-speech system that performs generative modeling directly on raw, uncompressed audio waveforms, eliminating the pre-trained autoencoders, neural codecs, and vocoders that dominate current diffusion-based TTS pipelines. The authors argue that the prevailing reliance on VAE latents or mel-spectrograms imposes an information-loss ceiling and introduces multi-stage error accumulation, and they demonstrate—apparently for the first time at this quality level—that a flow-matching Diffusion Transformer can operate tractably in the high-dimensional time domain while approaching state-of-the-art latent- and mel-space systems. The work is a collaboration between Shanghai Jiao Tong University, the Shanghai Innovation Institute, and ByteDance Seed (2606.03455).

The central technical obstacles are well known: raw 16 kHz audio yields sequences roughly two orders of magnitude longer than mel-spectrogram representations, and the model must simultaneously capture long-range linguistic structure and fine-grained phase, periodicity, and high-frequency content. WavTTS addresses these through waveform patchification, an $x$-prediction objective coupled to multi-scale mel-spectrogram supervision, and a pair of noise-scheduling interventions—signal-noise variance alignment and noise-shifted temporal scheduling—applied at both training and inference.

Architecture and training objective

WavTTS follows the rectified-flow formulation of flow matching under the text-conditioned speech-infilling task introduced by Voicebox. Given a clean waveform $x_1$, Gaussian noise $x_0$, and interpolation $x_t = (1-t)x_0 + t x_1$, the standard velocity-prediction objective requires regressing $v_t = x_1 - x_0$, a target containing the stochastic noise component. The authors instead adopt the $x$-prediction reformulation from JiT, in which the network directly predicts the clean waveform; the FM loss becomes $\| (x_\theta - x_1)/(1-t) \|_2^2$. This choice is consequential beyond numerical stability: it makes the predicted waveform directly available for auxiliary perceptual losses, which would otherwise require reconstructing the waveform from an estimated velocity field.

The backbone is a DiT with adaLN-Zero timestep conditioning, RMSNorm, and RoPE. The 1D waveform is patchified into non-overlapping blocks of length $F=160$ at 16 kHz sampling, yielding a 100 Hz token rate—a compression ratio sufficient to make Transformer attention computationally feasible while remaining lossless. Text is encoded by ConvNeXt V2 blocks using bilingual pinyin/alphabetic tokens padded to the audio length, enabling implicit alignment in the style of E2-TTS and F5-TTS. Classifier-free guidance is enabled by dropping both conditions with probability 0.1 during training.

Two auxiliary design choices complete the framework. First, a multi-scale mel-spectrogram $L_1$ loss (seven scales with window sizes 32–2048, following DAC's configuration) is applied to the masked target region, weighted by $\lambda_{\mathrm{mel}} = 0.05$. Second, inference uses Euler ODE integration with 50 NFEs, CFG scale $x_1$0, and the proposed PolyShift schedule.

Noise scheduling for waveform-space flow matching

This section contains what are arguably the paper's most instructive findings, because they expose how assumptions inherited from latent-space diffusion break down in the time domain.

Signal-noise variance alignment. Standard rectified flow implicitly assumes comparable variance between data and noise prior. Raw speech violates this: empirical waveform standard deviation is approximately 0.12 on Emilia and 0.07 on LibriTTS, versus $x_1$1 for the Gaussian prior. Under linear interpolation, this produces a Log-SNR trajectory shifted downward by 20 dB relative to the variance-aligned case, forcing the model to denoise from overwhelmingly noise-dominated states for most timesteps. The fix is simple: scale the target waveform by $x_1$2 before training so that $x_1$3, compute the mel loss strictly at the original amplitude scale, and apply the inverse scaling after generation. The ablation is stark—with no scaling ($x_1$4), SIM-o collapses to 0.32 and UTMOS to 2.40, versus 0.65 and 3.93 at $x_1$5. This single hyperparameter accounts for a larger performance delta than any other component studied.

Noise-shifted temporal scheduling. Under $x_1$6-prediction, uniform timestep sampling induces highly imbalanced implicit loss weighting $x_1$7 across timesteps. Drawing $x_1$8 from a logit-normal distribution with $x_1$9, $x_0$0 shifts density toward high-noise regions, which the authors show accelerates convergence substantially—WER drops below 2% within 400K steps—and improves final intelligibility. An aggressive variant ($x_0$1) degrades SIM-o and UTMOS, indicating that over-weighting coarse structure formation comes at the cost of speaker fidelity and fine acoustic detail. Notably, uniform sampling achieves the highest final UTMOS despite slower convergence, which the authors attribute to greater allocation toward low-noise refinement; they leave dynamic schedules that adjust the distribution over training as an open question.

For inference, the authors find Sway Sampling's shift insufficient for waveforms and propose PolyShift, combining a polynomial transform with a time shift ($x_0$2). PolyShift ($x_0$3) outperforms uniform sampling and Sway Sampling across all metrics; pushing further to $x_0$4 marginally lowers WER but introduces audible background noise from inadequate late-stage refinement.

Main results

On the Seed-TTS zero-shot benchmark, trained on 100K hours of Emilia with a 673M-parameter model, WavTTS achieves the best English WER (1.50%) and highest UTMOS (3.92) among all compared systems—including ZipVoice, F5-TTS, MaskGCT, LongCat-AudioDiT, and several AR baselines—while remaining competitive on Chinese (CER 1.59%). Speaker similarity (SIM-o 0.65 en / 0.73 zh), however, trails latent-space models such as LongCat-AudioDiT (0.76 / 0.81). The authors attribute this gap candidly to the difficulty of prioritizing timbre cloning within a high-dimensional signal space rich in phase and ambient detail, and identify speaker-oriented alignment as necessary future work rather than claiming parity.

Against prior end-to-end systems on LJSpeech and LibriSpeech-PC, WavTTS attains the lowest WER and highest UTMOS on both sets (e.g., 3.43% WER / 4.39 UTMOS on LJSpeech versus 3.72% / 4.37 for VITS$x_0$5), despite operating zero-shot with a fixed prompt against supervised in-domain baselines. This comparison also clarifies a terminological point: VITS and JETS, though labeled end-to-end, perform their core sequence modeling in lower-resolution intermediate spaces with adversarial decoders, whereas WavTTS models the waveform natively.

Ablations and scaling behavior

The prediction-target ablation shows $x_0$6-prediction modestly improving WER over $x_0$7-prediction (1.65% vs. 1.67%) with a clearer SIM-o gain (0.65 vs. 0.61). The mel-loss weight sweep demonstrates that perceptual supervision is essential for tractable optimization: without it, the model fails to produce intelligible speech even at 200K steps, while the supervised variant reaches sub-5% WER by that point; excessive weights ($x_0$8) degrade all metrics, suggesting perceptual supervision can distract from learning the underlying flow.

Scaling experiments reveal a strong data-dependence. On 585 hours of LibriTTS, both 340M and 673M variants generalize poorly out-of-domain (SIM-o ≈ 0.31–0.35), and the larger model provides no benefit—slightly hurting SIM-o. On 100K-hour Emilia, scaling to 673M parameters yields clear gains, particularly in speaker similarity (0.56 → 0.65). The conclusion that model scaling is effective only when supported by sufficient data is consistent with trends reported for BASE TTS and CosyVoice 3, but here it carries added weight because the waveform space is far less forgiving of underfitting than compressed representations.

A representation comparison against STFT and MDCT under the same flow-matching framework supports direct time-domain modeling: waveform diffusion converges fastest (4.10% vs. 9.76% WER at 200K steps relative to mel) and reaches better UTMOS (3.93 vs. 3.68 at 1M steps), while STFT and MDCT require 600K and 400K steps respectively before producing recognizable speech. The authors attribute MDCT's failure to a sharply peaked feature distribution (standard deviation ~0.005 with outliers up to 0.15) and STFT's to coupled magnitude-phase complexity—an honest acknowledgment that frequency-domain flow matching may need substantial preprocessing innovation.

Limitations and open questions

The paper concedes several boundaries explicitly. Speaker similarity remains below the strongest latent-space competitors, and the proposed explanation—capacity competition between timbre cloning and fine-grained acoustic detail—is a hypothesis rather than a demonstrated mechanism. All evaluations use model-based metrics (WER/CER via Whisper-large-v3 and Paraformer-zh, SIM-o, UTMOS); no large-scale human MOS study is reported, so claims about naturalness rest partly on a proxy predictor. The 16 kHz sampling rate limits high-fidelity applications, and extension to 24/48 kHz would multiply the already severe sequence-length burden. The uniform-sampling UTMOS anomaly and the optimal balance between high-noise and low-noise timestep allocation remain unresolved, as does the question of whether tailored speaker-alignment objectives can close the SIM-o gap without sacrificing the naturalness advantages of lossless modeling.

Conclusion

WavTTS establishes that direct raw-waveform diffusion TTS is practically viable at modern zero-shot scale, achieving best-in-class intelligibility and objective naturalness on Seed-TTS-en and surpassing prior end-to-end systems, while trailing only on speaker similarity. Its principal contributions are less architectural than diagnostic: the demonstration that variance misalignment between speech and the Gaussian prior, and imbalanced implicit loss weighting under $x_0$9-prediction, are first-order obstacles in waveform-space flow matching, each addressable with simple, well-motivated corrections. The result challenges the assumption that high-quality TTS requires lossy intermediate representations, though whether waveform-native models can match latent systems on speaker fidelity—and scale to higher sampling rates—remains open.