DiffWave: Non-Autoregressive Audio Synthesis

• DiffWave is a non-autoregressive diffusion probabilistic model that transforms Gaussian noise into high-fidelity audio through an iterative denoising process.
• It leverages a learned reverse Markov chain with a residual stack of dilated convolutions to efficiently perform both conditional and unconditional synthesis.
• The model demonstrates versatility in tasks such as neural vocoding, speech restoration, and non-speech audio generation, offering competitive quality and speed.

DiffWave is a non-autoregressive diffusion probabilistic model designed for high-fidelity waveform generation in audio synthesis. It operates by incrementally transforming Gaussian noise into structured audio through a learned denoising Markov process. Distinct for its versatility, DiffWave supports conditional (e.g., vocoding, class-conditional) and unconditional audio synthesis, with notable efficiency and competitive audio quality relative to existing autoregressive models. The architecture has seen extensions in diverse tasks, including neural vocoding, speech restoration from lossy transforms, and unconditional generation of non-speech vocalizations such as infant cries (Kong et al., 2020, Zhang et al., 2021, Hoq et al., 2024).

1. Diffusion Probabilistic Modeling for Audio

DiffWave frames audio synthesis as an iterative denoising process within the diffusion probabilistic modeling paradigm. The model consists of a forward “diffusion” process and a learned reverse process:

• Forward process: Progressive perturbation of data $x_0$ to noise $x_T$ using a Markov chain:

$$q(x_1,\ldots,x_T\mid x_0) = \prod_{t=1}^T \mathcal{N}\left(x_t ; \sqrt{1-\beta_t} x_{t-1}, \beta_t I \right)$$

with $\beta_t$ scheduled (usually linearly) across $T$ steps.

• Closed-form: $$x_t = \sqrt{\bar\alpha_t}x_0 + \sqrt{1-\bar\alpha_t}\epsilon$$, $\bar\alpha_t = \prod_{s=1}^t (1-\beta_s)$, $\epsilon \sim \mathcal{N}(0, I)$.
• Reverse process: Neural network $\epsilon_\theta$ parameterizes the denoising Markov chain to reconstruct $x_0$ from $x_T$:

$$p_\theta(x_{t-1}\mid x_t) = \mathcal{N}\bigl(x_{t-1}; \mu_\theta(x_t, t), \tilde{\beta}_t I\bigr)$$

where

$$\mu_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}}\left( x_t - \frac{\beta_t}{\sqrt{1 - \bar\alpha_t}} \epsilon_\theta(x_t, t) \right),\quad \alpha_t = 1-\beta_t$$

(Kong et al., 2020, Hoq et al., 2024).

2. Model Architecture

The DiffWave architecture employs a non-autoregressive residual stack with dilated convolutions inspired by WaveNet:

• Residual backbone: 30 to 80 residual layers (task-dependent), each a 1D dilated convolution (kernel size 3) with exponentially increasing dilation within blocks.
• Timestep embedding: Each diffusion step $t$ is encoded using a 128-dimensional sinusoidal embedding, injected as an additive bias in every residual layer.
• Conditioning mechanisms: For vocoder tasks, DiffWave uses an 80-channel mel-spectrogram conditioner upsampled via two 2D transposed convolutions to match waveform resolution. For class-conditional tasks, label embeddings are processed similarly. In unconditional tasks, no external conditioner is used (Kong et al., 2020, Hoq et al., 2024).

Table: Key architectural characteristics across applications

Task	Conditioning	Residual Layers	Conditioner Details
Neural vocoding	Mel-spectrogram	30–80	2x 2-D transposed conv upsampler
Speech restoration	Degraded mel	30	Deep 15-layer CNN upsampler
Unconditional audio	None	30–48	No conditioner
Non-speech (e.g. cry)	None	30	Sinusoidal time embedding only

3. Training Paradigm and Objectives

DiffWave is trained by minimizing a simplified denoising score-matching loss, a variant of the variational lower bound (ELBO):

$$L(\theta) = \mathbb{E}_{x_0\sim q_{\rm data},\,t\sim \{1..T\},\,\epsilon\sim \mathcal{N}(0,I)} \left\| \epsilon - \epsilon_\theta \left( \sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t}\epsilon,\, t\right)\right\|_2^2$$

Notable training setups:

• Optimizer: Adam, learning rate $2\times 10^{-4}$ is standard.
• Batch size: Typically 16.
• Diffusion steps: $T=20$–$200$; smaller $T$ for faster inference, larger $T$ for higher fidelity (Kong et al., 2020, Hoq et al., 2024).
• Unconditional setup: In unconditional models, the conditioner path is disabled and all randomness stems from the sampled noise vector.

4. Inference and Fast Sampling Techniques

DiffWave supports accelerated inference through step-collapsed sampling:

• Regular sampling: $T$ sequential Markov updates starting from $x_T \sim \mathcal{N}(0,I)$.
• Fast sampling: By specifying a reduced step schedule ($S \ll T$, e.g., $S=6$), and aligning fast-sample noise levels to those at training, the model can generate high-quality audio at $\sim 3-6\times$ real-time speed with minimal quality loss (MOS drop $<0.02$ in practical cases) (Kong et al., 2020, Hoq et al., 2024).
• Robustness: Fast sampling preserves perceptually salient structure even in non-speech vocalization tasks, e.g., infant cries, where S=6 suffices for viable synthesis (Hoq et al., 2024).

5. Applications and Empirical Results

DiffWave’s versatility spans a range of audio synthesis and restoration applications:

• Neural vocoding: Matches WaveNet in MOS (4.44 vs. 4.43 on LJ-speech) with higher sampling efficiency and a parameter count under 7M (Kong et al., 2020).
• Class-conditional generation: Achieves 91.2% classification accuracy, FID=1.113, IS=6.63, and MOS=3.50 on SC09 digits, outperforming autoregressive and GAN models (Kong et al., 2020).
• Unconditional synthesis: Demonstrates higher diversity and perceptual quality on unconstrained waveform synthesis (e.g., MOS 3.39 vs. 1.43 for WaveNet-256 on SC09) (Kong et al., 2020).
• Non-speech vocalizations: Capable of synthesizing diverse, high-fidelity infant-cry sounds without any conditioning, learning temporal and harmonic patterns directly from waveform-level supervision (Hoq et al., 2024).
• Speech restoration: By modifying the conditioner network (deep CNN upsampler), DiffWave excels in inverting deterministic degradations (e.g., LPC-10, AMR-NB compression, or signal clipping), yielding statistically significant improvements across perceptual and intelligibility metrics (PESQ, PFP, MOS) over both raw degraded signals and vanilla DiffWave (Zhang et al., 2021).

6. Extensions and Architectural Modifications

Enhancements of DiffWave have focused on adapting the conditioner and sampling mechanisms:

• Deep CNN conditioner: For restoration of spectrally-distorted signals, a 15-layer 2D CNN upsampler replaces the original two-layer transposed convolution, enabling more powerful mel-to-waveform mappings (Zhang et al., 2021).
• Training regime: Two-stage training for conditioner pretraining can stabilize learning and allow nontrivial nonlinear mappings in the acoustic domain.
• Unconditional adaptation: For tasks with no exogenous information (e.g., vocalization synthesis), DiffWave can maintain synthesis diversity and quality with only sinusoidal time-step embeddings and a moderate number of residual layers (Hoq et al., 2024).

7. Limitations and Future Directions

Current limitations include:

• Sampling speed: Though fast, DiffWave’s inference is still slower than flow-based models for large $T$, motivating ongoing research in adaptive step schedules, kernel optimization, and further reductions in step count.
• Restoration under stochastic degradation: Existing work focuses on deterministic spectrotemporal degradations; generalization to additive noise or stochastic channel effects is unproven (Zhang et al., 2021).
• Joint training: There is potential for efficiency gains by jointly training the diffusion backbone and complex upsamplers, or by end-to-end fine-tuning.

Future extensions are anticipated to leverage adaptive scheduling, longer-context modeling (e.g., for music), and integration with text-to-spectrogram pipelines for unified end-to-end TTS and inpainting (Kong et al., 2020, Zhang et al., 2021).

---

For foundational advances, implementation protocols, evaluation procedures, and a complete mathematical treatment, see the original DiffWave paper (Kong et al., 2020), as well as domain-specific adaptations (Hoq et al., 2024, Zhang et al., 2021).