Diffusion Models for Audio Restoration

• Diffusion models for audio restoration are probabilistic generative frameworks that iteratively denoise audio signals, recovering clarity from various degradations.
• They leverage learned priors over raw waveforms, time–frequency representations, or latent spaces to address tasks like declipping, inpainting, and denoising.
• Recent advances include conditioning strategies, hybrid architectures, and fast sampling techniques that significantly enhance both restoration quality and computational efficiency.

Diffusion models for audio restoration employ iterative probabilistic generative frameworks to recover clean or plausible audio signals from a wide range of degradations, including additive and non-Gaussian noise, reverberation, clipping, missing data, bandwidth limitation, and unknown nonlinear distortions. These models leverage learned priors over raw waveforms, time–frequency representations, or compressed latent codes, enabling both problem-agnostic and highly specialized restoration through conditional or posterior-guided sampling. Recent research demonstrates state-of-the-art restoration quality, versatility across audio domains (speech, music, environmental sounds), and adaptability to new restoration tasks without retraining underlying generative models.

1. Probabilistic Formulation of Diffusion Models for Audio Restoration

Diffusion models define a forward noising process—typically via a discrete or continuous-time Markov chain—where clean signals are incrementally perturbed by Gaussian (or non-Gaussian) noise, mapping data to an analytically tractable distribution such as a standard normal. Restoration is performed by learning a reverse process parameterized by a neural network, recovering clean samples from noisy observations. The fundamental construction in audio settings follows the DDPM paradigm:

• Forward process (standard DDPM): For clean $x_0$, recursively define

$$q(x_t \mid x_{t-1}) = \mathcal{N}\left(x_t; \sqrt{1-\beta_t} x_{t-1}, \beta_t I\right).$$

Cumulative noise schedules yield the closed-form marginal:

$$q(x_t \mid x_0) = \mathcal{N}\left(\sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t)I\right),\quad \bar{\alpha}_t = \prod_{s=1}^t (1-\beta_s).$$

• Reverse process: A neural model learns

$$p_\theta(x_{t-1} \mid x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \sigma_t^2 I)$$

with $\mu_\theta$ constructed via reparameterized noise estimation.

Restoration in inverse problems involves conditioning the reverse process to enforce consistency with the observed (damaged) signal, via conditioning features or explicit likelihood-based score correction (Lemercier et al., 2024).

Alternative frameworks extend the diffusion formalism:

• Cold diffusion replaces stochastic forward noise with arbitrary (deterministic or stochastic) degradations, supporting non-Gaussian or nonlinear artifacts (Yen et al., 2022).
• Latent diffusion applies the process within the code space of an autoencoder or neural codec (Dhyani et al., 2024, Santoso et al., 28 Aug 2025).
• Discrete diffusion operates over quantized token sequences produced by pretrained audio tokenizers (Dror et al., 11 Jul 2025).

2. Conditioning and Posterior Guidance Mechanisms

Restoration tasks require directing the generative process towards outputs that reconcile prior plausibility with observation fidelity. Key methods include:

• Direct conditioning: Concatenate observation-derived features (e.g., noisy STFT magnitudes, degraded Mel-specs) to model input, or inject them via cross-attention, as in DiffuSE’s use of noisy magnitude STFT in each residual block (Lu et al., 2021).
• Posterior sampling / reconstruction guidance: Compute the gradient of the data likelihood and add it to the reverse SDE/ODE or score function. The guidance term aligns sampling trajectories towards solutions consistent with the observed degraded signal, especially when the forward degradation is linear, nonlinear, or includes missing data (Moliner et al., 2022, Lemercier et al., 2024).
• Data-consistency projection: After each denoising step, directly overwrite the restored components with observed data in the unmasked regions for inpainting or bandwidth extension (Iashchenko et al., 2023, Moliner et al., 2022).
• Null-space correction: For underdetermined systems, e.g., erasures or missing packets, adjust the solution to fit known coordinates via pseudoinverse operations in latent space (Grassucci et al., 2023).
• Semantic and multimodal conditioning: Semantic tokens can be fused via cross-attention, enabling restoration jointly guided by audio and linguistic context, especially in semantic communication frameworks (Grassucci et al., 2023).
• Supportive reverse process: Reinject the noisy observation during each generation step (e.g., DiffuSE's SRP), stabilizing the early denoising dynamics and accelerating convergence (Lu et al., 2021).

3. Neural Architectures and Representation Domains

Diffusion models for audio restoration employ several neural backbones and input domains:

• Raw waveform U-Nets: Feedforward or residual 1D-convolutional networks (e.g., DiffWave), parameterizing reverse diffusion directly in the time domain (Lu et al., 2021, Yen et al., 2022).
• Spectrogram and complex domain models: 2D U-Nets in the STFT or complex Fourier domain facilitate phase-aware denoising and enable leveraging transformer variants for scalability—such as complex image-generative diffusion transformers (CIGDT) with sparse attention (Li et al., 2024).
• Constant-Q Transform (CQT) preconditioning: CQT-Diff leverages logarithmically spaced frequency axes and translational equivariance to optimally capture musical pitch structure and address inpainting, bandwidth extension, and declipping without retraining (Moliner et al., 2022, Moliner et al., 2023).
• Discrete token domains: Discrete diffusion using large-codebook quantizers (e.g., WavTokenizer) reduces sequence length, improving the stability and coherence of long-gap inpainting (Dror et al., 11 Jul 2025).
• Latent spaces: Restoration via diffusion in either VAE latents, as in semantic communication (Grassucci et al., 2023), or neural codec embeddings (Santoso et al., 28 Aug 2025, Dhyani et al., 2024), offers efficient computation and enables high-resolution speech generation.
• Variants for non-Gaussian noise: Joint U-Net parameterization of both diffusion reverse scores and Gaussian mixture model (GMM) noise components allows tractable denoising under real-world, non-Gaussian contamination (Wang et al., 2024).

Conditional diffusion, joint VAEs, and regularization strategies (e.g., RestoreGrad's jointly learned prior, stage-adaptive SSL guidance, structure-aware transformer regularization) further modulate restoration capability, efficiency, and convergence (Lee et al., 19 Feb 2025, Zhang et al., 6 May 2026).

4. Restoration Tasks: Bandwidth Extension, Inpainting, Declipping, Denoising, and Nonlinear Inverse Problems

Diffusion-based approaches address a broad range of audio restoration problems:

• Denoising: Classic additive noise removal is handled via conditional or unconditional DDPMs, often with supportive or guided sampling to accelerate and improve restoration (Lu et al., 2021, Wang et al., 2024, Li et al., 2024). Mixture models and transformers push the state-of-the-art on speech and bioacoustic denoising benchmarks (Wang et al., 2024, Li et al., 2024).
• Declipping: Both (a) explicit posterior-guided sampling with analytic gradient computation for hard/soft clipping models (Moliner et al., 2022, Iashchenko et al., 2023), and (b) blind nonlinear operator inversion using differentiable splines and joint likelihood+prior optimization in diffusion space are effective (Švento et al., 10 Jan 2025).
• Inpainting: Pitch-equivariant U-Nets in CQT space or transformer-based discrete diffusion enable reconstruction of missing segments up to 300–500 ms, with discrete approaches yielding more plausible long-range completions compared to continuous methods (Moliner et al., 2023, Dror et al., 11 Jul 2025).
• Bandwidth extension and generative equalization: CQT and frequency-domain U-Nets, combined with posterior and data-consistency guidance, produce plausible hallucination of missing frequency bands in both modern and historical audio (Moliner et al., 2022, Moliner et al., 2024). BABE-2 couples diffusion priors with filter parameter estimation for blind, joint magnitude response compensation.
• Dereverberation: Task-adapted SDEs and joint or cascaded model strategies allow simultaneous denoising and dereverberation, with optimal ordering depending on degradation dominance. A single “mixed-objective” diffusion model trained on both isolated and combined distortions provides robustness across diverse scenarios (Meise et al., 26 Aug 2025).
• Blind restoration of nonlinear systems: Diffusion-based posteriors, combined with differentiable operator models (e.g., cubic Catmull–Rom splines), solve for both original signal and unknown nonlinear distortion in a fully unsupervised fashion (Švento et al., 10 Jan 2025).
• Semantic communication and multi-condition restoration: Latent-diffusion-driven restoration in VAE spaces, guided by semantic embeddings, enables joint denoising and missing data imputation under communication channel degradations (Grassucci et al., 2023).

5. Training, Sampling, and Efficiency Considerations

Diffusion-based audio restoration is characterized by high sample-quality but inherent computational cost due to iterative sampling. Recent advances mitigate these burdens:

• Fast sampling: Reduced-step (e.g., 6-step) inference schedules with supportive reverse processes or learned priors preserve restoration quality, as in DiffuSE and RestoreGrad, with only minor perceptual quality loss relative to full-length chains (Lu et al., 2021, Lee et al., 19 Feb 2025).
• Stage-adaptive training: SSL-alignment variables, decayed semantic guidance, self-adaptive timestep selection, and structure-driven regularization optimize resource allocation between semantic acquisition and detail refinement, reducing overall training and sampling steps (Zhang et al., 6 May 2026).
• Latent-space diffusion and neural codecs: Operating in quantized or compressed latent spaces (e.g., WaveLLDM, Hi-ResLDM) yields significant reductions in input dimensionality and model size, facilitating faster and more memory-efficient restoration pipelines, albeit with quality trade-offs in perceptual clarity and fine structure (Dhyani et al., 2024, Santoso et al., 28 Aug 2025).
• Hybrid guidance and score-combination: Posterior sampling, data-consistency projection, and adaptive combination of unconditional and conditional scores further boost sampling efficiency and accuracy (Iashchenko et al., 2023, Moliner et al., 2022, Lemercier et al., 2024).
• Jointly-learned priors: VAE-integrated architectures (e.g., RestoreGrad) expedite convergence and allow for aggressive reduction in sampling iterations through prior adaptation to observations (Lee et al., 19 Feb 2025).
• Semantic domain guidance: Cross-attention-based fusion of linguistic or SSL-derived context during both training and inference supports multi-modal restoration and robustness under unseen degradations (Zhang et al., 6 May 2026, Grassucci et al., 2023).

6. Comparative Evaluation and Impact

Diffusion-based restoration methods consistently outperform classical signal processing and GAN/flow-based baselines in spectral, perceptual, and subjective metrics across tasks:

Method / Benchmark	PESQ (↑)	STOI (↑)	LSD (↓)	MOS/WER (↓/↑)	Notable Context
DiffGMM (Wang et al., 2024)	3.48	0.96	—	—	SOTA on speech denoising, robust to non-Gaussian noise
CIGDTN (Li et al., 2024)	3.55	0.964	—	4.24 (COVL)	Transformer diffusion in complex domain, outperforms GAN baselines
Hi-ResLDM (Dhyani et al., 2024)	—	—	—	WER: 0.05–0.12	Outperforms GAN/CFM for 48 kHz restoration, preferred by human experts
CQT-Diff (Moliner et al., 2022)	—	—	0.66	—	Music BWE, better than BEHM-GAN, outperforms STFT-Diff on harmonics
WaveLLDM (Santoso et al., 28 Aug 2025)	1.6–1.7	0.76–0.78	0.48–0.60	—	Latent space–efficient, but below best in perceptual metrics
AudioInpainting Discrete (Dror et al., 11 Jul 2025)	—	—	0.0824–0.3129	—	SOTA on 100–500 ms inpainting gaps, discrete diffusion over tokens
Cold Diffusion (Yen et al., 2022)	2.77	—	—	—	Handles arbitrary deterministic degradation, strong on unseen noise

A plausible implication is that diffusion models sidestep many distributional assumptions required by classical methods, robustly generalizing to new degradations, unseen noise types, and mismatched domains.

7. Current Limitations and Future Directions

Major limitations include the high inference cost of iterative denoising, the need for large high-fidelity training sets for best restoration quality, and challenges in scaling token/discrete models to long contexts or real-time deployment. Further research directions involve:

• Fast and adaptive samplers (DDIM, distillation, learned noise schedules).
• Multi-domain and multi-modal conditioning (integration with audio-visual or semantic context).
• Blind or self-tuning degradation operator estimation during inference (e.g., nonlinear, time-varying, spatially distributed artifacts).
• Perceptual and causal architectures suitable for low-latency, online deployment.
• Robustness to distributional shifts, non-stationarity, and challenging acoustic environments via self-supervised learning, uncertainty quantification, and multi-condition training.

Diffusion models constitute a versatile and unifying probabilistic framework for audio restoration, capable of high-fidelity, robust, and semantically consistent recovery across diverse audio restoration tasks, outperforming both traditional and contemporary competing methods (Lemercier et al., 2024, Lu et al., 2021, Moliner et al., 2022).