Speech-Shaped Noise (SSN)

• Speech-shaped noise (SSN) is a stationary Gaussian noise process whose power spectral density is shaped to match a target audio’s spectral envelope.
• It is integrated in diffusion-based GAN training to perturb audio samples, thus improving discriminator stability and overall audio quality.
• SSN employs FFT and iFFT methods for efficient spectral shaping, yielding measurable improvements in metrics like PESQ, STOI, and FAD.

Speech-shaped noise (SSN) is a class of noise used in speech and music synthesis systems, distinguished by its spectral properties being matched, or adaptively shaped, to the spectral envelope of a target audio signal—most notably, human speech. In neural vocoder and generative adversarial network (GAN) architectures, SSN is employed within a forward diffusion process to perturb both real and generated audio samples with colored noise whose frequency-domain power distribution is inversely proportional to the target spectral envelope. This adaptive injection of shaped noise regularizes GAN training, encourages refined discriminator behavior, and yields measurable improvements in both perceived audio quality and robustness across speech and music domains (Baoueb et al., 2024).

1. Mathematical Construction of Speech-Shaped Noise

SSN is formally defined as a zero-mean, stationary Gaussian noise process with covariance $\Sigma_{\mathrm{spec}}$ designed such that its power spectral density (PSD) matches a desired shaping function. For a waveform $x_0\in\mathbb{R}^n$, the SSN injection at a given diffusion step is

$$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\;\varepsilon, \qquad \varepsilon\sim\mathcal{N}(0,\Sigma_{\mathrm{spec}})$$

where $\bar\alpha_t$ is the cumulative product of the diffusion rate parameters up to step $t$. The covariance is factored as $$\Sigma_{\mathrm{spec}}=L_{\mathrm{spec}}L_{\mathrm{spec}}^\top$$ with

$L_{\mathrm{spec}} = G^+\,M_{\mathrm{spec}}\,G$

where $G$ and $G^+$ denote the short-time Fourier transform (STFT) matrix and its inverse, respectively. The diagonal complex filter $M_{\mathrm{spec}}$ is defined such that its magnitude $|M_{\mathrm{SG}}(f)|$ matches the (minimum-phase) spectral envelope $E(f)$ of the target audio, typically estimated from the mel-spectrogram. The shaping function $H(f)$ is

$H(f):=|M_{\mathrm{spec}}(f)| = 1 / E(f),$

yielding a shaped PSD $S_{\mathrm{noise}}(f) = H(f)^2$. Sampling from the SSN distribution is performed either in the frequency domain,

$$\varepsilon_f(f) \sim \mathcal{C}\mathcal{N}(0, S_{\mathrm{noise}}(f)),\qquad \varepsilon = G^+\varepsilon_f,$$

or by convolving white Gaussian noise $w\sim\mathcal{N}(0, I)$ with the minimum-phase impulse response $h = \mathrm{iFFT}\{H(f)\}$,

$\varepsilon = w*h.$

This ensures noise energy is concentrated inversely to the dominant spectral features of the target, perturbing less-energetic regions more strongly.

2. Algorithmic Integration in Diffusion-based GAN Training

In the SpecDiff-GAN framework, SSN is incorporated into a forward diffusion process applied to both real and generated waveforms prior to discriminator evaluation. The process follows these core steps:

1. For each minibatch with audio $x_{\mathrm{real}}$ and conditioning mel $c$, generate $x_{\mathrm{gen}}$ from the generator.
2. Sample a diffusion step $t$ and compute $\bar\alpha_t$.
3. For each sample, estimate the spectral envelope $E(f)$ from $c$, construct $H(f) = 1/E(f)$.
4. For $x_{\mathrm{real}}$ and $x_{\mathrm{gen}}$, sample independent SSN perturbations $\varepsilon_r$ and $\varepsilon_g$, respectively.
5. Construct perturbed samples: $$x_{\mathrm{real}, t} = \sqrt{\bar\alpha_t}\,x_{\mathrm{real}} + \sqrt{1-\bar\alpha_t}\,\varepsilon_r$$ (similarly for generated).
6. Update the discriminator via least-squares loss computed on perturbed samples.
7. Update the generator via adversarial loss, feature-matching loss, and mel-reconstruction loss.
8. Adjust the diffusion schedule adaptively by monitoring discriminator statistics.

All SSN sampling and envelope estimation operations are fully batched and vectorized for efficiency. Sampling routines leverage FFT/iFFT or convolution for noise shaping, and typically use overlap-add for windowed STFT-based processing (Baoueb et al., 2024).

3. Implementation Details and Computational Considerations

SSN generation requires estimating the spectral envelope $E(f)$, commonly via a small DNN or exponentiation of linearly scaled mel coefficients. The shaping filter $H(f)$ is produced per conditioning mel. In practice:

• Generate white Gaussian noise $w$ of the same length as the target audio.
• FFT of $w$ yields $W(f)$, which is multiplied by $H(f)$ for spectral shaping.
• Inverse FFT produces time-domain SSN $\varepsilon$.
• Overlap-add is used if windowed STFT is employed.
• All batched operations are implemented using GPU-accelerated libraries in PyTorch or TensorFlow.
• Overall, computational cost is dominated by two FFTs per audio sample, which is substantially less intensive than multi-step reverse diffusion sampling.

This approach supports real-time execution: empirical throughput is 180–220 times faster than the audio duration—a factor relevant for deployment in interactive systems or on-device inference (Baoueb et al., 2024).

4. Empirical Evaluation and Ablation Results

Comprehensive experiments evaluate SSN's impact within SpecDiff-GAN using several standard speech and music benchmarks with objective metrics: PESQ and STOI for perceptual quality and intelligibility, WARP-Q for speech quality, and FAD for music assessment. Key findings include:

• On the LJSpeech dataset, SpecDiff-GAN attains PESQ 3.758, STOI 0.985, and WARP-Q 1.018, outperforming both StandardDiff-GAN (white noise diffusion) and original HiFi-GAN baselines.
• On the VCTK dataset (out-of-domain speakers), SpecDiff-GAN maintains an advantage exceeding 0.15 PESQ over white noise variants.
• For piano music (MAPS), FAD achieves 0.080 for SpecDiff-GAN, better than standard approaches (0.108).
• Ablation demonstrates that omitting SSN in favor of unshaped diffusion noise reduces perceptual quality (PESQ drops by ~0.14), while disabling diffusion or replacing core modules yields further degradation.
• SSN leads to smoother discriminator loss curves, more controlled discriminator decision rates ($r_d$ stable near 0.6), and prevention of mode collapse even at elevated learning rates.

Model	PESQ (↑)	STOI (↑)	WARP-Q (↓)
HiFi-GAN	3.468	0.976	1.203
StandardDiff-GAN	3.621	0.982	1.086
SpecDiff-GAN	3.758	0.985	1.018

Empirical evidence thus establishes that SSN not only enhances generated audio quality but also improves training robustness across domains (Baoueb et al., 2024).

5. Functional Mechanisms and Theoretical Insights

Injecting SSN serves as an adaptive regularizer for GAN discriminators by amplifying perturbations in spectrally weak zones — such as formant valleys and high-frequency regions — while preserving dominant spectral features. This compels the discriminator to leverage higher-order, localized, fine-structured spectral cues, disincentivizing superficial shortcuts based on energy-dominated regions. Consequent effects include:

• Enhanced feature learning in generators,
• Stabilized adversarial training dynamics,
• Reduction in discriminator overfitting (manifest as well-controlled $r_d$ rates),
• Fewer artifacts such as “whistling” or “buzz”.

For music synthesis, the noise shaping can be designed according to domain-specific envelopes, e.g., those derived from the Constant-Q Transform (CQT) or instrument-specific mel templates, tailoring perturbation to instrument timbres. Trade-offs involve minor computational overhead and sensitivity to envelope estimation errors, though smoothing the envelope in practice mitigates potential coloration artifacts (Baoueb et al., 2024).

6. Practical Guidelines and Broader Implications

Effective deployment of SSN-based diffusion regularization requires precise, real-time extraction of the spectral envelope from conditioning features and low-latency FFT/batched operations integrated into the main training pipeline. Monitoring discriminator metrics is essential for adaptive diffusion scheduling.

The adaptivity and task-awareness conferred by SSN suggest a broader principle: regularization should be guided by the underlying signal structure, forcing discriminative models to account for domain-relevant details rather than relying on incidental spectral cues. This principle is extensible to music, environmental sound synthesis, and potentially beyond speech domains. A plausible implication is expanded applicability to other generative models where shaped noise may regularize or balance adversarial training.

Speech-shaped noise, as articulated and deployed in SpecDiff-GAN, delivers robust and efficient regularization for neural vocoder GANs, substantially improving both empirical performance and training stability at real-time inference rates (Baoueb et al., 2024).