ReSyn: Speech Enhancement by Resynthesis

• ReSyn is a speech enhancement approach that predicts a clean log mel-spectrogram from noisy audio and synthesizes a new waveform, effectively bypassing direct waveform filtering.
• The method employs a 3-layer bidirectional LSTM for feature prediction and leverages neural vocoders like WaveNet and WaveGlow to render high-fidelity speech based solely on clean representations.
• Empirical results indicate that ReSyn outperforms traditional mask-based systems in both objective metrics and subjective quality, while offering trade-offs between real-time inference and perceptual quality.

Searching arXiv for the specified paper and closely related speech-resynthesis/enhancement work. arXiv search: (Maiti et al., 2019) Parametric Resynthesis with neural vocoders related speech enhancement resynthesis neural vocoder ReSyn, in speech processing, denotes enhancement by resynthesis: a class of systems that infer an intermediate clean-speech representation from degraded audio and then synthesize a new waveform from that representation, rather than directly filtering the noisy waveform. In the formulation developed in "Parametric Resynthesis with neural vocoders," ReSyn is instantiated as parametric resynthesis (PR), in which a neural predictor maps noisy speech to a clean log mel-spectrogram and a neural vocoder such as WaveNet or WaveGlow renders the final waveform (Maiti et al., 2019). Its defining premise is that the system should model only clean speech; noise is removed implicitly because synthesis is conditioned on predicted clean parameters and ignores the noise during waveform generation.

1. Conceptual basis and problem setting

ReSyn addresses a central weakness of conventional speech enhancement. Standard noise suppression systems usually operate in the STFT or time domain by attenuating time-frequency regions estimated to be noise. The data identify two recurrent failure modes: over-suppression, in which masks or filters remove energy in speech-dominant regions and reduce naturalness and intelligibility, and under-suppression, in which intrusive residual noise remains and often sounds musical (Maiti et al., 2019).

The ReSyn alternative is to predict clean acoustic features from noisy input and then synthesize a waveform entirely from those features. This changes the modeling target. Instead of jointly modeling speech and noise, ReSyn methods explicitly model only clean speech. The vocoder therefore need not reconstruct a partially denoised version of the original waveform; it produces a new signal that is consistent with the estimated clean representation. In the terminology of the paper, this is enhancement-by-resynthesis rather than enhancement-by-masking (Maiti et al., 2019).

This distinction is methodologically important. Mask-based and end-to-end waveform denoisers are constrained by the corrupted waveform and therefore inherit many of its artifact modes. ReSyn relies on a parametric bottleneck and on the prior of a high-fidelity vocoder. A plausible implication is that the quality ceiling becomes tied less to mask estimation accuracy and more to how well the intermediate representation captures the clean speech manifold.

2. Signal representation and processing pipeline

The system operates on noisy waveforms $x(t)$ sampled at $22$ kHz. From each utterance, a noisy log mel-spectrogram $Y(\omega,t)$ is extracted, and the training target is the clean log mel-spectrogram $X(\omega,t)$ from the parallel clean recording. The configuration reported is an STFT window length of $46.4$ ms, hop size $11.6$ ms, and $80$ mel bins; the vocoders condition on normalized log mel-spectrograms (Maiti et al., 2019).

The feature construction follows the standard sequence of STFT, mel projection, and log compression:

$$X(k,n)=\sum_{t=0}^{T-1} x(t) w(t-nH)e^{-j2\pi kt/N},$$

$$M(m,n)=\sum_k B_m(k)\cdot |X(k,n)|, \qquad M_{\log}(m,n)=\log(\epsilon+M(m,n)).$$

The prediction stage learns the mapping $Y \mapsto \hat{X}$ with a 3-layer bidirectional LSTM having 400 units per layer. The input is the noisy log mel-spectrogram and the target is the clean log mel-spectrogram. Training uses Adam with learning rate $22$0, batch size $22$1, and $22$2 epochs. The loss is mean squared error over mel bins and time:

$22$3

The full ReSyn pipeline is therefore factorized into three stages: noisy waveform $22$4 noisy mel features, noisy mel features $22$5 predicted clean mel features, and predicted clean mel features $22$6 synthesized waveform $22$7 (Maiti et al., 2019).

That factorization is the core of the approach. The predictor is trained to remove corruption at the feature level, while the vocoder is trained only on clean speech paired with clean mel-spectrograms. At test time, the vocoder is exposed to predicted mel-spectrograms $22$8, not to the noisy waveform itself.

3. Neural vocoders, likelihoods, and optimization

Two vocoders are used: WaveNet and WaveGlow. Both are trained on clean speech paired with clean mel-spectrograms and later conditioned on predicted mel-spectrograms during enhancement (Maiti et al., 2019).

WaveNet is the autoregressive option. Its architecture uses dilated causal convolutions with gated activations, residual and skip connections, arranged as 24 layers grouped into 4 dilation cycles, with 512 residual channels, 512 gate channels, and 256 skip channels. The output distribution is a mixture of logistics with $22$9 components, and the conditional likelihood is

$Y(\omega,t)$0

Training maximizes log-likelihood on clean data, with $Y(\omega,t)$1 weight regularization at $Y(\omega,t)$2 and an exponential moving average with decay $Y(\omega,t)$3 used for inference. The implementation follows r9y9’s WaveNet vocoder with Tacotron2-like conditioning on normalized log mel-spectrograms (Maiti et al., 2019).

WaveGlow is the non-autoregressive alternative. It is a normalizing flow operating on blocks of 8 audio samples; each flow comprises a $Y(\omega,t)$4 convolution followed by an affine coupling layer conditioned on mel-spectrograms. The reported setup uses 12 coupling layers, and each coupling subnetwork has 8 layers of dilated convolution with 512 residual channels, 256 skip channels, and weight normalization. Its maximum-likelihood objective is written as

$Y(\omega,t)$5

WaveGlow samples are generated in parallel, which makes it substantially faster than WaveNet, but inference depends on the sampling parameter $Y(\omega,t)$6; the paper notes that lower $Y(\omega,t)$7 can cause speech dropouts when conditioning is imperfect (Maiti et al., 2019).

The work also explores joint training of predictor and vocoder after initializing both from pretrained models. Conceptually, the combined objective is

$Y(\omega,t)$8

where $Y(\omega,t)$9 is either the WaveNet or WaveGlow negative log-likelihood. In practice, the reported setup showed that naive end-to-end joint training was unstable: the predictor tended to inflate mel magnitudes rather than converge toward the clean target, producing a vocoder-condition mismatch and degraded quality (Maiti et al., 2019).

4. Experimental protocol and empirical results

The speech data are from LJSpeech, a single-female-speaker corpus with 13,100 clips of 1–10 seconds at 22 kHz, corresponding to roughly 24 hours of training audio; the test set contains 24 files. Noise comes from CHiME-3 environmental recordings—street, bus, pedestrian, and cafe. Training mixtures use SNRs from $X(\omega,t)$0 dB to $X(\omega,t)$1 dB, with average approximately $X(\omega,t)$2 dB; test mixtures use SNRs from $X(\omega,t)$3 dB to $X(\omega,t)$4 dB. Because CHiME-3 is originally at 16 kHz, white Gaussian noise is synthesized in the 8–11 kHz band and energy-matched to the 7–8 kHz band of the original recordings (Maiti et al., 2019).

The baselines are Oracle Wiener Mask, Chimera++, and PR-WORLD. Objective metrics, computed at 16 kHz after downsampling, are SIG, BAK, OVL, PESQ, and STOI. Subjective evaluation uses MUSHRA-style listening tests with 5 listeners and 12 utterances, rated from 0 to 100 for speech quality, noise suppression quality, overall quality, and intelligibility (Maiti et al., 2019).

The objective results establish that both neural ReSyn systems outperform Chimera++ across all reported metrics, while subjective evaluation is even more favorable to ReSyn: both PR-WaveNet and PR-WaveGlow are rated significantly better than the oracle Wiener mask, PR-WORLD, and Chimera++, and PR-WaveNet receives the highest subjective quality ratings overall (Maiti et al., 2019).

System	Objective snapshot	Notable observation
PR-WaveGlow	SIG 3.9, BAK 2.5, OVL 3.1, PESQ 2.58, STOI 0.87	Best objective PR-neural variant
PR-WaveNet	SIG 3.8, BAK 2.2, OVL 3.0, PESQ 2.46, STOI 0.87	Best subjective quality
Chimera++	SIG 3.7, BAK 2.1, OVL 2.8, PESQ 2.44, STOI 0.86	Strong masking baseline, below PR-neural
Oracle Wiener	SIG 4.0, BAK 2.4, OVL 3.2, PESQ 2.90, STOI 0.91	Slightly ahead objectively on average

The apparent objective advantage of the oracle Wiener mask does not contradict the listening results. The paper attributes part of the discrepancy to alignment: resynthesized audio is not perfectly time-aligned with the clean reference, which depresses objective scores, especially STOI and PESQ. This is a recurring methodological issue for enhancement-by-resynthesis systems, because the output is a newly synthesized waveform rather than a phase-preserving transformation of the input (Maiti et al., 2019).

Upper-bound experiments also clarify vocoder behavior. When conditioned on clean mel-spectrograms, WaveGlow achieves SIG 5.0, BAK 4.1, OVL 5.0, PESQ 3.81, and STOI 0.98, whereas WaveNet achieves SIG 4.9, BAK 2.8, OVL 4.0, PESQ 3.05, and STOI 0.94. Yet under predicted mel conditioning, WaveNet is preferred subjectively. This suggests that robustness to imperfect conditioning and subjective naturalness are not captured fully by the objective metrics used here (Maiti et al., 2019).

5. Failure modes, fine-tuning behavior, and runtime trade-offs

ReSyn systems exhibit characteristic error modes that differ from those of masking-based enhancers. WaveGlow, when conditioned on imperfect $X(\omega,t)$5, can mute uncertain regions and produce speech dropouts, especially at low $X(\omega,t)$6. WaveNet can instead produce smooth but garbled or mumbled speech, particularly after unsuccessful joint training. These behaviors follow directly from the resynthesis formulation: the system is not denoising an observed waveform but generating a new one under imperfect conditioning (Maiti et al., 2019).

Joint training proved especially delicate. The paper reports that end-to-end training with vocoder losses caused mel loudness drift and degraded objective scores. Fine-tuning only the vocoder on predicted mel-spectrograms also worsened performance, with more dropouts or garbling. By contrast, fixing the vocoder and fine-tuning only the predictor with a combined mel MSE plus vocoder loss yielded slight improvements; one example reported is a WaveGlow STOI increase from 0.87 to 0.90 when only the predictor is fine-tuned (Maiti et al., 2019).

The operational trade-off between WaveNet and WaveGlow is stark. WaveNet generates at approximately 95–98 samples per second, corresponding to about $X(\omega,t)$7 real time at 22 kHz; synthesizing one second of audio takes about 232 seconds. WaveGlow generates one second of speech in about one second on a GTX 1080 Ti, making it roughly real-time. WaveNet therefore offers the best perceived quality, whereas WaveGlow offers practical latency (Maiti et al., 2019).

The training costs are also asymmetrical. The paper notes that training WaveGlow from scratch can take approximately two months on a GTX 1080 Ti, which is why publicly released pretrained LJSpeech models were used. Joint models also remain costly: PR-WaveNet-Joint is reported at 355k iterations with batch size 1 and approximately 2.31 seconds per iteration on a GTX 1080, and PR-WaveGlow-Joint at 150k iterations with batch size 3 and more than 3 seconds per iteration on a GTX 1080 Ti (Maiti et al., 2019).

6. Position within speech resynthesis and prospective development

Within speech processing more broadly, ReSyn belongs to a larger family of resynthesis pipelines that separate intermediate linguistic or acoustic structure from final waveform generation. The enhancement setting in (Maiti et al., 2019) predicts clean log mel-spectrograms from noisy speech, whereas PPG-based speech resynthesis for voice conversion and speech editing predicts mel-spectrograms from phonetic posteriorgrams and then vocodes them with WaveGlow (Gaudier et al., 2024). The common architectural theme is disentanglement by intermediate representation: the waveform is regenerated from a parametric description rather than incrementally repaired in place.

The paper on parametric resynthesis positions its neural systems relative to PR-WORLD by noting that they predict log mel-spectrograms instead of WORLD parameters and leverage neural vocoders rather than classical parametric synthesis (Maiti et al., 2019). In adjacent work on PPG-based resynthesis, the same general principle is used for identity control: PPGs encode phonetic content at 100 frames per second over 40 English phone classes, while speaker-specific synthesis models impose target identity, and automatic speaker verification cannot recover the source speaker after resynthesis (Gaudier et al., 2024). This suggests that ReSyn is not merely a denoising technique but a broader factorization strategy for speech generation tasks in which some attributes are preserved and others are discarded or reassigned.

Several future directions are explicit in the source material. On the prediction side, the paper proposes improved mel prediction through self-supervised pretraining, multi-task learning with phonetic recognition, perceptual losses, and explicit time-alignment mechanisms. On the vocoder side, it proposes faster non-autoregressive models such as HiFi-GAN, WaveRNN, Parallel WaveGAN, and UnivNet, as well as robust conditioning with noisy or perturbed mel targets. Further directions include streaming and low-latency predictors, integration with ASR or neural codecs, multi-speaker conditioning and domain adaptation, and adaptive tuning of WaveGlow’s $X(\omega,t)$8 to trade speech dropouts against background intrusion (Maiti et al., 2019).

In that sense, ReSyn defines a research program rather than a single architecture. Its central claim is that enhancement quality can improve when the system abandons direct waveform repair and instead predicts a clean, synthesis-ready representation. The reported results support that claim: PR-WaveNet and PR-WaveGlow outperform a strong separation baseline objectively, and both achieve substantially better subjective quality than the oracle Wiener mask, with WaveNet maximizing perceptual quality and WaveGlow approaching real-time operation (Maiti et al., 2019).