HiFi-GAN: Efficient High-Fidelity Neural Vocoder

• HiFi-GAN is a high-fidelity, parallel, adversarial neural vocoder designed for efficient, real-time synthesis of audio from mel-spectrogram inputs.
• Its generator architecture employs multi-receptive-field fusion and transposed-convolution upsampling to accurately convert mel-spectrograms into high-quality waveforms.
• The integrated Multi-Period and Multi-Scale discriminator systems enhance pitch regularity, timbre fidelity, and temporal precision during adversarial training.

HiFi-GAN is a high-fidelity, parallel, adversarial neural vocoder architecture that synthesizes time-domain audio waveforms from mel-spectrogram or related acoustic representations. Initially introduced by Kong et al. (2020), the architecture emphasizes computational efficiency, pitch and timbre fidelity, and fast real-time inference. Since its introduction, HiFi-GAN has become a canonical vocoder backbone in both academic and applied speech synthesis, with a proliferation of methodological refinements, variants, and applications. This article provides a comprehensive technical overview of HiFi-GAN, including its core architecture, discriminator design, objective functions, key extensions, integration strategies, and advances in time-frequency discriminators.

1. Generator Architecture

HiFi-GAN employs a non-autoregressive generator $G$ conditioned on an $80$-channel mel-spectrogram $M\in\mathbb{R}^{80\times T}$, producing a waveform $\hat{x}\in\mathbb{R}^{T'}$. The design consists of an initial 1D convolutional frontend, a stack of transposed-convolutional upsampling blocks, and a final output convolution:

• Preprocessing:

$M$ is projected to 512 channels by a Conv1D ($\text{kernel}=7$, $\text{stride}=1$, $\text{padding}=3$), followed by LeakyReLU (slope 0.1).

• Upsampling Stack:
  • Block 1: 512$\to$256, kernel=16, stride=4
  • Block 2: 256$\to$128, kernel=16, stride=4
  • Block 3: 128$80$064, kernel=4, stride=4
  • Block 4: 64$80$132, kernel=4, stride=2
  • This leads to a total expansion factor of $80$2 (or higher for high-rate setups).
• Multi-Receptive-Field (MRF) Fusion:

After each upsampling, an MRF module with three parallel residual branches (kernel sizes 3, 5, 7 or {3, 7, 11} and dilation patterns {1, 3, 5}) processes the hidden activations. Each branch consists of cascaded 1D convolutions with dilations and weight normalization, summed and averaged before residual connection.

• Output:

A final LeakyReLU, a $80$3 Conv1D (maps channels to 1), and tanh activation constrain $80$4.

This topology achieves both high spectral precision and computational efficiency. Parameter counts for the generator are typically in the range 11–15M for mainstream configurations (Chary et al., 2 Sep 2025, Yoneyama et al., 2022).

2. Discriminator Systems

HiFi-GAN introduces adversarial discrimination through two principal discriminator ensembles:

• Multi-Period Discriminator (MPD):

Five sub-discriminators reshape the waveform into $80$5 2D blocks along periods $80$6, each processed via 2D convolutional stacks. MPD is particularly sensitive to periodic structure and pitch regularity.

• Multi-Scale Discriminator (MSD):

Three or more discriminators process the waveform at full, $80$7-downsampled, and $80$8-downsampled rates using 1D ConvNet stacks. This configuration promotes modeling of both fine and coarse spectral/temporal structure.

All internal layers use LeakyReLU (default 0.1 slope) and weight normalization. Each discriminator outputs a scalar logit score for real/fake classification, and the collection of all intermediate activations is used for feature-matching.

Numerous works have replaced or augmented the original discriminators with time-frequency discriminators, including multi-resolution STFT (Baoueb et al., 2024), MRD (Baoueb et al., 2024), Multi-Scale Sub-Band Constant-Q Transform (MS-SB-CQT) (Gu et al., 2023, Gu et al., 2024), and multi-basis CWT discriminators (Gu et al., 2024).

3. Adversarial and Auxiliary Objectives

HiFi-GAN training follows a multi-objective regime:

• Adversarial Loss (Hinge or LSGAN):

$80$9

$M\in\mathbb{R}^{80\times T}$0

where $M\in\mathbb{R}^{80\times T}$1 is each discriminator in the MPD/MSD ensemble.

• Feature Matching Loss:

$M\in\mathbb{R}^{80\times T}$2

Matching deep representations between real and generated audio at intermediate discriminator layers accelerates convergence and improves perceptual stability.

• Mel-Spectrogram Loss:

$M\in\mathbb{R}^{80\times T}$3

This L1 penalty ensures the generator preserves energy, formant, and timbre structure visible in the mel domain.

The total generator loss is typically

$M\in\mathbb{R}^{80\times T}$4

with $M\in\mathbb{R}^{80\times T}$5, $M\in\mathbb{R}^{80\times T}$6 as standard.

4. Methodological Extensions and Applications

A range of variants and extensions have adapted HiFi-GAN's methodology to new constraints:

• Spectrogram Patch Codec Vocoder:

HiFi-GAN can synthesize from quantized/codec-distorted mel-spectrograms, with training directly on VQ-VAE reconstructed features to facilitate robust neural speech coding at bitrates of $M\in\mathbb{R}^{80\times T}$7 kbit/s and maintain low real-time factors (Chary et al., 2 Sep 2025). No changes beyond upsampling factor adjustments are required, demonstrating the model's robustness to representation artifacts.

• Speaking-Rate-Controllable Vocoder:

Feature interpolation modules (linear or bandlimited) can be inserted at input or hidden feature positions for time-axis warping, enabling real-time speaking rate control without retraining or parameter changes (Xin et al., 2022). Linear mel-spectrogram warping achieves minimal mel cepstral distortion and maintains mean opinion score (MOS) comparable to ground truth for moderate rate changes.

• Source-Filter HiFi-GAN:

A hierarchical source-filter structure splits excitation and filtering, integrating sine-wave excitation and pitch-adaptive dilated convolutions into the upsampling workflow. This improves pitch controllability and robustness during singing voice synthesis, outperforming both baseline HiFi-GAN and uSFGAN in pitch RMSE and MOS under pitch scaling (Yoneyama et al., 2022).

• Phase-Coherent Vocoding:

Architectures directly predicting complex STFT frames, with explicit phase-aware loss terms and prosody-guided harmonic attention, outperform conventional HiFi-GAN on F0 RMSE, voiced/unvoiced error, and MOS by avoiding temporal smearing and improving pitch fidelity (Al-Radhi et al., 20 Jan 2026).

• SpecDiff-GAN:

Spectrally-shaped noise diffusion during adversarial training, along with replacement of the MSD by a Multi-Resolution Discriminator, stabilizes GAN learning and yields improvements in PESQ, STOI, and WARP-Q over vanilla HiFi-GAN in both speech and music synthesis (Baoueb et al., 2024).

5. Advances in Time-Frequency Discriminators

Enhancements to the basic MPD/MSD structure focus on discriminators optimized for pitch and fine-grained temporal dynamics:

• MS-SB-CQT Discriminator:

Applies the Constant-Q Transform (CQT) at multiple bins-per-octave scales; sub-band processing re-aligns octaves in frequency/time for Conv2D embedding. The CQT-based discriminator augments the standard MSD, significantly improving F0 tracking, harmonic sharpness, and overall MOS, especially on singing voice synthesis (Gu et al., 2023, Gu et al., 2024). Joint use with MS-STFT discriminators leverages the respective strengths: CQT for fine-grained harmonic detail, STFT for broad time localization.

• Multi-Scale Temporal-Compressed CWT Discriminator:

A continuous wavelet transform (CWT)-based critic yields dynamic time-frequency resolution, directly targeting both harmonic and onset content. Employing multiple mother wavelets and temporal compression, this module has demonstrated further MOS gains when combined with STFT and CQT-based discriminators (Gu et al., 2024).

These discriminators are only active during training; inference speed is unaffected.

6. Quantitative Performance and Evaluation

HiFi-GAN and its derivatives have been evaluated on wide-ranging benchmarks in speech, singing, and music. Key metrics include:

Model/Extension	Domain	MOS (seen/unseen)	F0 RMSE	PESQ	RTF
HiFi-GAN Baseline	Singing	3.27/3.40	–	~3.5	~0.01
+MS-SB-CQT+MS-STFT	Singing	3.87/3.78	↓	↑	~0.01
Source-Filter HiFi-GAN	Singing	3.66 (copy)	0.038	–	0.008
Prosody-Attn, ISTFT	Speech	4.45	16.8 Hz	–	0.002–.01
SpecDiff-GAN	Speech	–	–	3.76	×221 RT

All variants preserve or improve real-time synthesis capability, especially compared to autoregressive or flow-based alternatives.

7. Significance and Future Directions

HiFi-GAN has catalyzed a shift toward parallel, adversarial, upsampling-based vocoders, now standard in neural TTS, singing voice synthesis, neural speech codecs, and controllable TTS applications. Persistent challenges include explicit phase modeling, extreme pitch/time control, and perceptual quality at very low bitrates or with highly expressive inputs. Recent phase-coherent approaches and the integration of adaptive time-frequency discriminators suggest rich directions for further fidelity improvements and generalization (Al-Radhi et al., 20 Jan 2026, Gu et al., 2023, Gu et al., 2024). The architecture's modularity ensures continued adaptability as new generative paradigms and discriminators emerge.