---
title: 'Vocos: Fourier Neural Vocoder'
url: https://www.emergentmind.com/topics/vocos
type: topic
---

# Vocos: Fourier Neural Vocoder

Vocos is a neural vocoder in the time–frequency domain that directly generates Fourier spectral coefficients from acoustic features and reconstructs the waveform with inverse short-time Fourier transform (iSTFT), rather than predicting sample-domain waveforms through learned upsampling stacks [2306.00814]. In the original formulation, it is a generative adversarial model conditioned on mel-spectrograms and built from ConvNeXt-style blocks operating at fixed frame resolution; this design was introduced to narrow the quality gap between time-domain GAN vocoders and Fourier-based vocoders while substantially improving inference efficiency [2306.00814]. Subsequent work has treated Vocos both as a baseline and as a reusable architectural principle for low-energy spiking vocoding, bandwidth extension, low-bitrate codecs, and larger spectrogram-conditioned singing voice systems [2509.13049] [2603.07285] [2601.13055] [2508.01796].

## 1. Foundational architectural premise

Vocos was introduced as a Fourier-based neural vocoder that predicts complex short-time Fourier transform (STFT) coefficients directly and then reconstructs audio via iSTFT [2306.00814]. Its conditioning signal in the original speech-vocoding setting is a mel-spectrogram extracted at 24 kHz with $n\_fft = 1024$, hop $H = 256$, and 100 mel bins, and the model predicts the single-sideband complex spectrum for each frame [2306.00814]. Because the generator stays at the conditioning frame rate throughout the network, temporal resolution remains constant across depth and no transposed-convolution upsampling is required [2306.00814].

This architectural choice distinguishes Vocos from several major vocoder families. Compared with autoregressive models such as WaveNet, WaveRNN, and LPCNet, it is non-autoregressive and therefore avoids sequential synthesis latency; compared with time-domain GAN vocoders such as MelGAN, HiFi-GAN, and BigVGAN, it avoids the large upsampling stacks that dominate compute and memory bandwidth; compared with diffusion or flow-matching vocoders, it requires a single forward pass rather than tens or hundreds of iterative denoising steps [2509.13049]. The original motivation was that time-domain GAN systems inherit large upsampling factors, potential aliasing from learned upsamplers, and heavier final-stage computation, whereas Fourier-domain synthesis can exploit FFT/iFFT algorithms and the inductive bias of harmonic organization in time–frequency representations [2306.00814].

The generator backbone adapts ConvNeXt blocks to 1D sequences. The blocks use depthwise convolution for temporal context, pointwise convolutions for expansion and projection, GELU activations, Layer Normalization, and residual connections, all at fixed hop-rate resolution [2306.00814]. This fixed-resolution design is central to later Vocos-derived systems as well: Spiking Vocos explicitly preserves the same macro-architecture, input/output parameterization, and block topology while replacing the ANN backbone with a spiking variant [2509.13049], and VoCodec adopts an “improved Vocos” encoder/decoder architecture for causal low-bitrate coding in the time–frequency domain [2601.13055].

A recurrent misconception in neural vocoding has been that Fourier-domain models are structurally faster but intrinsically inferior in perceptual quality because of phase recovery difficulties. Vocos was proposed specifically to challenge that assumption by eliminating a separate phase-recovery stage and learning complex coefficients directly [2306.00814].

## 2. Spectral parameterization and waveform reconstruction

The defining technical mechanism of Vocos is its parameterization of complex STFT coefficients. If $h$ denotes the hidden representation projected to the output head, it is split into magnitude and phase parameters, $m$ and $p$, per frequency bin. Magnitude is produced as
$$
M = \exp(m),
$$
while phase is mapped onto the unit circle through
$$
x = \cos(p), \qquad y = \sin(p).
$$
The predicted complex coefficient is then
$$
\hat{X}(k,n) = M(k,n)\cdot (x(k,n) + j y(k,n)),
$$
which implies a wrapped phase $\phi = \operatorname{atan2}(y,x)\in(-\pi,\pi]$ by construction [2306.00814].

This parameterization is significant because it avoids direct regression of absolute wrapped phase with bounded activations, a strategy that the original paper reports as degraded relative to the unit-circle formulation [2306.00814]. In the broader literature around Vocos, this point became central to subsequent analyses of joint magnitude–phase estimation. A later comparative study categorized Vocos as a “single-stream” model in which a shared backbone processes features for both magnitude and phase and only the final output heads are separated; in that study, this topology was contrasted with “dual-stream” APNet2-style models, and Vocos’s single-stream formulation was associated with robust magnitude–phase coupling on LibriTTS [2509.18806].

Reconstruction is performed by iSTFT with overlap-add. In the original formulation, if $w[\cdot]$ is the analysis window, $N$ the FFT size, and $H$ the hop size, Vocos generates $\hat{X}(k,n)$ for single-sideband bins and reconstructs the waveform through the usual overlap-add iSTFT pipeline [2306.00814]. Because waveform-rate expansion is delegated to iSTFT rather than to learned transposed-convolution layers, the “upsampling” step is computationally cheap and lossless in the signal-processing sense used by the model design [2509.13049].

This STFT-centric view of generation has enabled multiple reinterpretations of Vocos. In bandwidth extension, a Vocos-style network maps mel features to complex STFT coefficients at a fixed 48 kHz grid and then merges generated high-frequency content with the original low band through a Linkwitz–Riley-inspired frequency-domain refiner [2603.07285]. In low-bitrate speech coding, VoCodec computes STFT, forms
$$
Z_{in} = \operatorname{Concat}(\log(|X|), \angle(X)),
$$
encodes the frame sequence, quantizes it with residual vector quantization, and decodes complex spectra that are again inverted through iSTFT [2601.13055]. These systems preserve the core Vocos principle that spectral prediction plus iSTFT can replace explicit in-network waveform upsampling.

## 3. Training objectives, discriminators, and learned representations

The original Vocos is trained adversarially with Multi-Period Discriminator (MPD) and Multi-Resolution Discriminator (MRD) components, using hinge GAN losses, an $L\_1$ mel reconstruction term, and discriminator feature matching [2306.00814]. The mel loss is
$$
L_{mel} = \lVert \mathcal{M}(\boldsymbol{x}) - \mathcal{M}(\hat{\boldsymbol{x}}) \rVert_1,
$$
and the feature-matching loss averages $L\_1$ distances between discriminator feature maps on real and generated audio [2306.00814]. The paper emphasizes that, unlike some other Fourier-domain systems, Vocos does not rely on multi-resolution STFT losses as an explicit reconstruction loss; MRD appears in the discriminator stack rather than as a separate spectral loss [2306.00814].

Training in the original speech-vocoding setting used LibriTTS at 24 kHz, AdamW with learning rate $2\times 10^{-4}$ and betas $(0.9, 0.999)$, cosine decay, random crops of 16384 samples, batch size 16, and up to 2M iterations, split as 1M generator and 1M discriminator iterations [2306.00814]. Random gain augmentation was applied so that peaks fell between $-1$ and $-6$ dBFS [2306.00814].

Later systems preserved the Vocos backbone while altering objectives for new tasks. Spiking Vocos distills from an ANN Vocos teacher and supervises both intermediate features and final magnitude/phase spectra. Its distillation losses include a layer-wise feature alignment term,
$$
L_{feat} = \sum_{n=1}^{N} \|F(z_{stu}^{(n)}) - z_{tea}^{(n)}\|_2^2,
$$
a log-magnitude loss,
$$
L_M = \|\log A_{stu} - \log A_{tea}\|_1,
$$
and phase-aware anti-wrapping losses for instantaneous phase, group delay, and phase time-difference, all combined as
$$
L_{KD} = \lambda_{feat}L_{feat} + \lambda_P L_P + \lambda_M L_M
$$
[2509.13049]. VoCodec, by contrast, adds codebook and commitment losses for residual vector quantization and uses least-squares GAN losses rather than the hinge setup reported for original Vocos [2601.13055]. Vocos-based bandwidth extension combines MRSTFT, mel $L_1$, adversarial training with an MRD discriminator, and feature matching, though the paper explicitly notes that some MRSTFT term details are standard or inferred rather than fully spelled out in the source [2603.07285].

The representation learned by Vocos has also been studied structurally. In the joint-estimation analysis of Vocos and APNet2, Vocos was treated as the canonical single-stream architecture with shared ConvNeXt v1 blocks and direct phase prediction, and the study argued that shared representations provide implicit information exchange between magnitude and phase throughout the network [2509.18806]. This interpretation is notable because it reframes Vocos not merely as a fast Fourier-domain vocoder, but also as a topology for stable joint estimation in complex spectral spaces.

## 4. Empirical quality and efficiency

In the original LibriTTS 24 kHz evaluation, Vocos achieved UTMOS 3.734, ViSQOL 4.66, PESQ 3.700, V/UV F1 0.9582, and periodicity 0.101 [2306.00814]. Among the reported baselines, it achieved the best ViSQOL, PESQ, V/UV F1, and lowest periodicity, while BigVGAN slightly led on UTMOS with 3.749 [2306.00814]. Subjective listening yielded MOS 3.62 ± 0.15 and SMOS 4.55 ± 0.15, matching BigVGAN without a statistically significant difference according to a Wilcoxon test with $p > 0.05$ [2306.00814].

The efficiency results are a central part of Vocos’s identity. On an NVIDIA A100 GPU and an AMD EPYC 7542 CPU, measured with batches of 16 one-second samples and no hardware-specific optimizations, Vocos reached 6696.52× real-time on GPU and 169.63× real-time on CPU, with 13.5M parameters [2306.00814]. In the same setup, HiFi-GAN achieved 495.54× on GPU and 5.84× on CPU, BigVGAN 98.61× on GPU and 0.40× on CPU, and iSTFTNet 1045.94× on GPU and 14.44× on CPU [2306.00814]. These numbers are the empirical basis for the claim that Vocos delivers an order-of-magnitude increase in speed over prevailing time-domain neural vocoding approaches [2306.00814].

A compact comparison is useful because later papers repeatedly position their own methods relative to this efficiency–quality profile.

| System | Role of Vocos | Selected reported result |
|---|---|---|
| Original Vocos [2306.00814] | Fourier-based GAN vocoder | GPU 6696.52× real-time; CPU 169.63× real-time |
| Spiking Vocos [2509.13049] | SNN reimplementation of Vocos backbone | Final 4-step model uses 14.7% of ANN energy |
| BWE via Vocos [2603.07285] | Backbone generator for bandwidth extension | RTF 0.0001 on A100, batch 32 |
| VoCodec [2601.13055] | Vocos-based speech codec backbone | 349.29M MACs/s and 30 ms latency |

Later evaluations reinforced both the quality and the topology arguments around Vocos. In the 2025 study on joint magnitude–phase estimation, baseline Vocos on LibriTTS achieved PESQ 3.487, UTMOS 3.356, VISQOL 4.861, MCD 3.260, M-STFT 0.921, V/UV F1 0.945, periodicity RMSE 0.124, pitch RMSE 32.846, and 13.53M parameters [2509.18806]. In the same study, a baseline APNet2* dual-stream system exhibited severe collapse on LibriTTS, with PESQ 2.556 and UTMOS 1.886, leading the authors to argue that Vocos’s single-stream design is comparatively robust on large, diverse datasets [2509.18806].

## 5. Major derivatives and application domains

Vocos has become a reusable design pattern rather than remaining only a standalone vocoder. Several later systems preserve its Fourier-domain synthesis pathway while changing the task definition or internal computation.

Spiking Vocos recasts the Vocos backbone as a spiking neural network for low-energy inference on computationally restricted edge devices [2509.13049]. It preserves the mel-input, constant-resolution, iSTFT-based macro-architecture of Vocos, but replaces the artificial-neuron ConvNeXt backbone with a spiking variant using Parametric Leaky Integrate-and-Fire neurons inserted before pointwise convolutions [2509.13049]. It adds three targeted mechanisms: a Spiking ConvNeXt design that moves dominant pointwise channel-mixing into sparse spike-driven accumulations, an amplitude shortcut path
$$
Z_{recover} = |Z_{in}| \odot Z_{out}
$$
to mitigate the binary-spike information bottleneck, and self-architectural distillation from an ANN Vocos teacher [2509.13049]. On LibriTTS test-clean, the final 4-step model reported UTMOS 3.74, PESQ 3.45, ViSQOL 4.65, V/UV F1 0.9558, periodicity 0.116, and an estimated energy of $8.5\times 10^9$ pJ versus $58.0\times 10^9$ pJ for ANN Vocos, corresponding to 14.7% of the ANN energy budget [2509.13049].

In audio bandwidth extension, a Vocos-style vocoder serves as the backbone generator that maps an 80-bin mel-spectrogram computed on a 48 kHz grid to complex STFT coefficients, followed by iSTFT and a frequency-domain refiner that blends the generated high band with the original low band around the input Nyquist [2603.07285]. The model uses 8 residual ConvNeXt-style blocks with channel width 512, depthwise separable temporal convolution with kernel $7\times 1$, and feed-forward expansion $512 \rightarrow 1536 \rightarrow 512$ [2603.07285]. On VCTK, the proposed system reported LSD 0.85/0.80/0.74 for 8/12/16→48 kHz and ViSQOL 3.51/3.53/3.69, with RTF 0.0053 on an 8-core CPU and 0.0001 on an A100 at batch 32 [2603.07285].

In speech coding, VoCodec uses Vocos as the vocoder backbone in both encoder and decoder, again operating entirely in the time–frequency domain [2601.13055]. It computes STFT with a 30 ms window, 10 ms hop, and FFT size 720, concatenates log-magnitude and phase, projects the result to 192 channels, encodes the frame sequence with ConvNeXt blocks and a lightweight attention module, applies 6-layer residual vector quantization, and decodes complex spectra through a Vocos-style decoder and iSTFT [2601.13055]. The reported system has 3.47M parameters, 349.29M MACs/s, receiver-side compute 144.82M MACs/s, 30 ms latency, and ranked fourth on Track 1 of the 2025 LRAC Challenge while achieving the highest subjective MUSHRA score on the clean speech test set [2601.13055].

Vocos has also been adapted to larger spectrogram representations. In singing voice synthesis, a redesigned “Vocos2D” follows a diffusion-based linear spectrogram estimation stage and replaces the original 1D mel-conditioned generator with a 2D ConvNeXt-style vocoder specialized for large linear spectrograms such as $F=592$ bins, using per-frequency learned embeddings, 2D convolutions, and persistent shortcut injection of the input condition into every block [2508.01796]. This was motivated by the observation that the original Vocos generator “sees” the conditioning spectrogram once and must propagate it through hidden states, which becomes brittle when the conditioning spectrogram has many more frequency bins [2508.01796].

## 6. Comparative analyses, modifications, and open questions

Subsequent literature has treated Vocos as both a reference model and a source of modifiable components. One line of work studied whether magnitude and phase should be estimated in a shared stream or in separate streams. In that comparison, Vocos represented the shared or “single-stream” paradigm, while APNet2 represented the dual-stream paradigm [2509.18806]. On LibriTTS, the paper reported that APNet2’s dual-stream baseline collapsed severely, and then proposed three stabilization strategies—topological sharing or shuffling between branches, a pseudo-inverse mel-spectrogram prior in the source space, and MI-RI output coupling in the output space—to narrow the gap with Vocos [2509.18806]. A plausible implication is that Vocos’s robustness is not only a matter of Fourier-domain synthesis, but also of shared feature topology.

A separate line of work addressed model size in frequency-domain GAN vocoders. FreeV compared itself against Vocos and APNet2 and introduced a pseudo-inverse mel filterbank prior for amplitude estimation,
$$
\hat{A}_0 = \max(\operatorname{Abs}(M^{+}m), 10^{-5}),
$$
turning amplitude prediction into a residual refinement problem [2406.08196]. In that study’s reported setup, Vocos had 13.5M parameters and GPU RTF 0.0006, APNet2 had 31.4M and 0.0011, and FreeV had 18.2M and 0.0006, while the authors argued that the pseudo-inverse trick also improved early-stage convergence when inserted into Vocos-like generators [2406.08196]. This does not alter the core Vocos design directly, but it shows that Vocos is compatible with source priors that partially offload spectral estimation from the neural backbone.

The broader ConvNeXt-vocoder literature has also positioned Vocos as a precursor to unified frameworks. WaveNeXt 2 describes Vocos as a ConvNeXt-based vocoder that predicts high-resolution STFT spectra from mel conditioning and reconstructs via iSTFT, then argues that such ConvNeXt generators had primarily been used in GAN frameworks and showed limited performance in multi-speaker settings unless carefully engineered [2605.25506]. WaveNeXt 2 responds by moving from direct spectrum prediction to residual denoising and sub-modeling across GAN and diffusion variants [2605.25506]. This suggests an ongoing diversification of the Vocos design ethos: the original efficiency-driven ConvNeXt-plus-spectral-reconstruction template remains influential, but later work increasingly reinterprets what the generator should predict.

Several limitations recur across the ecosystem. Original Vocos depends on consistent STFT configuration between training and inference; mismatches in sample rate, FFT size, hop, or window can degrade quality [2306.00814]. Spiking Vocos notes that binary spiking can introduce small degradations in signal-level metrics such as PESQ because of quantization-like effects, requiring amplitude shortcuts, temporal shift modules, and distillation to compensate [2509.13049]. Bandwidth-extension work notes that the model hallucinates missing high-band content and can produce overly bright textures or faint harmonic residues on some inputs, although the smooth crossover refiner mitigates abrupt spectral seams [2603.07285]. Singing-voice work points to the compute overhead of diffusion-based spectrogram estimation and the increased memory cost of 2D processing over large frequency grids [2508.01796].

Taken together, these developments frame Vocos as more than a single 2023 vocoder architecture. It is a family of design commitments: frame-rate processing, explicit complex spectral prediction, iSTFT-based synthesis, ConvNeXt-derived backbones, and a bias toward replacing learned waveform upsampling with signal-processing structure [2306.00814]. Later research has preserved those commitments while stretching them toward energy-efficient spiking computation, causal speech coding, arbitrary-rate bandwidth extension, and large-bandwidth singing synthesis [2509.13049] [2601.13055] [2603.07285] [2508.01796].

Source: https://www.emergentmind.com/topics/vocos