APNet2 Neural Vocoder for Speech Synthesis

• APNet2 is a dual-branch neural vocoder that predicts amplitude and phase spectra from mel-spectrograms using a ConvNeXt v2 backbone for improved efficiency at a 22.05 kHz sampling rate.
• It refines adversarial training by introducing a multi-resolution discriminator and switching from least-squares to hinge GAN loss, addressing phase prediction challenges with longer frame shifts.
• APNet2 balances quality and speed by leveraging direct time-frequency domain predictions, making it a competitive alternative to HiFi-GAN and iSTFTNet in modern speech synthesis.

Searching arXiv for APNet2 and closely related vocoder papers to ground the article. to=arxiv_search.search 彩票直属 彩神争霸 qq的天天中彩票ి 大发快三官网 code=json {"query":"APNet2 neural vocoder amplitude phase spectra (Du et al., 2023) Vocos FreeV", "max_results": 10} APNet2 is a neural vocoder for speech synthesis that predicts amplitude and phase spectra directly from acoustic features and reconstructs the waveform with inverse short-time Fourier transform (ISTFT). It is presented as an improved successor to APNet, with the stated objective of removing the earlier system’s limitations at higher sampling rates and longer spectral frame shifts. In its common configuration, APNet2 operates at a waveform sampling rate of 22.05 kHz with a spectral frame shift of 256 points, uses ConvNeXt v2 as the backbone for amplitude and phase prediction, introduces a multi-resolution discriminator (MRD) into the GAN-based losses, and replaces APNet’s negative-cosine anti-wrapping function with a linear form for phase loss. The reported outcome is synthesized speech quality comparable to HiFi-GAN and iSTFTNet, while retaining substantially faster inference than those waveform-oriented baselines (Du et al., 2023).

1. Origins and problem setting

APNet2 arises within the broader shift from direct waveform vocoders to time-frequency-domain neural vocoders. The motivating contrast is explicit: traditional autoregressive vocoders such as WaveNet are high quality but slow, whereas GAN-based waveform vocoders such as HiFi-GAN are much faster but still rely on multiple transposed convolutions to upsample low-rate acoustic features to sample-level waveforms. APNet2 follows the alternative strategy used by iSTFTNet, Vocos, and APNet: predict spectral representations at frame rate, then reconstruct the waveform by ISTFT (Du et al., 2023).

The motivation for the transition from APNet to APNet2 is tied to two constraints identified for the original model. First, APNet had been demonstrated mainly at 16 kHz, whereas the authors note that many modern benchmarks use 22.05 kHz. Second, APNet’s phase prediction becomes harder when the frame shift increases; the paper states that phase prediction is highly sensitive to frame shift, and that with longer frame shifts phase continuity deteriorates and phase modeling accuracy drops. APNet2 is therefore positioned as a higher-capacity, more robust all-frame-level vocoder for higher sampling rate and longer-frame-shift synthesis (Du et al., 2023).

This framing places APNet2 between waveform GAN vocoders and other frequency-domain systems. Relative to waveform generators, its computational advantage derives from avoiding sample-rate prediction. Relative to earlier spectral vocoders, its contribution is not a change in the basic ISTFT-based paradigm, but a stronger backbone, revised adversarial training, and revised phase-loss design. A plausible implication is that APNet2 should be understood less as a new representation family than as an optimization and architecture refinement of direct amplitude-phase prediction.

2. Dual-branch architecture and spectral reconstruction

APNet2 retains the APNet principle of direct prediction of amplitude and phase spectra in parallel. The full model contains two branches: an amplitude spectrum predictor (ASP) and a phase spectrum predictor (PSP). Both take acoustic features such as an 80-dimensional mel-spectrogram as input; ASP predicts the logarithmic amplitude spectrum, PSP predicts the wrapped phase spectrum, and the two outputs are combined into an STFT spectrum for waveform recovery by ISTFT (Du et al., 2023).

ASP is described as a cascade of an input convolution layer, a ConvNeXt v2 backbone, and an output convolution layer. PSP is similarly organized around input convolution layers and a ConvNeXt v2 backbone, but its output stage uses the phase parallel estimation architecture inherited from APNet. In that design, two parallel linear convolution outputs represent pseudo-real and pseudo-imaginary parts, $R$ and $I$, and the wrapped phase is computed as

$$\Phi(R,I)=\arctan\left(\frac{I}{R}\right)-\frac{\pi}{2}\cdot Sgn^*(I)\cdot\left[Sgn^*(R)-1\right],$$

where $Sgn^*(x)=1$ if $x\ge 0$, and $-1$ otherwise (Du et al., 2023).

The central architectural change relative to APNet is the adoption of ConvNeXt v2 in both branches. The paper characterizes ConvNeXt v2 as having stronger modeling capacity than the ResNet used in APNet, and notes that compared with ConvNeXt in Vocos it adds global response normalization (GRN), intended to improve feature diversity, contrast, and selectivity. A ConvNeXt v2 block is described as containing a large-kernel depth-wise convolution, layer normalization, a point-wise convolution expanding feature dimensions, GELU, GRN, another point-wise convolution restoring dimensionality, and a residual connection (Du et al., 2023).

Subsequent work recasts this same architecture in topological terms. In that literature, APNet2 is treated as a representative dual-stream time-frequency neural vocoder, contrasted with the single-stream design of Vocos. The distinction is that Vocos shares most of the deep backbone and only splits near the output heads, whereas APNet2 uses independent magnitude and phase branches with little or no interaction. This later reading emphasizes that APNet2’s “joint estimation” of magnitude and phase is implemented through two largely decoupled streams rather than a deeply shared backbone (Dai et al., 23 Sep 2025).

3. Objective function, phase treatment, and adversarial training

The APNet2 training objective preserves the multi-term structure of APNet while changing several specific components. The total generator loss is

$$\mathcal L_G=\lambda_A\mathcal L_A+\lambda_P\mathcal L_P+\lambda_S\mathcal L_S+\lambda_W\mathcal L_W,$$

where $\mathcal L_A$ is the amplitude spectrum loss, $\mathcal L_P$ the phase spectrum loss, $\mathcal L_S$ the reconstructed STFT spectrum loss, and $I$0 the waveform-related loss. The weights $I$1 are unchanged from APNet (Du et al., 2023).

The amplitude term $I$2 is the $I$3 distance between predicted and target log amplitude spectra. The phase term $I$4 is the sum of instantaneous phase loss, group delay loss, and phase time-difference loss. The key revision is the anti-wrapping function. APNet had used a negative cosine anti-wrapping function; APNet2 replaces it with the linear form

$I$5

The stated reason is that the linear form works better for phase prediction than the cosine form (Du et al., 2023).

The reconstructed STFT spectrum loss $I$6 includes STFT consistency loss between the reconstructed STFT and a consistent STFT, together with $I$7 losses on real and imaginary parts between reconstructed and natural STFT spectra. The waveform loss $I$8 includes mel-spectrogram loss, feature matching loss, and adversarial loss. In adversarial training, APNet2 uses a multi-period discriminator (MPD) and introduces MRD as a replacement for the multi-scale discriminator used in APNet. The paper states that the earlier MSD has a high parameter count, slows training, and can overfit; MRD instead uses strided 2-D convolutions and leaky ReLU activations, with multiple sub-discriminators operating at different temporal and frequency resolutions (Du et al., 2023).

Another explicit revision is the switch from least-squares GAN loss to hinge GAN loss. The generator and discriminator terms are given as

$I$9

and

$$\Phi(R,I)=\arctan\left(\frac{I}{R}\right)-\frac{\pi}{2}\cdot Sgn^*(I)\cdot\left[Sgn^*(R)-1\right],$$0

The discriminator loss is $$\Phi(R,I)=\arctan\left(\frac{I}{R}\right)-\frac{\pi}{2}\cdot Sgn^*(I)\cdot\left[Sgn^*(R)-1\right],$$1. In the paper’s ablations, replacing hinge GAN with least-squares GAN improves $$\Phi(R,I)=\arctan\left(\frac{I}{R}\right)-\frac{\pi}{2}\cdot Sgn^*(I)\cdot\left[Sgn^*(R)-1\right],$$2 somewhat but worsens waveform and spectral metrics, which is presented as evidence that hinge GAN is better for balanced overall quality (Du et al., 2023).

4. Experimental configuration and reported performance

The reported experiments use LJSpeech, described as 13,100 clips and about 24 hours of speech from a single English female speaker, split into 12,000 training, 100 validation, and 500 test utterances. The common configuration is a waveform sampling rate of 22.05 kHz, FFT size 1024, frame length 1024 samples, and frame shift 256 samples, approximately 11.6 ms. Input features are 80-dimensional mel-spectrograms over the frequency range 0–8000 Hz, so the vocoder implicitly performs bandwidth extension when generating a 22.05 kHz waveform (Du et al., 2023).

Training uses 8 ConvNeXt v2 blocks per ASP and PSP, depthwise convolution kernel size 7, channel size 512, and output pointwise convolution channels 1536. The training run length is 2 million steps with batch size 16, random crop length 8192 samples, AdamW optimization with $$\Phi(R,I)=\arctan\left(\frac{I}{R}\right)-\frac{\pi}{2}\cdot Sgn^*(I)\cdot\left[Sgn^*(R)-1\right],$$3, $$\Phi(R,I)=\arctan\left(\frac{I}{R}\right)-\frac{\pi}{2}\cdot Sgn^*(I)\cdot\left[Sgn^*(R)-1\right],$$4, weight decay 0.01, initial learning rate $$\Phi(R,I)=\arctan\left(\frac{I}{R}\right)-\frac{\pi}{2}\cdot Sgn^*(I)\cdot\left[Sgn^*(R)-1\right],$$5, and decay factor 0.999 per epoch, on a single Nvidia 2080Ti GPU. Baselines include HiFi-GAN, iSTFTNet, Vocos, and APNet, all trained in the same setting using open-source implementations (Du et al., 2023).

Two tasks are evaluated: analysis-synthesis, using natural mel-spectrograms as input, and TTS, using mel-spectrograms predicted by a FastSpeech2-based acoustic model. On analysis-synthesis at 22.05 kHz and 256-frame shift, APNet2 reports SNR 3.84, LAS-RMSE 6.34, MCD 1.73, F0-RMSE 44.33, V/UV error 5.31, RTF (GPU) 0.0015 = 665.05× real time, and RTF (CPU) 0.021 = 47.73× real time. The interpretation given in the source is that APNet2 is much faster than HiFi-GAN and iSTFTNet, close to them in objective quality, and clearly better than Vocos and APNet in the 80-dimensional mel setting (Du et al., 2023).

The subjective MOS results reinforce the same placement. In analysis-synthesis, APNet2 scores $$\Phi(R,I)=\arctan\left(\frac{I}{R}\right)-\frac{\pi}{2}\cdot Sgn^*(I)\cdot\left[Sgn^*(R)-1\right],$$6, compared with Natural $$\Phi(R,I)=\arctan\left(\frac{I}{R}\right)-\frac{\pi}{2}\cdot Sgn^*(I)\cdot\left[Sgn^*(R)-1\right],$$7, HiFi-GAN $$\Phi(R,I)=\arctan\left(\frac{I}{R}\right)-\frac{\pi}{2}\cdot Sgn^*(I)\cdot\left[Sgn^*(R)-1\right],$$8, iSTFTNet $$\Phi(R,I)=\arctan\left(\frac{I}{R}\right)-\frac{\pi}{2}\cdot Sgn^*(I)\cdot\left[Sgn^*(R)-1\right],$$9, Vocos $Sgn^*(x)=1$0, and APNet $Sgn^*(x)=1$1. In TTS, APNet2 scores $Sgn^*(x)=1$2, compared with Natural $Sgn^*(x)=1$3, HiFi-GAN $Sgn^*(x)=1$4, iSTFTNet $Sgn^*(x)=1$5, Vocos $Sgn^*(x)=1$6, and APNet $Sgn^*(x)=1$7. The paper further states that APNet2 is significantly better than Vocos and APNet and statistically comparable to HiFi-GAN and iSTFTNet in analysis-synthesis, with $Sgn^*(x)=1$8 versus iSTFTNet and $Sgn^*(x)=1$9 versus HiFi-GAN (Du et al., 2023).

5. Derivatives, baselines, and later reinterpretations

APNet2 became an immediate baseline for subsequent frequency-domain GAN vocoders. In "FreeV: Free Lunch For Vocoders Through Pseudo Inversed Mel Filter" (Lv et al., 2024), it serves as the reference architecture from which a streamlined derivative is built. That work keeps APNet2’s overall frequency-domain GAN formulation, retains the phase branch and the ASP/PSP separation, but makes the amplitude branch substantially cheaper by replacing a learned mel-to-amplitude mapping with a pseudo-inverse mel filter prior. The approximate amplitude spectrum is defined as

$x\ge 0$0

where $x\ge 0$1 is the precomputed pseudo-inverse of the mel filter. Relative to APNet2, FreeV replaces the ASP input convolution with the frozen matrix $x\ge 0$2, reduces ASP depth from 8 ConvNeXtV2 blocks to 1 block, matches ASP input-output dimensions to amplitude spectrum dimensions, and removes the ASP output convolution layer, while leaving PSP largely intact (Lv et al., 2024).

The reported comparison quantifies the computational weight of APNet2’s learned amplitude branch. FreeV has 18.2M parameters versus 31.4M for APNet2, corresponding to a roughly 42% reduction. On GPU, the paper reports APNet2 RTF 0.0011 and FreeV RTF 0.0006, described as a 1.8× speedup. On LJSpeech resynthesis, FreeV is reported to beat APNet2 on most listed metrics: MCD 3.112 versus 3.518, LAS-RMSE 0.779 versus 0.782, V/UV F1 0.956 versus 0.950, Periodicity 0.118 versus 0.132, F0-RMSE 26.40 versus 31.08, STOI 0.967 versus 0.950, and PESQ 3.431 versus 3.029 (Lv et al., 2024).

A separate later paper, "Rethinking the joint estimation of magnitude and phase for time-frequency domain neural vocoders" (Dai et al., 23 Sep 2025), treats APNet2 and Vocos as canonical examples of dual-stream and single-stream time-frequency vocoders, respectively. Its main empirical observation is that APNet2 performs reasonably on LJSpeech but suffers a severe performance collapse on LibriTTS, a much larger and more acoustically diverse corpus. The paper argues that the topological issue is not merely output representation or loss choice: even after aligning APNet2 variants with Vocos in unit choice, output type, and loss setup, a substantial gap remains, pointing to topological design as the key issue (Dai et al., 23 Sep 2025).

That work introduces three stabilization strategies targeting what it calls the topological space, source space, and output space. The topological interventions are partially shared structure and shuffled connections, both designed to increase interaction between the magnitude and phase streams. The source-space intervention replaces raw compressed mel input with a pseudo-inverse mel-spectrogram prior in the range space of the target STFT spectrum. The output-space intervention forces the phase branch to participate in magnitude formation through

$x\ge 0$3

so that magnitude-related quantities drive updates to both branches. The stated conclusion is that APNet2’s weakness is not simply that it is dual-stream, but that its original dual-stream form is too isolated for large-scale, diverse vocoding (Dai et al., 23 Sep 2025).

6. Technical significance, limits, and common points of confusion

Within the APNet lineage, APNet2’s significance is defined by a specific balance of representational choice and efficiency. It is an all-frame-level neural vocoder: it predicts amplitude and phase spectra at the same temporal resolution as the acoustic input, rather than upsampling to waveform resolution. This design is the source of its reported inference advantage over HiFi-GAN and iSTFTNet, because the model never predicts raw waveform samples directly and avoids the heavy transposed-convolution upsampling stacks used in waveform GAN vocoders (Du et al., 2023).

At the same time, the architecture is not presented as universally dominant across all conditions. In the original APNet2 paper, Vocos is fastest but much worse in quality in the 80-dimensional mel setting, whereas HiFi-GAN and iSTFTNet are higher quality but slower; APNet2 is positioned as a strong middle ground of high quality with very high efficiency (Du et al., 2023). Later work complicates that picture by showing that the same dual-stream decomposition can be fragile on large-scale and diverse data unless inter-branch coupling is strengthened (Dai et al., 23 Sep 2025). This suggests that APNet2’s empirical profile is dataset- and topology-sensitive rather than absolute.

A recurrent misunderstanding in the later literature is to treat APNet2 as merely a “phase-prediction” vocoder or, conversely, as a standard single-stream spectral model. Neither description matches the formulation in the cited works. APNet2 predicts both amplitude and phase spectra directly, but it does so through two separate branches, ASP and PSP, rather than a shared backbone with late output heads (Du et al., 2023). This distinction became central once later studies argued that strict branch separation reduces useful feature exchange, especially when phase estimation must generalize across diverse signals (Dai et al., 23 Sep 2025).

Another potential point of confusion is terminological rather than architectural. The vocoder APNet2 is distinct from "Attention-based Pyramid Aggregation Network" (Zhu et al., 2018), a visual place recognition model abbreviated as APANet, and from "Anchor-based Plain Net" (Du et al., 2021), a mobile image super-resolution model abbreviated as ABPN. In the speech-synthesis literature, APNet2 specifically denotes the direct amplitude-phase neural vocoder introduced as the successor to APNet (Du et al., 2023).