Vocos2D Neural Vocoder

• Vocos2D is a frequency-domain neural vocoder that processes spectral representations via inverse STFT and 2D convolutional upsamplers.
• It integrates spiking ConvNeXt blocks with PLIF neurons and amplitude shortcuts to preserve fine-grained audio features while optimizing computational efficiency.
• The architecture leverages self-distillation and TSM to achieve near-ANN perceptual quality, reducing energy usage dramatically for high-fidelity speech synthesis.

Vocos2D is a frequency-domain neural vocoder architecture designed for high-fidelity audio synthesis. It is notable for its use in neural speech generation tasks and for serving as the foundation of recent energy-efficient spiking neural vocoder (SNN) research. While the full technical details for Vocos2D itself are not public, its methodological lineage and performance benchmarks are well contextualized by work on the Spiking Vocos framework, which adapts and extends Vocos2D with spiking neural computation, knowledge distillation, and optimized upsampling pipelines (Chen et al., 16 Sep 2025).

1. Underlying Architecture

Vocos2D leverages a frequency-domain approach by operating directly on spectral representations of audio. The core workflow involves analysis and synthesis steps via inverse Short-Time Fourier Transform (iSTFT). The original Vocos2D utilizes 2D transposed convolutional upsamplers for time-frequency representation refinement. The Spiking Vocos framework introduces an architectural revision where these upsamplers are replaced by spiking ConvNeXt blocks, augmented by amplitude shortcut pathways and temporal information fusion, preserving key features for spectral audio reconstruction while optimizing computational efficiency.

A potential extension for Vocos2D is to insert Parametric Leaky Integrate-and-Fire (PLIF) spiking neurons prior to heavy 2D pointwise convolutions, together with amplitude shortcuts along both frequency and time axes. This addresses the information loss typically induced by the binary quantization inherent in spiking activations.

2. Spiking ConvNeXt Blocks with PLIF Neurons

The Spiking ConvNeXt module adapts ConvNeXt-style bottlenecks to event-driven computation. In place of conventional activation functions, PLIF neurons model the membrane dynamics, enabling energy-efficient, sparse spike propagation. The module is structured as follows:

• 1D depthwise convolution for local feature extraction.
• Layer normalization.
• PLIF activation yielding a sparse spike tensor.
• Sequential $1 \times 1$ pointwise convolutions on binary spike inputs, further sparsified by PLIF neurons.
• Amplitude shortcut connection re-injecting the absolute input amplitude at the block output.
• (Optionally) a Temporal Shift Module (TSM) for temporal context fusion.

The PLIF neuron captures state transitions via three phases—membrane charging, spike generation via a Heaviside step function, and membrane reset—accompanied by a learnable time constant $\tau$: $$H_t = V_{t-1} + \frac{1}{\tau}(X_t - (V_{t-1} - V^{\mathrm{re}}))$$

$S_t = \Theta(H_t - V^{\mathrm{th}})$

$$V_t = \begin{cases} V^{\mathrm{re}}, & \text{if } S_t = 1 \ H_t, & \text{if } S_t = 0 \end{cases}$$

3. Amplitude Shortcut for Information Preservation

Amplitude shortcut paths address the loss of fine-grained amplitude information due to the binary (spike) nature of SNNs. By injecting the modulus of the input activations into the block output, the architecture compensates for quantization artifacts: $$Z_{\mathrm{rec}} = |Z_{\mathrm{in}}| \odot W_2\,\mathrm{PLIF}(W_1\,\mathrm{PLIF}(\mathrm{LN}(\mathrm{DWConv}(Z_{\mathrm{in}}))))$$ This mechanism is essential both in the 1D (original Spiking Vocos) and proposed 2D extensions for Vocos2D, where amplitude shortcuts are applied across time and frequency axes to maintain signal dynamics.

4. Knowledge Distillation and Training

Spiking Vocos employs a self-architectural distillation paradigm, where an SNN (student) is supervised by its ANN (teacher, original Vocos) through multi-faceted loss functions:

• Intermediate Feature Loss: Alignment of internal representations via a learned adapter at each block.
• Magnitude Spectrum Loss: $$\mathcal{L}_M = \|\log A_{\mathrm{stu}} - \log A_{\mathrm{tea}}\|_1$$
• Phase Spectrum Loss: Encompasses instantaneous phase, group delay, and phase time difference, with anti-wrapping to mitigate phase discontinuities.

The combined knowledge distillation loss is: $$\mathcal{L}_{\mathrm{KD}} = \lambda_{\mathrm{feat}}\,\mathcal{L}_{\mathrm{feat}} + \lambda_{M}\,\mathcal{L}_{M} + \lambda_{P}\,\mathcal{L}_{P}$$ Training is conducted for 1M steps using AdamW on the LibriTTS dataset, with adapters inserted after TSM where present.

5. Temporal Shift Module

The Temporal Shift Module (TSM) augments temporal context without incurring additional MAC (multiply-accumulate) cost. Channels are subdivided and shifted along the temporal axis, then merged residually: $$Z_{\mathrm{out}} = Z_{\mathrm{org}} + \alpha Z_{\mathrm{shift}}$$ with $\alpha = 0.5$. This facilitates information fusion from neighboring time steps, crucial in sequence modeling.

6. Quantitative Evaluation and Energy Efficiency

The following table summarizes key metrics for the original ANN Vocos, Spiking Vocos in its optimal configuration, and characteristic Vocos2D results (where available):

Model	UTMOS	PESQ	Energy (pJ, $L$=1000)
Vocos (ANN baseline)	3.82	3.65	$58.0 \times 10^9$
Spiking Vocos (4-step, TSM+KD best)	3.74	3.45	$8.5 \times 10^9$
Vocos2D (ANN, approximate)	$\sim$3.80	–	$\sim 1\times$ ANN energy

Spiking Vocos achieves comparable perceptual quality (UTMOS 3.74 vs. 3.82 for ANN baseline) while requiring only 14.7% of the ANN’s energy consumption. 8-timestep SNN variants further improve fidelity (UTMOS 3.80) at increased inference latency.

7. Trade-offs, Applicability, and Extensions

The main trade-off centers on fidelity versus efficiency and latency. Higher temporal resolution SNNs approach ANN quality but at increased real-time cost, while reduced timestep configurations offer substantial energy savings at minimal perceptual degradation. The methodological insight for Vocos2D is that replacing transposed 2D convolutions with spiking ConvNeXt blocks and amplitude shortcuts, combined with self-architectural distillation and TSM, sustains high-quality synthesis in frequency-domain neural vocoders with marked reductions in MACs and energy usage. This suggests SNN-based architectures (PLIF + shortcut) with knowledge distillation are effective in bridging the performance gap between event-driven and conventional deep learning approaches in vocoding (Chen et al., 16 Sep 2025).