Spiking Vocos: Energy-Efficient Neural Vocoder

• Spiking Vocos is a spiking neural vocoder that applies PLIF neurons and spiking ConvNeXt modules for energy-efficient mel-to-waveform synthesis.
• It replaces heavy convolutional operations with a lightweight Temporal Shift Module and amplitude shortcut to retain spectral detail.
• Self-architectural distillation aligns spectral and phase outputs, achieving competitive performance with up to 15% ANN energy cost.

Spiking Vocos is a spiking neural vocoder introduced as an energy-efficient frequency-domain waveform synthesizer built upon the Vocos framework for deployment on computationally restricted edge devices. It takes a 100-dimensional mel-spectrogram as input, predicts the magnitude $A$ and phase $\phi$ of a complex spectrogram, and reconstructs the waveform with inverse STFT. Its core contribution is the conversion of the heavy ConvNeXt-style processing in Vocos into a spiking architecture centered on Parametric Leaky Integrate-and-Fire (PLIF) neurons, a Spiking ConvNeXt module, an amplitude shortcut path, a lightweight Temporal Shift Module (TSM), and a self-architectural distillation strategy. On LibriTTS test-clean, the reported system with TSM and distillation reaches UTMOS $=3.74$ and PESQ $=3.45$ while consuming $14.7\%$ of the energy of its ANN counterpart (Chen et al., 16 Sep 2025).

1. Origin, scope, and design objective

Spiking Vocos is motivated by a specific limitation of contemporary neural vocoders: although synthesis speed and fidelity have advanced substantially, high energy consumption remains a barrier to practical deployment on low-resource edge hardware. The system addresses this constraint by transferring a Vocos-style neural vocoder into the spiking domain, exploiting the event-driven computation of SNNs to reduce the cost of the most expensive operations while preserving perceptual quality (Chen et al., 16 Sep 2025).

The model is explicitly framed as a vocoder rather than a complete text-to-speech stack. Its input is a 100-dimensional mel-spectrogram at 24 kHz with $n\_fft=1024$ and $n\_hop=256$, and its output is a waveform reconstructed from predicted magnitude and phase. This places the method in the mel-to-waveform stage of speech synthesis or speech reconstruction pipelines. The source code is reported as available at https://github.com/pymaster17/Spiking-Vocos (Chen et al., 16 Sep 2025).

A central architectural decision is to preserve the efficient Vocos formulation while replacing the base ConvNeXt blocks with spiking counterparts. The result is not a generic SNN speech model, but a frequency-domain neural vocoder in which most $1\times1$ Multiply-Accumulate (MAC) operations are converted into spike-driven accumulation (AC) operations scaled by the firing rate $r$. This design target governs nearly every component of the system.

2. End-to-end architecture

The end-to-end pipeline is: mel-spectrogram $\rightarrow$ stacked Spiking ConvNeXt blocks with TSM $\phi$0 magnitude head and phase head $\phi$1 iSTFT $\phi$2 waveform (Chen et al., 16 Sep 2025).

Within this pipeline, the generator consists of a stack of $\phi$3 Spiking ConvNeXt blocks. Each block contains a depthwise convolution $\phi$4, a Temporal Shift Module, two point-wise $\phi$5 convolutions wrapped with spiking activations, and an amplitude shortcut path. The output stage uses two $\phi$6 convolutions to predict the magnitude $\phi$7 and phase $\phi$8 of the complex spectrogram, after which inverse STFT combines $\phi$9 and $=3.74$0 into the waveform.

Relative to the base ANN Vocos, the spiking version introduces four modifications. First, PLIF neurons are inserted before the heavy $=3.74$1 convolutions. Second, an amplitude shortcut path is added to restore erased amplitude dynamics. Third, TSM is used for lightweight temporal fusion. Fourth, self-architectural distillation transfers knowledge from the ANN Vocos teacher to the SNN student. The architecture is therefore hybrid in the sense that it preserves the original vocoder decomposition into spectrogram prediction and iSTFT reconstruction while reworking the internal feature processing around spiking dynamics.

The internal structure of one Spiking ConvNeXt block is fixed: depthwise convolution with kernel size $=3.74$2 and padding $=3.74$3, PLIF neuron, pointwise convolution expanding channels from $=3.74$4 to $=3.74$5, PLIF neuron, pointwise convolution projecting from $=3.74$6 back to $=3.74$7, and amplitude shortcut with residual addition. This structure directly targets the expensive channel-mixing operations in ConvNeXt-style blocks, which in an ANN are dominated by dense $=3.74$8 MACs.

3. Spiking computation, temporal fusion, and amplitude recovery

The spiking nonlinearity is the PLIF neuron. At discrete timestep $=3.74$9, its dynamics are defined by

$=3.45$0

$=3.45$1

$=3.45$2

Here $=3.45$3 is the Heaviside step, $=3.45$4 is the threshold, $=3.45$5 is the reset potential, and $=3.45$6 is a learnable membrane time constant. In Spiking Vocos, PLIF neurons convert continuous pre-activations $=3.45$7 into binary spikes $=3.45$8, sparsifying the expensive $=3.45$9 convolutions into cheap AC operations (Chen et al., 16 Sep 2025).

The energy accounting separates depthwise and pointwise convolutions:

$14.7\%$0

and

$14.7\%$1

with $14.7\%$2, $14.7\%$3, sequence length $14.7\%$4, SNN timesteps $14.7\%$5, average firing rate $14.7\%$6, and per-operation costs $14.7\%$7 pJ and $14.7\%$8 pJ in 45 nm CMOS. The total SNN energy is approximated by summing $14.7\%$9 over blocks.

The Temporal Shift Module provides explicit past and future context with zero MAC cost. For an intermediate feature tensor $n\_fft=1024$0, channels are split into three groups: $n\_fft=1024$1, $n\_fft=1024$2, and $n\_fft=1024$3, typically with $n\_fft=1024$4 and $n\_fft=1024$5. One group is shifted forward by one timestep, one is left unchanged, and one is shifted backward by one timestep. The shifted tensor is then fused residually:

$n\_fft=1024$6

This gives each block both past and future context without introducing additional MAC-heavy temporal operators.

The amplitude shortcut path addresses a different failure mode: binary spikes erase amplitude information, described in the paper as “saturation.” If $n\_fft=1024$7 and $n\_fft=1024$8 denote the features before and after the spiking layers, amplitude is recovered by

$n\_fft=1024$9

where $n\_hop=256$0 is element-wise multiplication. This explicitly injects raw amplitude into the stream and is intended to mitigate information loss caused by binarization.

Taken together, PLIF sparsification, zero-MAC temporal shifting, and amplitude recovery define the core computational logic of Spiking Vocos. The first reduces the cost of channel mixing, the second restores temporal context that would otherwise be weak in a short-step SNN, and the third compensates for spike-induced loss of analog magnitude structure.

4. Distillation strategy and training regime

Spiking Vocos uses self-architectural distillation to bridge the performance gap with its ANN counterpart. The distillation scheme includes intermediate feature alignment, magnitude-spectrum distillation, and phase distillation (Chen et al., 16 Sep 2025).

For intermediate feature alignment across $n\_hop=256$1 distilled blocks,

$n\_hop=256$2

Magnitude-spectrum distillation is

$n\_hop=256$3

Phase supervision is based on an anti-wrapping function

$n\_hop=256$4

with phase losses

$n\_hop=256$5

$n\_hop=256$6

$n\_hop=256$7

and

$n\_hop=256$8

The total knowledge-distillation loss is

$n\_hop=256$9

The training setup is fixed and relatively specific. The dataset is the LibriTTS train set, approximately $1\times1$0 hours of speech. Features use 100-dimensional mel bins at 24 kHz with $1\times1$1 and $1\times1$2. Both the ANN baseline and the SNN student are trained for $1\times1$3 M generator steps and $1\times1$4 M discriminator steps. Optimization uses AdamW with $1\times1$5, $1\times1$6, and default weight decay. The default SNN setting is $1\times1$7 timesteps for low latency, with an additional experiment at $1\times1$8 timesteps for a quality trade-off. Distillation applies $1\times1$9 at shifted intermediate points, specifically post-TSM, to avoid misalignment. The training recipe uses standard GAN losses and no extra data augmentation.

This training design reflects a recurrent theme in spiking generative modeling: direct ANN-to-SNN substitution is typically insufficient. In Spiking Vocos, the student is not merely trained to reconstruct waveforms; it is also trained to match the ANN teacher’s intermediate representation, magnitude response, and phase behavior.

5. Empirical performance, ablations, and energy profile

On LibriTTS test-clean, the ANN Vocos baseline reports UTMOS $r$0, PESQ $r$1, and ViSQOL $r$2. A $r$3-step full-SNN Spiking Vocos reports UTMOS $r$4 and PESQ $r$5. Adding only TSM yields UTMOS $r$6 and PESQ $r$7. Adding only distillation yields UTMOS $r$8 and PESQ $r$9. Combining TSM and distillation yields UTMOS $\rightarrow$0 and PESQ $\rightarrow$1 (Chen et al., 16 Sep 2025).

Variant	UTMOS	PESQ
ANN Vocos (baseline)	3.82	3.65
Spiking Vocos (4-step full SNN)	3.46	3.31
+ TSM only	3.71	3.36
+ Distillation only	3.70	3.43
+ TSM + Distillation	3.74	3.45

The energy results are reported per 1000-sample sequence. ANN Vocos consumes $\rightarrow$2 pJ. A $\rightarrow$3-step Spiking Vocos with firing rate $\rightarrow$4 consumes $\rightarrow$5 pJ, which is $\rightarrow$6 of the ANN cost. The TSM-plus-distillation model, with $\rightarrow$7, consumes $\rightarrow$8 pJ, or $\rightarrow$9 of the ANN cost. The overall reduction is approximately $\phi$00.

The ablations isolate two mechanisms. TSM adds approximately $\phi$01 UTMOS recovery, indicating that explicit temporal fusion is critical in SNNs. Self-distillation adds approximately $\phi$02 UTMOS and larger PESQ gains, indicating that layer-wise and spectral guidance help recover precision. The combined system yields near-baseline perceptual scores with minimal latency at $\phi$03 steps.

The paper’s discussion section presents the main strengths and limitations in operational terms. Reported strengths are ultra-low energy at $\phi$04 of ANN cost, competitive fidelity with UTMOS within $\phi$05 of ANN and human MOS approximately $\phi$06 versus $\phi$07 for Vocos, and low latency with $\phi$08 SNN timesteps, approximately $\phi$09 ms, enabling real-time operation on neuromorphic hardware. Reported limitations are that PESQ remains approximately $\phi$10 below ANN, implying minor spectral detail loss due to binary quantization, periodicity error and V/UV F1 degrade slightly, and TSM and distillation increase firing rates, creating a trade-off against pure sparsity.

A common misconception is that energy-efficient spiking synthesis necessarily requires a large quality collapse. The reported results do not support that generalization for this architecture: the quality gap narrows substantially once temporal shifting and self-distillation are included. A second misconception is that spiking efficiency is obtained for free. The firing-rate increase under TSM and distillation shows that quality recovery and sparsity can pull in opposite directions.

6. Position within spiking speech research and prospective developments

Spiking Vocos belongs to a broader but heterogeneous set of spiking speech-generation and speech-reconstruction systems. Its closest point of comparison in the supplied literature is SpikeVoice, which addresses text-to-speech via SNNs and introduces Spiking Temporal-Sequential Attention to overcome “partial-time dependency.” In SpikeVoice, the spiking model outputs Mel spectrograms and a standard neural vocoder such as HiFi-GAN converts them into waveforms; its reported total SNN energy is $\phi$11 of ANN energy (Wang et al., 2024). Relative to that design, Spiking Vocos occupies a different layer of the synthesis stack: it is itself the vocoder, and it uses a Vocos-style magnitude/phase decoder with iSTFT rather than an external ANN vocoder.

A second adjacent system is NeuroVoc, which reconstructs acoustic waveforms from simulated neural activity patterns using an inverse Fourier transform. NeuroVoc operates on neurograms derived from auditory-nerve fiber models, performs downsampling, pseudo-dB and power conversion, Mel-to-linear non-negative least squares, Griffin-Lim phase reconstruction, and inverse STFT, and is intended for auditory perception and cochlear implant simulation rather than low-power neural vocoding (Nobel et al., 4 Jun 2025). Relative to NeuroVoc, Spiking Vocos is not a hearing-model inversion framework; it is a learned neural vocoder optimized for energy efficiency under mel-spectrogram conditioning.

These distinctions are important because the phrase “spiking vocoder” can refer to materially different objects: a biologically grounded neurogram inversion framework, a text-to-speech system with a spiking acoustic model and external vocoder, or a fully neural vocoder built from spiking modules. Spiking Vocos denotes the third of these categories.

The future directions stated for Spiking Vocos are fourfold: investigation of mixed-precision spikes or multi-level spiking to capture more amplitude detail; hardware co-design and deployment on event-driven neuromorphic chips such as Loihi to validate real power savings; extension to multilingual or noisy-speech vocoding scenarios; and combination with diffusion priors or flow-based teacher models for further quality improvement (Chen et al., 16 Sep 2025). These directions follow directly from the system’s present trade-offs. Mixed-precision or multi-level spikes target the remaining spectral-detail loss, hardware co-design targets the gap between estimated and realized energy savings, and stronger teacher models target the residual performance gap to ANN baselines.

In that sense, Spiking Vocos marks a specific development in spiking speech synthesis: a demonstration that a frequency-domain neural vocoder can be transformed into a spiking architecture with ultra-low estimated energy consumption while maintaining performance comparable to its ANN counterpart under a short-timestep, low-latency regime.