Learnable Residual Speech-to-Spike Encoder

• The paper demonstrates that replacing fixed thresholding with a two-stage residual encoding improves spike sparsity and discrimination.
• It leverages coarse and fine passes to adaptively quantize log-mel features into signed spikes, optimizing spiking neural network performance.
• Empirical benchmarks on Google Speech Commands reveal that task-aligned spike representations yield higher accuracy with parameter-efficient architectures.

A learnable residual speech-to-spike encoder is a speech front-end for spiking neural networks in which the conversion from continuous acoustic features to discrete spikes is itself optimized jointly with the downstream model, rather than being fixed a priori. In the most explicit formulation currently available, the encoder replaces static thresholding with a two-stage residual mechanism that emits coarse and fine signed spikes from log-mel speech frames and is trained end-to-end with a recurrent leaky integrate-and-fire backbone for keyword spotting. The central premise is that a learnable encoder can produce task-aligned spike representations that are sparser and more discriminative than those obtained from fixed speech-to-spike transforms (Anon et al., 17 Jun 2026).

1. Definition and problem setting

The learnable residual speech-to-spike encoder addresses a specific bottleneck in neuromorphic speech processing: the mismatch between continuous acoustic signals and discrete event-driven processing. In conventional pipelines, fixed spike encoders such as step-forward mechanisms with manually chosen thresholds apply the same quantization behavior to every utterance. The formulation in current work identifies three consequences of this design choice: it generates excessive spikes in “easy” regions, misses subtle but task-relevant fluctuations in “hard” regions, and forces the downstream spiking neural network to learn input adaptation, thereby inflating model size (Anon et al., 17 Jun 2026).

The defining architectural response is a two-stage residual encoder with learnable coarse and fine step sizes. The coarse stage tracks large-scale changes, while the fine stage operates on the residual left after the coarse pass. The stated purpose is to adjust sensitivity to different spectral bands and time frames, use the coarse pass to capture dominant variation, use the fine pass to correct residual errors, and minimize redundant spikes while preserving discriminative features. This formulation positions residual encoding not as a generic architectural motif, but as a mechanism for adaptive quantization of speech features into event streams.

A common misconception is that a speech-to-spike encoder is principally a reconstruction device. The encoder analysis reported for the learnable residual model argues otherwise: the encoder does not target faithful signal reconstruction but instead learns task-aligned spike representations that enhance class separability (Anon et al., 17 Jun 2026).

2. Two-stage residual encoding mechanism

The encoder operates on an 80-bin log-mel frame $x_t \in \mathbb{R}^{80}$ at time $t$ and produces signed spikes

$s_t \in \{0,1\}^{4 \times 80}.$

Its first phase is a coarse step-forward process. The tracking error is

$e^{(1)}_t = x_t - \hat x_{t-1}.$

Positive and negative coarse spikes are generated by thresholding this error: $$s^{(1,+)}_t = H\bigl(e^{(1)}_t - \delta^{(1)}\bigr),\quad s^{(1,-)}_t = H\bigl(-\,e^{(1)}_t - \delta^{(1)}\bigr).$$ The coarse trace is then updated as

$$\hat x_t = \hat x_{t-1} + \delta^{(1)}\bigl(s^{(1,+)}_t - s^{(1,-)}_t\bigr).$$

The second phase explicitly encodes the residual after the coarse pass. The residual and its tracking error are

$$r_t = x_t - \hat x_t,\quad e^{(2)}_t = r_t - \hat r_{t-1}.$$

Fine residual spikes are emitted as

$$s^{(2,+)}_t = H\bigl(e^{(2)}_t - \delta^{(2)}\bigr),\quad s^{(2,-)}_t = H\bigl(-\,e^{(2)}_t - \delta^{(2)}\bigr),$$

and the fine trace is updated via

$$\hat r_t = \hat r_{t-1} + \delta^{(2)}\bigl(s^{(2,+)}_t - s^{(2,-)}_t\bigr).$$

The final encoder output is the concatenation

$$s_t = \bigl[s^{(1,+)}_t,\;s^{(1,-)}_t,\;s^{(2,+)}_t,\;s^{(2,-)}_t\bigr]\in\{0,1\}^{320}.$$

The learnable parameters are the step sizes, constrained by

$t$0

where $t$1 and $t$2 are global scalars, $t$3, and $t$4 is the logistic sigmoid. The Heaviside function $t$5 is paired with a straight-through surrogate derivative during training (Anon et al., 17 Jun 2026).

This construction makes “residual” precise at the encoder level: the second stage is not an auxiliary branch but an explicit residual quantizer that corrects the coarse trace. A plausible implication is that the encoder can allocate spike budget across scales, reserving fine spikes for residual structure that remains useful after coarse tracking.

3. Integration with recurrent spiking networks and learning rules

The learnable encoder is coupled to a recurrent leaky integrate-and-fire network. For each layer $t$6, the discrete-time R-LIF dynamics are

$t$7

$t$8

Here $t$9 is the membrane potential, $s_t \in \{0,1\}^{4 \times 80}.$0 is the post-reset state, $s_t \in \{0,1\}^{4 \times 80}.$1 and $s_t \in \{0,1\}^{4 \times 80}.$2 are feed-forward and recurrent weights, $s_t \in \{0,1\}^{4 \times 80}.$3 is a learnable leak, and $s_t \in \{0,1\}^{4 \times 80}.$4 is a learnable threshold. The encoder’s 320-channel spike output is fed as $s_t \in \{0,1\}^{4 \times 80}.$5 into the first layer. After $s_t \in \{0,1\}^{4 \times 80}.$6 layers, the model averages the final-layer spikes,

$s_t \in \{0,1\}^{4 \times 80}.$7

and applies a small MLP head to obtain class logits (Anon et al., 17 Jun 2026).

Training is end-to-end under the objective

$s_t \in \{0,1\}^{4 \times 80}.$8

where the first term is cross-entropy and the second penalizes average spike rate. This makes the encoder part of the same optimization problem as the recurrent spiking backbone.

Two credit-assignment regimes are reported. Under backpropagation through time, the network is unrolled in time and the step nonlinearity is differentiated with a fast-sigmoid surrogate. Under Direct Feedback Alignment, fixed random matrices $s_t \in \{0,1\}^{4 \times 80}.$9 map the global error $e^{(1)}_t = x_t - \hat x_{t-1}.$0 to local errors

$e^{(1)}_t = x_t - \hat x_{t-1}.$1

The stated significance of the DFA comparison is that it benchmarks a bio-inspired, hardware-friendly alternative to surrogate-gradient BPTT under identical architectures and training conditions (Anon et al., 17 Jun 2026).

4. Empirical performance and scaling

The reported benchmark is Google Speech Commands v2 under the 35-class protocol. Audio is sampled at 16 kHz and converted to an 80-bin log-mel spectrogram using a 25 ms window, a 10 ms hop, and $e^{(1)}_t = x_t - \hat x_{t-1}.$2. Each time step therefore supplies an 80-dimensional frame to the encoder, which expands it to 320 spike channels (Anon et al., 17 Jun 2026).

The main quantitative results are summarized below.

System	Parameters	Accuracy
Fixed encoder + R-LIF (baseline)	—	90.70 %
Learnable encoder + R-LIF (BPTT, Large)	≈1.8 M	94.97 %
Learnable encoder + R-LIF (DFA, Large)	≈1.8 M	91.50 %
Learnable encoder + R-LIF (Small)	≈0.7 M	92.64 %
Learnable encoder + R-LIF (Tiny, BPTT)	35 k	89.80 %

The encoder is described as highly parameter-efficient in the compact regime: the 35k-parameter variant reaches 89.8%, matching or exceeding prior baselines that require an order of magnitude more parameters. In the broader spiking keyword spotting comparison, the reported reference points are DCLS-Delays at 95.3% with 2.5 M parameters, SIDC-KWS at 94.7% with 403 k parameters, ED-sKWS at 93.1% with 307 k parameters, SRNN+ALIF at 92.5% with 222 k parameters, and Speech2Spikes at 89.5% with 410 k parameters (Anon et al., 17 Jun 2026).

These results frame the learnable residual encoder as a front-end-centered alternative to scaling the recurrent backbone alone. A plausible implication is that part of the performance gain normally attributed to deeper or larger spiking networks can instead be captured by adaptive input eventization.

5. Representation geometry and interpretability

The encoder analysis is notable because it evaluates not only accuracy but also what kind of spike representation the encoder learns. In the linear probing experiment, the encoder is frozen and a linear classifier is trained on its spikes. The learned encoder attains 71.63% probe accuracy, compared with 63.72% for a fixed encoder. The reported interpretation is direct: learned spikes form more linearly separable features (Anon et al., 17 Jun 2026).

A second analysis studies the relation between gradient magnitude, spike rate, and reconstruction error. The reported correlations are

$e^{(1)}_t = x_t - \hat x_{t-1}.$3

and

$e^{(1)}_t = x_t - \hat x_{t-1}.$4

The stated conclusion is that encoder updates are driven by downstream discriminative loss, not just rate control or faithful spectrogram reconstruction (Anon et al., 17 Jun 2026).

This result matters because it clarifies the role of residual encoding in the speech-to-spike setting. The residual stage is not simply repairing coarse quantization in the signal-processing sense; it is being shaped by task loss. That observation constrains how the method should be interpreted: the encoder is best regarded as a supervised representation learner operating under spiking constraints, rather than as a generic neuromorphic codec.

6. Relation to earlier speech-to-spike models and learnable auditory front-ends

Earlier work on speech-to-spike conversion in spiking networks used a markedly different training paradigm. A 2016 model learned to convert speech into spike train signatures using a small, nonrecurrent SNN with Izhikevich Regular-Spiking neurons, spike timing dependent plasticity, and biologically realistic synapses. Raw speech was divided into $e^{(1)}_t = x_t - \hat x_{t-1}.$5 overlapping frames with 50% overlap and a Hamming window; each frame was converted to a log-power spectrum and reduced to $e^{(1)}_t = x_t - \hat x_{t-1}.$6 band energies using Fibonacci-scaled bands over 0–4 kHz, yielding a $e^{(1)}_t = x_t - \hat x_{t-1}.$7-dimensional feature vector per utterance. The network contained 200 $e^{(1)}_t = x_t - \hat x_{t-1}.$8-layer neurons and 10 $e^{(1)}_t = x_t - \hat x_{t-1}.$9-layer neurons, for 2000 adaptive synapses in total. Reported performance was 90.9% on clean speech and 70.9% at 10 dB SNR on a spoken digit task. The published system was purely feedforward, but its technical summary explicitly proposed residual or skip pathways as possible extensions, including direct identity current paths and skip sums in deeper multilayer SNNs (Tavanaei et al., 2016).

A more recent line of work moved residual ideas into the auditory front-end itself. Spiking-LEAF combines a learnable Gabor filter bank with PCEN compression and a two-compartment spiking neuron called IHC-LIF, with dendritic and somatic states, lateral feedback, and spike regularization. On keyword spotting with Google Speech Commands V2, the reported best accuracy is 93.95% with a recurrent SNN, compared with 93.58% using fbank features. On speaker identification with VoxCeleb1, the reported best accuracy is 32.45%, compared with 31.76% using fbank. Noise robustness was evaluated on NOISEX-92 and CHiME-3, and encoding efficiency improved when lateral inhibition and spike regularization were used, with spike rates reduced from $$s^{(1,+)}_t = H\bigl(e^{(1)}_t - \delta^{(1)}\bigr),\quad s^{(1,-)}_t = H\bigl(-\,e^{(1)}_t - \delta^{(1)}\bigr).$$0 to $$s^{(1,+)}_t = H\bigl(e^{(1)}_t - \delta^{(1)}\bigr),\quad s^{(1,-)}_t = H\bigl(-\,e^{(1)}_t - \delta^{(1)}\bigr).$$1. Its “Blueprint for a Learnable Residual Speech-to-Spike Encoder” proposed residual filter-bank pathways from raw $$s^{(1,+)}_t = H\bigl(e^{(1)}_t - \delta^{(1)}\bigr),\quad s^{(1,-)}_t = H\bigl(-\,e^{(1)}_t - \delta^{(1)}\bigr).$$2 directly to neuron somatic inputs, residual neuron layers that pass a fraction of dendritic state forward, and cross-compartment residuals from soma back to dendrite. Those residual scalars are learned via backpropagation through time using the surrogate gradient of $$s^{(1,+)}_t = H\bigl(e^{(1)}_t - \delta^{(1)}\bigr),\quad s^{(1,-)}_t = H\bigl(-\,e^{(1)}_t - \delta^{(1)}\bigr).$$3, and the residual paths are described as bypassing thresholds and resets while preserving event-driven spiking behavior (Song et al., 2023).

Taken together, these works outline three distinct regimes. The 2016 system learned spike signatures through local STDP with supervised gating; Spiking-LEAF learned a front-end composed of filter-bank, compression, and compartmental spiking dynamics; and the 2026 learnable residual speech-to-spike encoder made residual quantization itself the principal adaptive mechanism. This suggests that “residual” in neuromorphic speech can refer either to residual errors between successive traces, as in two-stage step-forward encoding, or to residual pathways that bypass parts of the spiking dynamics, as in auditory front-end blueprints.