Adaptive Speech-to-Spike Encoding for Spiking Neural Networks

Abstract: The mismatch between continuous acoustic signals and discrete event-driven processing remains a fundamental bottleneck for neuromorphic speech processing. Current systems typically rely on fixed spike encoders, forcing downstream Spiking Neural Networks (SNNs) to compensate for non-adaptive input representations. To address this, we present a learnable residual speech-to-spike encoder jointly trained end-to-end with a Recurrent Leaky Integrate-and-Fire (R-LIF) backbone. We validate this approach on the Google Speech Commands v2 (GSC-v2) benchmark, achieving up to 94.97% accuracy. Notably, the learned encoder remains highly parameter-efficient with a compact 35k-parameter variant that reaches 89.8%, matching or exceeding prior baselines that require an order of magnitude more parameters. Our encoder-focused analysis, including linear probing and gradient-residual inspection, indicates that the encoder does not target faithful signal reconstruction but instead learns task-aligned spike representations that enhance class separability. Finally, we benchmark bio-inspired, hardware-friendly credit assignment by comparing Direct Feedback Alignment (DFA) with surrogate-gradient BPTT under identical architectures and training conditions. We find that DFA reaches 91.5% accuracy, quantifying the performance trade-off of bio-inspired learning rules for modern neuromorphic audio.

• The paper presents a novel adaptive speech-to-spike encoder that uses learnable coarse and fine quantization to optimize sparse, event-driven representations.
• It demonstrates improved classification accuracy (up to 94.97%) and significant spike reduction, leading to enhanced hardware efficiency on edge devices.
• It also compares DFA and BPTT learning rules, highlighting trade-offs between energy efficiency and state-of-the-art discriminability.

Adaptive Speech-to-Spike Encoding for Spiking Neural Networks

Introduction

The paper "Adaptive Speech-to-Spike Encoding for Spiking Neural Networks" (2606.19039) addresses the core challenge of mapping dense acoustic signals to the sparse, event-driven format required by Spiking Neural Networks (SNNs) in neuromorphic speech processing. Standard approaches rely on fixed encoding schemes with static thresholds, leading to suboptimal representations and inflated model sizes. This work proposes a fully differentiable, learnable residual speech-to-spike (S2S) encoder, jointly trained with a recurrent Leaky Integrate-and-Fire (R-LIF) backbone, and rigorously benchmarks both accuracy and hardware efficiency across learning rules and compute proxies.

Overview of Proposed Architecture

The architecture consists of three core modules: (1) an adaptive S2S encoder that transforms log-mel spectrograms into signed spike trains using learnable coarse and fine step sizes, (2) a multi-layer R-LIF backbone capturing temporal dependencies, and (3) a nonlinear readout mapping spike activity to output logits.

The S2S encoder operates via a hierarchical two-phase quantization where input magnitudes are tracked with coarse steps, and the residual error is further encoded with finer granularity. Both step sizes are optimized via global trainable scalars, ensuring $\delta^{(2)} < \delta^{(1)}$ via softplus and sigmoid parameterization. Critical to neuromorphic training, the encoder utilizes a surrogate gradient (straight-through estimator) to propagate error signals through the non-differentiable spike emission function.

Figure 1: Overview of the learnable step-forward speech-to-spike encoder with a spiking backbone converting log-mel features into signed spike trains.

Encoder Analysis and Task-Aligned Representations

A principal contribution is the demonstration that the learnable S2S encoder does not prioritize signal reconstruction. Encoder-side probing reveals that the resulting spike trains establish superior class separability compared to baselines using fixed thresholds. Freezing the encoder and training a linear probe classifier on its output shows an 8.24-point gain in accuracy (71.63\% vs. 63.72\%) relative to the fixed alternative, indicating robust discriminative feature extraction rather than faithful transformation. Gradient-residual correlation analysis ($r \approx 0.22$) suggests that encoder adaptation is dominated by downstream optimization objectives rather than mere rate control or fidelity, aligning the spike budget with task-relevant information.

Benchmarking Against Prior Art

Performance is evaluated on the Google Speech Commands v2 (GSC-v2) dataset across multiple backbone scales. The adaptive encoder, when paired with the R-LIF backbone, achieves up to 94.97\% test accuracy with a large configuration, surpassing recent SNN-based keyword spotting (KWS) models like SIDC-KWS, ED-sKWS, and Speech2Spikes. The compact Tiny variant, with only 35k parameters, reaches 89.8\% accuracy, matching or exceeding alternatives demanding an order of magnitude more parameters. This result underscores the value of adaptive encoding in achieving high accuracy with minimal network complexity and event activity, reducing spikes per utterance from 2982 (fixed) to 2119 (learnable) and raising input sparsity to 93.4\%.

Sparsity and Hardware Efficiency

A hardware-agnostic energy and compute proxy is used to estimate operational efficiency. The large model achieves 95.5\% global sparsity with only 3.9M active synaptic operations (SynOps) compared to 82.9M dense operations, yielding a projected active energy of 16$\mu$J at 45\,nm CMOS. These metrics provide convincing evidence that learnable encoding, combined with spike-activity regularization, enables aggressive reduction of on-chip compute loads while preserving performance, a decisive advantage for extreme edge neuromorphic deployment.

Learning Rule Trade-Offs

The paper also systematically benchmarks Direct Feedback Alignment (DFA) against surrogate-gradient Backpropagation Through Time (BPTT). DFA, which avoids symmetric weight transport and supports parallel layer updates, achieves 91.5\% accuracy under identical architecture and training conditions, quantifying the expected penalty for bio-inspired, hardware-friendly learning rules. The accuracy gap (94.97\% for BPTT vs. 91.5\% for DFA) motivates further research toward closing this performance gap through refined architectures or layerwise credit assignment schemes.

Implications and Future Directions

Practically, the results indicate that end-to-end learnable speech encoding can yield compact, energy-efficient SNNs with minimal loss in performance, highly relevant for edge devices with stringent power budgets. Theoretically, the observed decoupling of fidelity and class separability at the encoder level suggests that adaptive, task-aligned quantization may be a general principle for neuromorphic sensory processing. The DFA–BPTT trade-off solidifies hardware-aware training rules as central to future SNN research, calling for novel local learning algorithms capable of matching BPTT-level discriminability with sustainable compute footprints. Future directions include layer-level compute and energy proxies with finer granularity, optimization of sparsity constraints in learning rules, and exploration of meta-learning or reward-modulated STDP variants for further reducing the DFA–BPTT gap.

Conclusion

The paper establishes that jointly optimized, adaptive speech-to-spike encoding with learned coarse and fine step sizes yields robust gains in classification accuracy and significant reductions in spike activity, enabling compact SNNs for real-world keyword spotting and neuromorphic speech tasks. Hardware-friendly learning rules such as DFA provide moderate accuracy with substantial efficiency, but closer investigation and improvement are needed to reach BPTT-level performance without inflating spike budgets. The work strongly indicates that principled, end-to-end learnable encoding is critical to the practical deployment of neuromorphic models on demanding edge audio platforms.