Spiking SpeechMamba: Neuromorphic ASR Insights
- Spiking SpeechMamba is a neuromorphic adaptation of SpeechMamba that incorporates binary spiking (LIF) neurons into state-space models for efficient long-context speech recognition.
- The model leverages a hybrid architecture combining Mamba blocks with self-attention, introducing both event-driven graded activations (FATReLU) and spiking mechanisms to balance sparsity and hardware gains.
- Empirical evaluations on LibriSpeech reveal a trade-off between higher activation sparsity and word error rate, underlining challenges in translating idealized reductions into real-world hardware efficiencies.
Searching arXiv for the cited paper and closely related prior work to ground the article. Spiking SpeechMamba most directly denotes S-SpeechMamba, the spiking variant of SpeechMamba introduced in “Spiking and Event-driven Neuromorphic Mamba Models for Efficient Speech Recognition” (Ahmed et al., 31 May 2026). In that work, SpeechMamba is treated as a strong automatic speech recognition (ASR) backbone because Mamba/state-space models provide long-context sequence modeling with lower complexity than attention-heavy transformers, while the original architecture had already shown excellent LibriSpeech performance with a relatively compact parameter budget. The paper studies two neuromorphic reformulations of this backbone—an event-driven SpeechMamba using thresholded graded activations and a spiking SpeechMamba using binary spikes—and asks whether activation sparsity in a Mamba-based ASR model translates into realistic hardware gains rather than only idealized synaptic-operation reductions (Ahmed et al., 31 May 2026).
1. SpeechMamba as the substrate for neuromorphic ASR
The starting point is the original SpeechMamba from Gao and Chen, described as an encoder-decoder ASR model composed of repeated units that combine Mamba blocks with self-attention. Within each unit, two Mamba blocks are interleaved with self-attention layers. The Mamba blocks capture long-range temporal dependencies through selective state-space dynamics, while attention models lower-level temporal relationships (Ahmed et al., 31 May 2026).
The paper repeatedly refers to a block structure containing linear layers, convolutional layers, SSM/Mamba blocks, residual connections, and normalization, plus the usual ASR output projection head. The inserted sparsity points are placed after each linear layer, convolutional layer, and SSM block, while preserving internal Mamba dynamics where necessary. The output metric is word error rate (WER), and the end of the network is an ASR decoding head mapping sequence features to token predictions, although the paper does not provide further tokenization or output-head specifics (Ahmed et al., 31 May 2026).
The selective state-space formulation underlying SpeechMamba is given in the paper’s supporting discussion as the standard Mamba recurrence: with the discretized form
$\mathbf{h}_t = \overline{\mathbf{A}_t \mathbf{h}_{t-1} + \overline{\mathbf{B}_t x_t, \qquad y_t = \mathbf{C}_t \mathbf{h}_t + \mathbf{D}x_t .$
These equations express the use of a recurrent latent state with input-dependent selective parameterization. The paper specifically notes that activation insertions inside SSM blocks must preserve the learning dynamics of , because those matrices can require both positive and negative values and correspond to structured bases such as Legendre-polynomial-related parameterizations (Ahmed et al., 31 May 2026).
A broader speech-specific context comes from “Mamba in Speech: Towards an Alternative to Self-Attention,” which argues that Mamba is a potential alternative to Transformer self-attention for speech because self-attention has quadratic complexity in sequence length, while speech often involves very long frame sequences (Zhang et al., 2024). That paper also shows that for ASR, Mamba works best when embedded in a richer speech architecture rather than used as an isolated stack. This suggests why SpeechMamba, already designed as a hybrid Mamba-attention ASR model, became a plausible baseline for neuromorphic sparsification.
2. Neuromorphic reformulations: event-driven and spiking SpeechMamba
The first reformulation is E-SpeechMamba, an event-driven SpeechMamba that keeps real-valued activations but forces many of them to zero using FATReLU. The authors first create a sparsity-friendly version of SpeechMamba by inserting standard ReLUs “at key bottleneck points.” According to the paper and Fig. 1, these ReLUs are inserted after each linear layer, convolutional layer, and SSM block in both encoder and decoder. After this pretraining stage, every inserted ReLU is replaced with a FATReLU layer, which behaves like a ReLU with a positive threshold. The resulting network remains fundamentally an ANN with real-valued activations, but now many activations are exactly zero, making it suitable for event-driven execution where zero events are skipped (Ahmed et al., 31 May 2026).
The second reformulation is S-SpeechMamba, the model most closely corresponding to the name Spiking SpeechMamba. It replaces thresholded graded activations with explicit leaky integrate-and-fire (LIF) neurons. The paper states that S-SpeechMamba substitutes the event-driven Mamba modules of E-SpeechMamba with the spiking Mamba architecture from SpikMamba, and uses the Spikformer attention design. In the attention block, input features are linearly projected into query, key, and value; these are then normalized and passed through LIF layers to produce binary spike trains. Attention is computed from spike-based query-key similarity without softmax and applied to spiking values (Ahmed et al., 31 May 2026).
This distinction is central. E-SpeechMamba inserts thresholded real-valued activations after dense operators, whereas S-SpeechMamba makes the core sequence-processing modules themselves spiking. At the same time, the design is hybrid rather than purely binary end to end: the simulator description makes clear that binary activations interact with real-valued weights, so dense weights remain while activity becomes sparse (Ahmed et al., 31 May 2026).
The experiments include four practical variants:
| Variant | Description | Reported role |
|---|---|---|
| SpeechMamba | Dense reference | Original baseline |
| base model with naive sparsification | ReLU “only at the end of each module” | Weak sparse baseline |
| E-SpeechMamba | FATReLU at many bottleneck points | Main event-driven model |
| E-SpeechMamba (Optimized) | Additional FATReLU points inside the Mamba block | Simulator-guided event-driven variant |
| S-SpeechMamba | LIF neurons with spiking Mamba and spike-based attention | Spiking model |
A useful historical precursor is “Mamba-Spike: Enhancing the Mamba Architecture with a Spiking Front-End for Efficient Temporal Data Processing,” which proposed a spiking front-end in front of a Mamba backbone and validated the idea on TIDIGITS, where Mamba-Spike reached 99.2% accuracy and LIF with time constant 30 ms gave the best result (Qin et al., 2024). That work was not a full ASR system, but it established a speech-relevant foundation for combining spiking computation with Mamba-style temporal modeling.
3. Activation sparsification, spike regularization, and state dynamics
For the event-driven model, the paper gives the FATReLU equation explicitly: where is a threshold, initialized manually and then finetuned (Ahmed et al., 31 May 2026).
Threshold initialization is data-driven. The authors first run a trained ReLU-pretrained E-SpeechMamba on representative training data, record a reference loss , and collect activation statistics at every FATReLU insertion point. For each layer, the initial threshold is set to the mean of the lowest 10% of activations. They then sweep upward through higher activation deciles and accept threshold increases so long as the normalized loss ratio remains below a tolerance (Ahmed et al., 31 May 2026).
After initialization, thresholds are finetuned with an added sparsity loss. The PDF extraction is damaged, but the intended form is reported as
$L_{\mathrm{spar} = \sum_i \left( \text{FATReLU}(\mathbf{x}_i) + \left(\frac{1}{T_i}\right)^2 \right),$
where $\mathbf{h}_t = \overline{\mathbf{A}_t \mathbf{h}_{t-1} + \overline{\mathbf{B}_t x_t, \qquad y_t = \mathbf{C}_t \mathbf{h}_t + \mathbf{D}x_t .$0 indexes FATReLU layers. The textual explanation is explicit: one term penalizes activation magnitude, and the other penalizes small thresholds, so training is pushed toward larger thresholds and smaller surviving activations, increasing zero activity and hence event sparsity (Ahmed et al., 31 May 2026).
For the spiking model, the paper states that leaky integrate-and-fire neurons are used and that surrogate-gradient learning is used by reference to standard SNN training literature, but it does not print the full LIF membrane equation, threshold-and-reset equation, or surrogate-gradient derivative. The only explicit equations given for S-SpeechMamba are the firing-rate regularizers: $\mathbf{h}_t = \overline{\mathbf{A}_t \mathbf{h}_{t-1} + \overline{\mathbf{B}_t x_t, \qquad y_t = \mathbf{C}_t \mathbf{h}_t + \mathbf{D}x_t .$1
$\mathbf{h}_t = \overline{\mathbf{A}_t \mathbf{h}_{t-1} + \overline{\mathbf{B}_t x_t, \qquad y_t = \mathbf{C}_t \mathbf{h}_t + \mathbf{D}x_t .$2
where $\mathbf{h}_t = \overline{\mathbf{A}_t \mathbf{h}_{t-1} + \overline{\mathbf{B}_t x_t, \qquad y_t = \mathbf{C}_t \mathbf{h}_t + \mathbf{D}x_t .$3 is the average firing rate in spiking layer $\mathbf{h}_t = \overline{\mathbf{A}_t \mathbf{h}_{t-1} + \overline{\mathbf{B}_t x_t, \qquad y_t = \mathbf{C}_t \mathbf{h}_t + \mathbf{D}x_t .$4, and $\mathbf{h}_t = \overline{\mathbf{A}_t \mathbf{h}_{t-1} + \overline{\mathbf{B}_t x_t, \qquad y_t = \mathbf{C}_t \mathbf{h}_t + \mathbf{D}x_t .$5 are target lower and upper bounds. $\mathbf{h}_t = \overline{\mathbf{A}_t \mathbf{h}_{t-1} + \overline{\mathbf{B}_t x_t, \qquad y_t = \mathbf{C}_t \mathbf{h}_t + \mathbf{D}x_t .$6 prevents layers from becoming silent and untrainable, while $\mathbf{h}_t = \overline{\mathbf{A}_t \mathbf{h}_{t-1} + \overline{\mathbf{B}_t x_t, \qquad y_t = \mathbf{C}_t \mathbf{h}_t + \mathbf{D}x_t .$7 suppresses over-firing, promoting sparsity (Ahmed et al., 31 May 2026).
A relevant antecedent outside ASR is “Spiking Structured State Space Model for Monaural Speech Enhancement,” which combined an S4 backbone with LIF dynamics in a speech pipeline and showed that state-space sequence modeling and spiking dynamics are complementary (Du et al., 2023). That work did not use Mamba, but it made the same broader point that long-range sequence memory can remain continuous and structured while spiking units provide nonlinear temporal integration and sparsity.
4. Training configuration and hardware-aware simulation
The training setup is specified unusually clearly for the neuromorphic SpeechMamba study. All experiments use LibriSpeech, approximately 1,000 hours of 16 kHz read English speech, and train on all training splits, approximately 980 hours. Training is done in SpeechBrain on a single NVIDIA A100, with FP32 throughout. The random seed is fixed to 74443. All models are trained for 100 epochs using Adam with
$\mathbf{h}_t = \overline{\mathbf{A}_t \mathbf{h}_{t-1} + \overline{\mathbf{B}_t x_t, \qquad y_t = \mathbf{C}_t \mathbf{h}_t + \mathbf{D}x_t .$8
an initial learning rate of $\mathbf{h}_t = \overline{\mathbf{A}_t \mathbf{h}_{t-1} + \overline{\mathbf{B}_t x_t, \qquad y_t = \mathbf{C}_t \mathbf{h}_t + \mathbf{D}x_t .$9, and a Noam schedule with 25{,}000 warm-up steps. They use gradient accumulation = 4, gradient clipping at 5.0, and dynamic batching with maximum batch length 1024 frames. For E-SpeechMamba and E-SpeechMamba (Optimized), threshold initialization is done on an Intel Xeon CPU, then the models are finetuned for 20 additional epochs (Ahmed et al., 31 May 2026).
The paper also states what is not specified: there is no mention of knowledge distillation, ANN-to-SNN conversion, CTC vs seq2seq criterion, tokenizer type, label units, feature frontend such as log-mel or fbank, or initialization details beyond what is inherited from SpeechBrain/SpeechMamba (Ahmed et al., 31 May 2026). This omission becomes one of the main reproducibility limits of the work.
A major contribution is the cycle-accurate event-driven simulator. Rather than relying on synaptic-operation counts or idealized sparse-MAC estimates, the authors simulate operation-level, event-driven dataflow execution on a simple RISC-V Ibex RV32IMC core using Verilator. The model is lowered to this processor-like environment and compiled with the lowRISC GCC-based toolchain. The simulator assumes a system with a unified instruction/data memory and tracks detailed counters (Ahmed et al., 31 May 2026).
Its event-driven schedule is fine-grained: each layer emits partial outputs as soon as atomic input events arrive, and these partial results are immediately forwarded to downstream layers. For SNNs, spikes and non-spikes are encoded as integers 1 and 0, letting the simulator skip multiplications by zero and replace multiplications by one with direct weight propagation. The reported hardware metrics are CPU cycles, CPU instructions, memory accesses, and latency per sample, all measured relative to the naively sparsified base model (Ahmed et al., 31 May 2026).
This simulator-centered methodology differs from several earlier spiking speech papers that primarily reported idealized energy proxies. “IML-Spikeformer,” for example, reports theoretical inference energy using 0, 1, timestep count 2, and firing-rate terms rather than wall-power or processor-cycle measurements (Song et al., 10 Jul 2025). The SpeechMamba paper therefore shifts emphasis from abstract spike efficiency to realized execution cost.
5. Recognition results, sparsity, and hardware trade-offs
The LibriSpeech results define the empirical profile of Spiking SpeechMamba. The dense baseline SpeechMamba reports WERs of 2.16 / 5.13 / 2.32 / 5.23 on dev-clean / dev-other / test-clean / test-other with 67.6M parameters. The base model with naive sparsification degrades to 2.30 / 5.51 / 2.47 / 5.86 and reaches only 20% sparsity (Ahmed et al., 31 May 2026).
The main event-driven model, E-SpeechMamba, reaches 62% average activation sparsity with 67.6M parameters and WER 2.90 / 7.40 / 3.20 / 7.80. The spiking model, S-SpeechMamba, reaches 72% sparsity, with 67.8M parameters and WER 4.27 / 9.34 / 4.71 / 9.98. The simulator-guided E-SpeechMamba (Optimized) reaches 64% sparsity and WER 3.10 / 7.80 / 3.60 / 8.30 (Ahmed et al., 31 May 2026).
These results show a clear trade-off. Relative to the dense SpeechMamba, E-SpeechMamba loses about 0.88 absolute WER on test-clean for 62% sparsity; S-SpeechMamba loses 2.39 absolute WER on test-clean for 72% sparsity. The paper also reports strong heterogeneity across insertion points: sparsity ranges from below 40% to above 90% inside encoder and decoder blocks (Ahmed et al., 31 May 2026).
The hardware results are more revealing than sparsity alone. Relative to the naively sparsified base model:
| Model | Cycles | Instructions | Memory accesses | Latency |
|---|---|---|---|---|
| E-SpeechMamba | 32.32% | 14.30% | 17.57% | 29.78% |
| S-SpeechMamba | 19.58% | 14.0% | 7.63% | 17.9% |
| E-SpeechMamba (Optimized) | 46.13% | 26.9% | 28.50% | 37.5% |
The optimized event-driven variant improves fewer headline sparsity points than the spiking version, yet it is the strongest hardware design because the extra FATReLU placements target computational hotspots, especially inside the SSM Scan. The paper states that this delivers additional >10% efficiency improvement over the original event-driven design (Ahmed et al., 31 May 2026).
One especially important conclusion is that higher sparsity does not automatically imply better hardware efficiency. S-SpeechMamba has higher activation sparsity than E-SpeechMamba, but its memory-access improvement is over 10 percentage points lower. The stated reason is membrane-state overhead: LIF neurons are stateful and require extra loads and stores for membrane potentials across time, so memory traffic erodes the benefits of binary sparsity (Ahmed et al., 31 May 2026). This is a direct challenge to the common assumption that binary spikes are necessarily the most hardware-efficient regime.
6. Relation to prior neuromorphic speech models, limits, and implications
Within the speech literature, Spiking SpeechMamba occupies a specific position. It is not merely a spiking front-end attached to a dense backbone, as in Mamba-Spike (Qin et al., 2024), and it is not a Transformer-centered spiking ASR model such as IML-Spikeformer (Song et al., 10 Jul 2025). Its novelty is the combination of a state-space ASR architecture, two distinct neuromorphic sparsification strategies, and a hardware-aware simulator that exposes the mismatch between algorithmic sparsity and realized execution cost (Ahmed et al., 31 May 2026).
The comparison with earlier speech–state-space work is also instructive. “Mamba in Speech” shows that for ASR, plain Mamba/BiMamba stacks perform poorly, while Mamba works best as a replacement for MHSA inside Transformer/Conformer blocks, with FFNs, residuals, and convolution providing the extra nonlinear and local modeling capacity (Zhang et al., 2024). This suggests that SpeechMamba’s hybrid Mamba-attention structure is not incidental: it may be part of why the dense backbone remains strong enough that neuromorphic sparsification is worth studying at all.
The paper’s own limitations are explicit. Several implementation details for full reproducibility are omitted, including exact feature extraction, target/tokenization scheme, precise surrogate-gradient function and time-step setting for the SNN, and exact dense-vs-spike boundary inside every module (Ahmed et al., 31 May 2026). The optimized event-driven model also gives slightly worse WER than the original event-driven model, so the efficiency–accuracy trade-off remains nontrivial.
A broader implication is that event-driven graded activations may be more hardware-friendly than binary spiking when memory bandwidth and state maintenance dominate. This is not a generic claim about all neuromorphic speech systems, but it is the explicit lesson of the SpeechMamba study (Ahmed et al., 31 May 2026). A plausible implication is that future “Spiking SpeechMamba” research may split into two lines: one that pursues genuinely spike-native Mamba modules, and another that treats thresholded graded event sparsity as the more deployable compromise for edge ASR.
In that sense, Spiking SpeechMamba names both a concrete model—S-SpeechMamba—and a broader research direction: adapting Mamba-based speech systems to neuromorphic execution while recognizing that activation sparsity, state-space dynamics, and hardware efficiency are related but not identical objectives.