Spiking and Event-driven Neuromorphic Mamba Models for Efficient Speech Recognition

Abstract: Deep learning has greatly advanced automatic speech recognition (ASR), enabling widespread deployment on edge devices such as smartphones and smart home systems. However, the computational and energy demands of deep neural networks pose significant challenges for such resource-constrained deployments, introducing latency and limiting real-time interaction. Neuromorphic computing offers a promising solution by introducing activation sparsity through spiking neural networks (SNNs) and event-driven neural networks, converting dense operations into sparse computations. However, a study that evaluates the hardware benefits of different neuromorphic strategies remains lacking for ASR. This paper explores spiking and event-driven neuromorphic neural networks to improve activation sparsity in the state-of-the-art SpeechMamba model for ASR. We introduce an event-driven SpeechMamba with FATReLU activation, achieving over 60% activation sparsity with less than 1% accuracy degradation on LibriSpeech. We also propose a spiking SpeechMamba that attains over 70% sparsity while using 30% fewer parameters than comparable SNNs. Finally, we develop a cycle-accurate event-driven simulator enabling flexible algorithm-hardware co-exploration, which helps us identify computational bottlenecks and yields over 10% additional efficiency improvements.

• The paper introduces neuromorphic Mamba models leveraging FATReLU-based sparsification and LIF spiking neurons to enhance edge ASR efficiency.
• The paper systematically optimizes activation sparsity, achieving over 60% to 70% sparsity with minimal WER degradation and significant reductions in CPU cycles and memory accesses.
• The paper presents a cycle-accurate event-driven simulator that enables joint algorithm-hardware co-exploration, providing critical insights for neuromorphic ASR design.

Spiking and Event-driven Neuromorphic Mamba Models for Efficient Speech Recognition

Introduction and Motivation

This paper addresses the computational constraints associated with deploying state-of-the-art automatic speech recognition (ASR) models on resource-limited edge devices. It explores neuromorphic computing strategies—specifically activation sparsity via spiking and event-driven neural paradigms—as a means to lower energy and latency overheads in deep learning-based ASR. The focal point is the SpeechMamba model, which leverages Selective State Space Models (SSMs) and attention, offering competitive accuracy with a compact architecture.

The manuscript systematically investigates both event-driven (E-SpeechMamba) and spiking (S-SpeechMamba) variants, quantifies their sparsification and hardware execution advantages, and presents a cycle-accurate simulator for joint algorithm-hardware co-exploration.

Event-driven SpeechMamba: Architecture and Sparsification Mechanism

Event-driven SpeechMamba is constructed by interleaving FATReLU activation functions at critical junctures in the encoder-decoder architecture, thereby inducing activation sparsity that translates into computation skipping for negligible accuracy loss. The sparsification process comprises three major stages: (1) initial ReLU-based pre-training with a sparsity regularizer, (2) threshold initialization for FATReLU using quantile-based data sweeps, and (3) threshold finetuning via regularization to optimize the trade-off between sparsity and accuracy.

Figure 1: Encoder and decoder blocks of the E-SpeechMamba architecture with inserted FATReLU points for activation sparsification.

Furthermore, simulation-guided profiling identified submodules amenable to additional FATReLU insertions, yielding the "Optimized" E-SpeechMamba variant. The approach explicitly targets hardware-exploitable sparsity while avoiding significant information loss at sensitive layers (e.g., SSM Scan modules), where sparsification would otherwise impair recognition fidelity.

Figure 2: Internal structure of an E-SpeechMamba Mamba block with optimized FATReLU activation placement.

Spiking SpeechMamba: SNN Integration and Regularization

The S-SpeechMamba model extends the event-driven approach by substituting graded activations with leaky integrate-and-fire (LIF) spiking neurons and adopting spiking Mamba blocks and spiking self-attention. The attention mechanism forgoes softmax in favor of scaled dot-product operations between binary spike encodings, yielding further computational savings at inference.

To prevent trivial solutions during training (all silent or overly active neurons), the loss function incorporates both minimum and maximum firing rate penalties, which are critical for stable convergence and useful sparsity levels in SNNs.

Simulator Infrastructure for Neuromorphic Co-Design

A major contribution is the development of a RISC-V Ibex–based event-driven simulator, enabling fine-grained evaluation of latency, cycles, and memory operations in digital neuromorphic settings. Unlike analytical or chip-specific simulators, this platform supports operation-level flexibility and captures execution costs relevant to real digital hardware. Event propagation is triggered as soon as atomic (nonzero) activations occur, aligning simulation with the event-driven paradigm.

Simulator-guided analysis provided empirical evidence guiding architectural refinements—including new FATReLU insertion points—that resulted in measurable improvements in computation intensity and resource utilization.

Experimental Results and Analysis

On the LibriSpeech dataset, the E-SpeechMamba model achieved over 60% average activation sparsity with less than 1% WER degradation relative to the dense baseline. The S-SpeechMamba exceeded 70% sparsity, outperforming previous spiking models while using 30% fewer parameters.

Hardware-aware simulation revealed nuanced aspects of algorithm-hardware interplay. Notably:

• E-SpeechMamba reduces CPU cycles by 32.3%, memory accesses by 17.6%, and latency by 29.8% relative to a naive sparsified baseline.
• S-SpeechMamba, despite higher sparsity, achieved lower hardware efficiency gains (19.6% cycle reduction) due to state maintenance overhead in LIF neurons (evidenced by a more modest 7.6% reduction in memory accesses).
• Optimized E-SpeechMamba, with simulator-driven FATReLU placements, achieved a 46.1% reduction in CPU cycles and 28.5% reduction in memory accesses.

  Figure 3: Distribution of activation sparsity at various encoder/decoder locations in E-SpeechMamba and S-SpeechMamba, showing heterogeneous hardware-exploitable sparsity points.

Detailed block-level cycle breakdown confirmed that after optimization, dominant computational hotspots (SSM Scan, matrix multiplications) saw substantial load reduction. The findings corroborate the thesis that not all algorithmically sparse models deliver comparable hardware speedup; both microarchitectural placement of sparsity points and state tracking overhead must be quantified on realistic simulators.

Figure 4: Cycle breakdown for submodules within the encoder Mamba block in the base model, highlighting naive sparsification inefficiencies.

Implications and Outlook

The research demonstrates that event-driven and spiking neuromorphic approaches are compatible with large-vocabulary ASR tasks and that, when carefully engineered, they deliver substantial resource benefits at near-baseline recognition performance. The key empirical result is that sparsity—if concentrated at high-dimensional, computation-intensive junctures—translates into measurable compute and memory savings. At the same time, hardware constraints such as memory bandwidth and neuron state retention delimit achievable efficiency, vindicating the need for operation-level simulator-driven evaluation.

Theoretically, this establishes Mamba-based architectures as viable substrates for neuromorphic computation, given their favorable balance of expressivity and stateful compression. Practically, these findings offer a blueprint for co-designing neuromorphic networks and digital hardware for real-time, low-latency speech applications. Future directions include extending the event-driven simulation framework to emerging non-von Neumann substrates, investigating trade-offs with quantization and structured sparsity, and generalizing architectural insights to other sequential real-world tasks.

Conclusion

This work delivers a comprehensive exploration of spiking and event-driven neuromorphic instantiations of the SpeechMamba ASR model, supported by both algorithmic and hardware-aware assessments. By integrating FATReLU-based sparsification and LIF spiking mechanisms, and leveraging a cycle-accurate event-driven simulator, the study elucidates the benefits and systemic trade-offs inherent to neuromorphic ASR. The contributions pave the way toward efficient edge deployment and inform future co-design strategies for neuromorphic architectures in sequence modeling and beyond.