Event-Driven SpeechMamba

• Event-Driven SpeechMamba is a family of architectures that employs selective state-space models to process acoustic inputs based on temporally significant events.
• It supports applications such as streaming ASR, duplex conversational decoding, keyword spotting, and neuromorphic speech processing by dynamically updating its state.
• Empirical results demonstrate high performance with nearly 98% accuracy in keyword detection, reduced latency, and significant hardware efficiency gains.

Event-Driven SpeechMamba denotes a family of speech architectures built around selective state space models, especially Mamba, in which computation is organized around temporally meaningful events rather than global self-attention over entire sequences. In the current literature, the term spans several closely related usages: causal or chunked speech modeling in which the recurrent state is updated online, token-activity–driven streaming ASR, duplex conversational decoding driven by explicit input-state events, activation-sparse neuromorphic SpeechMamba that processes only non-zero events, and event-camera–conditioned speech generation. Across these variants, the unifying principle is a fixed-size, input-selective latent state that evolves with the incoming signal while preserving linear or near-linear sequence scaling (Lu et al., 16 Feb 2025).

1. Conceptual lineage and scope

Mamba entered speech research as a selective state space alternative to Transformer self-attention, with empirical studies showing comparable or better performance across ASR, text-to-speech, spoken language understanding, and speech summarization, together with efficiency in long-form speech processing (Miyazaki et al., 2024). In parallel, work on speech-specific Mamba transfer reported that bidirectional Mamba consistently outperforms vanilla Mamba and that BiMamba can replace self-attention in Transformer and Conformer architectures, particularly for semantic-aware tasks such as ASR (Zhang et al., 2024).

Within that broader trajectory, “event-driven” has acquired more than one technical meaning. In keyword spotting, the term refers to always-on, stateful temporal scanning over speech frames, where the Mamba recurrence is naturally suited to streaming keyword or event detection (Ding et al., 10 Aug 2025). In streaming ASR, it refers to token emission triggered by unimodal aggregation peaks and valleys rather than fixed-frame decoding (Fang et al., 2024). In duplex conversational systems, it refers to branching and switching behavior triggered by <response>, <incomplete>, and <ignore> state tokens while the model listens and speaks simultaneously (Lu et al., 16 Feb 2025). In neuromorphic ASR, it refers to thresholded activations that become sparse events so hardware can skip zero-valued computation (Ahmed et al., 31 May 2026). A common misconception is that “event-driven” in this area denotes only spiking computation; the literature instead uses it for both streaming decision logic and sparse neuromorphic execution.

2. State-space basis and selective temporal updates

The technical core is the continuous-time linear state space model

$h'(t) = A h(t) + B x(t), \quad y(t) = C h(t),$

which, after discretization with step $\Delta$, yields

$$h(t) = \overline{A} h(t-1) + \overline{B} x(t), \quad y(t) = C h(t).$$

When unrolled over a sequence, this admits a convolutional view, but Mamba departs from classical fixed-parameter SSMs by making the effective parameters input-dependent (Ding et al., 10 Aug 2025).

In the formulation used for speech, selectivity is introduced through

$$\overline{B} = s_B(x), \quad \overline{C} = s_C(x), \quad \Delta = \tau_A(\text{Parameter} + s_A(x)),$$

or, equivalently in another speech formulation,

$$B_t,\, C_t = W_B x_t,\ (W_C x_t)^{\mathsf{T}}, \qquad \Delta_t = \text{softplus}(W_{\Delta} x_t).$$

This makes the recurrence time-varying and input-driven, so the state can selectively retain, attenuate, or overwrite information depending on the current acoustic frame (Miyazaki et al., 2024).

From an event-driven perspective, the key recurrence is the online update

$h_t = \overline{A}_t h_{t-1} + \overline{B}_t x_t.$

This supports a stateful interpretation in which the model does not need to revisit the entire past; it needs only the current input and the previous state. This suggests a natural mapping from “event” to “new frame,” “new chunk,” “token boundary,” or “non-zero activation,” depending on the application. The same state-space logic also explains why explicit positional encodings are often unnecessary in Mamba-based speech systems, since ordering is carried by the recurrent state evolution rather than by a separate positional embedding (Miyazaki et al., 2024).

3. Streaming, duplex, and token-triggered speech processing

The most direct streaming reinterpretation appears in keyword spotting. “Keyword Mamba” processes 40-dimensional MFCCs extracted from 16 kHz audio with a 30 ms window and 10 ms stride, pads 1-second clips to 98 frames, and applies Mamba only along the time axis. Its Bidirectional Mamba block uses forward and backward SSM-plus-Conv1D chains with gating, and the paper explicitly notes that this architecture is essentially a blueprint for an event-driven, streaming speech model for always-on keyword or event detection (Ding et al., 10 Aug 2025). Empirically, KWM-192 reaches 98.01% on Speech Commands V1-12 and 98.79% on V2-12, while KWM-T-192 reaches 98.05% and 98.91%, respectively, with fewer parameters than Transformer KWT-3 in comparable settings (Ding et al., 10 Aug 2025).

For strict streaming ASR, the event logic becomes explicit. “Mamba for Streaming ASR Combined with Unimodal Aggregation” uses a causal Mamba encoder, a small convolutional lookahead layer, and a streaming-style Unimodal Aggregation module that computes scalar weights

$\alpha_t = \sigma(W \mathbf{e}_t + b).$

Local minima of $\alpha_t$ act as UMA valleys, local maxima as UMA peaks, and each token embedding is formed by weighted aggregation between consecutive valleys: $$\mathbf{c}_i = \frac{\sum_{t=\tau_i}^{\tau_{i+1}} \alpha_t \, \mathbf{e}_t}{\sum_{t=\tau_i}^{\tau_{i+1}} \alpha_t}.$$ Decoder computation is therefore triggered by learned boundary events rather than by every frame (Fang et al., 2024). On AISHELL-1, Mamba+UMA with 256 ms lookahead and Early Termination achieves 5.55% CER with 494 ms average latency; without lookahead and with Early Termination it achieves 6.82% CER with 196 ms average latency (Fang et al., 2024).

Duplex conversational modeling generalizes the event concept from token boundaries to input-state decisions. DuplexMamba slices speech into 2–3 s chunks, uses a Mamba-based speech encoder and a Mamba-based LLM, and introduces three special state tokens—<response>, <incomplete>, and <ignore>—that determine whether an auxiliary decoding branch should take over, wait, or be discarded (Lu et al., 16 Feb 2025). Because each Mamba layer maintains a fixed-size hidden state, branch duplication is cheap, enabling simultaneous input processing and output generation. On the interruption/non-awakening benchmark, DuplexMamba reports precision 99.46, recall 99.23, and F1 99.35, while GPU memory remains about 6.8 GB from $2^6$ to $\Delta$0 tokens (Lu et al., 16 Feb 2025).

Setting	Event signal	Triggered action
Keyword spotting	Incoming speech frames	Stateful keyword/event detection
Streaming ASR	UMA valleys and peaks	Token aggregation and emission
Duplex conversation	<response>, <incomplete>, <ignore>	Branch keep, switch, or discard

4. Neuromorphic Event-Driven SpeechMamba

The most literal use of the term appears in “Spiking and Event-driven Neuromorphic Mamba Models for Efficient Speech Recognition,” where Event-driven SpeechMamba is a neuromorphic variant of the SpeechMamba ASR model that converts dense operations into sparse computations by inserting thresholded activations and exploiting activation sparsity in hardware-aware execution (Ahmed et al., 31 May 2026). Here, “event-driven” means that many activations are forced to zero and downstream operations involving those zeros are skipped.

The key nonlinearity is FATReLU: $\Delta$1 with learnable per-layer threshold $\Delta$2. Thresholds are initialized by activation-statistics sweeping and then fine-tuned jointly with weights using

$\Delta$3

The first term suppresses large non-zero activations; the second penalizes small thresholds, encouraging more zeros (Ahmed et al., 31 May 2026).

On LibriSpeech, the dense SpeechMamba baseline reports test-clean WER 2.32 and test-other 5.23. E-SpeechMamba reports 3.20 and 7.80 with 62% sparsity, and the optimized variant reports 3.60 and 8.30 with 64% sparsity. The abstract states that the event-driven model achieves over 60% activation sparsity with less than 1% accuracy degradation on LibriSpeech (Ahmed et al., 31 May 2026). Hardware co-exploration via a cycle-accurate simulator shows that E-SpeechMamba yields 32.32% CPU cycle reduction, 14.30% CPU instruction reduction, 17.57% memory-access reduction, and 29.78% latency reduction relative to the base model with naive sparsification; the optimized variant improves these to 46.13%, 26.9%, 28.50%, and 37.5%, respectively (Ahmed et al., 31 May 2026).

The same paper contrasts this with Spiking SpeechMamba, which reaches about 72% activation sparsity and uses 67.8M parameters, about 30% fewer than comparable SNN baselines at 99.4M, but degrades to 4.71% test-clean WER and shows smaller cycle savings than the event-driven ANN because LIF neurons require additional state management (Ahmed et al., 31 May 2026). This distinction is important: event-driven SpeechMamba in the neuromorphic sense is not necessarily a spiking network.

5. Broader speech and multimodal variants

Several adjacent speech architectures extend the same design logic—long-context state propagation, selective temporal updates, and efficient sequence scaling—without using the term “event-driven” as their primary label. Speech-Mamba couples Mamba and attention in Mamba–attention–Mamba encoder and decoder blocks and shows dramatic long-context gains: on LibriSpeech long-context test-clean-L, Transformer reports 48.66% WER while Speech-Mamba reports 7.71%; on dev-clean-100, Transformer reaches 67.87 while Speech-Mamba reaches 10.74 (Gao et al., 2024). This suggests that persistent state-space memory is particularly effective when the “events” of interest are distributed across very long acoustic contexts.

In speech enhancement, MambaDC combines a Mamba layer for global modeling with a depth-wise 1-D convolution for local refinement: $\Delta$4 The paper reports that MambaDC outperforms Transformer, Conformer, and the standard Mamba across all training targets, and in the MP-SENet framework BiMambaDC-9 reaches PESQ 3.48, STOI 0.96, CSIG 4.69, CBAK 3.87, and COVL 4.17 on VoiceBank+DEMAND (Chen et al., 2024). Although this is not event-driven in the neuromorphic sense, the selective SSM plus gating structure is directly compatible with conditional temporal computation.

Speaker diarization provides another related case. A Mamba-based segmentation module replacing the pyannote BiLSTM backend uses WavLM Base features, a 768→256 projection, and seven External BiMamba blocks with $\Delta$5. In full-pipeline DER, the best LSTM configuration reaches 17.7% at $\Delta$6 s, whereas the best Mamba configuration reaches 16.1% at $\Delta$7 s, indicating that Mamba enables longer windows whose richer speaker embeddings outweigh the harder segmentation problem (Plaquet et al., 2024). A plausible implication is that event-driven SpeechMamba designs for meeting analysis could use Mamba states to track change points or activity transitions over much longer horizons than conventional recurrent backends.

The multimodal extension appears in EventSpeech, which uses neuromorphic events from event cameras for expressive speech generation. Its Audio Encoder explicitly uses Mamba as the SSM backbone within a Hierarchical Wavelet Contextualizer, and the full system combines event features with acoustic features through bidirectional cross-attention (Fang et al., 26 May 2026). On EVT-SPK-Synth, the events-based model reaches MCD 3.67, LSE-C 0.843, F0-RMSE 0.156, KL 0.126, and WER 0.038; on EVT-SPK-Real it reaches MCD 3.18, LSE-C 0.843, F0-RMSE 0.124, KL 0.108, and WER 0.028 (Fang et al., 26 May 2026). The same study shows that events outperform even RGB 120 FPS on MCD, LSE-C, F0-RMSE, and KL, extending the idea of “event-driven speech” from internal computation to event-sensor input.

6. Limitations, misconceptions, and research directions

A central limitation across the literature is the trade-off between bidirectionality and strict causality. Keyword Mamba reports that BiMamba-Bi-Bi reaches 98.01% while purely unidirectional Mamba-Fo-Fo collapses to 78.29% in offline KWS, showing that full bidirectionality is critical in that setting (Ding et al., 10 Aug 2025). DuplexMamba addresses this by chunked streaming alignment rather than by fully causalizing its bidirectional ConMamba encoder (Lu et al., 16 Feb 2025). Streaming ASR work instead retains causality and adds bounded lookahead, making latency an explicit control variable (Fang et al., 2024). The broader lesson is that Event-Driven SpeechMamba is not a single fixed architecture; it is a design space defined by how much future context is permitted and what constitutes an event.

Another recurring limitation is evaluation scope. Keyword Mamba is evaluated only on Google Speech Commands; DuplexMamba uses a 2.8B Mamba v1 LLM and notes that scale constrains some QA behavior; Event-driven SpeechMamba for ASR is tested on LibriSpeech; EventSpeech’s real event-camera subset contains 4 hours and 15 actors (Ding et al., 10 Aug 2025, Lu et al., 16 Feb 2025, Ahmed et al., 31 May 2026, Fang et al., 26 May 2026). This suggests that robustness under far-field noise, multilingual speech, spontaneous conversation, and open-world continuous deployment remains only partially characterized.

The literature also points toward several concrete research directions. DuplexMamba proposes streaming training on continuous audio streams and migration to Mamba-2 (Lu et al., 16 Feb 2025). The neuromorphic SpeechMamba work identifies simulator-guided hotspot sparsification as a productive algorithm–hardware co-design loop (Ahmed et al., 31 May 2026). EventSpeech explicitly proposes moving beyond voxelization toward continuous modeling with Neural Ordinary Differential Equations (Fang et al., 26 May 2026). Taken together, these results suggest that future Event-Driven SpeechMamba systems are likely to combine causal selective SSMs, explicit event labels or triggers, sparse conditional computation, and continuous-time multimodal inputs in a single stateful framework.