SpikingMamba: Spike-Driven Mamba LLM

• SpikingMamba is a hybrid architecture that integrates spiking neural computation with Mamba’s state-space model to achieve energy-efficient inference.
• It replaces dense projections with TI-LIF neurons and applies Smoothed Gradient Compensation during training to retain semantic polarity and improve performance.
• Empirical results show significant energy savings with modest accuracy drops, positioning SpikingMamba as a bridge between dense LLMs and spike-native systems.

SpikingMamba denotes a class of architectures that combine spiking neural computation with Mamba-style selective state-space modeling, but the term has two distinct uses in the recent literature. In the strict sense, it refers to the spiking LLM introduced in "SpikingMamba: Towards Energy-Efficient LLMs via Knowledge Distillation from Mamba," which distills a pretrained Mamba2 teacher into an SNN-based recurrent model that replaces expensive dense projections with spike-driven computation while preserving $O(L)$ sequence processing and constant-memory autoregressive structure (Huang et al., 6 Oct 2025). In a broader sense, nearby work uses closely related names for hybrid SNN–Mamba systems in neuromorphic temporal processing, temporal video grounding, point cloud analysis, event-based human action recognition, and spike-native vision backbones, with substantial variation in where spiking is applied and whether the Mamba core itself is modified (Qin et al., 2024, Li et al., 2024, Wu et al., 19 Apr 2025, Chen et al., 2024, Bai et al., 28 Apr 2026).

1. Terminology and scope

The literature does not treat “SpikingMamba” as a single canonical architecture. Some systems are best described as hybrid pipelines in which spikes appear only in the encoder or proposal generator, while a conventional continuous-valued Mamba backbone performs the downstream sequence modeling. Other systems integrate spike events more deeply into the state-space computation itself.

This distinction is explicit in "Mamba-Spike," which does not present a fully spiking Mamba model. Its architecture is a spiking front-end, a spike-to-activation interface, and a standard Mamba backbone. The paper states that it is closer to “Mamba with a spiking input/feature front-end” than to a fully spiking Mamba cell (Qin et al., 2024). By contrast, the 2025 LLM paper titled "SpikingMamba" preserves the recurrent SSM core of Mamba2 but replaces the dominant dense projections with spike-driven equivalents and trains the student by single-stage distillation plus optional reinforcement learning (Huang et al., 6 Oct 2025).

A broader survey of related models shows that the “spiking + Mamba” design space is modality-specific rather than uniform.

System	Domain	Spiking locus
Mamba-Spike	Temporal and neuromorphic data	Spiking front-end; standard Mamba backbone
SpikeMba	Temporal video grounding	Spiking saliency detector; SSM/Mamba contextual reasoning
Spiking Point Mamba	3D point cloud analysis	Spiking embedding and Spiking Mamba Block
SpikMamba	Event-based HAR	Spike-form patch embedding, SpikeSLA, spike-form Mamba
Vision SmolMamba	Vision backbone	Spike-guided token pruning and spiking state-space recurrence

This suggests that “SpikingMamba” functions both as the name of a specific LLM architecture and as a broader umbrella for spike-aware Mamba variants whose common theme is the attempt to combine sparse event-driven computation with linear-time state-space modeling.

2. SpikingMamba as a spiking LLM

The specific model named SpikingMamba was proposed to make Mamba-style LLM inference substantially more energy efficient while preserving as much of the original model’s zero-shot capability as possible. Its starting point is that even Mamba, despite avoiding quadratic self-attention and KV caching, still relies heavily on dense matrix multiplications, especially in the input and output projections. The paper states that in Mamba2, more than 90% of parameters in the 1.3B model sit in the input/output linear layers, and these are the operations targeted for spiking replacement (Huang et al., 6 Oct 2025).

The architecture combines four elements. First, it keeps Mamba’s recurrent SSM backbone for linear-time inference. Second, it introduces TI-LIF, a ternary-integer spiking neuron designed to preserve semantic polarity through signed multi-level spike representations. Third, it adds a training-exclusive Smoothed Gradient Compensation (SGC) path to mitigate quantization loss without changing inference-time computation. Fourth, it uses single-stage distillation from pretrained Mamba2, followed by optional reinforcement learning with DPO or KTO (Huang et al., 6 Oct 2025).

TI-LIF addresses a limitation of ordinary LIF and prior integer spiking neurons. Standard LIF emits binary spikes, while I-LIF typically emits non-negative integer-valued activations during training. The paper argues that non-negative-only integer neurons are problematic for language modeling because negative activation values often encode meaningful semantic polarity. TI-LIF therefore quantizes to a symmetric signed range:

$$\boldsymbol{s}_t = \mathrm{Clip}\bigl(\mathrm{Round}(\boldsymbol{x}_t), -D, D\bigr).$$

It uses a surrogate gradient

$$\frac{\partial \boldsymbol{s}_t}{\partial \boldsymbol{x}_t} = \begin{cases} \alpha, & -D \le \boldsymbol{x}_t \le D, \ 0, & \text{otherwise}, \end{cases}$$

so that magnitude and polarity are both retained during training (Huang et al., 6 Oct 2025).

At inference time, TI-LIF remains spike-driven. The input token embedding is injected once as $|\boldsymbol{x}_t|$ at the first micro-step, followed by zero input at later micro-steps. With $\beta=1$, the final output is recovered by summing spike-driven accumulations and restoring the sign with a single sign bit:

$$\boldsymbol{y}_t = \mathrm{sgn}(\boldsymbol{x}_t)\cdot \sum_{i=1}^{D}\boldsymbol{W}\mathbf{s}[i].$$

The paper emphasizes that sign retrieval is cheap in hardware because a single XOR with the sign bit is sufficient, so polarity recovery adds essentially no arithmetic overhead (Huang et al., 6 Oct 2025).

The second key component, SGC, is explicitly training-only. The spiking branch output is written as

$$\boldsymbol{y}_t = f_{\text{SN}}(\boldsymbol{x}_t)\boldsymbol{W},$$

while the auxiliary path uses

$$\boldsymbol{y}'_t = f_m(\boldsymbol{x}_t)\boldsymbol{W}', \qquad f_m(\boldsymbol{x}_t)=D\times \tanh(\boldsymbol{x}_t).$$

A hidden-state alignment loss encourages agreement between the spiking and smooth branches:

$$\mathcal{L}_{\text{Hidden}} = \frac{1}{2T} \sum_{t=1}^{T} \left\| \mathrm{softmax}(\boldsymbol{y}_t) - \mathrm{softmax}(\boldsymbol{y}'_t) \right\|_2^2.$$

Because the SGC branch is removed after training, inference remains fully spike-driven (Huang et al., 6 Oct 2025).

3. Distillation, alignment, and optimization

SpikingMamba is not pretrained from scratch. The paper instead adopts a single-stage self-distillation strategy from a pretrained Mamba2 teacher of the same size. The total distillation objective combines output-distribution matching and hidden-state alignment:

$$\mathcal{L} = \mathcal{L}_{\text{KL}} + \mathcal{L}_{\text{Hidden}}.$$

The KL term is defined as

$$\boldsymbol{s}_t = \mathrm{Clip}\bigl(\mathrm{Round}(\boldsymbol{x}_t), -D, D\bigr).$$0

where $$\boldsymbol{s}_t = \mathrm{Clip}\bigl(\mathrm{Round}(\boldsymbol{x}_t), -D, D\bigr).$$1 and $$\boldsymbol{s}_t = \mathrm{Clip}\bigl(\mathrm{Round}(\boldsymbol{x}_t), -D, D\bigr).$$2 are teacher and student parameters. In practice, only three layers use SGC: the first, middle, and last layers (Huang et al., 6 Oct 2025).

The supervised fine-tuning stage uses GenQA, InfinityInstruct, and OpenHermes 2.5, with a single-epoch strategy. The embedding layer is frozen. For the 1.3B model, distillation takes about 42 hours on 8 A100 GPUs (Huang et al., 6 Oct 2025).

After distillation, the model is further improved with preference optimization on UltraFeedback. The paper describes both DPO and KTO. For DPO, the stated objective is

$$\boldsymbol{s}_t = \mathrm{Clip}\bigl(\mathrm{Round}(\boldsymbol{x}_t), -D, D\bigr).$$3

with

$$\boldsymbol{s}_t = \mathrm{Clip}\bigl(\mathrm{Round}(\boldsymbol{x}_t), -D, D\bigr).$$4

KTO is presented as a more stable alternative that replaces pairwise comparison with a reward baseline:

$$\boldsymbol{s}_t = \mathrm{Clip}\bigl(\mathrm{Round}(\boldsymbol{x}_t), -D, D\bigr).$$5

The paper emphasizes that reinforcement learning does not change the architecture, so inference energy cost is preserved while the distilled policy is improved (Huang et al., 6 Oct 2025).

This training recipe is a defining characteristic of the LLM version of SpikingMamba. Earlier spiking LLMs such as SpikeGPT, SpikingSSMs, and SpikeSSM are described as scratch-trained, whereas conversion-style approaches such as SpikeLLM typically rely on repeated spike accumulation over many steps. SpikingMamba is positioned as a bridge between pretrained dense Mamba2 models and spike-driven deployment (Huang et al., 6 Oct 2025).

4. Empirical performance and ablation results

The principal empirical claim is that SpikingMamba can reduce energy substantially while keeping accuracy reasonably close to Mamba2. For the 1.3B model, the reported average zero-shot accuracy is 59.40 for Mamba2 and 54.62 for SpikingMamba with TI-LIF + SGC, corresponding to a 4.78% absolute gap from the teacher. After reinforcement learning, the reported averages rise to 56.91 with DPO and 57.17 with KTO, amounting to about 2.29% and 2.55% improvement over the distilled model, respectively (Huang et al., 6 Oct 2025).

The headline efficiency result is an energy ratio of about 4.76× for the 1.3B model. The paper reports 6150.1188 total energy for Mamba2 and (1291.3147) for SpikingMamba (TI-LIF + SGC), yielding an energy benefit of 4.7627× (Huang et al., 6 Oct 2025).

On WikiText-103, a 130M SpikingMamba achieves 26.32 PPL, compared with 39.75 for SpikeGPT and 33.94 for SpikingSSMs. The paper notes that this is achieved despite not being specially tuned on WikiText (Huang et al., 6 Oct 2025).

The ablations isolate the importance of both the neuron model and the training scaffold. In the neuron ablation, the reported zero-shot averages are 54.62 for TI-LIF + SGC, 53.86 for TI-LIF without SGC, 53.21 for I-LIF without SGC, and 48.35 for plain LIF without SGC. The paper’s interpretation is that signed multi-level spikes matter, SGC helps even with TI-LIF, and plain LIF loses substantial semantic information (Huang et al., 6 Oct 2025).

A second ablation concerns the interaction between SGC and reinforcement learning. The paper reports 57.17 for KTO with SGC and 56.73 for KTO without SGC, and states that SGC not only helps distillation but also improves the effectiveness of downstream RL. It also notes that the gain from RL is larger than the gain observed for Mamba2 itself, which suggests that the spiking student benefits particularly strongly from post-distillation alignment (Huang et al., 6 Oct 2025).

5. Related spiking–Mamba architectures across modalities

Outside the LLM setting, the spiking–Mamba literature is distributed across several modalities and problem formulations. These systems are related by design philosophy rather than by a single shared implementation.

"Mamba-Spike" targets neuromorphic and temporal data such as DVS Gesture, TIDIGITS, Sequential MNIST, and CIFAR10-DVS. Its architecture is explicitly

$$\boldsymbol{s}_t = \mathrm{Clip}\bigl(\mathrm{Round}(\boldsymbol{x}_t), -D, D\bigr).$$6

The front-end uses biologically plausible neuron models such as LIF or SRM, with learnable membrane time constants and thresholds, while the interface accumulates spikes over a fixed window and normalizes by firing rates to produce continuous-valued activations for the standard Mamba backbone. The paper reports 97.8% on DVS Gesture, 99.2% on TIDIGITS, 99.4% with 15 ms latency on Sequential MNIST, and 92.5% on CIFAR10-DVS, alongside a reduction to 785 spikes/sample on DVS Gesture (Qin et al., 2024).

"SpikeMba" is a multi-modal spiking saliency mamba for temporal video grounding. It combines a Spiking Saliency Detector (SSD), a Contextual Moment Reasoner (CMR) with relevant slots, and a Multi-modal Relevant Mamba (MRM). Here the spiking module generates a dynamic and binary saliency proposal set, while the SSM/Mamba components perform efficient long-range contextual reasoning. The paper reports strong results on QVHighlights, TACoS, Charades-STA, TVSum, and Youtube-HL, including 64.13 [email protected] and 43.79 mAP Avg. on the QVHighlights test split (Li et al., 2024).

"Spiking Point Mamba" is introduced as the first Mamba-based SNN in the 3D domain. It proposes Hierarchical Dynamic Encoding (HDE), a Spiking Mamba Block (SMB), and an asymmetric SNN-ANN architecture for spike-based pre-training and finetune. Relative to the previous SOTA SNN models, it improves OA by +6.2%, +6.1%, and +7.4% on three variants of ScanObjectNN, boosts instance mIOU by +1.9% on ShapeNetPart, and reports energy consumption at least 3.5× lower than its ANN counterpart (Wu et al., 19 Apr 2025).

"SpikMamba" addresses event-based human action recognition with a spiking 3D patch embedding, window-based spike linear attention (SpikeSLA), and a spike-form Mamba module. It reports 96.28% on PAF, 97.32% on HARDVS, 99.01% on DVSGesture, and 71.02% on E-FAction, while using 0.12 GFLOPs and 0.18M parameters. The abstract states gains over the previous state of the art of 1.45%, 7.22%, 0.15%, and 3.92% on those four datasets (Chen et al., 2024).

"Vision SmolMamba" extends the design space further by integrating a Spike-Guided Spatio-Temporal Token Pruner (SST-TP) with bidirectional selective state-space recurrence. The paper states that it is the first spiking state-space architecture for vision tasks and reports at least 1.5× lower estimated energy cost than prior spiking Transformer baselines and a Spiking Mamba variant, while maintaining competitive or improved accuracy across ImageNet-1K, CIFAR10/100, CIFAR10-DVS, and DVS128 Gesture (Bai et al., 28 Apr 2026).

Taken together, these works suggest that the field has moved from spiking front-ends attached to standard Mamba backbones toward more spike-native state-space blocks, with the exact degree of spiking integration depending strongly on the modality and the dominant computational bottleneck.

6. Misconceptions, limitations, and adjacent non-spiking Mamba models

A common misconception is that any model whose name combines “spike” and “Mamba” is a fully spiking state-space model. The literature does not support that simplification. "Mamba-Spike" is explicitly not a fully spiking Mamba architecture, because the spiking part is only the front-end and interface, while the Mamba backbone remains a conventional continuous-valued sequence model (Qin et al., 2024). "SpikeMba" is likewise a coordinated system in which SNNs are used for proposal generation and SSMs for contextual reasoning, rather than a wholesale replacement of the state-space core by spiking dynamics (Li et al., 2024).

A second misconception is that the “Mamba” label alone implies spiking computation. Several closely named architectures are not spiking at all. "SpineMamba" is a 3D spinal segmentation network with residual visual Mamba layers and a vertebrae shape prior module; the paper explicitly states that it does not use spikes, membrane potentials, neuron firing dynamics, or event-driven spiking computation (Zhang et al., 2024). "SP-Mamba" for unsupervised medical anomaly detection is also stated to be not a spiking model and instead combines a CNN encoder, prototype learning, and Mamba-based decoding (Pan et al., 25 Jul 2025). "MSCrackMamba" uses Vision Mamba + UperNet for multispectral crack detection and is relevant only in the broader sense of Mamba-based efficient vision modeling, not as an SNN system (Zhu et al., 2024).

The main limitations of the LLM version of SpikingMamba are equally explicit. TI-LIF is more accurate but slightly less energy-efficient than simpler LIF or I-LIF neurons. SGC improves training but is removed at inference, so it cannot help runtime beyond enabling better learned weights. The reported gains are strongest on the targeted Mamba2-style architecture and may not transfer directly to all LLM backbones. Most importantly, the model still trails the dense teacher in zero-shot accuracy, so the efficiency gain comes with a measurable performance cost (Huang et al., 6 Oct 2025).

The related literature exposes further caveats. In "Mamba-Spike," robustness to perturbations and noise is claimed in the abstract and conclusion, but the supplied details note that there are no detailed robustness tables or specific noise-injection experiments shown (Qin et al., 2024). In "SpikMamba," some reported improvement numbers are inconsistent across the abstract, table, and text, although the paper’s overall conclusion remains that it is state of the art on the evaluated benchmarks (Chen et al., 2024). In "Vision SmolMamba," energy is estimated analytically from operation counts rather than measured directly, so the reported energy savings should be interpreted accordingly (Bai et al., 28 Apr 2026).

The broader direction is nevertheless clear. A plausible implication is that future work will continue to differentiate between three regimes: hybrid spiking preprocessor + dense Mamba, spike-aware Mamba blocks with partial state-space redesign, and fully spike-native state-space architectures. The present literature shows active progress in all three regimes, with SpikingMamba in the strict sense occupying the LLM-specific branch centered on distillation from pretrained Mamba2 teachers (Huang et al., 6 Oct 2025).