---
title: 'SpikingMamba: Spike-Driven Mamba LLM'
url: https://www.emergentmind.com/topics/spikingmamba
type: topic
---

# SpikingMamba: Spike-Driven Mamba LLM

SpikingMamba denotes a class of architectures that combine spiking neural computation with Mamba-style selective state-space modeling, but the term has two distinct uses in the recent literature. In the strict sense, it refers to the spiking large language model introduced in "SpikingMamba: Towards Energy-Efficient Large Language Models via Knowledge Distillation from Mamba," which distills a pretrained Mamba2 teacher into an SNN-based recurrent model that replaces expensive dense projections with spike-driven computation while preserving $O(L)$ sequence processing and constant-memory autoregressive structure [2510.04595]. In a broader sense, nearby work uses closely related names for hybrid SNN–Mamba systems in neuromorphic temporal processing, temporal video grounding, point cloud analysis, event-based human action recognition, and spike-native vision backbones, with substantial variation in where spiking is applied and whether the Mamba core itself is modified [2408.11823] [2404.01174] [2504.14371] [2410.16746] [2604.25570].

## 1. Terminology and scope

The literature does not treat “SpikingMamba” as a single canonical architecture. Some systems are best described as **hybrid pipelines** in which spikes appear only in the encoder or proposal generator, while a conventional continuous-valued Mamba backbone performs the downstream sequence modeling. Other systems integrate spike events more deeply into the state-space computation itself.

This distinction is explicit in "Mamba-Spike," which does **not** present a fully spiking Mamba model. Its architecture is a spiking front-end, a spike-to-activation interface, and a standard Mamba backbone. The paper states that it is closer to “Mamba with a spiking input/feature front-end” than to a fully spiking Mamba cell [2408.11823]. By contrast, the 2025 LLM paper titled "SpikingMamba" preserves the recurrent SSM core of Mamba2 but replaces the dominant dense projections with spike-driven equivalents and trains the student by single-stage distillation plus optional reinforcement learning [2510.04595].

A broader survey of related models shows that the “spiking + Mamba” design space is modality-specific rather than uniform.

| System | Domain | Spiking locus |
|---|---|---|
| Mamba-Spike | Temporal and neuromorphic data | Spiking front-end; standard Mamba backbone |
| SpikeMba | Temporal video grounding | Spiking saliency detector; SSM/Mamba contextual reasoning |
| Spiking Point Mamba | 3D point cloud analysis | Spiking embedding and Spiking Mamba Block |
| SpikMamba | Event-based HAR | Spike-form patch embedding, SpikeSLA, spike-form Mamba |
| Vision SmolMamba | Vision backbone | Spike-guided token pruning and spiking state-space recurrence |

This suggests that “SpikingMamba” functions both as the name of a specific LLM architecture and as a broader umbrella for spike-aware Mamba variants whose common theme is the attempt to combine sparse event-driven computation with linear-time state-space modeling.

## 2. SpikingMamba as a spiking large language model

The specific model named **SpikingMamba** was proposed to make Mamba-style LLM inference substantially more energy efficient while preserving as much of the original model’s zero-shot capability as possible. Its starting point is that even Mamba, despite avoiding quadratic self-attention and KV caching, still relies heavily on dense matrix multiplications, especially in the input and output projections. The paper states that in Mamba2, more than 90% of parameters in the 1.3B model sit in the input/output linear layers, and these are the operations targeted for spiking replacement [2510.04595].

The architecture combines four elements. First, it keeps **Mamba’s recurrent SSM backbone** for linear-time inference. Second, it introduces **TI-LIF**, a ternary-integer spiking neuron designed to preserve semantic polarity through signed multi-level spike representations. Third, it adds a **training-exclusive Smoothed Gradient Compensation (SGC) path** to mitigate quantization loss without changing inference-time computation. Fourth, it uses **single-stage distillation from pretrained Mamba2**, followed by optional reinforcement learning with DPO or KTO [2510.04595].

TI-LIF addresses a limitation of ordinary LIF and prior integer spiking neurons. Standard LIF emits binary spikes, while I-LIF typically emits non-negative integer-valued activations during training. The paper argues that non-negative-only integer neurons are problematic for language modeling because negative activation values often encode meaningful semantic polarity. TI-LIF therefore quantizes to a symmetric signed range:
$$
\boldsymbol{s}_t = \mathrm{Clip}\bigl(\mathrm{Round}(\boldsymbol{x}_t), -D, D\bigr).
$$
It uses a surrogate gradient
$$
\frac{\partial \boldsymbol{s}_t}{\partial \boldsymbol{x}_t} =
\begin{cases}
\alpha, & -D \le \boldsymbol{x}_t \le D, \\
0, & \text{otherwise},
\end{cases}
$$
so that magnitude and polarity are both retained during training [2510.04595].

At inference time, TI-LIF remains spike-driven. The input token embedding is injected once as $|\boldsymbol{x}_t|$ at the first micro-step, followed by zero input at later micro-steps. With $\beta=1$, the final output is recovered by summing spike-driven accumulations and restoring the sign with a single sign bit:
$$
\boldsymbol{y}_t = \mathrm{sgn}(\boldsymbol{x}_t)\cdot \sum_{i=1}^{D}\boldsymbol{W}\mathbf{s}[i].
$$
The paper emphasizes that sign retrieval is cheap in hardware because a single XOR with the sign bit is sufficient, so polarity recovery adds essentially no arithmetic overhead [2510.04595].

The second key component, SGC, is explicitly training-only. The spiking branch output is written as
$$
\boldsymbol{y}_t = f_{\text{SN}}(\boldsymbol{x}_t)\boldsymbol{W},
$$
while the auxiliary path uses
$$
\boldsymbol{y}'_t = f_m(\boldsymbol{x}_t)\boldsymbol{W}',
\qquad
f_m(\boldsymbol{x}_t)=D\times \tanh(\boldsymbol{x}_t).
$$
A hidden-state alignment loss encourages agreement between the spiking and smooth branches:
$$
\mathcal{L}_{\text{Hidden}} = \frac{1}{2T} \sum_{t=1}^{T}
\left\| \mathrm{softmax}(\boldsymbol{y}_t) - \mathrm{softmax}(\boldsymbol{y}'_t) \right\|_2^2.
$$
Because the SGC branch is removed after training, inference remains fully spike-driven [2510.04595].

## 3. Distillation, alignment, and optimization

SpikingMamba is not pretrained from scratch. The paper instead adopts a **single-stage self-distillation** strategy from a pretrained Mamba2 teacher of the same size. The total distillation objective combines output-distribution matching and hidden-state alignment:
$$
\mathcal{L} = \mathcal{L}_{\text{KL}} + \mathcal{L}_{\text{Hidden}}.
$$
The KL term is defined as
$$
\mathcal{L}_{\text{KL}} =
\frac{1}{T} \sum_{t=1}^{T}
\mathrm{KL}\bigl(
p(\cdot|\hat{y}_{1:t}, x, \theta_T)
\;\|\;
p(\cdot|\hat{y}_{1:t}, x, \theta_S)
\bigr),
$$
where $\theta_T$ and $\theta_S$ are teacher and student parameters. In practice, only three layers use SGC: the first, middle, and last layers [2510.04595].

The supervised fine-tuning stage uses **GenQA**, **InfinityInstruct**, and **OpenHermes 2.5**, with a single-epoch strategy. The embedding layer is frozen. For the 1.3B model, distillation takes about **42 hours on 8 A100 GPUs** [2510.04595].

After distillation, the model is further improved with preference optimization on **UltraFeedback**. The paper describes both **DPO** and **KTO**. For DPO, the stated objective is
$$
\pi_{\theta} = \max_{\theta} \mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}
\log \sigma\left( \beta\cdot (f_w - f_l) \right),
$$
with
$$
f_w = \log \frac{p(y_w|x;\theta)}{p(y_w|x;\theta_T)},
\qquad
f_l = \log \frac{p(y_l|x;\theta)}{p(y_l|x;\theta_T)}.
$$
KTO is presented as a more stable alternative that replaces pairwise comparison with a reward baseline:
$$
r(x,y)=\beta \log \frac{p(y|x;\theta)}{p(y|x;\theta_T)}.
$$
The paper emphasizes that reinforcement learning does **not** change the architecture, so inference energy cost is preserved while the distilled policy is improved [2510.04595].

This training recipe is a defining characteristic of the LLM version of SpikingMamba. Earlier spiking LLMs such as SpikeGPT, SpikingSSMs, and SpikeSSM are described as scratch-trained, whereas conversion-style approaches such as SpikeLLM typically rely on repeated spike accumulation over many steps. SpikingMamba is positioned as a bridge between pretrained dense Mamba2 models and spike-driven deployment [2510.04595].

## 4. Empirical performance and ablation results

The principal empirical claim is that SpikingMamba can reduce energy substantially while keeping accuracy reasonably close to Mamba2. For the **1.3B** model, the reported average zero-shot accuracy is **59.40** for Mamba2 and **54.62** for SpikingMamba with TI-LIF + SGC, corresponding to a **4.78% absolute gap** from the teacher. After reinforcement learning, the reported averages rise to **56.91** with DPO and **57.17** with KTO, amounting to about **2.29%** and **2.55%** improvement over the distilled model, respectively [2510.04595].

The headline efficiency result is an energy ratio of about **4.76×** for the 1.3B model. The paper reports **6150.1188** total energy for Mamba2 and **1291.3147** for SpikingMamba (TI-LIF + SGC), yielding an **energy benefit of 4.7627×** [2510.04595].

On **WikiText-103**, a **130M** SpikingMamba achieves **26.32 PPL**, compared with **39.75** for SpikeGPT and **33.94** for SpikingSSMs. The paper notes that this is achieved despite not being specially tuned on WikiText [2510.04595].

The ablations isolate the importance of both the neuron model and the training scaffold. In the neuron ablation, the reported zero-shot averages are **54.62** for TI-LIF + SGC, **53.86** for TI-LIF without SGC, **53.21** for I-LIF without SGC, and **48.35** for plain LIF without SGC. The paper’s interpretation is that signed multi-level spikes matter, SGC helps even with TI-LIF, and plain LIF loses substantial semantic information [2510.04595].

A second ablation concerns the interaction between SGC and reinforcement learning. The paper reports **57.17** for KTO with SGC and **56.73** for KTO without SGC, and states that SGC not only helps distillation but also improves the effectiveness of downstream RL. It also notes that the gain from RL is larger than the gain observed for Mamba2 itself, which suggests that the spiking student benefits particularly strongly from post-distillation alignment [2510.04595].

## 5. Related spiking–Mamba architectures across modalities

Outside the LLM setting, the spiking–Mamba literature is distributed across several modalities and problem formulations. These systems are related by design philosophy rather than by a single shared implementation.

"Mamba-Spike" targets neuromorphic and temporal data such as **DVS Gesture**, **TIDIGITS**, **Sequential MNIST**, and **CIFAR10-DVS**. Its architecture is explicitly
$$
\text{Raw temporal input} \rightarrow \text{Spiking front-end} \rightarrow \text{Spike-to-activation interface} \rightarrow \text{Mamba backbone} \rightarrow \text{Prediction}.
$$
The front-end uses biologically plausible neuron models such as **LIF** or **SRM**, with learnable membrane time constants and thresholds, while the interface accumulates spikes over a fixed window and normalizes by firing rates to produce continuous-valued activations for the standard Mamba backbone. The paper reports **97.8%** on DVS Gesture, **99.2%** on TIDIGITS, **99.4%** with **15 ms** latency on Sequential MNIST, and **92.5%** on CIFAR10-DVS, alongside a reduction to **785 spikes/sample** on DVS Gesture [2408.11823].

"SpikeMba" is a **multi-modal spiking saliency mamba for temporal video grounding**. It combines a **Spiking Saliency Detector (SSD)**, a **Contextual Moment Reasoner (CMR)** with relevant slots, and a **Multi-modal Relevant Mamba (MRM)**. Here the spiking module generates a dynamic and binary saliency proposal set, while the SSM/Mamba components perform efficient long-range contextual reasoning. The paper reports strong results on **QVHighlights**, **TACoS**, **Charades-STA**, **TVSum**, and **Youtube-HL**, including **64.13** R1@0.5 and **43.79** mAP Avg. on the QVHighlights test split [2404.01174].

"Spiking Point Mamba" is introduced as the **first Mamba-based SNN in the 3D domain**. It proposes **Hierarchical Dynamic Encoding (HDE)**, a **Spiking Mamba Block (SMB)**, and an asymmetric **SNN-ANN** architecture for spike-based pre-training and finetune. Relative to the previous SOTA SNN models, it improves OA by **+6.2%**, **+6.1%**, and **+7.4%** on three variants of **ScanObjectNN**, boosts instance mIOU by **+1.9%** on **ShapeNetPart**, and reports energy consumption at least **3.5× lower** than its ANN counterpart [2504.14371].

"SpikMamba" addresses **event-based human action recognition** with a **spiking 3D patch embedding**, **window-based spike linear attention (SpikeSLA)**, and a **spike-form Mamba** module. It reports **96.28%** on **PAF**, **97.32%** on **HARDVS**, **99.01%** on **DVSGesture**, and **71.02%** on **E-FAction**, while using **0.12 GFLOPs** and **0.18M parameters**. The abstract states gains over the previous state of the art of **1.45%**, **7.22%**, **0.15%**, and **3.92%** on those four datasets [2410.16746].

"Vision SmolMamba" extends the design space further by integrating a **Spike-Guided Spatio-Temporal Token Pruner (SST-TP)** with bidirectional selective state-space recurrence. The paper states that it is the **first spiking state-space architecture for vision tasks** and reports at least **1.5×** lower estimated energy cost than prior spiking Transformer baselines and a Spiking Mamba variant, while maintaining competitive or improved accuracy across **ImageNet-1K**, **CIFAR10/100**, **CIFAR10-DVS**, and **DVS128 Gesture** [2604.25570].

Taken together, these works suggest that the field has moved from **spiking front-ends attached to standard Mamba backbones** toward **more spike-native state-space blocks**, with the exact degree of spiking integration depending strongly on the modality and the dominant computational bottleneck.

## 6. Misconceptions, limitations, and adjacent non-spiking Mamba models

A common misconception is that any model whose name combines “spike” and “Mamba” is a fully spiking state-space model. The literature does not support that simplification. "Mamba-Spike" is explicitly **not** a fully spiking Mamba architecture, because the spiking part is only the front-end and interface, while the Mamba backbone remains a conventional continuous-valued sequence model [2408.11823]. "SpikeMba" is likewise a coordinated system in which SNNs are used for proposal generation and SSMs for contextual reasoning, rather than a wholesale replacement of the state-space core by spiking dynamics [2404.01174].

A second misconception is that the “Mamba” label alone implies spiking computation. Several closely named architectures are not spiking at all. "SpineMamba" is a 3D spinal segmentation network with residual visual Mamba layers and a vertebrae shape prior module; the paper explicitly states that it does **not** use spikes, membrane potentials, neuron firing dynamics, or event-driven spiking computation [2408.15887]. "SP-Mamba" for unsupervised medical anomaly detection is also stated to be **not a spiking model** and instead combines a CNN encoder, prototype learning, and Mamba-based decoding [2507.19076]. "MSCrackMamba" uses **Vision Mamba + UperNet** for multispectral crack detection and is relevant only in the broader sense of Mamba-based efficient vision modeling, not as an SNN system [2412.06211].

The main limitations of the LLM version of SpikingMamba are equally explicit. TI-LIF is more accurate but slightly less energy-efficient than simpler LIF or I-LIF neurons. SGC improves training but is removed at inference, so it cannot help runtime beyond enabling better learned weights. The reported gains are strongest on the targeted **Mamba2-style architecture** and may not transfer directly to all LLM backbones. Most importantly, the model still trails the dense teacher in zero-shot accuracy, so the efficiency gain comes with a measurable performance cost [2510.04595].

The related literature exposes further caveats. In "Mamba-Spike," robustness to perturbations and noise is claimed in the abstract and conclusion, but the supplied details note that there are no detailed robustness tables or specific noise-injection experiments shown [2408.11823]. In "SpikMamba," some reported improvement numbers are inconsistent across the abstract, table, and text, although the paper’s overall conclusion remains that it is state of the art on the evaluated benchmarks [2410.16746]. In "Vision SmolMamba," energy is **estimated analytically from operation counts rather than measured directly**, so the reported energy savings should be interpreted accordingly [2604.25570].

The broader direction is nevertheless clear. A plausible implication is that future work will continue to differentiate between three regimes: **hybrid spiking preprocessor + dense Mamba**, **spike-aware Mamba blocks with partial state-space redesign**, and **fully spike-native state-space architectures**. The present literature shows active progress in all three regimes, with SpikingMamba in the strict sense occupying the LLM-specific branch centered on distillation from pretrained Mamba2 teachers [2510.04595].

Source: https://www.emergentmind.com/topics/spikingmamba