Spiking LSTM: Energy-Efficient Temporal Modeling

• Spiking LSTM is a recurrent architecture that transposes LSTM gating and memory mechanisms into an event-driven, sparse spiking regime.
• It employs LIF-based neurons, two-compartment models, and surrogate gradients to effectively manage long-range temporal dependencies.
• Empirical benchmarks reveal high accuracy and significant energy savings across tasks like time-series classification and sequence modeling.

Spiking Long Short-Term Memory (Spiking LSTM, SLSTM) networks are recurrent neural architectures that transpose the gating, memory, and credit-assignment mechanisms of classical LSTM units into the sparse, event-driven regime characteristic of spiking neural networks (SNNs). These models achieve long-range temporal dependencies with orders-of-magnitude greater energy efficiency and inherent neuronal sparsity, enabling deployment on neuromorphic hardware for sequence modeling, time-series classification, and temporally complex decision-making tasks. Multiple lines of research have established theoretically principled mappings, diverse neuron-level implementations, and practical training recipes for such systems, validating their empirical competitiveness with traditional LSTM architectures across a variety of sequential benchmarks.

1. Spiking LSTM: Foundations and Motivation

At their core, Spiking LSTM architectures seek to bridge two domains: the temporal processing power of LSTM (which leverages gated, persistent cell states for learning long-term dependencies) and the event-driven, sparse communication and processing of SNNs. Standard SNNs, based on biophysically inspired models such as the Leaky Integrate-and-Fire (LIF) neuron, naturally encode temporal and spatial sparseness but typically lack effective mechanisms for sequence-level memory comparable to gradient-based LSTMs. The Spiking LSTM paradigm is driven by two primary goals: (i) to replicate LSTM’s gating and state retention with binary spikes and biophysical currents, and (ii) to do so in a manner suitable for low-power neuromorphic hardware, by exploiting temporal and neuronal sparsity (Plank et al., 2021, Rezaabad et al., 2020).

2. Core Neuron and Gate Mechanisms

Spiking LSTM models instantiate a variety of neuron and gating mechanisms to translate the continuous-valued operations of conventional LSTMs into the spike-based regime. Canonical formulations include:

• LIF-based Gated Units: The hidden state is encoded as the membrane potential of an LIF neuron, which integrates synaptic currents, leaks over time, and emits a spike when crossing a threshold. The cell state is maintained as in the standard LSTM by gated accumulation and is coupled to spike generation (Moustakas et al., 16 May 2025, Henkes et al., 2022).
• Two-Compartment Models: Architectures such as LSTM-LIF employ somatic (short-term, reset-on-spike) and dendritic (long-term, leaky or near-leakless) compartments, with gating implemented via inter-compartment currents and learned gains. These designs support long-term temporal integration with event-driven memory control (Zhang et al., 2023, Plank et al., 2021).
• AHP-Current Augmentation: The addition of a slow after-hyperpolarizing (AHP) current, as in AHP-LIF neurons, provides a controllable memory channel that mimics the LSTM cell state, allowing networks to emulate persistent memory with sparse firing (Plank et al., 2021).

A representative update for an AHP-LIF neuron is:

$$\begin{align*} i_I[j,t+1] &= \alpha_I i_I[j,t] + \sum_{i\to j} w_{ij} z_i[t-d_{ij}] \ i_\mathrm{AHP}[j,t+1] &= \alpha_\mathrm{AHP} i_\mathrm{AHP}[j,t] - \beta z_j[t] \ V_j[t+1] &= \alpha_V V_j[t] + (i_I[j,t+1] + i_\mathrm{AHP}[j,t+1]) \ z_j[t+1] &= 1 \quad \mathrm{if}\ V_j[t+1] > V_\mathrm{thr};\ \mathrm{else}\ 0 \end{align*}$$

For a two-compartment LSTM-LIF neuron (Zhang et al., 2023):

$$\begin{align*} U^\mathrm{D}[t] &= U^\mathrm{D}[t-1] + \beta_1 U^\mathrm{S}[t-1] + I[t] - \gamma S[t-1] \ U^\mathrm{S}[t] &= U^\mathrm{S}[t-1] + \beta_2 U^\mathrm{D}[t] - V_\mathrm{th} S[t-1] \ S[t] &= \Theta(U^\mathrm{S}[t] - V_\mathrm{th}) \end{align*}$$

3. Training Methodologies and Surrogate Gradients

Due to the non-differentiability of the spike function, Spiking LSTM networks rely on surrogate-gradient methods for backpropagation through time (BPTT). The typical scheme replaces the derivative of the Heaviside step or threshold function with a smooth surrogate (e.g., fast sigmoid, arctangent, or pdf-shaped function). For example, the surrogate derivative might be:

$$\frac{\partial S}{\partial U} \approx \frac{1}{1 + (\pi U)^2} \quad \text{or} \quad \sigma'_\text{surr}(u) = \frac{1}{1 + (\pi u)^2}$$

(Moustakas et al., 16 May 2025, Henkes et al., 2022).

Training proceeds by unrolling the network in time, accumulating gradients through the gates, cell state updates, and spiking nonlinearity, and updating parameters (weights, thresholds, leak rates) using optimizers such as Adam (Rezaabad et al., 2020, Datta et al., 2022).

Some conversion-based approaches begin from continuous-valued LSTM weights and activations, apply hard or piecewise-linear approximations to gate nonlinearities, and then initialize spiking IF/LIF neurons with matched thresholds, optionally adding optimal bias shifts to align time-averaged firing rates (Datta et al., 2022).

4. Theoretical Correspondence and Memory Analysis

Theoretical work has established that SNNs with multiple time constants (e.g., via AHP-currents or multi-compartment models) can uniformly approximate causal fading-memory filters, including those used by LSTM cells. Key insights include:

• Filter Approximation Theory: Any causal filter with fading memory (LSTM’s cell update) can be approximated by a pool of spiking neurons with appropriate exponential kernels, provided sufficient diversity of time constants (Plank et al., 2021).
• Gradient Flow Preservation: Two-compartment models, by preserving gradients through separate dendritic (long-term) and somatic (short-term) states, alleviate vanishing/exploding gradient problems. The maximal singular value of the Jacobian for the coupled compartments can be tuned to prevent gradient decay or blow-up, supporting stable long-range credit assignment. In practice, LSTM-LIF achieves a gradient norm that does not vanish for empirically selected parameters (Zhang et al., 2023).
• Mapping Between LSTM and SNN Parameters: There are explicit correspondences between LSTM gates and spiking neuron variables, e.g., the AHP current as analogue of cell state, synaptic filter decay as forget gate, and output gate mapping to readout threshold and weights (Plank et al., 2021).

5. Empirical Benchmarks and Hardware Implementations

Across studies, Spiking LSTM and related architectures demonstrate strong performance on standard sequential and temporal tasks:

Architecture	Dataset	Accuracy / Error	Energy Efficiency	Hardware
AHP-SNN	sMNIST	94.3%	~1000× lower E/infer vs GPU LSTM	Loihi
LSTM-LIF	sMNIST	99.18% (vs LIF: 89.3%)	28.2 nJ/sample (vs 2834.7 nJ LSTM)	CMOS, generic
Spiking-LSTM	sMNIST	98.3%	Low (sparse, event-driven)	Sim., Python
SLSTM	Flow Cyt.	98.43%	High sparsity, fast inference	event-based CMOS
SLSTM	Mech. Reg.	MRE ~2.87e-3 (final step)	×120–×238 (Loihi vs GPU)	Loihi, V100
SNN Conv. LSTM	GSC (T=2–8)	up to 95.02% (0.3% from ANN)	4.1× lower E than standard LSTM	FPGA

• Energy savings are consistently observed, with Loihi and similar neuromorphic substrates providing 1–3 orders of magnitude reductions over standard GPU/CPU implementations (Plank et al., 2021, Henkes et al., 2022, Datta et al., 2022).
• Classification accuracy of Spiking LSTM networks generally reaches within 1–2% of standard LSTM baselines on challenging benchmarks, such as sMNIST, PS-MNIST, speech commands, flow cytometry, and nonlinear regression (Zhang et al., 2023, Moustakas et al., 16 May 2025, Henkes et al., 2022).
• Event-based input pipelines, aggressive spiking sparsity regularization, and parallel pipelined execution further reduce effective latency and energy, especially in vision and biomedical domains (Datta et al., 2022, Moustakas et al., 16 May 2025).

6. Architectural Variants and Implementation Recipes

Diverse instantiations of Spiking LSTM exist:

• Explicit Spiking Gate Models: Each LSTM gate (f, i, o, g) is realized as a spiking neuron or IF/LIF block, with binary outputs replaced into the standard update equations. Architectures employ hard thresholding, and employ surrogates for backpropagation (Rezaabad et al., 2020, Henkes et al., 2022).
• Hybrid Continuous-Spiking Formulations: Gates retain continuous outputs for gating operations, while the hidden state, membrane potential, or output is discretized into spikes (as in many SLSTM models) (Moustakas et al., 16 May 2025, Henkes et al., 2022).
• Two-Compartment Neurons: The separation of memory into dendritic (persistent) and somatic (fast, resettable) mimics the LSTM split between cell and hidden state, with inter-compartment gates mapping to input, forget, and output interactions (Zhang et al., 2023).
• AHP-Enhanced LIF: Augmenting standard LIF neurons with slow self-inhibitory currents, allowing effective cell-state emulation on neuromorphic hardware such as Loihi with multi-compartment support (Plank et al., 2021).

Standard training practice is to first initialize network weights (random or from ANN conversion), then perform BPTT with surrogate gradients for all spike-generating gates, learning internal parameters (weights, thresholds, leak constants) and architecture-specific adaptation/time constants (Plank et al., 2021, Henkes et al., 2022, Moustakas et al., 16 May 2025).

7. Limitations, Applications, and Ongoing Directions

Spiking LSTM networks remain an active area of research with several frontiers:

• Limitations: Hardware core and connectivity constraints limit scalability (notably on current Loihi); some SNN variants lack sub-threshold or refractory dynamics for precise timing; training remains computationally expensive, primarily due to BPTT's memory demands (Plank et al., 2021, Rezaabad et al., 2020).
• Target Applications: Empirical successes span time-series classification (sMNIST, speech), biomedical event streams (flow cytometry), regression in physical systems, and large-scale relational reasoning (e.g., bAbI RelNet embedding) (Plank et al., 2021, Moustakas et al., 16 May 2025, Henkes et al., 2022).
• Neuromorphic Suitability: Designs directly map to multi-compartment digital spikes (Loihi), analog event-based devices, and FPGA implementations. Ultra-sparse firing (<1 Hz/neuron) is routinely achieved, maximizing longevity and energy savings (Plank et al., 2021, Zhang et al., 2023).
• Algorithmic Advances: Prospective improvements include on-chip biologically motivated learning (e.g., e-prop), integration of additional slow currents (GLIF3), improved routing/placement for large networks, and combined adaptation mechanisms (Plank et al., 2021, Moustakas et al., 16 May 2025, Zhang et al., 2023).

Spiking LSTM models crucially demonstrate that the memory, gating, and credit assignment features central to LSTM functionality can be realized in a spike-efficient, hardware-amenable manner. This closes a longstanding gap between the theoretical expressive power of conventional RNNs and the energy-efficient event-driven computation of third-generation neural networks.