LSTM-LIF: Biologically-Inspired Spiking Memory

• LSTM-LIF is a two-compartment spiking neural architecture that combines LSTM's long-term memory capabilities with LIF's biological efficiency.
• It uses coupled dendritic and somatic compartments with tunable feedback to mitigate vanishing gradients and achieve superior temporal classification performance.
• Experimental results demonstrate that LSTM-LIF outperforms traditional SNNs while maintaining low power consumption and faster convergence.

A Long Short-Term Memory Leaky Integrate-and-Fire (LSTM-LIF) model is a spiking neural architecture synthesizing the long-term temporal processing strengths of LSTM-style recurrent units with the biological and computational efficiencies of leaky integrate-and-fire (LIF) neurons. The fundamental innovation is a two-compartment spiking neuron—the “LSTM-LIF neuron”—that emulates key features of real pyramidal cells. By coupling dendritic and somatic membrane dynamics with tunable feedback and reset mechanisms, LSTM-LIF achieves extended temporal memory, resilience to vanishing gradients during training, low-power spike-based computation, and achieves state-of-the-art results on challenging temporal classification tasks among spiking networks (Zhang et al., 2023).

1. Biological and Computational Motivation

The two-compartment architecture of LSTM-LIF is directly motivated by the structure and function of cortical pyramidal neurons. In biological systems, dendritic compartments integrate long-lasting synaptic inputs, providing a slow “memory reservoir,” while the soma integrates rapidly varying signals and emits spikes in response to contextually salient patterns. Single-compartment LIF neurons, by contrast, have an intrinsic rapid decay of memory due to exponential leakage, limiting their ability to bridge long temporal intervals.

To address this, the LSTM-LIF model implements two separate but coupled compartments:

• Dendritic compartment: Receives inputs and encodes long-term memory via slower dynamics, analogous to sustained dendritic currents in biological neurons.
• Somatic compartment: Acts as a fast, volatile read-out, generating spikes and providing short-term integration.

Opposite-signed, learnable couplings between the compartments allow the network to retain historical inputs in the dendrite while resetting only the somatic compartment after each spike. This design prevents rapid signal decay and enhances memory capacity for long temporal dependencies (Zhang et al., 2023).

2. Discrete-Time Governing Equations

The LSTM-LIF neuron evolves as follows:

Let at time $t$:

• $\mathcal{U}^D[t]$: dendritic membrane potential
• $\mathcal{U}^S[t]$: somatic membrane potential
• $\mathcal{S}[t]\in\{0,1\}$: output spike
• $\mathcal{I}[t]$: synaptic input current
• $\mathcal{V}_{th}$: firing threshold
• $\beta_1=-\sigma(c_1)\in(-1,0)$, $\beta_2=+\sigma(c_2)\in(0,1)$: learnable inter-compartment gains
• $\gamma>0$: dendritic reset scale (learnable)

The recursive updates are: $$\begin{align*} &\mathcal{U}^{D}[t]=\mathcal{U}^{D}[t-1] + \beta_1\,\mathcal{U}^{S}[t-1] + \mathcal{I}[t] - \gamma\,\mathcal{S}[t-1]\ &\mathcal{U}^{S}[t]=\mathcal{U}^{S}[t-1] + \beta_2\,\mathcal{U}^{D}[t] - \mathcal{V}_{th}\,\mathcal{S}[t-1]\ &\mathcal{S}[t]=\Theta\bigl(\mathcal{U}^S[t]-\mathcal{V}_{th}\bigr)\ &\mathcal{I}[t]=\sum_{i}\omega_i\,\mathcal{S}_i[t-1]+b \end{align*}$$ where $\mathcal{U}^D[t]$0 is the Heaviside function. Typically, $\mathcal{U}^D[t]$1 is negative, $\mathcal{U}^D[t]$2 positive, and both are tuned via gradient descent for stability and efficient temporal credit assignment (Zhang et al., 2023).

3. Gradient Dynamics and Training Stability

Vanishing gradients are a well-documented challenge for training recurrent SNNs with BPTT. The LSTM-LIF exploits carefully parameterized recurrent Jacobians to circumvent rapid gradient decay. The system Jacobian at time $\mathcal{U}^D[t]$3 is: $\mathcal{U}^D[t]$4 The $\mathcal{U}^D[t]$5-norm of $\mathcal{U}^D[t]$6 controls gradient propagation: $\mathcal{U}^D[t]$7 By learning suitable $\mathcal{U}^D[t]$8, $\mathcal{U}^D[t]$9, the norm can be maintained at or above unity, ensuring gradient flow is sustained over long timescales and the vanishing gradient problem is suppressed. Empirical optimization of the inter-compartment gains is required to enforce this property throughout training (Zhang et al., 2023).

Backpropagation-through-time is facilitated by adopting surrogate gradients for the non-differentiable thresholding function: $\mathcal{U}^S[t]$0 This enables continuous gradient-based optimization despite the spiking discontinuities (Zhang et al., 2023).

4. Experimental Protocol, Architectures, and Results

Datasets employed:

• S-MNIST and PS-MNIST (sequential/permuted variants)
• Google Speech Commands v2 (GSC)
• Spiking Heidelberg Digits (SHD)
• Spiking Speech Commands (SSC)

Architectural arrangements:

• Feedforward: Two hidden layers (e.g., 256–128 units) with readout
• Recurrent: Single recurrent layer of 200–256 LSTM-LIF neurons

Baselines for comparison:

• Standard single-compartment LIF, PLIF, GLIF
• LSNN (adaptive threshold)
• Non-spiking LSTM
• SRNN+ALIF

Classification accuracy (recurrent networks):

Dataset	LIF	SRNN+ALIF	LSTM-LIF
S-MNIST	89.3%	98.7%	99.18%
PS-MNIST	—	94.30%	95.07%
GSC	—	92.10%	94.14%
SHD	—	84.40%	88.91%
SSC	—	57.30%	61.09%

LSTM-LIF achieves highest reported accuracy among all SNNs evaluated, and approaches or surpasses non-spiking LSTM performance in these benchmarks. Network convergence is substantially accelerated (≈25 epochs vs. 75–100 for single-compartment LIF). Loss landscapes exhibit smoother, broader minima, indicating improved generalization and gradient stability (Zhang et al., 2023).

5. Energy Efficiency and Computational Complexity

LSTM-LIF provides significant theoretical and empirical energy savings relative to traditional LSTM networks and only modestly increased consumption over single-compartment SNNs. On a 45 nm CMOS process, operation counts and energy per operation are:

• Accumulate (AC): $\mathcal{U}^S[t]$1 pJ per operation (used for synaptic updates in SNNs)
• Multiply-Accumulate (MAC): $\mathcal{U}^S[t]$2 pJ per operation (pervasive in ANNs/LSTMs)

Per-sample energy cost (S-MNIST):

Model	Energy (nJ)
LSTM	2834.7
LIF	23.8
LSTM-LIF	28.2

Thus, LSTM-LIF networks are approximately $\mathcal{U}^S[t]$3 more energy efficient than standard LSTM while providing superior accuracy, with only $\mathcal{U}^S[t]$4 energy overhead relative to standard LIF due to additional dendritic computations (Zhang et al., 2023).

6. Implications, Significance, and Future Directions

LSTM-LIF establishes a new paradigm for SNNs by demonstrating that architectural features inspired by biological neurons—specifically, two-compartment dynamics—can be precisely parameterized and optimized to provide extended memory, enhanced credit assignment, and high practical utility for temporally extended tasks. The model demonstrates that nontrivial gating and retention mechanisms can be incorporated into spike-driven architectures without negating their inherent computational and energy advantages.

It opens several research directions:

• Scale-up for neuromorphic hardware: Highly efficient memory and energy properties render LSTM-LIF directly attractive for deployment on edge or large-scale neuromorphic substrates.
• Bridging SNNs and conventional RNNs: LSTM-LIF narrows the gap between biological plausibility and practical machine learning performance.
• Long-horizon temporal reasoning: The capacity to learn dependencies across hundreds of steps without exploding parameter counts or energy costs addresses a key bottleneck in contemporary SNN design (Zhang et al., 2023).

A plausible implication is that future SNNs for temporal modeling and low-power AI may increasingly adopt compartmental neuron formulations analogous to LSTM-LIF. Continuous exploration of trainable inter-compartmental couplings, advanced surrogate gradient methods, and hybrid ANN-SNN architectures is also prompted by these findings. State-of-the-art performance on both image and speech temporal classification, coupled with low power consumption, positions LSTM-LIF as a foundational component for next-generation neuromorphic and edge-AI systems (Zhang et al., 2023).