Memory-Augmented Spiking Neural Networks

• Memory-Augmented SNNs are neuromorphic models that integrate event-driven spiking architectures with explicit memory mechanisms for durable associative and sequential processing.
• They employ diverse memory strategies—from distributed associative networks and spiking attractor dynamics to logic-based RAM and gated mechanisms—to overcome long-range dependency challenges.
• Practical implementations span Nengo, SpiNNaker, and custom CMOS platforms, enabling applications such as image-label recall, pattern completion, and energy-efficient sequential learning.

Memory-Augmented Spiking Neural Networks (SNNs) are a class of neuromorphic models that integrate spiking architectures with explicit memory mechanisms for storing, retrieving, and manipulating information. These systems leverage the event-driven, temporally precise computation of SNNs and augment them with components such as associative memories, random-access memory blocks, attractor networks, and gating architectures, yielding biological plausibility and energy efficiency on neuromorphic substrates. Various memory-augmentation strategies have been demonstrated: from biologically inspired distributed memories implemented with Hebbian plasticity, to fully neuromorphic logic-based RAM modules, to hardware-fused storage arrays and recurrent spiking gate architectures. These approaches enable SNNs to tackle long-range dependencies, robust pattern storage, online working memory, and cognitive-level associative tasks.

1. Distributed Associative Memories in Spiking Networks

A prominent memory-augmentation paradigm exploits distributed associative memories such as Sparse Distributed Memory (SDM) and Correlation Matrix Memories (CMM), implemented in SNNs via N-of-M encoding and Hebbian plasticity. In the SDM framework (Ajwani et al., 2021), binary address and data vectors are encoded as sparse N-of-M codes, where exactly N bits of an M-bit vector are "on." The associative memory consists of two layers: an address decoder with fixed random binary weights, and a data memory layer embodying the CMM through a plastic all-to-all weight matrix. Storage ("write" phase) accumulates outer products of address and data codes, and recall ("read" phase) involves a thresholded top-d selection over the postsynaptic summed input.

These associative SNNs are implemented in Nengo with various spiking neuron models, including Leaky Integrate-and-Fire (LIF), Adaptive-LIF, Spiking Rectified Linear Unit (ReLU-like), and Izhikevich neurons, and with learning rules such as BCM and Oja. The empirical memory capacity—quantified as the number of exact address–data associations retrievable with zero Hamming errors—reveals near parity between spiking and non-spiking SDM/CMM under N-of-M coding, demonstrating robustness to neuron and plasticity model variation. For a 256→256 SDM, capacity peaks at ≈400 address–data pairs, scaling roughly linearly with the number of hidden units in the address decoder layer. Integration into Nengo supports applications as diverse as MNIST image–label association, with linear capacity scaling and gradual degradation beyond peak load (Ajwani et al., 2021).

2. Spiking Attractor Networks and Representation Learning

Another trajectory in memory-augmented SNNs centers on dynamical attractor networks for robust pattern storage, completion, and cognitive operations. An example is the neocortically inspired architecture of Ravichandran et al. (Ravichandran et al., 5 Jun 2024), featuring populations organized into hypercolumns and minicolumns that mimic columnar cortical organization. This architecture comprises feedforward Input→Hidden (representation learning), recurrent Hidden→Hidden (associative attractor memory), and feedback Hidden→Input-Reconstruction projections.

Neurons employ Poisson spike generators with softmax-based local WTA competition. Synaptic plasticity follows the BCPNN (Hebbian-Bayesian) rule, combining short- and long-term spike traces to adapt weights as $W_{ij} = \log \frac{P_{ij}}{P_i P_j}$, promoting both feature selectivity and associative storage. Structural plasticity enforces sparse, mutually informative connectivity. The recurrent symmetry in Hidden→Hidden supports attractor states, enabling pattern completion, perceptual rivalry, distortion resistance, and prototype extraction.

Experimental evaluation using occluded, rival, and distorted MNIST patterns demonstrates that recurrent attractor memory gives significant accuracy boosts for difficult pattern completion tasks (5–10% over feedforward only at high distortion). The hidden representations become increasingly orthogonal during recurrent settling (orthogonality ratio S_ortho: 1.04 → 7.0), and the network is capable of clustering input space into interpretable prototypes. The system thus captures key properties of associative memory within a wholly spiking and synaptically plastic substrate (Ravichandran et al., 5 Jun 2024).

3. Spiking Logic-Gate-Based RAM and Working Memory Modules

In contrast to distributed, continuous-valued associative recall, some memory-augmented SNNs realize digital-style random-access memory using spiking logic gates. On the SpiNNaker platform, an entire decoder+latch matrix for RAM is constructed from LIF neuron-based NOT, AND, and OR gates with static synapses (Ayuso-Martinez et al., 2022). The memory cell is a neural equivalent of the conventional D-latch, and read/write operations are implemented by compositional networks of spiking gates, with a 1 ms integration window per clock cycle.

Write cycles are triggered by one-hot decoder outputs, with D-latch cells in the selected row capturing data spikes; readout is direct unless a gating stage is introduced. In hardware, a 3×3 memory (9 bits) demonstrated 4 ms per write, 0% error rate, and energy cost of ∼10–20 nJ/bit per operation, enabled by precise event-driven transmission and robust to spike jitter. This approach allows seamless integration into SNN controllers, temporally sequenced working-memory buffers, and neuromorphic microcontrollers—enabling SNNs to implement registers, FIFOs, or hierarchical state machines (Ayuso-Martinez et al., 2022).

Component	Neuron Type / Mechanism	Memory Functionality
Associative Layer	LIF/Adaptive-LIF/Others	SDM, CMM (Hebbian)
Attractor Network	Poisson/softmax	Pattern completion
Logic Gate RAM	LIF/static gates	Bit-precise storage

4. Hardware Memory Augmentation: Compute-In-Memory SNNs

At the hardware level, memory-augmented SNNs benefit from fused storage of synaptic weights and neuron states. The IMPULSE architecture (Agrawal et al., 2021) demonstrates a physically fused 10T-SRAM compute-in-memory macro, interleaving weight (WMEM) and membrane-potential (VMEM) arrays on the same bitlines. SNN inference operates by in-memory accumulation of weights and potentials (e.g., $VMEM := VMEM + w_i$), bypassing conventional SRAM/ECPU shuttling. Reconfigurable column peripherals allow dynamic adjustment to neuron functionality (IF, LIF, RMP), with staggered mapping supporting mixed-precision storage.

Measured on a 65nm CMOS prototype, the system achieves energy efficiency up to 0.99 TOPS/W and a 97.4% reduction in energy-delay-product (EDP) at realistic SNN sparsity (85%). Evaluation on sequential sentiment classification for the IMDB dataset demonstrates SNN accuracy within 1% of LSTM baselines, with an 8.5× reduction in parameters due to event-driven, memory-augmented computation (Agrawal et al., 2021).

5. Recurrent and Gated Memory Mechanisms: Spiking LSTM Networks

Recurrent spiking memory modules are operationalized with LSTM-like units. In this architecture, spiking gates (forget, input, output) and binary spikes govern stored cell states, mirroring standard LSTM memory propagation but in a spiking format (Rezaabad et al., 2020). The discrete-time "cell state" is updated as $$\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \mathbf{g}_t$$, with gates computed as thresholded affine transforms of input and previous hidden states. Surrogate-gradient methods enable training by backpropagation through time, circumventing the non-differentiability of spikes.

Empirical results indicate that spiking LSTM-SNNs approximate the temporal memory capacity of conventional LSTMs: e.g., sequential MNIST accuracy of 98.23% vs. 99.10% for non-spiking LSTMs of comparable size, with competitive results in language modeling and sequence classification. This suggests that gated spiking memory mechanisms support long-term dependencies with sparsity and energy-efficiency advantages (Rezaabad et al., 2020).

6. Implementation Considerations and Practical Applications

Memory-augmented SNNs can be instantiated on a range of platforms and for diverse tasks:

• Nengo: Supports SDM/CMM implementations with various neuron and learning models (Ajwani et al., 2021).
• SpiNNaker: Enables spiking logic gate RAM for low-power working memory (Ayuso-Martinez et al., 2022).
• Custom CMOS (IMPULSE): Embeds fused, bit-precise, compute-in-memory for efficient inference (Agrawal et al., 2021).
• Representation Learning/Attractor Domains: Naturally suitable for unsupervised representation learning, robust prototype extraction, and perceptual or pattern-completion tasks (Ravichandran et al., 5 Jun 2024).
• Recurrent Sequence Tasks: Spiking LSTM frameworks permit sequence modeling and language tasks without traditional backpropagation bottlenecks (Rezaabad et al., 2020).

Applications demonstrated include associative image–label recall, working memory for controllers, online learning and pattern restoration, and energy-efficient inference for sequential and perceptual tasks.

7. Comparative Performance, Robustness, and Open Challenges

Empirical studies across these paradigms consistently report that spiking memory-augmented architectures match or approach the performance of classical, non-spiking baselines on capacity, recall accuracy, and robustness to input noise. Memory capacity in SDM/CMM systems scales linearly with the size of the hidden layer and degrades gracefully beyond peak storage load (Ajwani et al., 2021). Spiking logic-based RAM achieves precise, error-free operation and low latency, with performance bounded by physical core count (Ayuso-Martinez et al., 2022). Hardware co-location of memory and computation yields substantial energy savings without compromise to accuracy (Agrawal et al., 2021). Recurrent and attractor dynamics afford additional robustness in completion and segmentation tasks (Ravichandran et al., 5 Jun 2024).

A plausible implication is that the integration of structured memory into SNNs, via distributed, local, or architectural means, is sufficiently mature to support a broad range of associative, sequential, and control tasks at low energy cost. Nevertheless, scaling to very large datasets, efficient continual learning, and direct integration with event-based sensors remain active research topics.

---

References

• (Ajwani et al., 2021) Sparse Distributed Memory using Spiking Neural Networks on Nengo
• (Ravichandran et al., 5 Jun 2024) Spiking representation learning for associative memories
• (Ayuso-Martinez et al., 2022) Construction of a spike-based memory using neural-like logic gates based on Spiking Neural Networks on SpiNNaker
• (Agrawal et al., 2021) IMPULSE: A 65nm Digital Compute-in-Memory Macro with Fused Weights and Membrane Potential for Spike-based Sequential Learning Tasks
• (Rezaabad et al., 2020) Long Short-Term Memory Spiking Networks and Their Applications