GeNN Simulator: GPU-Accelerated SNN Simulation

• GeNN Simulator is a code-generation framework that enables efficient simulation of large-scale spiking neural networks on NVIDIA GPUs.
• It employs advanced sparse data structures and CUDA kernel optimization to achieve significant reductions in simulation time and memory usage.
• The framework integrates synaptic and structural plasticity via techniques like e-prop and DEEP R, supporting novel learning algorithms and unsupervised topographic mapping.

The GeNN (GPU Enhanced Neural Network) simulator is a code generation-based framework for the simulation and training of large-scale spiking neural networks (SNNs) on general purpose NVIDIA GPUs. GeNN employs advanced sparse data structures, a high-level C++ and Python interface, and an automated code-generation paradigm to facilitate efficient simulation of both standard synaptic plasticity and dynamically rewiring, structurally plastic networks. GeNN provides a platform for computational neuroscience and neuromorphic machine learning, optimizing simulation speed and memory usage, and supports novel learning algorithms operating on sparse, dynamically changing connectivity (Knight et al., 22 Oct 2025, Balaji et al., 2014).

1. System Architecture and Code Generation Workflow

GeNN is structured around a code generation core that emits CUDA (and optionally reference C++) kernels, integrating user-defined neuron, synapse, and plasticity models at the source code level. The architecture features:

• Backends: A CUDA backend for GPU acceleration and a C++ backend for reference CPU execution, with planned HIP support.
• Frontends: A high-level C++ API and a Python interface (PyGeNN), allowing definition of model structure, plasticity rules, and runtime control (Balaji et al., 2014).
• Workflow:

1. Model Specification: Users instantiate models, define neuron/synapse populations, connectivity patterns, and equations.
2. Intermediate Representation: The input is parsed into a computation graph (populations, synapses, equations, parameters).
3. Code Generation: For each element, the framework emits CUDA C++ kernels, inlining user equations, with host-side code for simulation orchestration.
4. Compilation: The CUDA sources are compiled via nvcc and linked into an executable.
5. Run-Time: Host code allocates device memory, configures CUDA launch parameters, and orchestrates kernel invocation over simulation steps.

• Structural Plasticity Integration: GeNN implements a “ragged-matrix” format for sparse connectivity, custom update generators for “connectivity update models”, and runtime primitives for connection creation/removal without memory reallocation (Knight et al., 22 Oct 2025).

2. Modeling Structural Plasticity: e-prop and DEEP R

GeNN supports both synaptic and structural plasticity, including:

• e-prop (Eligibility Propagation): An online learning algorithm for SNNs using local eligibility traces. For adaptive LIF (ALIF) neurons, the subthreshold voltage update and readout dynamics are:

$$\begin{aligned} v_j^{t+1} &= \alpha\bigl(v_j^t - z_j^t\,v_{\rm thr}\bigr) + \sum_i W_{ji}^{\rm rec}\,z_i^t + \sum_i W_{ji}^{\rm in}\,x_i^t,\ a_j^{t+1} &= \rho\,a_j^t + z_j^t,\ z_j^t &= H\bigl(v_j^t - (v_{\rm thr} + \beta\,a_j^t)\bigr), \end{aligned}$$

with eligibility and weight update rules:

$$\begin{aligned} \epsilon_{ji}^{t+1} &= \psi_j^t\,\bar z_i^t + (\rho - \psi_j^t\,\beta)\,\epsilon_{ji}^t,\ \Delta W_{ji}^{\rm rec} &= \sum_t \left( \sum_k B_{jk}(\pi_k^t - \pi_k^{*,t}) \right) \bar e_{ji}^t, \end{aligned}$$

where $B=(W^{\rm out})^T$ (Knight et al., 22 Oct 2025).

• DEEP R (Dynamic Evolution of Efficient Plasticity for Rewiring): A rewiring algorithm that maintains strict sparsity throughout training by removing "dormant" synapses (those violating fixed sign constraints) and randomly forming new connections. Operations use GPU-parallel bitfields for connection presence and sign, and custom row- and host-level update code for elimination and formation cycles.
• Integration with mlGeNN/EPropCompiler: Sparse layers can be specified in a Keras-like interface, and DEEP R rewiring is automatically invoked within supervised e-prop training loops.

3. Sparse Data Structures and GPU Kernel Optimization

• Ragged-Matrix Format: The primary sparse storage in GeNN is a per-row, fixed-length padded structure where each presynaptic neuron has slots for postsynaptic indices, weights, and delays up to a predefined maximum out-degree $K_\text{max}$.
• Parallelism: Weight and target arrays are processed in parallel, with each CUDA thread responsible for propagating spikes or updating synapses. Memory accesses are optimized for coalescence.
• Connectivity Updates: Custom GPU "row_update_code" iterates over active connections, leveraging append/swap strategies for efficient addition/removal. Host-side "host_update_code" can compute random indices or counters for GPU.
• Auxiliary Bitfields: GPU device arrays encode presence and sign per possible connection, enabling rapid formation/elimination checks with bitwise operations, especially in structural plasticity (DEEP R, topographic map models).
• Compressed Sparse Row (CSR) Alternative: Earlier versions used CSR (arrays for values, postsynaptic indices, row pointers), allowing for memory reduction of 5–20× compared to dense storage (Balaji et al., 2014).

4. Performance Benchmarks: Sparse Versus Dense SNNs

Benchmarks conducted on single NVIDIA A100 GPUs demonstrate:

Sparsity	Relative Training Time	Parameter Count	N-MNIST Accuracy	DVS-gesture Accuracy
100% (Dense)	1×	1×	99.19 ± 0.1 %	88.94 ± 0.6 %
5% (Fixed)	1/8×	1/4×	98.2 ± 0.3 %	83.5 ± 1.0 %
5% + DEEP R	1/8×	1/4×	99.15 ± 0.1 %	88.86 ± 0.3 %
1% (Fixed)	1/15×	1/11×	96.8 ± 0.4 %	75.2 ± 2.5 %
1% + DEEP R	1/15×	1/11×	99.12 ± 0.2 %	88.30 ± 0.5 %

Sparse DEEP R models match dense accuracy provided input connections remain $\gtrsim5\%$. Training time reductions approach 10–15× with up to 90-fold parameter saving for recurrent classifiers, while DEEP R overhead remains below 1% of step time (Knight et al., 22 Oct 2025).

5. Structural Plasticity in Unsupervised Topographic Map Formation

GeNN supports unsupervised learning scenarios such as spatial topographic map formation:

• Model Structure: Two grid-based layers with Poisson input sources deliver spatially-correlated stimuli; target layer employs all-to-all spike-timing-dependent plasticity (STDP) combined with structural rewiring at fixed time intervals.
• Plasticity Rules:
  • Elimination Probability: For synapse $(i,j)$,

  $$p_{\rm elim} = \begin{cases} p_{\rm elim\_dep} & \text{if }g_{ij}<g_\theta \ p_{\rm elim\_pot} & \text{if }g_{ij}\geq g_\theta \end{cases}$$ - Formation Probability:

  $$p_{\rm form}(d) = p_{\rm peak} \exp\left({-d_{ij}^2/(2\sigma^2_{\rm form})}\right)$$

• Dynamics: Formation/elimination stabilizes after $\sim$200 ms of biological time. Mean in-/out-degree converge; receptive fields sharpen spatially. For $L_0\times L_0$ layers ($$\begin{aligned} \epsilon_{ji}^{t+1} &= \psi_j^t\,\bar z_i^t + (\rho - \psi_j^t\,\beta)\,\epsilon_{ji}^t,\ \Delta W_{ji}^{\rm rec} &= \sum_t \left( \sum_k B_{jk}(\pi_k^t - \pi_k^{*,t}) \right) \bar e_{ji}^t, \end{aligned}$$0), simulation remains faster-than-realtime for smaller scales, with runtime dominated ($$\begin{aligned} \epsilon_{ji}^{t+1} &= \psi_j^t\,\bar z_i^t + (\rho - \psi_j^t\,\beta)\,\epsilon_{ji}^t,\ \Delta W_{ji}^{\rm rec} &= \sum_t \left( \sum_k B_{jk}(\pi_k^t - \pi_k^{*,t}) \right) \bar e_{ji}^t, \end{aligned}$$190%) by connectivity updates at high structural plasticity frequency.

Scale $$\begin{aligned} \epsilon_{ji}^{t+1} &= \psi_j^t\,\bar z_i^t + (\rho - \psi_j^t\,\beta)\,\epsilon_{ji}^t,\ \Delta W_{ji}^{\rm rec} &= \sum_t \left( \sum_k B_{jk}(\pi_k^t - \pi_k^{*,t}) \right) \bar e_{ji}^t, \end{aligned}$$2	Neurons/Layer	Real-Time Factor	Connectivity Update Share
1	256	2× faster	~90%
7	12,544	~0.2× real-time	~90%

The step time increases only by $$\begin{aligned} \epsilon_{ji}^{t+1} &= \psi_j^t\,\bar z_i^t + (\rho - \psi_j^t\,\beta)\,\epsilon_{ji}^t,\ \Delta W_{ji}^{\rm rec} &= \sum_t \left( \sum_k B_{jk}(\pi_k^t - \pi_k^{*,t}) \right) \bar e_{ji}^t, \end{aligned}$$310× for a $$\begin{aligned} \epsilon_{ji}^{t+1} &= \psi_j^t\,\bar z_i^t + (\rho - \psi_j^t\,\beta)\,\epsilon_{ji}^t,\ \Delta W_{ji}^{\rm rec} &= \sum_t \left( \sum_k B_{jk}(\pi_k^t - \pi_k^{*,t}) \right) \bar e_{ji}^t, \end{aligned}$$4 network size increase (Knight et al., 22 Oct 2025).

6. Guidelines and Practical Trade-offs for Sparse Simulation

• Initialization: Effective learning and performance require each neuron’s input degree to remain at least $$\begin{aligned} \epsilon_{ji}^{t+1} &= \psi_j^t\,\bar z_i^t + (\rho - \psi_j^t\,\beta)\,\epsilon_{ji}^t,\ \Delta W_{ji}^{\rm rec} &= \sum_t \left( \sum_k B_{jk}(\pi_k^t - \pi_k^{*,t}) \right) \bar e_{ji}^t, \end{aligned}$$5 of possible connections.
• DEEP R Sparsity: Maintains constant connection density with sub-1% computational overhead. Sparse models often require longer training to reach the same accuracy (consistent with lottery-ticket hypothesis).
• Memory and Performance: Ragged-matrix padding yields sublinear memory savings as sparsity increases. For high-frequency rewiring (topomap scenario), connectivity updates may dominate overall runtime.
• Optimization: Code-generated CUDA kernels are automatically tuned for occupancy $$\begin{aligned} \epsilon_{ji}^{t+1} &= \psi_j^t\,\bar z_i^t + (\rho - \psi_j^t\,\beta)\,\epsilon_{ji}^t,\ \Delta W_{ji}^{\rm rec} &= \sum_t \left( \sum_k B_{jk}(\pi_k^t - \pi_k^{*,t}) \right) \bar e_{ji}^t, \end{aligned}$$6; block sizes are multiples of 32 (warp size) (Balaji et al., 2014).
• Scalability: A single GPU with GeNN can emulate the performance of small CPU clusters for SNN tasks, reducing hardware cost and power consumption (Balaji et al., 2014).

The GeNN simulator combines code-generation strategies with advanced data structures, delivering an extensible platform for high-throughput investigation of sparse, plastic, and structurally adaptive spiking neural networks at large scale (Knight et al., 22 Oct 2025, Balaji et al., 2014).