LongSpike: Fractional SNN Framework

• LongSpike is a spiking neural network framework that integrates fractional-order state-space modeling to introduce non-Markovian, power-law memory into long-sequence learning.
• It replaces the standard first-order Markovian state transition with a Caputo fractional system, using a sum-of-exponentials approximation and FFT-based convolutions for computational efficiency.
• Empirical results on LRA, WikiText-103, and Speech Commands demonstrate that LongSpike outperforms previous SNN baselines while preserving sparse synaptic computation.

LongSpike is a spiking neural network framework for efficient long-sequence learning that integrates fractional-order State-Space Modeling, or f-SSM, from control theory into the spiking domain (He et al., 11 Jun 2026). It replaces the usual first-order Markovian state transition with a Caputo fractional-order system, thereby introducing non-Markovian, power-law memory into a sparse spiking architecture. The model combines a fractional state-space block, a discrete-time leaky integrate-and-fire neuron, and a Surrogate Dynamic Network (SDN) that supports efficient, parallel training on long contexts. Empirical evaluations were reported on Long Range Arena (LRA), WikiText-103, and Speech Commands, where LongSpike outperformed prior spiking state-space baselines while preserving sparse synaptic computation (He et al., 11 Jun 2026).

1. Research context and motivation

LongSpike emerged within a line of work that adapts structured state-space models to spiking computation for long sequences. SpikingSSM developed spiking state space models by hierarchically integrating neuronal dynamics with an SSM block, realizing sparse synaptic computation and using a light-weight surrogate dynamic network to predict the after-reset membrane potential; on LRA it reached an average accuracy of approximately $84.33\%$ at a $10\%$ spiking rate, and on WikiText-103 it reported perplexity $33.94$ with $75$M parameters (Shen et al., 2024). SPikE-SSM addressed similar constraints through Parallel Max–Min Boundary Compression (PMBC), a reset-refractory neuron model, and trainable thresholds and refractory magnitudes; it reported $84.18\%$ average accuracy on LRA at an overall spiking rate of $\sim 8\%$, and perplexity $33.18$ on WikiText-103 with a $24.5\%$ spike rate (Zhong et al., 2024).

The specific motivation for LongSpike is the limitation that dominant SNN architectures typically rely on first-order Ordinary Differential Equations to govern neuronal state transitions. In the LongSpike formulation, this first-order assumption imposes a “memoryless” bottleneck that limits the capacity to capture the complex, long-range dependencies inherent in long-sequence tasks (He et al., 11 Jun 2026). This emphasis on richer temporal state is consistent with broader efforts in SNNs to move beyond single-compartment memory mechanisms; for example, LSTM-LIF introduced somatic and dendritic compartments tailored to retain short- and long-term memories and provided a vanishing-gradient analysis for that design (Zhang et al., 2023). LongSpike’s distinctive contribution is to import fractional-order dynamics into a spiking state-space architecture while retaining parallel sequence processing (He et al., 11 Jun 2026).

2. Fractional-order state-space formulation

LongSpike replaces the standard continuous-time state-space system

$$\frac{d h(t)}{dt} = A\,h(t) + B\,x(t), \qquad y(t)=C\,h(t),$$

with the Caputo fractional-order system of order $\alpha\in(0,1)$:

$10\%$0

where $10\%$1 is the input, $10\%$2 the latent state, and $10\%$3 the output (He et al., 11 Jun 2026). The Caputo derivative is defined as

$10\%$4

Its Volterra-integral equivalent is

$10\%$5

This kernel introduces long-memory behavior through a power-law history term. Direct history accumulation is $10\%$6, so LongSpike uses the diffusive representation

$10\%$7

and truncates it with a quadrature-based sum-of-exponentials (SOE) approximation:

$10\%$8

With auxiliary states

$10\%$9

the model obtains $33.94$0 coupled first-order ODEs,

$33.94$1

together with

$33.94$2

After discretization with step $33.94$3, the recurrence becomes

$33.94$4

$33.94$5

$33.94$6

with

$33.94$7

Because each $33.94$8 evolves linearly, the block can be written as a global convolution

$33.94$9

and computed in $75$0 via FFT (He et al., 11 Jun 2026). This formulation is central to LongSpike’s claim that fractional operators can be made tractable and parallel in an SNN setting.

3. Spiking dynamics and block architecture

The current output $75$1 from the fractional SSM drives a discrete-time leaky integrate-and-fire neuron:

$75$2

$75$3

$75$4

where $75$5 is the Heaviside step, $75$6 is the membrane decay, $75$7 is a learnable threshold, and soft-reset is used (He et al., 11 Jun 2026). The binary spike train $75$8 provides the sparse event-driven output of the block.

To avoid sequential BPTT through $75$9 steps, LongSpike uses a small Surrogate Dynamic Network, typically a causal CNN, that learns a nonautoregressive mapping

$84.18\%$0

The surrogate derivative is piecewise quadratic,

$84.18\%$1

which the paper describes as ensuring stable gradient flow near threshold (He et al., 11 Jun 2026).

A single LongSpike block is organized as follows. First, it computes

$84.18\%$2

Second, for $84.18\%$3, it convolves $84.18\%$4 with the exponential kernel

$84.18\%$5

Third, it aggregates

$84.18\%$6

Fourth, it predicts spikes $84.18\%$7 and updates the membrane via the soft-reset rule (He et al., 11 Jun 2026). The architecture therefore hierarchically integrates neuronal dynamics with long-memory kernels, extending the hierarchical SSM-plus-spike pattern introduced in earlier spiking SSMs (Shen et al., 2024).

4. Training procedure, parallelism, and computational cost

LongSpike is trained end-to-end by BPTT over the surrogate spiking network and the linear f-SSM (He et al., 11 Jun 2026). Gradients through the f-SSM block reduce to gradients through convolutions and are computed via standard FFT backprop. The parameters $84.18\%$8, the SOE nodes $84.18\%$9 optionally, and the SDN weights are updated by Adam. Weight decay, dropout, and layer- or batch-norm are applied as in standard SSM literature (He et al., 11 Jun 2026).

The computational advantage comes from recasting the fractional dynamics into linear time-invariant convolutions. In the reported formulation, steps $\sim 8\%$0–$\sim 8\%$1 of the block are cast as FFTs or parallel scanned recurrences, achieving $\sim 8\%$2 time (He et al., 11 Jun 2026). Relative to SpikingSSM’s forward cost $\sim 8\%$3, LongSpike adds $\sim 8\%$4 to build the SOE kernels but leaves the FFT convolution unchanged. With $\sim 8\%$5, this is described as a small constant-factor overhead (He et al., 11 Jun 2026).

The memory trade-off is explicit. The vectorized implementation uses $\sim 8\%$6 memory, while a loop-based implementation uses $\sim 8\%$7 memory (He et al., 11 Jun 2026). On Speech Commands, adding $\sim 8\%$8 increased per-epoch time by approximately $\sim 8\%$9 but reduced GPU memory by approximately $33.18$0 through kernel-fusion optimizations (He et al., 11 Jun 2026). This suggests that the additional burden of fractional memory was operationally modest in the reported setting.

5. Empirical performance

LongSpike was evaluated with $33.18$1 terms on LRA, WikiText-103, and Speech Commands, and compared directly to SpikingSSM (He et al., 11 Jun 2026).

Benchmark	LongSpike	SpikingSSM
LRA six-task average	$33.18$2	$33.18$3
WikiText-103	$33.18$4M params, perplexity $33.18$5, spiking rate $33.18$6	$33.18$7M params, perplexity $33.18$8, spiking rate $33.18$9
Speech Commands	$24.5\%$0K params, accuracy $24.5\%$1	$24.5\%$2K params, accuracy $24.5\%$3

On the six LRA tasks, the reported per-task results for LongSpike were: ListOps $24.5\%$4, Text $24.5\%$5, Retrieval $24.5\%$6, Image $24.5\%$7, Path $24.5\%$8, and Path-X $24.5\%$9. The corresponding SpikingSSM numbers were $$\frac{d h(t)}{dt} = A\,h(t) + B\,x(t), \qquad y(t)=C\,h(t),$$0, $$\frac{d h(t)}{dt} = A\,h(t) + B\,x(t), \qquad y(t)=C\,h(t),$$1, $$\frac{d h(t)}{dt} = A\,h(t) + B\,x(t), \qquad y(t)=C\,h(t),$$2, $$\frac{d h(t)}{dt} = A\,h(t) + B\,x(t), \qquad y(t)=C\,h(t),$$3, $$\frac{d h(t)}{dt} = A\,h(t) + B\,x(t), \qquad y(t)=C\,h(t),$$4, and $$\frac{d h(t)}{dt} = A\,h(t) + B\,x(t), \qquad y(t)=C\,h(t),$$5 (He et al., 11 Jun 2026). On language modeling, LongSpike improved perplexity over SpikingSSM while keeping the parameter count fixed at $$\frac{d h(t)}{dt} = A\,h(t) + B\,x(t), \qquad y(t)=C\,h(t),$$6M; on audio classification, it improved Speech Commands accuracy from $$\frac{d h(t)}{dt} = A\,h(t) + B\,x(t), \qquad y(t)=C\,h(t),$$7 to $$\frac{d h(t)}{dt} = A\,h(t) + B\,x(t), \qquad y(t)=C\,h(t),$$8 at the same $$\frac{d h(t)}{dt} = A\,h(t) + B\,x(t), \qquad y(t)=C\,h(t),$$9K parameter scale (He et al., 11 Jun 2026).

The paper summarizes these results by stating that LongSpike outperforms state-of-the-art SNNs in accuracy while preserving sparse synaptic computation (He et al., 11 Jun 2026). A plausible implication is that the additional fractional-memory mechanism contributed more to long-context modeling than it cost in constant-factor overhead, at least in the $\alpha\in(0,1)$0 regime used in the experiments.

6. Interpretation, limitations, and outlook

LongSpike’s central interpretive claim is that fractional-order dynamics with $\alpha\in(0,1)$1 can be made tractable and parallel for SNNs by SOE approximations and convolution-based state-space formulations (He et al., 11 Jun 2026). The resulting heavy-tailed (Mittag–Leffler) impulse response is reported to capture long-range dependencies far beyond first-order ODE kernels, while preserving the spike-driven efficiency of SNNs (He et al., 11 Jun 2026). In that sense, LongSpike can be viewed as a non-Markovian extension of earlier spiking SSMs: SpikingSSM introduced sparse, parallel spiking state-space learning (Shen et al., 2024), and LongSpike modifies the state dynamics themselves to encode power-law memory (He et al., 11 Jun 2026).

The reported limitations are also specific. The paper notes the current lack of deployment on neuromorphic chips, such as Intel Loihi, and a slight constant overhead in kernel construction (He et al., 11 Jun 2026). Future directions named in the work include adaptive $\alpha\in(0,1)$2, variable-order calculus $\alpha\in(0,1)$3, hardware-friendly sparse SOE, and fractional dynamics in graph or attention layers (He et al., 11 Jun 2026). The availability of code at the project repository was also noted in the original report (He et al., 11 Jun 2026).

Taken together, LongSpike occupies a distinct position in the literature on long-sequence SNNs. It is neither a purely neuronal modification, such as two-compartment memory or reset-refractory dynamics, nor merely a sparse replacement of analog activations. Its defining feature is the insertion of fractional-calculus memory into a spiking state-space backbone, with a training formulation designed to remain compatible with modern parallel accelerators (He et al., 11 Jun 2026).