---
title: 'LongSpike: Fractional SNN Framework'
url: https://www.emergentmind.com/topics/longspike
type: topic
---

# LongSpike: Fractional SNN Framework

LongSpike is a spiking neural network framework for efficient long-sequence learning that integrates fractional-order State-Space Modeling, or f-SSM, from control theory into the spiking domain [2606.12895]. It replaces the usual first-order Markovian state transition with a Caputo fractional-order system, thereby introducing non-Markovian, power-law memory into a sparse spiking architecture. The model combines a fractional state-space block, a discrete-time leaky integrate-and-fire neuron, and a Surrogate Dynamic Network (SDN) that supports efficient, parallel training on long contexts. Empirical evaluations were reported on Long Range Arena (LRA), WikiText-103, and Speech Commands, where LongSpike outperformed prior spiking state-space baselines while preserving sparse synaptic computation [2606.12895].

## 1. Research context and motivation

LongSpike emerged within a line of work that adapts structured state-space models to spiking computation for long sequences. SpikingSSM developed spiking state space models by hierarchically integrating neuronal dynamics with an SSM block, realizing sparse synaptic computation and using a light-weight surrogate dynamic network to predict the after-reset membrane potential; on LRA it reached an average accuracy of approximately \(84.33\%\) at a \(10\%\) spiking rate, and on WikiText-103 it reported perplexity \(33.94\) with \(75\)M parameters [2408.14909]. SPikE-SSM addressed similar constraints through Parallel Max–Min Boundary Compression (PMBC), a reset-refractory neuron model, and trainable thresholds and refractory magnitudes; it reported \(84.18\%\) average accuracy on LRA at an overall spiking rate of \(\sim 8\%\), and perplexity \(33.18\) on WikiText-103 with a \(24.5\%\) spike rate [2410.17268].

The specific motivation for LongSpike is the limitation that dominant SNN architectures typically rely on first-order Ordinary Differential Equations to govern neuronal state transitions. In the LongSpike formulation, this first-order assumption imposes a “memoryless” bottleneck that limits the capacity to capture the complex, long-range dependencies inherent in long-sequence tasks [2606.12895]. This emphasis on richer temporal state is consistent with broader efforts in SNNs to move beyond single-compartment memory mechanisms; for example, LSTM-LIF introduced somatic and dendritic compartments tailored to retain short- and long-term memories and provided a vanishing-gradient analysis for that design [2307.07231]. LongSpike’s distinctive contribution is to import fractional-order dynamics into a spiking state-space architecture while retaining parallel sequence processing [2606.12895].

## 2. Fractional-order state-space formulation

LongSpike replaces the standard continuous-time state-space system
$$
\frac{d h(t)}{dt} = A\,h(t) + B\,x(t), \qquad y(t)=C\,h(t),
$$
with the Caputo fractional-order system of order \(\alpha\in(0,1)\):
$$
D^\alpha h(t) = A\,h(t) + B\,x(t), \qquad y(t)=C\,h(t),
$$
where \(x(t)\) is the input, \(h(t)\) the latent state, and \(y(t)\) the output [2606.12895]. The Caputo derivative is defined as
$$
D^\alpha y(t)=\frac{1}{\Gamma(1-\alpha)}\int_0^t (t-\tau)^{-\alpha} y'(\tau)\,d\tau.
$$
Its Volterra-integral equivalent is
$$
h(t)=\int_0^t k_\alpha(t-\tau)\,[A\,h(\tau)+B\,x(\tau)]\,d\tau,
\qquad
k_\alpha(t)=\frac{t^{\alpha-1}}{\Gamma(\alpha)}.
$$

This kernel introduces long-memory behavior through a power-law history term. Direct history accumulation is \(O(T^2)\), so LongSpike uses the diffusive representation
$$
k_\alpha(t)=\frac{t^{\alpha-1}}{\Gamma(\alpha)}
=\frac{\sin \pi \alpha}{\pi}\int_0^\infty e^{-\omega t}\,\omega^{-\alpha}\,d\omega,
$$
and truncates it with a quadrature-based sum-of-exponentials (SOE) approximation:
$$
k_\alpha(t)\approx \sum_{i=1}^M \eta_i e^{-\omega_i t}.
$$
With auxiliary states
$$
z_i(t)=\int_0^t e^{-\omega_i(t-\tau)}g(\tau)\,d\tau,
\qquad
g(\tau)=A\,h(\tau)+B\,x(\tau),
$$
the model obtains \(M\) coupled first-order ODEs,
$$
\dot z_i(t)=-\omega_i z_i(t)+g(t), \qquad z_i(0)=0,
$$
together with
$$
h(t)\approx \sum_{i=1}^M \eta_i z_i(t), \qquad y(t)=C\,h(t).
$$

After discretization with step \(\Delta\), the recurrence becomes
$$
g[k]=A\,h[k-1]+B\,x[k],
$$
$$
z_i[k]=a_i\,z_i[k-1]+b_i\,g[k],
$$
$$
h[k]=\sum_{i=1}^M \eta_i z_i[k], \qquad y[k]=C\,h[k],
$$
with
$$
a_i=e^{-\omega_i\Delta}, \qquad b_i=\frac{1-e^{-\omega_i\Delta}}{\omega_i}.
$$
Because each \(z_i\) evolves linearly, the block can be written as a global convolution
$$
y = x * \left(\sum_i \eta_i k_i\right),
\qquad
k_i[\ell]=C\,A^{\ell-1}B * (b_i a_i^{\ell-1}), \quad \ell=1\ldots L,
$$
and computed in \(O(L\log L)\) via FFT [2606.12895]. This formulation is central to LongSpike’s claim that fractional operators can be made tractable and parallel in an SNN setting.

## 3. Spiking dynamics and block architecture

The current output \(y[k]\) from the fractional SSM drives a discrete-time leaky integrate-and-fire neuron:
$$
\bar u[k]=\beta\,u[k-1]+y[k],
$$
$$
s[k]=H(\bar u[k]-v_{th}),
$$
$$
u[k]=\bar u[k]-s[k]\,v_{th},
$$
where \(H\) is the Heaviside step, \(\beta\in(0,1)\) is the membrane decay, \(v_{th}\) is a learnable threshold, and soft-reset is used [2606.12895]. The binary spike train \(s[k]\) provides the sparse event-driven output of the block.

To avoid sequential BPTT through \(T\) steps, LongSpike uses a small Surrogate Dynamic Network, typically a causal CNN, that learns a nonautoregressive mapping
$$
s_{1:L} = SDN(y_{1:L}).
$$
The surrogate derivative is piecewise quadratic,
$$
g'(x)=\max(0,\,1-\alpha |x|)\cdot \alpha,
$$
which the paper describes as ensuring stable gradient flow near threshold [2606.12895].

A single LongSpike block is organized as follows. First, it computes
$$
g[k]=A\,h[k-1]+B\,x[k].
$$
Second, for \(i=1\ldots M\), it convolves \(g\) with the exponential kernel
$$
z_i = g * k_i, \qquad k_i[\ell]=b_i a_i^{\ell-1}.
$$
Third, it aggregates
$$
h[k]=\sum_i \eta_i z_i[k], \qquad y[k]=C\,h[k].
$$
Fourth, it predicts spikes \(s[1:L]=SDN(y[1:L])\) and updates the membrane via the soft-reset rule [2606.12895]. The architecture therefore hierarchically integrates neuronal dynamics with long-memory kernels, extending the hierarchical SSM-plus-spike pattern introduced in earlier spiking SSMs [2408.14909].

## 4. Training procedure, parallelism, and computational cost

LongSpike is trained end-to-end by BPTT over the surrogate spiking network and the linear f-SSM [2606.12895]. Gradients through the f-SSM block reduce to gradients through convolutions and are computed via standard FFT backprop. The parameters \(\{A,B,C\}\), the SOE nodes \(\{\omega_i,\eta_i\}\) optionally, and the SDN weights are updated by Adam. Weight decay, dropout, and layer- or batch-norm are applied as in standard SSM literature [2606.12895].

The computational advantage comes from recasting the fractional dynamics into linear time-invariant convolutions. In the reported formulation, steps \(1\)–\(3\) of the block are cast as FFTs or parallel scanned recurrences, achieving \(O(L\log L)\) time [2606.12895]. Relative to SpikingSSM’s forward cost \(O(HNL + BHL\log L)\), LongSpike adds \(O(MHNL)\) to build the SOE kernels but leaves the FFT convolution unchanged. With \(M=2\), this is described as a small constant-factor overhead [2606.12895].

The memory trade-off is explicit. The vectorized implementation uses \(O(MHNL)\) memory, while a loop-based implementation uses \(O(HNL)\) memory [2606.12895]. On Speech Commands, adding \(M=2\) increased per-epoch time by approximately \(3.8\%\) but reduced GPU memory by approximately \(10\%\) through kernel-fusion optimizations [2606.12895]. This suggests that the additional burden of fractional memory was operationally modest in the reported setting.

## 5. Empirical performance

LongSpike was evaluated with \(M=2\) terms on LRA, WikiText-103, and Speech Commands, and compared directly to SpikingSSM [2606.12895].

| Benchmark | LongSpike | SpikingSSM |
|---|---|---|
| LRA six-task average | \(86.33\) | \(84.33\) |
| WikiText-103 | \(75\)M params, perplexity \(32.31\), spiking rate \(27.05\%\) | \(75\)M params, perplexity \(33.94\), spiking rate \(26.40\%\) |
| Speech Commands | \(305\)K params, accuracy \(96.31\%\) | \(305\)K params, accuracy \(96.09\%\) |

On the six LRA tasks, the reported per-task results for LongSpike were: ListOps \(60.95\), Text \(88.19\), Retrieval \(89.82\), Image \(89.28\), Path \(94.31\), and Path-X \(95.41\). The corresponding SpikingSSM numbers were \(60.23\), \(80.41\), \(88.77\), \(88.21\), \(93.51\), and \(94.82\) [2606.12895]. On language modeling, LongSpike improved perplexity over SpikingSSM while keeping the parameter count fixed at \(75\)M; on audio classification, it improved Speech Commands accuracy from \(96.09\%\) to \(96.31\%\) at the same \(305\)K parameter scale [2606.12895].

The paper summarizes these results by stating that LongSpike outperforms state-of-the-art SNNs in accuracy while preserving sparse synaptic computation [2606.12895]. A plausible implication is that the additional fractional-memory mechanism contributed more to long-context modeling than it cost in constant-factor overhead, at least in the \(M=2\) regime used in the experiments.

## 6. Interpretation, limitations, and outlook

LongSpike’s central interpretive claim is that fractional-order dynamics with \(\alpha<1\) can be made tractable and parallel for SNNs by SOE approximations and convolution-based state-space formulations [2606.12895]. The resulting heavy-tailed (Mittag–Leffler) impulse response is reported to capture long-range dependencies far beyond first-order ODE kernels, while preserving the spike-driven efficiency of SNNs [2606.12895]. In that sense, LongSpike can be viewed as a non-Markovian extension of earlier spiking SSMs: SpikingSSM introduced sparse, parallel spiking state-space learning [2408.14909], and LongSpike modifies the state dynamics themselves to encode power-law memory [2606.12895].

The reported limitations are also specific. The paper notes the current lack of deployment on neuromorphic chips, such as Intel Loihi, and a slight constant overhead in kernel construction [2606.12895]. Future directions named in the work include adaptive \(M\), variable-order calculus \(\alpha(t)\), hardware-friendly sparse SOE, and fractional dynamics in graph or attention layers [2606.12895]. The availability of code at the project repository was also noted in the original report [2606.12895].

Taken together, LongSpike occupies a distinct position in the literature on long-sequence SNNs. It is neither a purely neuronal modification, such as two-compartment memory or reset-refractory dynamics, nor merely a sparse replacement of analog activations. Its defining feature is the insertion of fractional-calculus memory into a spiking state-space backbone, with a training formulation designed to remain compatible with modern parallel accelerators [2606.12895].

Source: https://www.emergentmind.com/topics/longspike