LongSpike: Fractional Order Spiking State Space Models for Efficient Long Sequence Learning

Abstract: Spiking Neural Networks (SNNs) are well-regarded for their biological plausibility and energy efficiency in processing sequential data. However, dominant SNN architectures typically rely on first-order Ordinary Differential Equations (ODEs) to govern neuronal state transitions. This first-order assumption imposes a "memoryless" bottleneck, limiting the model's capacity to capture the complex, long-range dependencies inherent in long-sequence tasks. In this work, we propose LongSpike, a novel SNN framework that integrates fractional-order State-Space Modeling, or f-SSM, from control theory into the spiking domain. By extending traditional integer-order SSMs to the fractional-calculus regime, LongSpike enables the hierarchical integration of neuronal dynamics with long-memory kernels. To mitigate the computational overhead and parallelization challenges typically associated with fractional operators, we leverage a state-space formulation that supports efficient, parallel training. Empirical evaluations on challenging benchmarks, including Long Range Arena (LRA), large-scale WikiText-103, and Speech Commands, demonstrate that LongSpike outperforms state-of-the-art SNNs in accuracy while preserving sparse synaptic computation. The code is available at https://github.com/xinruihe389-commits/LongSpike.

• The paper introduces LongSpike, a framework using fractional calculus to overcome the memory limitations of conventional first-order spiking models.
• The paper employs a sum-of-exponentials approximation to make fractional derivatives computationally efficient and scalable on GPUs.
• The paper demonstrates improved performance on benchmarks by achieving long-range dependency learning with reduced spiking activity and enhanced energy efficiency.

LongSpike: Fractional Order Spiking State Space Models for Efficient Long Sequence Learning

Introduction and Motivation

Long-range sequence modeling remains a central challenge across domains such as language modeling, speech recognition, and time-series analysis, due to the need for architectures capable of retaining information over extended horizons. Spiking Neural Networks (SNNs) are lauded for their event-driven computation and biological plausibility, leading to substantial gains in energy efficiency. However, conventional SNNs—and even recently proposed Spiking State Space Models (SpikingSSMs)—are fundamentally constrained by first-order dynamics, which enforce implicit Markovianity and a memoryless regime, inhibiting their ability to exploit long-range dependencies.

This work proposes LongSpike, a novel SNN framework that leverages fractional-order State Space Modeling (f-SSM) to replace the standard first-order updates with dynamics governed by fractional calculus, thereby introducing power-law memory kernels with theoretical long-memory properties. This enables the system to retain information with heavy-tailed, algebraic decay, in contrast to the conventional fast-exponential relaxation. LongSpike addresses the traditional inefficiencies associated with fractional operators by introducing a sum-of-exponentials (SOE) parameterization, enabling highly efficient parallel training and inference.

Fractional State Space Modeling in Spiking Networks

Traditional state space models (SSMs) use integer-order ODEs, resulting in impulse responses that decay exponentially. By contrast, fractional-order differential equations (FDEs)—particularly in the Caputo sense—imbue the system with non-local, history-dependent dynamics. Specifically, a fractional derivative of order $\alpha \in (0,1)$ at time $t$ depends on the entire history of the input, weighted by a kernel with power-law decay. This distinguishes the memory characteristics of the system, crucial in long-context learning where input influence must persist far into the future.

The core innovation in LongSpike is the direct parametrization of the SSM's state evolution by a fractional differential equation:

$$D^\alpha h(t) = A h(t) + B x(t), \quad y(t) = C h(t),$$

where $D^\alpha$ is the Caputo fractional derivative. The Volterra integral form shows that the state is given by a continuum of historical influences, which, when solved via a sum-of-exponentials approximation, decomposes the memory kernel into $M$ exponential timescales. This mapping enables computational tractability and GPU compatibility.

These fractional-state traces drive the input to Leaky Integrate-and-Fire (LIF) spiking neurons, realizing a stacking of fractional-order temporal dynamics and event-driven computation. To circumvent inefficiencies in standard LIF simulation, a convolutional Surrogate Dynamic Network (SDN) is employed for direct, parallel prediction of the spike train.

Theoretical Properties and Memory Dynamics

Fractional SSMs introduce fundamentally distinct memory properties compared to integer-order SSMs. The solution to a fractional SSM with constant input diverges from the simple exponential relaxation and is characterized by Mittag-Leffler relaxation:

$$h(t) = \frac{x_c}{\lambda} - \frac{x_c}{\lambda} E_\alpha(-\lambda t^\alpha),$$

where $E_\alpha(\cdot)$ is the Mittag-Leffler function. For $\alpha=1$ this recovers the exponential case, but for $\alpha<1$ the decay is algebraic, meaning the influence of remote history decays polynomially rather than exponentially. This creates a pronounced heavy-tailed memory effect, a property absent in standard spiking and SSM architectures.

Empirical Evaluation

LongSpike was extensively evaluated on standard benchmarks including Long Range Arena (LRA), WikiText-103, and Speech Commands. Across these tasks, LongSpike achieves superior or competitive accuracy compared to both spiking and non-spiking SSM baselines, while maintaining the hallmark sparse activity of SNNs.

On LRA, LongSpike achieved the highest average accuracy among spiking models, demonstrating clear improvements in tasks requiring extended temporal credit assignment. Notably, in the TEXT task, LongSpike substantially increased classification accuracy to 88.19%, exceeding the SpikingSSM's 80.41%. Similar improvements were also evidenced on the RETRIEVAL and PATH-X tasks.

Energy efficiency analysis reveals that these gains were NOT realized by a significant increase in overall neural activity. On the LRA TEXT task, LongSpike exhibited an average spiking rate of 6.46%, lower than SpikingSSM's 6.75%.

Figure 1: Layer-wise spiking rates on the TEXT task of the LRA benchmark. LongSpike maintains a comparable or lower average spiking rate (6.46%) than SpikingSSM (6.75%), indicating that performance improvements do not rely on increased neural activity.

For WikiText-103, LongSpike outperformed all prior SNNs and showed further reduction in perplexity, thus narrowing the gap with ANN-based models significantly. On Speech Commands and neuromorphic datasets, LongSpike set new best or competitive results for SNN models, validating the practical benefit of the fractional-order approach.

Computational Efficiency and Scalability

By utilizing the sum-of-exponentials approximation, LongSpike inherits the efficient $O(L \log L)$ convolutional computation regime of modern SSMs such as S4 and Mamba, and does so without introducing prohibitive overhead. For typical configurations with $t$0 exponential terms, the memory and time cost increase is modest and parallelizable, supporting practical scaling to large sequence lengths and extensive parameter sizes.

Importantly, the parallelizable state space structure and non-sequential spike generation pipeline make LongSpike directly compatible with GPU acceleration. A full spiking LIF block is replaced with a convolutional surrogate for the spike output, thus achieving $t$1 parallel depth and eliminating the need for sequential simulation even for long sequence lengths. The framework is inherently suitable for hardware-efficient inference and neuromorphic deployment.

Implications and Future Directions

By bridging fractional calculus and SNN sequence modeling, LongSpike provides a new mechanism for integrating long-memory effects into event-driven neural models. Theoretical and empirical results collectively indicate that non-Markovian, power-law temporal dynamics confers distinct benefits in sequence tasks with dependencies extending over large timescales. The methodology is broadly extensible: fractional kernels could be introduced into other domains (e.g., attention mechanisms, graph models, or diffusion processes) wherever historical context is of operational importance.

The compatibility of LongSpike with neuromorphic hardware systems is direct, owing to its reliance on spike sparsity and recurrent updates over dense matrix multiplications. Future work includes deployment and benchmarking on next-generation neuromorphic processors, as well as advanced training pipelines that optimize the number of exponential terms or leverage learnable fractional orders.

On the theoretical front, fractional-order modeling invites the integration of more general, potentially variable-order kernels, and offers a unified perspective to relate sequence model expressivity to the order and structure of memory kernels. Connections with self-similar processes, anomalous diffusion, and non-local neural computations may also be explored.

Conclusion

LongSpike introduces a scalable, efficient, and memory-augmented spiking state space model by embedding fractional-order dynamics at the network level. The framework breaks the Markovian constraint inherent in first-order SNNs and SSMs, achieving robust long-range sequence modeling while retaining the energy-efficient, sparse computation regime critical for large-scale and hardware-aware deployment. The demonstrated performance across diverse benchmarks and the fidelity of energy-efficient computation underpin LongSpike's position as a principled advancement in sequence modeling architectures and spiking neural computation (2606.12895).