Generalizable, real-time neural decoding with hybrid state-space models (2506.05320v1)

Abstract: Real-time decoding of neural activity is central to neuroscience and neurotechnology applications, from closed-loop experiments to brain-computer interfaces, where models are subject to strict latency constraints. Traditional methods, including simple recurrent neural networks, are fast and lightweight but often struggle to generalize to unseen data. In contrast, recent Transformer-based approaches leverage large-scale pretraining for strong generalization performance, but typically have much larger computational requirements and are not always suitable for low-resource or real-time settings. To address these shortcomings, we present POSSM, a novel hybrid architecture that combines individual spike tokenization via a cross-attention module with a recurrent state-space model (SSM) backbone to enable (1) fast and causal online prediction on neural activity and (2) efficient generalization to new sessions, individuals, and tasks through multi-dataset pretraining. We evaluate POSSM's decoding performance and inference speed on intracortical decoding of monkey motor tasks, and show that it extends to clinical applications, namely handwriting and speech decoding in human subjects. Notably, we demonstrate that pretraining on monkey motor-cortical recordings improves decoding performance on the human handwriting task, highlighting the exciting potential for cross-species transfer. In all of these tasks, we find that POSSM achieves decoding accuracy comparable to state-of-the-art Transformers, at a fraction of the inference cost (up to 9x faster on GPU). These results suggest that hybrid SSMs are a promising approach to bridging the gap between accuracy, inference speed, and generalization when training neural decoders for real-time, closed-loop applications.

• The paper introduces POSSM, a hybrid model combining a cross-attention module for spike tokenization with a recurrent state-space backbone for high-speed, millisecond-level decoding.
• The paper demonstrates that POSSM achieves accuracy comparable to Transformer models while operating up to nine times faster on GPUs in both motor and clinical decoding tasks.
• The paper reveals significant cross-species transfer, showing that pretraining on monkey recordings enhances decoding performance in human tasks like handwriting and speech.

Generalizable, Real-time Neural Decoding with Hybrid State-space Models

The paper introduces POSSM, a pioneering framework for neural decoding designed to balance accuracy, inference speed, and generalization capabilities. POSSM addresses the demands of real-time applications such as closed-loop neuroscience experiments and brain-computer interfaces (BCIs), where latency constraints are critical. Traditional neural decoding models, including recurrent neural networks (RNNs), excel in speed but lack generalization to unseen data, while Transformer-based approaches offer strong generalization due to large-scale pretraining but are computationally intensive and not suited for low-resource or real-time environments.

Key Contributions

1. Hybrid Architecture: POSSM combines a cross-attention module for spike tokenization with a recurrent state-space model (SSM) backbone. This fusion enables millisecond-level resolution and throughput, crucial for real-time decoding. A significant innovation is the tokenization of individual spikes, allowing the model to handle variable-length sequences without reliance on time-binned inputs. This enhances flexibility and adaptability across diverse sessions, subjects, and tasks.
2. Performance Efficiency: POSSM is evaluated against established models on non-human primates performing motor tasks and clinical applications, such as human handwriting and speech decoding. It achieves decoding accuracy comparable to state-of-the-art Transformers but at reduced computational costs—up to nine times faster on a GPU. This achieves the delicate balance needed for practical deployment in online BCI systems.
3. Cross-species Transfer: Notably, POSSM demonstrates significant cross-species transfer learning capabilities. When pretrained on monkey motor-cortical recordings, it improves decoding performance in human tasks like handwriting. This suggests the potential for leveraging extensive data from animal studies to enhance performance in human clinical applications, a considerable advantage given the limitations in collecting large-scale human electrophysiology datasets.

Implications and Future Directions

The introduction of POSSM represents a significant advancement in neural decoding, particularly in applications requiring real-time interaction. The hybrid architecture's ability to manage individual spikes and leverage flexible tokenization supports efficient model adaptation and transfer across different species and tasks. These outcomes suggest broad applications in clinical settings, where closed-loop control systems could benefit from lower latency and enhanced accuracy.

Future work could explore POSSM's application across additional neural data modalities—such as EEG or calcium imaging—potentially leading to a universal model for neural decoding. There's also room to investigate more sophisticated self-supervised pretraining frameworks, aiming to further enhance generalization capabilities without reliance on task-specific annotations.

In summary, POSSM is a promising approach that merges state-of-the-art techniques from machine learning, offering a robust solution to the challenges inherent in neural decoding for real-time, closed-loop applications. It exemplifies how hybrid models can bridge the gap between computational efficiency and effective generalization, opening new avenues for research and development in AI-driven neurotechnologies.