--- title: Spiking Self-Attention (SSA) in SNNs url: https://www.emergentmind.com/topics/spiking-self-attention-ssa type: topic --- # Spiking Self-Attention (SSA) in SNNs Spiking Self-Attention (SSA) is a class of attention mechanisms specifically designed for spiking neural networks (SNNs), integrating the sparsity and event-driven computation paradigms of SNNs with the long-range dependency modeling capabilities of transformer self-attention. SSA, as exemplified by Spikformer and its successors, replaces the floating-point, softmax-normalized, quadratic-complexity operations of classical self-attention with sparse, non-multiplicative, and biologically inspired spike-based computations. This yields both high energy efficiency and the ability to transfer transformer-like architectures to static and event-based data domains, with growing adoption across vision, language, and graph applications [2209.15425][2401.02020][2505.14719][2403.15480][2510.00133]. ## 1. Mathematical Formulation and Core Mechanism SSA operates by encoding input tokens as spike trains and using Leaky Integrate-and-Fire (LIF) neurons to project these spikes into query, key, and value representations: \[ Q = \mathrm{SN}(\mathrm{BN}(X W_Q)), \quad K = \mathrm{SN}(\mathrm{BN}(X W_K)), \quad V = \mathrm{SN}(\mathrm{BN}(X W_V)) \] where $X \in \{0,1\}^{T \times N \times D}$ is the input spike tensor, $W_Q, W_K, W_V$ are linear projection matrices, BN denotes batch normalization, and SN is a spiking neuron operator (LIF dynamics) [2209.15425][2505.14719]. The attention map is computed by binary AND and accumulation: \[ A = Q K^\top \] This results in a sparse, integer-valued $A$ without softmax. The output is generated by weighing $V$ with $A$, possibly scaling, and applying a spiking neuron nonlinearity: \[ \text{SSA}(Q, K, V) = \mathrm{SN}(A \otimes V \cdot s) \] where $\otimes$ denotes element-wise multiplication and $s$ is a scale factor. The absence of softmax, exponentials, and floating-point multiplication makes the mechanism event-driven and energy efficient [2209.15425][2401.02020][2505.14719]. ## 2. Structural Variants and Architectural Integration SSA is implemented in a hierarchical fashion in spiking transformers. The canonical Spikformer block applies SSA per time step independently, feeding its output into a spike-domain MLP and then through residual connections and normalization [2209.15425][2401.02020]. In MSViT, a hybrid approach is used: early stages employ Multi-Scale Spiking Self-Attention (MSSA) for linear-complexity multi-scale aggregation, while deeper blocks revert to quadratic-complexity SSA to maximize abstraction power [2505.14719]. Several enhancements appear in recent architectures: - **Multi-Scale (MSSA):** Fuses low-level and high-level features via column sums, reducing complexity to $O(ND)$ [2505.14719]. - **Spatial-Temporal (STAtten):** Introduces block-wise computation to integrate local spatio-temporal dependencies at unchanged complexity [2409.19764]. - **Saccadic Spike Attention (SSSA):** Models spatial relevance using distributional similarity metrics robust to variable spike sparsity and introduces learnable, temporally aggregating “saccadic” neurons to achieve linear time/space complexity [2502.12677]. SSA forms the computational backbone in various modalities: vision (Spikformer, MSViT, Spikformer V2, SNN-ViT), language models (NeurTransformer), and graph transformers (SpikeGraphormer, with adapted SSA modules). ## 3. Comparison With Classical Self-Attention SSA fundamentally departs from analog self-attention (ASA) in several respects: - **No Softmax Normalization:** SSA operates on nonnegative, often binary, integers without normalization, leveraging the event-driven nature of spikes [2209.15425][2505.14719]. - **Sparsity:** Q, K, and V are spikes; thus, computation is predominantly zero-skipping. This introduces significant memory and energy savings [2401.02020][2510.00133]. - **Masking and Locality:** Many variants (e.g., MSSA, SSSA) avoid construction of full $N \times N$ attention matrices, reducing memory and latency [2505.14719][2502.12677][2403.15480]. - **Spectral Properties:** SSA acts as a high-pass filter in the frequency domain, emphasizing high-frequency, event-driven content typical of neuromorphic data, unlike the low-pass bias of classic ViTs [2510.12102]. - **Computation:** SSA replaces matrix multiplication and floating-point additions with binary logical ops and integer counting or addition on event presence, yielding linear or near-linear complexity in advanced versions [2403.15480][2502.12677]. ## 4. Performance, Complexity, and Energy Benefits SSA-based spiking transformers are empirically validated across vision and language tasks. The table below summarizes key efficiency and accuracy results: | Model | Params (M) | Top-1 Acc (ImageNet) | Energy (mJ) | Complexity | |----------------------|------------|----------------------|-------------|-------------| | DeiT-B (ANN, float) | 86.6 | 81.80% | 254.84 | $O(N^2D)$ | | Spikformer-8-768 | 66.3 | 74.81% | 20.00 | $O(N^2D)$ | | MSViT-10-768 | 69.8 | 85.06% | 45.88 | $O(ND)$/$O(N^2D)$| | SNN-ViT-8-512 | 54 | 80.23% | 35.75 | $O(ND)$ | SSA yields up to 130$\times$ per-layer energy reduction over softmax-VSA analogs [2401.02020]. NeurTransformer demonstrates 64.71%-85.28% energy reduction in large language models at modest cost in perplexity and accuracy [2510.00133]. Adaptive and multi-scale schemes further reduce both computation and hardware cost via token pruning and linear attention [2505.14719][2502.12677][2508.14138][2403.15480]. ## 5. Extensions: Spatio-Temporal, Saccadic, and Graph SSA SSA mechanisms have been generalized to capture richer dependencies: - **Spatio-Temporal SSA:** Modules such as DISTA and STAtten employ either learnable membrane time constants (intrinsic attention) or explicit block-wise spatio-temporal correlation to integrate spikes across time and space, providing multi-scale temporal memory and denoising [2311.09376][2409.19764]. - **Saccadic SSA:** SSSA replaces unreliable dot-products with distribution-based (cross-entropy) similarity and gates value tokens using a temporally aggregating saccadic module, achieving linear complexity [2502.12677]. - **Graph SSA (SGA):** SSA is reformulated for graphs by replacing $N^2$ interactions with per-channel, per-node sparse masks, reducing both computation and GPU memory (10-20$\times$ reduction) [2403.15480]. ## 6. Implementation, Hardware, and Training Considerations SSA is realized using LIF neuron dynamics for Q, K, V projections, with surrogate gradients for backpropagation through the spiking nonlinearity [2209.15425][2505.14719][2401.02020]. Column and row-wise operations in MSSA and SSSA are highly amenable to event-driven neuromorphic hardware architectures (e.g., Loihi), minimizing routing and storage demands [2505.14719][2403.15480]. Key practical aspects include: - **Surrogate Gradients:** Piecewise-linear or exponential approximations are necessary for $\partial \theta/\partial H$ to ensure trainability. - **Temporal Token Stability:** For temporal pruning/adaptive halting (STAS), architectures require integrated patch splitting to maintain token similarity across timesteps [2508.14138]. - **Fusion Strategies:** Multi-scale and saccadic variants fuse low- and high-level information; ablation studies identify the optimal design for context retention [2505.14719][2502.12677]. - **Energy Estimation:** SOPs (synaptic operations) are costed at 0.9 pJ, compared to 4.6 pJ for MACs in ANN attention; implementations measure firing rates for hardware energy profiling [2505.14719][2510.00133][2403.15480]. ## 7. Limitations and Research Directions SSA—while markedly efficient—has several important limitations: - **Loss of Analog Softmax:** Lack of normalization can yield unstable or noisy activations, especially on non-spiking or low-frequency data [2510.12102]. - **Limited Expressiveness:** Binary or ternary spikes reduce representational power; hybrid schemes (e.g., A$^2$OS$^2$A with partial ReLU/ternary neurons) partially address this [2503.00226]. - **Sensitivity to Spike Noise:** High-pass filter tendencies can amplify spurious events; band-pass or pooling modifications may be required for real-world event streams [2510.12102]. - **Training Instabilities:** Surrogate gradient shapes, thresholds, and firing rate tuning present open challenges for convergence stability [2505.14719][2311.09376]. - **Quadratic Bottleneck:** While linear-complexity variants (MSSA, SSSA, SGA) exist, deep networks or late-stage layers may retain $O(N^2)$ behavior for full expressiveness [2505.14719][2403.15480][2502.12677]. Current research explores adaptive surrogate learning, further reduction in inference timesteps, and extension beyond classification into dense prediction (detection, segmentation), as well as the biological plausibility and neuromorphic alignment of advanced attention structures [2505.14719][2508.14138]. --- **References:** [2209.15425] Spikformer: When Spiking Neural Network Meets Transformer [2401.02020] Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticket [2505.14719] MSVIT: Improving Spiking Vision Transformer Using Multi-scale Attention Fusion [2502.12677] Spiking Vision Transformer with Saccadic Attention [2510.12102] SpikePool: Event-driven Spiking Transformer with Pooling Attention [2409.19764] Spiking Transformer with Spatial-Temporal Attention [2503.00226] Spiking Transformer: Introducing Accurate Addition-Only Spiking Self-Attention for Transformer [2403.15480] SpikeGraphormer: A High-Performance Graph Transformer with Spiking Graph Attention [2311.09376] DISTA: Denoising Spiking Transformer with intrinsic plasticity and spatiotemporal attention [2510.00133] Large Language Models Inference Engines based on Spiking Neural Networks [2508.14138] STAS: Spatio-Temporal Adaptive Computation Time for Spiking Transformers Source: https://www.emergentmind.com/topics/spiking-self-attention-ssa