Accurate Addition-Only Spiking Self-Attention

• Accurate Addition-Only Spiking Self-Attention (A²OS²A) is a neural mechanism that replaces resource-intensive multiplications with addition-only spiking operations to enhance energy and memory efficiency.
• It leverages a hybrid spiking scheme with binary, ternary, and full-precision representations to maintain competitive accuracy while simplifying the computation.
• The approach reduces computational complexity from quadratic to linear and has been empirically validated in graph, vision, and language applications on neuromorphic hardware.

Accurate Addition-Only Spiking Self-Attention (A$^2$OS$^2$A) is a neural attention mechanism that eliminates all multiplicative operations from self-attention in Transformer and Graph Transformer architectures, leveraging event-driven spiking computations for energy and memory efficiency. By replacing conventional dot-product and softmax-based attention with addition-only operations on (hybrid) spiking activations, A$^2$OS$^2$A enables integration into spiking neural networks (SNNs) and supports scalable deployment on neuromorphic hardware. Two recent independent lines of work, "SpikeGraphormer: A High-Performance Graph Transformer with Spiking Graph Attention" (Sun et al., 2024) and "Spiking Transformer: Introducing Accurate Addition-Only Spiking Self-Attention for Transformer" (Guo et al., 28 Feb 2025), have formalized and demonstrated A$^2$OS$^2$A in graph and vision/language contexts.

1. Motivation and Foundations

Self-attention is the core computational primitive underlying the Transformer architecture, but is dominated by energy- and memory-intensive floating-point multiplications (matrix multiplications and softmax normalization). In large-scale graphs or high-resolution vision problems, standard self-attention scales as $O(n^2d)$ in both time and space (for $n$ tokens/nodes and $d$ dimensions), and is poorly suited to event-driven hardware. The A$^2$OS$^2$0A mechanism reimagines self-attention: all matrix multiply–accumulate and softmax operations are replaced with sparse, addition-only interactions between spiking activations—primarily binary or ternary, produced by Leaky Integrate-and-Fire (LIF) neurons.

A$^2$1OS$^2$2A is motivated by the need to:

• Reduce computational and energy complexity from quadratic to linear in sequence or node count
• Eliminate hardware-intensive multipliers and exponential functions
• Retain competitive representational power and accuracy via hybrid precision (binary/ternary/real) representations

This approach enables practical, large-scale graph and sequence processing on SNNs and is highly compatible with neuromorphic hardware that excels at logic and addition-based operations (Sun et al., 2024, Guo et al., 28 Feb 2025).

2. Spiking Encoding and Hybrid Neuron Representation

A$^2$3OS$^2$4A replaces the conventional floating-point Q/K/V projections with spike-encoded features, leveraging both binary, ternary, and full-precision non-negative representations:

• Binary Q ("query"): $^2$5; outputs $^2$6 via a binary LIF neuron.
• Full-precision non-negative K ("key"): $^2$7; full-precision via ReLU.
• Ternary V ("value"): $^2$8; outputs $^2$9 via a ternary LIF neuron.

Here, $^2$0 (binary LIF) and $^2$1 (ternary LIF) define spiking neuron dynamics with discrete output sets, Heaviside thresholding in the forward pass, and surrogate gradients during backpropagation for optimization stability: $^2$2 This hybrid scheme mitigates the loss of representational entropy otherwise incurred by pure binary SNN attention—e.g., full-precision tensors $^2$3 have $^2$4 bits of capacity versus $^2$5 for binary; ternarizing $^2$6 and retaining ReLU for $^2$7 recovers much of this capacity loss (Guo et al., 28 Feb 2025).

3. Addition-Only Spiking Self-Attention: Algorithmic Details

The A$^2$8OS$^2$9A mechanism replaces the attention computation

$^2$0

with addition-only operations, eliminating all multiplies, exponentials, and divisions.

3.1 Matrix-Free Spiking Attention

For graph attention (Sun et al., 2024):

1. Encoding: Project features into spike-encoded $^2$1 via LIF neurons over $^2$2 timesteps.
2. Mask construction: For each node-pair $^2$3 and channel $^2$4,

$^2$5

Aggregated as $^2$6.

1. Masked summation: Attention for query node $^2$7,

$^2$8

Only integer additions, binary ANDs, and thresholding are used.

For sequence attention (Guo et al., 28 Feb 2025):

2. Compute $^2$9: For binary $^2$0 and non-negative $^2$1,

$^2$2

This reduces to addition of selected rows of $^2$3.

1. Weighted value sum: $^2$4; summation over ternary $^2$5—again addition-only.
2. Spike output: Output passed through a spiking neuron: $^2$6.

Softmax and scaling by $^2$7 are eliminated, as $^2$8 is always non-negative and bounded, given $^2$9, $^2$0 (Guo et al., 28 Feb 2025).

3.2 Graph Sparsity and Adjacency

For Graph Transformers, the binary attention mask is further sparsified by the adjacency matrix $^2$1: mask bits are zeroed for non-adjacent node pairs, ensuring only $^2$2 nonzero elements per channel when $^2$3 in sparse graphs.

4. Computational Complexity and Energy Efficiency

A$^2$4OS$^2$5A achieves significant improvements in both computational and memory efficiency:

• Time complexity: For both graph and sequence, $^2$6 per layer, versus $^2$7 for conventional attention.
• Space complexity: $^2$8 for graphs or $^2$9 for sequences.
• Operational cost: All operations reduce to integer adds and bitwise AND, which on neuromorphic hardware are %%%%55$^2$056%%%% less energy-consuming than multiply-accumulate (MAC) operations; aggregate energy reduction is 10–200$O(n^2d)$2 per layer (Sun et al., 2024, Guo et al., 28 Feb 2025).

This approach enables all-pair interactions in large-scale settings with limited hardware resources and supports deployment on architectures such as Intel Loihi and IBM TrueNorth.

5. Integration with Transformer and Graph Transformer Architectures

5.1 Spiking Graphormer Dual-Branch Design

SpikeGraphormer (Sun et al., 2024) incorporates A$O(n^2d)$3OS$O(n^2d)$4A (“Spiking Graph Attention”—SGA) in a dual-branch architecture:

• Global branch: SGA-driven Transformer layers enable all-pair node interactions using spike-based self-attention.
• Local branch: A lightweight sparse GNN (e.g., GCN) captures fine-grained neighborhood structure.
• Fusion: At each layer,

$O(n^2d)$5

with $O(n^2d)$6.

5.2 Spiking Transformer Encoder

(Guo et al., 28 Feb 2025) applies A$O(n^2d)$7OS$O(n^2d)$8A within each encoder block of a vision transformer:

• Spiking patch splitting provides spike-encoded local features and positional embeddings.
• Encoder blocks alternate A$O(n^2d)$9OS$n$0A attention, ReLU-free MLPs, and binary/ternary spike processing, with residual pre-activation and global average pooling for classification.

6. Empirical Performance and Ablative Analysis

6.1 Graph and Sequence Classification

Key empirical results for A$n$1OS$n$2A-based models:

• OGB-Proteins: SpikeGraphormer achieves 79.62% ROC-AUC (vs. Nodeformer at 77.45%), train memory 3.7 GB (Sun et al., 2024).
• Amazon2M: 88.12% test accuracy (vs. Nodeformer at 87.85%), with large-batch, full-graph inference feasible on CPU.
• CIFAR-10/100: Spiking Transformer with A$n$3OS$n$4A achieves 94.91% (CIFAR-10) and 76.96% (CIFAR-100), surpassing spike-driven transformer baselines of equivalent size (Guo et al., 28 Feb 2025).
• ImageNet-1K: Spiking Transformer-10-512 (A$n$5OS$n$6A) attains 78.66% accuracy with only 4 timesteps and 36 M parameters.

6.2 Efficiency Gains

• On Cora, SpikeGraphormer reduces per-epoch training/inference times and cuts GPU memory by 10–20$n$7 (e.g., 93 MB vs. 239 MB for Nodeformer) (Sun et al., 2024).
• Spiking Transformer reduces per-layer energy by an order of magnitude compared to SNN-Transformer hybrids that retain dot-products (Guo et al., 28 Feb 2025).

6.3 Ablation and Information Capacity

• Fully binarized SNN attention loses most representational power (entropy); the hybrid binary/ReLU/ternary design recovers accuracy competitive with vanilla attention.
• Ablative removal of softmax or re-introduction of scaling leads to overfitting and loss of energy efficiency.

7. Broader Significance and Outlook

A$n$8OS$n$9A establishes a rigorous framework for enabling spiking self-attention at scale, reconciling the expressivity and flexibility of Transformer models with the operational and energy benefits of SNNs. The hybrid (binary/relu/ternary) encoding scheme is shown to be essential for practical accuracy. Deployments in cross-domain contexts (graph, image, text) indicate versatility. The architecture is poised to benefit deployments where memory or energy constraints are paramount, particularly in neuromorphic, edge, or battery-powered computing environments.

A$d$0OS$d$1A is referenced in SGA (“Spiking Graph Attention”) within SpikeGraphormer (Sun et al., 2024) and powers the self-attention sub-blocks of Spiking Transformer (Guo et al., 28 Feb 2025). Empirical results consistently demonstrate state-of-the-art SNN-Transformer accuracy with 10–200$d$2 reductions in energy and memory cost relative to conventional self-attention.

---

Key References:

• "SpikeGraphormer: A High-Performance Graph Transformer with Spiking Graph Attention" (Sun et al., 2024)
• "Spiking Transformer:Introducing Accurate Addition-Only Spiking Self-Attention for Transformer" (Guo et al., 28 Feb 2025)