Ge$^\text{2}$mS-T: Multi-Dimensional Grouping for Ultra-High Energy Efficiency in Spiking Transformer

Abstract: Spiking Neural Networks (SNNs) offer superior energy efficiency over Artificial Neural Networks (ANNs). However, they encounter significant deficiencies in training and inference metrics when applied to Spiking Vision Transformers (S-ViTs). Existing paradigms including ANN-SNN Conversion and Spatial-Temporal Backpropagation (STBP) suffer from inherent limitations, precluding concurrent optimization of memory, accuracy and energy consumption. To address these issues, we propose Ge$^\text{2}$mS-T, a novel architecture implementing grouped computation across temporal, spatial and network structure dimensions. Specifically, we introduce the Grouped-Exponential-Coding-based IF (ExpG-IF) model, enabling lossless conversion with constant training overhead and precise regulation for spike patterns. Additionally, we develop Group-wise Spiking Self-Attention (GW-SSA) to reduce computational complexity via multi-scale token grouping and multiplication-free operations within a hybrid attention-convolution framework. Experiments confirm that our method can achieve superior performance with ultra-high energy efficiency on challenging benchmarks. To our best knowledge, this is the first work to systematically establish multi-dimensional grouped computation for resolving the triad of memory overhead, learning capability and energy budget in S-ViTs.

• The paper introduces multi-dimensional grouping, combining temporal ExpG-IF neurons and GW-SSA modules for energy-efficient spiking transformer architecture.
• It achieves 79.82% top-1 accuracy on ImageNet-1k with under 15M parameters and sub-3 mJ energy per inference, outperforming existing S-ViT models.
• The approach generalizes across static and neuromorphic datasets, offering robust performance for on-device, real-time SNN applications.

Ge$^\text{2}$mS-T: Multi-Dimensional Grouping for Ultra-High Energy Efficiency in Spiking Transformers

Introduction and Motivation

The pursuit of energy-efficient yet high-performing spiking neural architectures has intensified as resource-constrained edge and mobile deployments come into sharper focus. Spiking Neural Networks (SNNs) inherently exploit event-driven, sparse communication dynamics to achieve substantial reductions in inference energy relative to conventional Artificial Neural Networks (ANNs). However, existing approaches for training and deploying Spiking Vision Transformers (S-ViTs)—notably ANN-SNN conversion and spatial-temporal backpropagation (STBP)—entail critical limitations. These include significant memory overhead, degraded accuracy under low-latency or realistic hardware constraints, and elevated energy consumption, especially when adapting Transformer's spatiotemporal mechanisms to spike-based computation.

The "Ge$^\text{2}$mS-T" architecture addresses these intertwined challenges by orchestrating grouped computation across temporal, spatial, and architectural/network structure dimensions. Its innovations include a novel Grouped-Exponential-Coding-based Integrate-and-Fire (ExpG-IF) neuron, and a Group-wise Spiking Self-Attention (GW-SSA) module, each engineered for precise spike regulation, constant memory, and multiplication-free, multi-scale attention. The proposed solution demonstrates substantial improvements in accuracy-energy tradeoffs on ImageNet-1k and generalizes robustly across static and neuromorphic datasets.

Figure 1: Parameter count, inference accuracy, and energy comparison of Ge$^\text{2}$mS-T versus prior S-ViT/S-CNN models on ImageNet-1k; upper-left denotes superior energy efficiency.

Architecture Overview

Ge$^\text{2}$mS-T is constructed hierarchically from several key building blocks: ExpG-IF neurons for efficient temporal coding, GW-SSA for energy-optimized global and locality-aware attention, hybrid convolutional feed-forward units, and systematic multi-stage spatial grouping. The architecture applies these mechanisms both within and across layers, compressing memory and computation requirements by constraining and grouping spike activity spatially and temporally.

Figure 2: Ge$^\text{2}$mS-T end-to-end architecture: composed of stage-wise blocks with SSA, SConv, and SFFN, leveraging multi-dimensional grouping to outperform conversion/STBP-based methods in energy efficiency.

ExpG-IF: Temporal Dimension Grouping

The ExpG-IF model maps inputs onto a non-uniform, exponential quantization space, enabling neurons to emit spikes only on strategically aggregated time-step subsets. The core properties include:

• Lossless conversion with $\mathcal{O}(1)$ training overhead (i.e., constant memory usage, independent of inference time-steps).
• Precise, implicit regulation of firing patterns, where neuron spikes are not uniformly distributed, but grouped and encoded exponentially, substantially limiting the spike count.
• Computational parity with standard Integrate-and-Fire, as binary search allows for $O(T)$ complexity spike pattern decoding, but the spike groupings deliver strong upper bounds on inference cost.

GW-SSA: Multi-Scale Spatial Grouping

GW-SSA divides the token set into groups both globally and locally (windows) to confine attention computation within selected clusters, efficiently reducing the quadratic scaling with respect to input spatial size ($\mathcal{O}(N^2)$ to $\mathcal{O}(N^2/|G_s|)$). Key technical details include:

• Dual-branch architecture: One branch conducts group/global attention via spatial pooling; the other extracts localized features via convolution.
• Multiplication-free operations: Utilizing addition-based similarity and event streams for SNN compatibility.
• Synergistic grouping with ExpG-IF: The combined effect of spatial and temporal groupings achieves sublinear growth in total synaptic operations (SOPs), strongly constraining overall energy draw.

Experimental Results and Analysis

ImageNet-1k: Accuracy–Energy–Parameter Tradeoffs

Ge$^\text{2}$mS-T delivers competitive or superior accuracy on ImageNet-1k, achieving 79.82% top-1 with $^\text{2}$0M parameters and $^\text{2}$1 mJ energy expenditure—substantially outperforming state-of-the-art S-ViT and S-CNN models, which require many times these resources for comparable accuracy. Notably:

• Ge$^\text{2}$2mS-T Small achieves 75.12% accuracy with only $^\text{2}$3 mJ, dwarfing ResNet-34 in efficiency.
• Ge$^\text{2}$4mS-T Large surpasses SNN-ViT and Spikingformer backbones, with $^\text{2}$520–80% reduction in parameters and energy per inference.

These results confirm the claim that multi-dimensional grouping offers a unique resolution to the trilemma of accuracy, memory, and energy in S-ViT architectures.

Generalization to Downstream Tasks and Neuromorphic Data

The proposed method maintains its advantage across CIFAR-10/100 and CIFAR10-DVS, surpassing competitive SNN learning algorithms and architectures (e.g., Spikingformer, GAC-SNN) by 3–11% in accuracy at identical or reduced inference time-steps. Notably, Ge$^\text{2}$6mS-T variants achieve up to 98.59% on CIFAR-10 with only 4 time-steps and consistently outperform recent transformer-based SNNs in both static and event-based visual domains.

Fine-Grained Energy Profiling

Figure 3: Module-level breakdown of firing rate, average SOPs, and theoretical upper bounds for SSA, SConv, and SFFN modules in Ge$^\text{2}$7mS-T on ImageNet-1k.

The clustered spatial grouping strategy maintains stable and low SOPs per attention layer, effectively counteracting the exponential growth in tokens. Regularization of spike activity and the adoption of lightweight separable convolutions in SFFNs further compound energy savings at all scales.

Theoretical and Practical Implications

The Ge$^\text{2}$8mS-T design paradigm generalizes beyond Vision Transformers, providing a blueprint for energy-efficient spike-based processing in architectures with pronounced spatiotemporal and token multiplicity characteristics. In contrast to conversion and surrogate-gradient-based approaches, the explicit grouping of spike events enables joint optimization of efficiency and learning capacity with provable bounds on overhead and precision loss. This has immediate implications for on-device and real-time inference on neuromorphic hardware, where both accuracy and strict power budgets are non-negotiable. Furthermore, the introduction of exponential group encoding for temporal spike patterns opens a direction for coding-theory-inspired SNN dynamics, potentially synergizing with SNN self-distillation, efficient quantization, and hybrid analog-digital co-processing in future research.

Conclusion

Ge$^\text{2}$9mS-T articulates a comprehensive solution to the central bottlenecks in Spiking Vision Transformers through coordinated grouping over multiple computation axes. The experimental and theoretical analyses validate that multi-dimensional grouped computation, when entangled with spike-efficient coding and multi-scale attention, enables simultaneous achievement of high accuracy, constant-order memory, and ultra-low energy consumption. This approach is poised to facilitate scalable, deployable SNN systems for advanced edge and neural-inspired computing tasks.

(2604.08894)