BSViT: A Burst Spiking Vision Transformer for Expressive and Efficient Visual Representation Learning

Abstract: Spiking Vision Transformers (S-ViTs) offer a promising framework for energy-efficient visual learning. However, existing designs remain limited by two fundamental issues: the restricted information capacity of binary spike coding and the dense token interactions introduced by global self-attention. To address these challenges, this work proposes BSViT, a burst spiking-driven Vision Transformer featuring a Dual-Channel Burst Spiking Self-Attention (DBSSA) mechanism. DBSSA encodes queries with binary spikes and keys with burst spikes to enhance representational capacity. The value pathway adopts dual excitatory and inhibitory binary channels, enabling signed modulation and richer spike interactions. Importantly, the entire attention operation preserves addition-only computation, ensuring compatibility with energy-efficient neuromorphic hardware. To further reduce spike activity and incorporate spatial priors, a patch adjacency masking strategy is introduced to restrict attention to local neighborhoods, resulting in structure-aware sparsity and reduced computational overhead. In addition, burst spike coding is systematically integrated across the network to increase spike-level representational capacity beyond conventional binary spiking. Extensive experiments on both static and event-based vision benchmarks demonstrate that BSViT consistently outperforms existing spiking Transformers in accuracy while maintaining competitive energy efficiency.

• The paper introduces a novel Dual-Channel Burst Spiking Self-Attention mechanism that leverages multilevel burst-coded keys and dual-channel values to enhance spike representation.
• The paper demonstrates improved performance on CIFAR-10/100 and ImageNet-1K datasets through energy-efficient, addition-only computations and structured patch adjacency masking.
• The paper shows that burst spiking coding and localized self-attention not only reduce computational latency but also boost accuracy in neuromorphic vision applications.

BSViT: A Burst Spiking Vision Transformer for Expressive and Efficient Visual Representation Learning

Introduction and Motivation

Spiking Vision Transformers aim to reconcile the modeling power of ViTs with the neuromorphic, event-driven processing paradigms of SNNs for extreme energy efficiency. Prior Spiking Transformer approaches suffer from either limited information capacity due to binary spike coding or excessive computational burden when retaining dense global self-attention. The BSViT architecture addresses both by introducing burst spiking-driven Transformer blocks and a Dual-Channel Burst Spiking Self-Attention (DBSSA) mechanism.

Dual-Channel Burst Spiking Self-Attention (DBSSA)

DBSSA fundamentally augments prior spiking self-attention by encoding keys with multi-level burst spikes, enhancing representational richness over binary-only coding. Furthermore, a dual-channel value path modulates excitatory and inhibitory spike streams independently, increasing expressiveness and enabling signed interactions. All core attention operations are addition-only, maintaining compatibility with neuromorphic hardware and event-driven processing.

Figure 1: Comparison between vanilla Spiking Self-Attention (SSA) and Dual-channel Burst Spiking Self-Attention (DBSSA); DBSSA integrates multilevel burst-coded keys, dual-channel values, and spatial adjacency masking for addition-only computation.

The mechanism preserves binary spike queries but activates keys via a burst-coded LIF neuron, supporting multilevel spike transmission and avoiding the information bottleneck inherent in binary spike maps. Dual value channels represent excitatory and inhibitory pathways, harmonizing computational efficiency with signal richness. The entire attention matrix is masked using a patch adjacency structure, restricting spike propagation to local neighborhoods and ensuring structure-aware sparsity.

Burst Spiking Coding and Patch Adjacency Masking

Burst spiking emulates biological burst firing with a single LIF neuron supporting multiple thresholds, quantizing membrane potential and emitting multiple spikes per timestep. This increases spike-level representational capacity and is hardware friendly for neuromorphic deployment. Spatial priors are enforced through patch adjacency masking that constrains attention to an 8-connected local neighborhood, reducing redundant spike activity, total energy cost, and enabling inductive bias for visual tasks.

Figure 2: Architecture overview of BSViT, demonstrating four stages: bursting patch splitting, DBSSA attention, bursting MLP, and linear classification head, all integrated for energy-efficient visual encoding.

Figure 3: Visual depiction of patch neighborhood structure for adjacency masking, which restricts spike attention at every stage within the local context.

Architecture and Experimental Results

BSViT comprises Bursting Spiking Patch Splitting (BSPS), DBSSA, Bursting Multi-Layer Perceptron (BMLP), and a linear classifier head. BSPS converts inputs to burst spike-coded patch sequences; DBSSA constructs expressive attention maps; BMLP amplifies burst coding in feature processing; and local masking constrains spike interactions. Residual connections and positional embedding are employed, with burst levels and optimal masking tuned for performance.

Empirical assessment on CIFAR-10/100 and ImageNet-1K datasets shows:

• BSViT achieves 96.25% (masked) and 96.55% (no-masked) accuracy on CIFAR-10, surpassing competing SNN Transformers, notably with only 2 time steps, reflecting low inference latency.
• On CIFAR-100, the adjacency-masked BSViT attains 80.32%, up to +0.63% over prior SoTA models, demonstrating improved generalization to complex tasks.
• ImageNet-1K evaluations show BSViT-10-512 reaches 78.98% Top-1 accuracy, eclipsing previous architectures with fewer parameters and minimal latency.
• Event-based benchmarks (CIFAR10-DVS) reveal BSViT achieves 81.5% (masked), outclassing all evaluated methods for neuromorphic data streams.

Ablation studies isolate the contribution of burst-coded keys (+0.25%), dual-channel values (+0.32%), masked DBSSA, and full architectural integration (+1.41%), confirming each component's efficacy.

Practical Implications and Computational Efficiency

The masked DBSSA transforms the quadratic complexity $\mathcal{O}(N^2d)$ of vanilla self-attention into a linear $\mathcal{O}(Nkd)$ regime, where $k \ll N$ owing to spatial locality. Event-driven, addition-only attention enables deployment on neuromorphic hardware with meaningful computational cost savings. Burst spiking coding, when appropriately tuned ($n=20$), maximizes information flow without introducing quantization noise or overfitting, as demonstrated by sensitivity analysis.

Energy estimation demonstrates BSViT can outperform prior models with 14.1% higher cost (BSViT-2-step vs Spikingformer-4-step), but achieves this with notably fewer timesteps and superior accuracy. Structure-aware spike sparsity from adjacency masking directly translates to lower SOP counts and tangible reductions in total energy cost.

Theoretical Implications and Future Directions

BSViT advances the biological plausibility and scalability of Transformer-based SNNs by integrating burst spiking, excitatory/inhibitory modulation, and spatial inductive bias. The addition-only attention paradigm and masked locality open avenues for real-time neuromorphic vision tasks requiring low-latency and high-fidelity encoding. The diminished information bottleneck, especially under restricted spike representations, sets the stage for further exploration of adaptive burst coding and compositional local/global attention mechanisms.

Scaling BSViT to larger vision corpora, fine-grained temporal event streams, and multimodal inputs presents compelling opportunities for neuromorphic computing research. Hardware co-design of spike-based attention mechanisms, aligned with the energy-efficient principles illustrated here, may further enhance deployment feasibility for edge AI, robotics, and real-time perception scenarios.

Conclusion

BSViT presents a burst spiking-driven Vision Transformer architecture, achieving a robust balance of accuracy, expressiveness, and energy efficiency in visual representation learning. The dual-channel DBSSA mechanism, burst coding, and structure-aware adjacency masking collectively advance the state of spiking Transformer models, opening new prospects for scalable, neuromorphic, and biologically inspired AI vision systems.