Elastic Spiking Transformers for Efficient Gesture Understanding

Abstract: Spiking Neural Networks (SNNs), particularly Spiking Transformers, offer energy-efficient processing of event-based sensor data for healthcare applications. Yet current architectures are rigid: they are trained and deployed as static networks with fixed parameter counts and computational graphs. This limits deployment on neuromorphic hardware such as Loihi and SpiNNaker, where on-chip constraints often require smaller models that trade accuracy for feasibility. We introduce the Elastic Spiking Transformer, a runtime-adaptive architecture that brings elasticity into the spiking paradigm. Inspired by Matryoshka-style representation learning, it embeds nested elasticity in the Feature Extractor, Spiking Self-Attention, and Feed-Forward blocks. Through granularity-aware weight sharing, a single universal model can dynamically slice network width and attention heads at inference time without retraining. This design provides two key advantages for SNNs. First, it allows the model to adjust its parameter footprint to different hardware memory budgets. Second, reducing active neurons also lowers spike firing rates, yielding proportional reductions in synaptic operations, an energy benefit not directly available in standard artificial neural networks. We evaluate the approach on CIFAR10/100, CIFAR10-DVS, and the EHWGesture clinical gesture understanding dataset. Results show that one Elastic Spiking Transformer spans a broad range of complexity-accuracy trade-offs, matching or surpassing independently trained baselines while supporting adaptive, real-time gesture recognition on resource-constrained edge devices.

• The paper introduces NESTformer, a universal elastic spiking transformer that combines nested weight sharing, scalable patch embedding, spiking MLPs, and row-wise attention for neuromorphic deployment without CPU-offloaded matrix multiplication.
• NESTformer scales across four granularities from 0.68M to 2.59M parameters, reducing estimated Loihi energy from 46.1 to 13.3 μJ and total spikes by 71%, while retaining 99.8% of peak EHWGesture accuracy at its practical g2 setting.
• The model achieves 75.98% accuracy on EHWGesture and generalizes across DVS128 Gesture, CIFAR10-DVS, and CIFAR-100, although its energy advantages remain analytically estimated and require validation on physical neuromorphic hardware.

Overview

The paper introduces NESTformer, an elastic spiking transformer designed for event-based gesture understanding on neuromorphic hardware. The central problem it addresses is architectural rigidity: existing spiking transformers (Spikformer, QKFormer, Spike-driven Transformer) are trained and deployed as fixed-size networks, so adapting to a new neuromorphic chip's memory or power budget requires training an entirely new model. Moreover, many of these architectures rely on batched matrix multiplications that current neuromorphic processors cannot execute natively, forcing offload to a CPU and eroding the energy advantages of spike-driven computation (2605.13869).

NESTformer embeds runtime adaptability directly into the spiking paradigm by combining Matryoshka-style nested weight sharing with three elastic components: a spiking MLP, a spiking self-attention block, and the patch-embedding/feature-extraction stage. A single "universal" model, trained once, can be sliced at inference time across four granularity levels ($g0$–$g3$), spanning 0.68M–2.59M parameters and a 3.8× energy range without retraining.

Motivation: energy does not scale with parameters

A key empirical finding motivates the architecture design. When the authors scale Spikformer and QKFormer across S/B/L variants, parameter counts vary by over 80% (5.3M to 0.7M), but estimated energy on Intel Loihi's model (23.6 pJ per synaptic operation) drops only ~12% (60.7 μJ to 53.0 μJ). Spike-count profiling reveals why: the patch-embedding stage, which operates at the highest spatial resolution, generates over $1.0 \times 10^6$ spikes per inference regardless of downstream model size, dominating total activity. Standard scaling strategies that shrink transformer depth or width therefore leave the dominant energy cost untouched.

This observation has a direct implication for how elasticity must be implemented in SNNs: reducing width must also reduce active neurons in the high-resolution early stages, not merely the token-processing blocks. NESTformer addresses this with XiSPS, an elastic spiking patch-splitting backbone adapted from XiConv, which uses a tunable compression factor $\gamma$ to trade feature granularity against compute at runtime while remaining purely event-driven via LIF neurons.

Elastic architecture components

The design follows three principles: nested architecture (smaller configurations are strict subsets of larger ones), spike-aware elasticity (size reduction minimizes both parameters and spike activity), and hardware-aware design (no CPU-offloaded operations such as BMM).

Elastic MLP: hidden dimensions follow log-spaced granularities $\{64, 160, 416, 1024\}$, following MatFormer. Since MLPs account for roughly 60% of transformer parameters, this yields large footprint reductions.

Elastic attention with row-wise computation: head counts are sliced dynamically from $\{Q, K, V\}$ tensors to the active count $h_g$. More significantly, the authors reformulate attention as a row-wise decomposition, $O_h = Q_h(K_h^T V_h)$ computed per query row, where each row reduces to two sequential LIF layers using $K_h^T$ and $V_h$ as weights. An explicit intermediate LIF activation between $g3$0 and the $g3$1 projection binarizes attention scores so the second operation is strictly spike-driven accumulation. This formulation is mathematically equivalent to Spikformer's SSA plus the intermediate LIF, allowing GPU training in parallel GEMM form and conversion to the row-wise form at deployment. The cost is a 2.6–10.2× slowdown on GPUs due to loss of BLAS parallelization — a training-time inconvenience rather than a deployment limitation, but one the paper concedes explicitly.

Training procedure: rather than sandwich training, the authors sample a single granularity per step from a distribution biased toward larger configurations, since uniform sampling left large subnetworks under-trained. Tensors are allocated at maximum capacity with binary masks zero-padding inactive regions, preserving static shapes; per-granularity batch normalization handles configuration-specific statistics. Convergence required roughly 50% more training steps than a single-size network — a nontrivial overhead the paper acknowledges.

Granularity–energy relationship

Across granularities, moving from g3 to g0 reduces total spikes per inference by 71% (1.95M → 0.56M) and energy proportionally (46.1 → 13.3 μJ). This linear coupling between size reduction and energy is the paper's central claim of advantage over prior spiking transformers, whose patch-embedding stages did not scale down.

One nuance deserves note: within the elastic MLP blocks, firing rates exhibit an inverse relationship with width. Shrinking the hidden dimension from 1024 to 64 raises average firing rates of surviving neurons from ~4.38% to ~21.01% (e.g., block.1.mlp.lif2 rises from 11.31% to 35.52%). Total energy still falls because fewer neurons fire, but per-neuron activity intensifies — a compaction effect that could matter for routing congestion on real chips, though the paper does not analyze this beyond noting its existence.

Results on EHWGesture

On EHWGesture (9,708 samples, 11 clinical gesture classes, T=16), NESTformer sets the best reported accuracy among compared baselines:

Model	Energy (μJ)	Params (M)	Accuracy (%)
PhiNet (CNN)	6,900	4.91	71.25
QKFormer-5-320	100.9	2.35	75.28
Spikformer-3-320	70.8	5.27	74.16
NESTformer (g3)	46.1	2.59	75.98
NESTformer (g2)	28.7	1.46	75.86
NESTformer (g0)	13.3	0.68	71.01

The g3 configuration outperforms the strongest baseline (QKFormer-5-320) while consuming less than half its energy, and orders of magnitude less than CNN baselines. The g2 configuration retains 99.8% of peak accuracy at ~40% lower energy, which the authors identify as the practical deployment point. Energy figures are estimates based on Loihi's per-synaptic-operation cost rather than measurements on physical hardware — an assumption worth bearing in mind when interpreting absolute numbers.

Generalization benchmarks

On DVS128 Gesture, NESTformer (g3) matches Spike-driven Transformer's 99.30% and exceeds QKFormer's 98.60%; even g0 retains 97.57% with 5.3× fewer parameters than g3. On CIFAR10-DVS, g3 reaches 83.61%, competitive with QKFormer's 84.00%. On static CIFAR-100, g3 achieves 81.30%, surpassing QKFormer (81.15%) and Spikformer (78.21%) — evidence that the elastic framework generalizes beyond event-based data. Across all four granularities, accuracy degrades gracefully, supporting the claim that a single trained model covers a wide operating range.

Timestep and task-dependent trade-offs

An analysis over timesteps ($g3$2) and granularities reveals opposing temporal preferences between the two EHWGesture tasks. Action quality assessment (AQA) benefits monotonically from higher temporal resolution, peaking at $g3$3 (80.7%), whereas gesture recognition peaks at $g3$4–32 and degrades at $g3$5 (96.7% → 94.4% at g3), which the authors attribute to sensor dynamics capturing spatial hand-position information at longer windows. For energy-constrained gesture recognition, $g3$6 achieves 96.24% at 10.49 μJ, while raising $g3$7 to 64 triples energy and lowers accuracy. The balanced $g3$8 point delivers 96.32% gesture / 78.92% AQA accuracy at 28.7 μJ. Notably, at low timesteps, increasing granularity yields larger gains than increasing timesteps: moving from g0 to g1 at $g3$9 lifts gesture accuracy from 93.1% to 96.0%.

Spike dynamics show total spikes scaling roughly 2× per doubling of $1.0 \times 10^6$0, while firing rates decrease at higher timesteps (1.78% at T=64 vs. 2.64% at T=8), suggesting more sparse distributed computation. The authors speculate this could reduce routing congestion on real neuromorphic hardware, potentially making actual efficiency better than energy-model estimates predict — a hypothesis left unverified.

Limitations and open questions

Several caveats are stated or implicit in the work. Energy results rely on Loihi's analytical energy model rather than on-chip measurement, and the claimed compatibility with native neuromorphic execution is not demonstrated on Loihi, SpiNNaker, or Spiker+ hardware in this paper. The row-wise attention reformulation incurs substantial GPU training slowdowns if used directly, requiring the train-parallel/deploy-rowwise conversion path. Training costs rise ~50% relative to single-size models, and the biased granularity sampling is an empirical choice whose sensitivity is not ablated. The inverse firing-rate behavior in narrow MLP configurations suggests that extreme slicing may concentrate activity in ways not captured by total-spike accounting. Finally, whether the observed firing-rate decrease at long timesteps translates into real routing-efficiency gains on silicon remains an open empirical question.

Conclusion

NESTformer demonstrates that Matryoshka-style elasticity transfers productively to spiking transformers, provided elasticity is applied to the spike-dominant patch-embedding stage and attention is reformulated for neuromorphic-native execution. Its principal contributions are the first fully elastic spiking transformer trainable as a universal model, a row-wise spiking attention scheme eliminating CPU-offloaded matrix multiplies, and the empirical demonstration that — unlike prior spiking transformers — granularity reduction yields proportional energy savings. With state-of-the-art accuracy on EHWGesture (75.98%) and CIFAR-100 among spiking transformers (81.30%), and a 3.8× energy range from a single checkpoint, the work substantiates its core claim, while validation on physical neuromorphic hardware remains the most significant outstanding step.