Spiking QKFormer: Hierarchical Spiking Transformer

• The paper presents a Q-K attention mechanism that bypasses quadratic complexity using binary spike vectors.
• It employs a hierarchical architecture with patch embedding and deformed shortcuts to extract multi-scale spiking representations.
• The method achieves high accuracy on ImageNet and neuromorphic datasets while significantly enhancing energy efficiency.

Searching arXiv for the primary and closely related papers to ground the article in current literature. Spiking QKFormer denotes the QKFormer family of hierarchical spiking transformers introduced in "QKFormer: Hierarchical Spiking Transformer using Q-K Attention" (Zhou et al., 2024). It was proposed to address two coupled constraints in directly trained spiking transformers: the mismatch between conventional self-attention and spike-based computation, and the memory cost of quadratic-token attention under temporal unfolding. The architecture combines a spike-form Q-K attention mechanism with linear complexity, a hierarchical multi-stage backbone that produces multi-scale spiking representations, and a patch embedding module with a deformed shortcut tailored to residual learning across resolution changes. In its reported high-resolution ImageNet-1K setting, QKFormer reached 85.65% top-1 and 97.74% top-5 accuracy with 64.96M parameters, which the authors described as the first time directly trained SNNs exceeded 85% top-1 on ImageNet-1K (Zhou et al., 2024).

1. Concept and research context

QKFormer emerged from a specific limitation of earlier spiking transformers. Directly trained SNNs offered low latency and high energy efficiency, but training remained difficult because of non-differentiable spikes, gradient instability, and the space cost of unfolding in time. In spiking transformers, these difficulties were compounded by spiking self-attention variants that still retained the quadratic token dependence of conventional attention. QKFormer therefore repositioned the attention module itself as the bottleneck: instead of adapting full QKV attention to spikes, it proposed a Q-K-centric mechanism that models token or channel importance through binary vectors and avoids explicit $QK^\top V$ computation (Zhou et al., 2024).

This placement is important within the spiking-transformer lineage. The original paper contrasts QKFormer with Spikformer, Spikingformer, SDSA, STSA, and CML, arguing that prior directly trained spiking transformers were largely single-scale and therefore poorly matched to hierarchical, multi-resolution processing. Later work preserved that characterization. TP-Spikformer described QKFormer as a "feature pyramid-based" spiking transformer with spike-based $Q$ and $K$ attention and stagewise changes in token count and spatial resolution, distinguishing it from "feature-map invariant" baselines such as Spikformer and SDT-V1 (Wei et al., 28 Feb 2026).

Subsequent literature generalized the term beyond the exact 2024 architecture. The 2025 A$^2$OS$^2$A paper explicitly described its addition-only spiking attention core as realizing a "spiking QKFormer" in the sense of a spike-friendly QK interaction and attention path. This suggests that, in later usage, "Spiking QKFormer" may denote both the specific hierarchical model of Zhou et al. and a broader class of Q-K-centric spiking attention designs (Guo et al., 28 Feb 2025).

2. Hierarchical architecture and patch embedding

QKFormer is organized as a three-stage hierarchical spiking transformer. For static RGB images, the input has shape $(T_0 \times H \times W \times n)$ with $T_0 = 1$ and $n = 3$; for event-based datasets, the same form is used with $T_0 = T$ and $n = 2$. Time is explicit in spiking neuron layers, whereas Conv and Linear layers merge time with batch. This permits standard tensor operators to coexist with explicit temporal spike dynamics (Zhou et al., 2024).

The model uses patch embedding with a deformed shortcut, denoted PEDS in the original paper. For Stage 2 and Stage 3, the activation-before-addition form is

$Q$0

where $Q$1 is implemented as either

$Q$2

or

$Q$3

and $Q$4 is a 1×1 convolution with stride $Q$5 that matches both the new spatial resolution and the new channel dimension. A pre-activation alternative,

$Q$6

is also discussed, but the main QKFormer configuration uses the activation-before-addition residual form (Zhou et al., 2024).

Stage 1 prepends spike encoding,

$Q$7

The hierarchy then proceeds through Stage 1 with 4×4 patches and stride 4, Stage 2 with 2×2 downsampling, and Stage 3 with another 2×2 downsampling. In the main ImageNet configuration HST-10-768, the stages contain $Q$8 QKFormer blocks, and the maximum embedding dimension is 768 in Stage 3. For 224×224 input, the token grids are $Q$9, $K$0, and $K$1 across the three stages (Zhou et al., 2024).

The structural role of PEDS is more than a patch tokenizer. In conventional ViT-style patch embedding, a single projection changes resolution without an explicit residual path. QKFormer instead treats patch embedding as a residual downsampling operator. TP-Spikformer later referred to each hierarchical stage as containing a "Spiking Patch Embedding with Deformed Shortcut (SPEDS)" module and a QKFormer block, reflecting how central this component became to the model's identity (Wei et al., 28 Feb 2026).

3. Spike-form Q-K attention

The defining mechanism of QKFormer is spike-form Q-K attention. Given spike-form input $K$2, the query and key streams are

$K$3

with binary outputs

$K$4

Unlike standard attention, there is no value projection inside the attention core. The resulting mechanism is not a low-precision approximation of softmax attention; it is a different operator that uses $K$5 to generate a binary importance vector and applies that vector to $K$6 (Zhou et al., 2024).

QKFormer defines two variants. In Q-K Token Attention (QKTA), token importance is computed by summing $K$7 across channels and passing the result through a spiking neuron:

$K$8

Here $K$9 is a spike-form token-importance vector, and $^2$0 denotes broadcast Hadamard masking along the channel dimension. In Q-K Channel Attention (QKCA), the same principle is applied across tokens:

$^2$1

The masked representation is then post-processed by

$^2$2

Experiments in the original paper use QKTA by default, although QKCA and mixed QKTA/QKCA settings were also studied (Zhou et al., 2024).

The original paper reports the following complexity comparison:

Method	Time complexity	Space
VSA	$^2$3	$^2$4
SSA	$^2$5	$^2$6
SDSA	$^2$7	$^2$8
QKTA	$^2$9	$^2$0
QKCA	$^2$1	$^2$2

Within the paper's formulation, the essential distinction is that QKTA and QKCA avoid explicit $^2$3 formation and replace dense attention maps with binary importance vectors. The authors argue that this linear-complexity structure is what makes hierarchical spiking transformers feasible under direct training, because token counts are large in early stages and temporal unfolding amplifies memory cost (Zhou et al., 2024).

The broader research direction did not stop at binary Q-K masking. "Spiking Transformer: Introducing Accurate Addition-Only Spiking Self-Attention for Transformer" reformulated a spiking QK-style core by using binary $^2$4, ReLU $^2$5, and ternary $^2$6, with

$^2$7

and

$^2$8

That work removes softmax and scaling, while preserving addition-only realization through spike-gated accumulation and ternary sign operations. It explicitly presents this as a "spiking QKFormer" core, indicating a later shift from binary-mask Q-K attention toward richer hybrid Q-K/V formulations (Guo et al., 28 Feb 2025).

4. Spiking dynamics, block structure, and direct training

Each QKFormer block follows a transformer-like residual organization. If $^2$9 is the block input, then

$(T_0 \times H \times W \times n)$0

followed by a spiking MLP:

$(T_0 \times H \times W \times n)$1

The SMLP is described as a spike-form MLP, typically Linear → BN → SN → Linear → BN → SN. Attention thus updates token relations first, after which the SMLP performs channel mixing under residual control (Zhou et al., 2024).

The neuron model is a standard Leaky Integrate-and-Fire unit:

$(T_0 \times H \times W \times n)$2

$(T_0 \times H \times W \times n)$3

$(T_0 \times H \times W \times n)$4

Here $(T_0 \times H \times W \times n)$5 is the input current, $(T_0 \times H \times W \times n)$6 is the pre-spike membrane potential, $(T_0 \times H \times W \times n)$7 is the emitted spike, and $(T_0 \times H \times W \times n)$8 is reset to $(T_0 \times H \times W \times n)$9 when firing occurs. The model uses explicit time steps in spiking layers but treats non-spiking layers by merging time with batch (Zhou et al., 2024).

Training is direct rather than conversion-based. To address the non-differentiability of the Heaviside spike function, the paper uses a sigmoid surrogate

$T_0 = 1$0

Backpropagation replaces the derivative of $T_0 = 1$1 with that of the smooth surrogate. On ImageNet-1K, the reported training setup uses AdamW, a base learning rate of $T_0 = 1$2 scaled by BatchSize/256, an effective batch size of 512 via gradient accumulation across 8 V100 GPUs, and 200 epochs with RandAugment, random erasing, and stochastic depth. CIFAR10/100 uses 400 epochs and batch size 64. The common time-step setting for static vision is $T_0 = 1$3, while neuromorphic datasets use $T_0 = 1$4 or $T_0 = 1$5 (Zhou et al., 2024).

A later theory paper treated QKFormer as an explicit instantiation of spiking self-attention with LIF neurons and spike-rate encoding. In that formulation, query and key spike tensors are

$T_0 = 1$6

and the attention scores are approximated by spike coincidence:

$T_0 = 1$7

This later reinterpretation places QKFormer inside a more general LIF-based spiking-attention theory rather than treating it as an isolated architecture (Guo et al., 17 Apr 2026).

5. Empirical performance

The original QKFormer paper reports strong gains on static and event-based benchmarks. On CIFAR10, HST-4-384 achieves 96.18% with 6.74M parameters and $T_0 = 1$8; on CIFAR100, the same model reaches 81.15%, compared with 78.21% for Spikformer-4-384 and 80.02% for CML-Spikformer-4-384. On ImageNet-1K, HST-10-384 reaches 78.80%, HST-10-512 reaches 82.04%, and HST-10-768 reaches 81.69% at $T_0 = 1$9, 84.22% at $n = 3$0 and 224×224, 85.25% at $n = 3$1 and 288×288, and 85.65% top-1 with 97.74% top-5 at $n = 3$2 and 384×384. The principal comparison in the paper is that Spikformer-8-768 obtains 74.81% with 66.34M parameters, whereas QKFormer-HST-10-768 obtains 85.65% with 64.96M parameters (Zhou et al., 2024).

On neuromorphic datasets, QKFormer also performs strongly with smaller models. HST-2-256, with 1.50M parameters, reaches 98.3/98.6% on DVS128 Gesture at $n = 3$3 and 83.8/84.0% on CIFAR10-DVS at $n = 3$4. These results exceed the corresponding Spikformer-2-256 numbers of 96.9/98.3% on DVS128 Gesture and 78.9/80.9% on CIFAR10-DVS, while using fewer parameters (Zhou et al., 2024).

Ablation studies clarify how the gains are distributed. Replacing Spikformer's Spiking Patch Splitting with PEDS increases CIFAR100 accuracy from 78.21% to 80.26% and CIFAR10-DVS from 80.9% to 82.2%. Time-step ablation on CIFAR100 reports 78.51% at $n = 3$5, 80.08% at $n = 3$6, 81.15% at $n = 3$7, and 81.30% at $n = 3$8, indicating that most of the improvement is realized by $n = 3$9. The paper also reports ImageNet firing rates inside QKTA blocks, for example Stage 1 values of 0.0432 for $T_0 = T$0, 0.1784 for $T_0 = T$1, 0.3477 for $T_0 = T$2, 0.0832 for $T_0 = T$3, and 0.1478 for $T_0 = T$4, which the authors use to argue for sparsity and energy efficiency (Zhou et al., 2024).

Later work reused QKFormer as a benchmark and, in some cases, reported slightly different headline numbers under altered evaluation protocols. "Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension" reports QKFormer-4-256 at 94.8% ± 0.2 on CIFAR-10 and 78.9% ± 0.3 on CIFAR-100, QKFormer-4-384 at 95.2% ± 0.2 and 80.1% ± 0.3, and an ImageNet-1K result of 85.7% with 0.40 mJ per inference on Loihi-class 14nm assumptions, corresponding to 44× energy efficiency relative to ViT-B/16 at 17.6 mJ. The same paper states more generally that spiking transformers, including QKFormer, achieve 38–57× energy efficiency on neuromorphic hardware (Guo et al., 17 Apr 2026).

6. Compression, pruning, and deployment variants

Because QKFormer is hierarchical and feature-variant, token pruning affects it differently from sequence-invariant spiking transformers. TP-Spikformer argues that direct token dropping is incompatible with QKFormer because each stage includes a Spiking Patch Embedding with Deformed Shortcut module containing structure-sensitive convolution and pooling operations, and because these stages reduce the number of tokens by a 2×2 patch size before each stage. Removing tokens outright breaks the square-grid structure required by subsequent convolutions and downsampling (Wei et al., 28 Feb 2026).

TP-Spikformer therefore introduces a block-level early stopping strategy rather than token deletion. For each block and time step, feature maps $T_0 = T$5 are scored by IRToP, a spatiotemporal information-retaining criterion. The spatial score is

$T_0 = T$6

with $T_0 = T$7 the local-window mean, while the temporal score is

$T_0 = T$8

The combined criterion is

$T_0 = T$9

Top-$n = 2$0 informative tokens are processed through SSA and MLP, while uninformative tokens are early-stopped and then reassembled into the original $n = 2$1 grid (Wei et al., 28 Feb 2026).

On ImageNet-1K, TP-Spikformer reports QKFormer-10-768 at $n = 2$2 and $n = 2$3 with 15.08 G block operations, 32.12 mJ power, and 85.56% top-1 accuracy. With $n = 2$4, the numbers become 10.7 G, 28.18 mJ, and 84.45%; with $n = 2$5, 9.61 G, 27.19 mJ, and 84.32%; and with $n = 2$6, 7.97 G, 25.71 mJ, and 82.53%. The same study reports training-time reductions on a single 4090, from 22.80 GB memory and 8.78 h/epoch at $n = 2$7 to 18.40 GB and 5.93 h/epoch at $n = 2$8 (Wei et al., 28 Feb 2026).

This deployment literature also reveals a structural property of QKFormer. In the zero-finetuning ablation at $n = 2$9, "Random + Drop" fails for QKFormer, whereas "Random + IR-Arc" yields 74.45%, "Spatial-only + IR-Arc" 58.93%, "Temporal-only + IR-Arc" 79.69%, and "IRToP + IR-Arc" 81.16%. That pattern indicates that temporal scoring is particularly important for QKFormer and that preserving hierarchical spatial structure is non-negotiable under pruning (Wei et al., 28 Feb 2026).

7. Critiques, reinterpretations, and future directions

A recurrent explanation for the gap between spiking transformers and conventional ANNs has been that binary sparse activation transmission causes information loss. "Spiking Transformers Need High Frequency Information" challenges that framing. It argues that the more fundamental issue is that spiking neurons preferentially propagate low-frequency information, so high-frequency components such as edges, textures, and local details are rapidly attenuated. In its signal-analysis treatment, the membrane dynamics of LIF neurons form a first-order IIR low-pass filter, and this low-pass term compounds across depth (Fang et al., 24 May 2025).

That critique has direct implications for spiking QK attention. The same paper states that spiking self-attention inherits this low-pass bias because $Q$00, $Q$01, and $Q$02 are generated via LIF plus linear layers, and the SSA output is again passed through LIF. It therefore argues that the QK mechanism itself does not inherently restore lost high-frequency information. On CIFAR-100, replacing Avg-Pooling token mixing with Max-Pooling raises top-1 accuracy from 76.73% to 79.12%, and the proposed Max-Former reaches 82.65% on CIFAR-100 and 82.39% on ImageNet, outperforming reported QKFormer* baselines in that study. This suggests that high-performance spiking QKFormer variants may need explicit frequency-preserving front ends or local high-frequency mixers rather than relying on Q-K attention alone (Fang et al., 24 May 2025).

A second reinterpretation is theoretical rather than spectral. "Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension" presents the first comprehensive expressivity theory for spiking self-attention and uses QKFormer as a primary validation architecture. The paper states that spiking attention with LIF neurons is a universal approximator of continuous permutation-equivariant functions, derives a softmax normalization circuit via lateral inhibition with proven $Q$03 convergence, and gives a spike-count lower bound of $Q$04. Its key practical claim is that measured effective dimensions, reported as $Q$05--$Q$06 for CIFAR/ImageNet, explain why $Q$07 timesteps suffice despite worst-case predictions of $Q$08. In that framework, QKFormer operates within 2–3× of the theoretical spike-efficiency frontier (Guo et al., 17 Apr 2026).

The architectural design space has also broadened. A$Q$09OS$Q$10A proposes binary $Q$11, full-precision non-negative ReLU $Q$12, and ternary $Q$13, removing softmax and scaling while preserving non-multiplicative computation. It reports 78.66% on ImageNet-1K for Spiking Transformer-10-512 and presents its core as an accurate addition-only spiking QK module. A plausible implication is that later work increasingly treats "Spiking QKFormer" as a family of Q-K-centric spike-attention strategies rather than a single binary-mask architecture (Guo et al., 28 Feb 2025).

Across these later analyses, two points recur. First, QKFormer established that hierarchical spiking transformers can match or exceed strong ANN backbones on vision benchmarks under direct training. Second, its successors and critics have shifted the central design question from whether attention can be made spike-compatible to how Q-K-centric attention should be encoded, normalized, pruned, and spectrally balanced in order to preserve both spike efficiency and representational richness (Zhou et al., 2024).