---
title: 'QKFormer: Hierarchical Spiking Transformer'
url: https://www.emergentmind.com/topics/qkformer
type: topic
---

# QKFormer: Hierarchical Spiking Transformer

QKFormer is a hierarchical spiking transformer architecture that introduces a spike-form Query-Key (Q-K) attention mechanism tailored for spiking neural networks (SNNs). It is designed for energy-efficient, large-scale neuromorphic vision tasks, achieving state-of-the-art direct-training SNN performance through hierarchical decomposition, sparse binary attention, and a bespoke patch embedding with deformed shortcuts. QKFormer enables linear-complexity attention, multi-scale spike-coded representation, and efficient neuromorphic hardware deployment [2403.16552][2509.15036].

## 1. Spike-Form Q–K Attention Mechanism

QKFormer innovates by replacing the standard Query-Key-Value (QKV) attention triplet with a pure spike-based Q-K formulation. Let $Q, K \in \{0,1\}^{T \times N \times D}$ denote the spike-coded queries and keys, with $T$ time steps, $N$ tokens, and $D$ channels per head. There are two principal modes:

- **Token-Wise Attention (QKTA):**
  $Q$ is summed across channels for each token and thresholded via a spiking neuron function $\mathrm{SN}$:
  $$
  A_t[j] = \mathrm{SN}\left(\sum_{i=1}^D Q_{i,j}\right) \in \{0,1\}, \quad j = 1, \dots, N
  $$
  The output is a binary mask applied to $K$ via Hadamard product:
  $$
  X'_{i, j} = A_t[j] \otimes K_{i, j}
  $$
  A spiking MLP (two-layer, with batch normalization and $\mathrm{SN}$) projects this output back to the residual stream.

- **Channel-Wise Attention (QKCA):**
  Analogously, tokens are summed to produce an attention vector over channels. Each form enables efficient modeling of importance across either token or channel dimension.

This spike-form attention avoids full softmax and value projection, yielding $O(D)$ (or $O(N)$ for QKCA) per-head computational complexity, a sharp reduction compared to $O(N^2D)$ for vision self-attention (VSA) or spike self-attention (SSA), and reducing total memory from $O(N^2 + ND)$ to $O(ND)$. All operations are confined to simple $\{0, 1\}$ addition and masking, optimizing deployment on neuromorphic hardware [2403.16552].

## 2. Hierarchical Spiking Transformer Architecture

QKFormer employs a three-stage hierarchy to generate multi-scale spike-based representations. 

- **Stage 1:** Processes $4 \times 4$ image patches, yielding feature maps at $(H/4) \times (W/4)$ resolution with width $C$.
- **Stage 2:** Downsamples by $2 \times 2$ patch embedding, further halving spatial dimensions and doubling channel width.
- **Stage 3:** Repeats downsampling to $(H/16) \times (W/16)$, quadrupling the channels relative to the first stage.

Each stage comprises $N_s$ QKFormer blocks, each constructed as follows:
$$
X'_{\ell} = \mathrm{QKTA}(X_{\ell-1}) + X_{\ell-1}, \quad
X_{\ell} = \mathrm{SMLP}(X'_\ell) + X'_\ell
$$
This residual topology maintains membrane potentials and spike timing fidelity across all scales [2403.16552].

## 3. Deformed-Shortcut Patch Embedding (PEDS)

Typical patch embedding in transformers disrupts residual connections due to mismatched dimensions. QKFormer introduces a deformed-shortcut mechanism (PEDS) that learns a $1 \times 1$ convolutional shortcut, $W_d$, parallel to the main path:
$$
Y = \mathcal{F}(X; \{W_i\}) + \mathrm{SN}(W_d X)
$$
Here, $\mathcal{F}$ comprises convolution–BN–pool–SN–conv–BN–SN, and $W_d$ enables dimensional and stride adaptation. This design ensures spike timing information is propagated, which empirically improves classification accuracy, as observed via ablation (e.g., on CIFAR100, $78.21\% \to 80.26\%$) [2403.16552].

## 4. Mathematical Foundations and Training Methods

The core neuronal unit is a leaky integrate-and-fire (LIF) spiking neuron, with the following update at timestep $t$:
$$
H[t] = V[t-1] + \frac{1}{\tau}(X[t] - (V[t-1] - V_{\text{reset}})),
$$
$$
S[t] = \Theta(H[t] - V_{\text{th}}),
$$
$$
V[t] = H[t] (1 - S[t]) + V_{\text{reset}} S[t]
$$
The non-differentiable spike $\Theta$ is handled by a surrogate gradient:
$$
\sigma(x) = \frac{1}{1 + \exp(-\alpha x)}, \quad \alpha = 4
$$

QKFormer is directly trained via backpropagation through time (BPTT) across $T$ steps, using AdamW with a learning rate scaled by batch size. For large-scale benchmarks (e.g., ImageNet), ImageNet-1K training uses batch size 512 on 8 $\times$ V100 GPUs over 200 epochs, with augmentations including RandAugment, random erasing, and stochastic depth [2403.16552].

## 5. Empirical Performance and Ablations

QKFormer attains superior results over all prior SNNs:
- On ImageNet-1K, HST-10-768 ($64.96$M params, $T=4$) achieves $85.65\%$ top-1 accuracy, surpassing Spikformer-8-768 ($74.81\%$) by $+10.84\%$. This is the first instance of direct-trained SNNs exceeding $85\%$ top-1 accuracy on ImageNet.
- On CIFAR10, HST-4-384 ($6.74$M, $T=4$): $96.18\%$ (vs. $95.51\%$); on CIFAR100: $81.15\%$ (vs. $78.21\%$).
- On DVS128 Gesture ($1.50$M): $98.6\%$ ($T=16$); CIFAR10-DVS: $84.0\%$, exceeding Spikformer by $+3.1\%$.

Ablations show:
- PEDS consistently boosts accuracy across benchmarks.
- QKTA, QKCA, and their combination provide comparable accuracy, with QKCA favored for layers with large $D$.
- Memory reduction is marked: at $\sqrt{N}=50$, SSA blocks use $\sim$26MB, QKTA uses $\sim$2.5MB.
- Firing rates (Stage 1, $64.96$M model): $Q:4.3\%$, $K:17.8\%$, $A_t:34.8\%$, $X':8.3\%$, $X'':14.8\%$, indicating strong event-driven sparsity [2403.16552].

## 6. Neuromorphic Hardware Integration and NEURAL Accelerator

QKFormer is efficiently realized in hardware, as demonstrated by the NEURAL architecture, a hybrid data-event neuromorphic accelerator hosting QKFormer blocks natively [2509.15036]. Key features include:
- **On-the-fly spike-driven QKFormer:** Operations are embedded within a conventional spiking convolutional pipeline via a pair of elastic FIFOs per processing element and an OR register, eliminating the need for specialized transformer hardware.
- **Window-to-Time-to-First-Spike (W2TTFS):** Average pooling is replaced by a fully spike-based downsampling mechanism that encodes the first spike event.
- **Single-timestep training:** Using knowledge distillation (KD) from a high-accuracy ANN teacher, followed by quantization-aware and spike-only fine-tuning, enables low-latency inference.

NEURAL, implemented on a Xilinx Virtex-7 FPGA, supports QKFormer at minimal overhead: halving logic and memory use versus prior SNN accelerators, supporting real-time (68 FPS) and low-power (0.79 W, 52.37 GSOPS/W) operation. Deploying spiking QKFormer blocks yields accuracy gains with only $\sim$2 ms extra latency and negligible spike cost (energy per image $<$10 mJ) [2509.15036].

## 7. Limitations and Prospects

Current QKFormer models rely on surrogate gradient methods and still require moderate time-steps ($T \leq 4$) for large-scale image tasks such as ImageNet, while some downstream hardware and KD implementations achieve $T=1$. Future research may address latency reduction, self-supervised or contrastive pre-training tailored for SNNs, and end-to-end hardware-software co-design. *A plausible implication is that architectural advances in spike-form attention and multi-scale SNNs, as demonstrated by QKFormer, will influence both algorithmic and neuromorphic system development for efficient, large-scale event-driven learning* [2403.16552][2509.15036].

Source: https://www.emergentmind.com/topics/qkformer