Reparameterized Spiking Self-Attention (RepSSA)

• The paper introduces RepSSA, which preserves continuous-valued query-key interactions during training to overcome precision loss in binary spiking attention.
• It reparameterizes separate query and key projections into a fused matrix for efficient, spike-driven inference in speech processing.
• Empirical ablations show that RepSSA combined with a Hierarchical Decay Mask significantly improves token mixing and reduces word error rates compared to traditional spiking methods.

Searching arXiv for the cited papers and closely related spiking self-attention work to ground the article. I’ll look up the RepSSA paper and the two related spiking self-attention papers on arXiv. arXiv search: "(Song et al., 10 Jul 2025) IML-Spikeformer Input-aware Multi-Level Spiking Transformer for Speech Processing" Reparameterized Spiking Self-Attention (RepSSA) is a spiking self-attention mechanism introduced in “IML-Spikeformer: Input-aware Multi-Level Spiking Transformer for Speech Processing” (Song et al., 10 Jul 2025). It is proposed as a replacement for standard spike-driven self-attention modules such as SDSA-3. Its defining idea is to compute a more precise attention map from continuous-valued query and key projections during training, while reparameterizing those projections into a single fused matrix at inference so that deployment remains spike-driven. In the full IML-Spikeformer architecture, RepSSA is combined with a Hierarchical Decay Mask (HDM) to form HD-RepSSA, which serves as the token-mixing component in each Transformer-style block (Song et al., 10 Jul 2025).

1. Definition and motivation

RepSSA is motivated by a limitation in prior spiking self-attention: queries and keys are converted into binary spike matrices before attention is computed. In the formulation discussed for SDSA-3, this takes the form

$$\mathbf{Q}_s = \mathcal{SN}(\mathrm{BN}(\mathbf{Q})), \qquad \mathbf{K}_s = \mathcal{SN}(\mathrm{BN}(\mathbf{K})),$$

followed by attention based on spike matrices such as

$\mathbf{A}_s = \mathbf{Q}_s \mathbf{K}_s^{T}.$

The stated problem is that binary spike conversion makes the attention map less precise, so temporal relationships between tokens are not captured accurately enough (Song et al., 10 Jul 2025).

RepSSA addresses this by separating the representational regime used in training from the execution regime used in inference. During training, it preserves a richer continuous-valued query–key interaction; during inference, it fuses the query and key projections into a single matrix so that the model can still operate efficiently in a spike-driven manner. The paper presents this as a way to improve the representational capacity of spiking attention maps while keeping spike-driven inference feasible (Song et al., 10 Jul 2025).

Within the speech setting, this design is tied to the claim that standard spiking self-attention lacks sufficient representational power for signals with fine local structure, hierarchical temporal dependencies, and long-range context. RepSSA is therefore not an isolated attention modification; it is one of the central mechanisms by which IML-Spikeformer is adapted to large-scale speech processing (Song et al., 10 Jul 2025).

2. Mathematical formulation and reparameterization

The core RepSSA attention map is defined by a training–inference split:

$$\mathbf{A} = \begin{cases} X W_Q W_K^T X^T, & \text{Training} \ X W_{QK} X^T, & \text{Inference} \end{cases}$$

with

$W_{QK} = W_Q W_K^T,$

and

$\mathbf{V_s} = \mathcal{SN}(XW_V).$

Here, $X$ is the binary spike input, $W_Q$, $W_K$, and $W_V$ are the learnable query, key, and value projection matrices, and $W_{QK}$ is the reparameterized fused matrix (Song et al., 10 Jul 2025).

This formulation makes the reparameterization explicit. During training, the attention map is factorized through separate query and key projections:

$\mathbf{A}_s = \mathbf{Q}_s \mathbf{K}_s^{T}.$0

During inference, the same interaction is collapsed into the fused bilinear form

$\mathbf{A}_s = \mathbf{Q}_s \mathbf{K}_s^{T}.$1

so the deployed model does not need to retain separate query and key projections in the attention path (Song et al., 10 Jul 2025).

The value path remains spiking throughout:

$\mathbf{A}_s = \mathbf{Q}_s \mathbf{K}_s^{T}.$2

Accordingly, RepSSA is neither a purely spike-only attention rule nor a conventional ANN attention rule. Its attention map is trained with a richer continuous-valued parameterization, whereas its deployment path is organized around a fused spike-driven computation. A plausible implication is that the method targets the specific representational bottleneck induced by premature query–key binarization rather than abandoning spike-driven inference.

3. Variants and computational regime

The paper defines two RepSSA variants, one linear and one softmax-based (Song et al., 10 Jul 2025).

Variant	Definition	Role in the paper
RepSSA$\mathbf{A}_s = \mathbf{Q}_s \mathbf{K}_s^{T}.$3	$\mathbf{A}_s = \mathbf{Q}_s \mathbf{K}_s^{T}.$4	Linear spiking self-attention
RepSSA$\mathbf{A}_s = \mathbf{Q}_s \mathbf{K}_s^{T}.$5	Softmax-based attention using $\mathbf{A}_s = \mathbf{Q}_s \mathbf{K}_s^{T}.$6, $\mathbf{A}_s = \mathbf{Q}_s \mathbf{K}_s^{T}.$7, and $\mathbf{A}_s = \mathbf{Q}_s \mathbf{K}_s^{T}.$8	Primary choice in IML-Spikeformer

The linear form is written as

$\mathbf{A}_s = \mathbf{Q}_s \mathbf{K}_s^{T}.$9

The softmax form, $$\mathbf{A} = \begin{cases} X W_Q W_K^T X^T, & \text{Training} \ X W_{QK} X^T, & \text{Inference} \end{cases}$$0, uses $$\mathbf{A} = \begin{cases} X W_Q W_K^T X^T, & \text{Training} \ X W_{QK} X^T, & \text{Inference} \end{cases}$$1 together with $$\mathbf{A} = \begin{cases} X W_Q W_K^T X^T, & \text{Training} \ X W_{QK} X^T, & \text{Inference} \end{cases}$$2 and the output projection $$\mathbf{A} = \begin{cases} X W_Q W_K^T X^T, & \text{Training} \ X W_{QK} X^T, & \text{Inference} \end{cases}$$3 (Song et al., 10 Jul 2025).

The paper states that, although linear attention can be theoretically attractive in some settings, the sequence length $$\mathbf{A} = \begin{cases} X W_Q W_K^T X^T, & \text{Training} \ X W_{QK} X^T, & \text{Inference} \end{cases}$$4 in this work is typically smaller than the feature dimension $$\mathbf{A} = \begin{cases} X W_Q W_K^T X^T, & \text{Training} \ X W_{QK} X^T, & \text{Inference} \end{cases}$$5, so linear attention does not provide a clear complexity advantage. It also states that linear attention can suffer from instability and unbounded-gradient issues. For these reasons, the final IML-Spikeformer primarily adopts RepSSA$$\mathbf{A} = \begin{cases} X W_Q W_K^T X^T, & \text{Training} \ X W_{QK} X^T, & \text{Inference} \end{cases}$$6 (Song et al., 10 Jul 2025).

This distinguishes RepSSA from earlier SDSA-style attention at the level of computational philosophy. SDSA effectively spikes first and computes attention from spike matrices. RepSSA instead learns a richer attention mapping during training and only then reparameterizes it for efficient deployment. That training–inference asymmetry is the mechanism’s defining characteristic.

4. HD-RepSSA and integration into IML-Spikeformer

RepSSA is used in the full model together with the Hierarchical Decay Mask (HDM), yielding the module called HD-RepSSA. The mask is defined as

$$\mathbf{A} = \begin{cases} X W_Q W_K^T X^T, & \text{Training} \ X W_{QK} X^T, & \text{Inference} \end{cases}$$7

with layer-dependent decay

$$\mathbf{A} = \begin{cases} X W_Q W_K^T X^T, & \text{Training} \ X W_{QK} X^T, & \text{Inference} \end{cases}$$8

The attention map is then modulated as

$$\mathbf{A} = \begin{cases} X W_Q W_K^T X^T, & \text{Training} \ X W_{QK} X^T, & \text{Inference} \end{cases}$$9

and the resulting module is

$W_{QK} = W_Q W_K^T,$0

The interpretation given in the paper is that shallow layers have stronger decay and thus more local attention, whereas deeper layers approach broader context (Song et al., 10 Jul 2025).

This mask introduces a structured inductive bias for speech. Early layers are biased toward local acoustic relationships, and deeper layers toward longer-range dependencies. The paper presents this as particularly important for speech signals with multiscale temporal structure (Song et al., 10 Jul 2025).

At the block level, each IML-Spikeformer block consists of two major submodules: HD-RepSSA for token mixing and ChannelMLP for channel mixing. The spiking neuron layers inside both modules use the proposed Input-aware Multi-Level Spike (IMLS) firing mechanism. Training uses IMLS multi-level spikes to simulate multi-timestep spike firing within a single timestep, while inference uses the reparameterized $W_{QK} = W_Q W_K^T,$1 attention path and converts IMLS multi-level spikes into equivalent binary spike trains for neuromorphic deployment (Song et al., 10 Jul 2025).

Taken together, these design choices place RepSSA at the center of the model’s token-interaction pathway. It is the attention core; HDM supplies the temporal prior; IMLS supplies the single-timestep spiking regime.

5. Empirical behavior, ablations, and reported performance

The ablation results reported for IML-Spikeformer are especially informative because they isolate the effect of RepSSA and HD-RepSSA on recognition accuracy (Song et al., 10 Jul 2025).

Replacement	Test CER	Change from HD-RepSSA$W_{QK} = W_Q W_K^T,$2
HD-RepSSA$W_{QK} = W_Q W_K^T,$3	5.9	—
RepSSA$W_{QK} = W_Q W_K^T,$4	6.5	+0.6
SDSA-3	9.0	+3.1
HD-RepSSA$W_{QK} = W_Q W_K^T,$5	6.7	+0.8
RepSSA$W_{QK} = W_Q W_K^T,$6	8.7	+2.8

These ablations support three claims made in the paper. First, replacing HD-RepSSA$W_{QK} = W_Q W_K^T,$7 with RepSSA$W_{QK} = W_Q W_K^T,$8 increases test CER from 5.9 to 6.5, indicating that HDM contributes additional benefit beyond RepSSA alone. Second, replacing HD-RepSSA$W_{QK} = W_Q W_K^T,$9 with SDSA-3 increases test CER from 5.9 to 9.0, which the paper treats as strong evidence that RepSSA is the major contributor to improved attention-map quality. Third, the softmax-based variants outperform the linear variants in this setting, with HD-RepSSA$\mathbf{V_s} = \mathcal{SN}(XW_V).$0 better than HD-RepSSA$\mathbf{V_s} = \mathcal{SN}(XW_V).$1 and RepSSA$\mathbf{V_s} = \mathcal{SN}(XW_V).$2 better than RepSSA$\mathbf{V_s} = \mathcal{SN}(XW_V).$3 (Song et al., 10 Jul 2025).

The paper also reports qualitative attention-map analysis. RepSSA$\mathbf{V_s} = \mathcal{SN}(XW_V).$4 without HDM produces near-uniform attention distributions across token pairs, and this behavior persists across shallow, intermediate, and deep layers. By contrast, HD-RepSSA$\mathbf{V_s} = \mathcal{SN}(XW_V).$5 yields structured, layer-dependent attention, in which shallow layers focus locally and deeper layers expand the receptive field to capture broader structure. This is used to argue that HDM supplies the hierarchical inductive bias needed for speech (Song et al., 10 Jul 2025).

At the system level, the paper reports that IML-Spikeformer achieves word error rates of 6.0\% on AiShell-1 and 3.4\% on Librispeech-960, comparable to conventional ANN transformers while reducing theoretical inference energy consumption by 4.64$\mathbf{V_s} = \mathcal{SN}(XW_V).$6 and 4.32$\mathbf{V_s} = \mathcal{SN}(XW_V).$7 respectively. It also reports an AiShell-1 ablation setting with test CER 5.9 for HD-RepSSA$\mathbf{V_s} = \mathcal{SN}(XW_V).$8, 30.35M parameters, and 6 timesteps. The paper does not isolate standalone compute numbers for RepSSA alone as cleanly as it does for the full model, but it explicitly ties the end-to-end gains to the improved attention module (Song et al., 10 Jul 2025).

6. Relation to adjacent spiking self-attention designs

RepSSA is one instance of a broader trend toward reparameterized or redesigned spiking attention, but the term is not used uniformly across the literature. Two nearby examples illustrate this point.

In “Spiking Transformer:Introducing Accurate Addition-Only Spiking Self-Attention for Transformer” (Guo et al., 28 Feb 2025), the term RepSSA does not appear explicitly. The closest correspondence is Accurate Addition-Only Spiking Self-Attention, A$\mathbf{V_s} = \mathcal{SN}(XW_V).$9OS$X$0A. That mechanism replaces all-binary $X$1, $X$2, and $X$3 with a hybrid assignment:

$X$4

and defines

$X$5

Its stated novelty is the hybrid neuron assignment—binary query, ReLU key, ternary value—together with removal of softmax and scaling. The paper describes this as a directly designed hybrid spiking attention rather than a classical post-training ANN-to-SNN conversion pipeline (Guo et al., 28 Feb 2025).

In “Neural Dynamics Self-Attention for Spiking Transformers” (Zhang et al., 9 Mar 2026), the exact term RepSSA also does not appear. The closest mechanisms are LRF-SSA and LRF-Dyn. LRF-SSA adds localized receptive fields to SSA, while LRF-Dyn reparameterizes the resulting computation into charge–fire–reset-like dynamics so that the attention matrix need not be explicitly stored. The dynamical form is written as

$X$6

The paper presents this as a way to reduce inference storage from approximately $X$7 to $X$8 while strengthening locality bias (Zhang et al., 9 Mar 2026).

These comparisons delimit the meaning of RepSSA. In the strict sense, RepSSA refers to the fused-query/key mechanism in IML-Spikeformer. In a broader descriptive sense, the term may be used for spiking self-attention blocks that are reformulated to recover expressiveness, locality, or efficiency under SNN constraints. This suggests that “reparameterized spiking self-attention” is better understood as a design pattern than as a single canonical architecture.

7. Conceptual significance and common points of confusion

RepSSA is best understood as a response to a specific failure mode of spiking Transformers: the loss of attention-map precision caused by early binary conversion of queries and keys. Its intervention is narrow but consequential. Rather than abandoning spike-driven inference, it relocates where precision is preserved—into the training-time attention parameterization—and where efficiency is enforced—into the inference-time fused matrix $X$9 (Song et al., 10 Jul 2025).

A common source of confusion is to equate RepSSA with any nonstandard spiking attention block. That identification is too broad. A$W_Q$0OS$W_Q$1A and LRF-Dyn are closely related in spirit, but they solve different bottlenecks. A$W_Q$2OS$W_Q$3A is organized around addition-only computation and mixed neuron types; LRF-Dyn is organized around locality bias and memory reduction through dynamical reformulation. RepSSA, by contrast, is centered on train-time continuous-valued query–key interactions and inference-time query–key fusion (Guo et al., 28 Feb 2025).

Another misconception is that reparameterization here implies a generic parameter-compression routine. In RepSSA, the reparameterization is specific:

$W_Q$4

Its purpose is not merely to reduce the number of matrices, but to preserve a richer attention map during training while maintaining spike-driven inference. A plausible implication is that the method should be read less as a generic compression device and more as an attention-specific deployment transformation.

Within the literature surveyed here, RepSSA therefore occupies a distinct position. It is a speech-oriented spiking attention mechanism whose main contribution lies in reconciling expressive attention-map formation with efficient spike-driven deployment, and whose strongest reported empirical support comes from the ablations showing a large gap between RepSSA-based attention and SDSA-3 in automatic speech recognition (Song et al., 10 Jul 2025).