---
title: S2A Self-Attention Mechanisms
url: https://www.emergentmind.com/topics/s2a-self-attention
type: topic
---

# S2A Self-Attention Mechanisms

In contemporary machine learning literature, “S2A self-attention” is not a single standardized mechanism but an overloaded label applied to several distinct constructions. The term appears in at least four technically different settings: “Stand-Alone Self-Attention” in vision backbones [1906.05909], “System 2 Attention” as a prompting wrapper for instruction-tuned LLMs [2311.11829], self-attention modules for feature-wise, temporal, and joint 2D weighting in Neural Bag-of-Features pipelines [2201.11092], and “Strip Self-Attention” in efficient Vision Transformers [2505.22195]. Across these usages, the common substrate is the standard self-attention operator, which for an input sequence $X \in \mathbb{R}^{T \times d}$ forms $Q = XW^Q$, $K = XW^K$, and $V = XW^V$, computes $A = \operatorname{softmax}(QK^\top/\sqrt{d_k})$ row-wise, and returns $Z = AV$ [2006.03265].

## 1. Terminological scope and common formal basis

The shared mathematical core of these methods is scaled dot-product attention, but the intervention point differs substantially across papers. In one line of work, self-attention is the primitive layer itself; in another, it is left untouched internally and is instead preceded by an external relevance-filtering stage; in another, it is specialized to feature and temporal axes before NBoF quantization; and in another, it is compressed spatially and channel-wise for efficient vision inference [1906.05909, 2311.11829, 2201.11092, 2505.22195].

| Usage of “S2A” | Domain | Defining operation |
|---|---|---|
| Stand-Alone Self-Attention | Vision backbones | Replaces $3 \times 3$ spatial convolutions with local self-attention |
| System 2 Attention | Instruction-tuned LLMs | Regenerates context to keep only relevant or unbiased portions before final generation |
| Self-Attention Neural Bag-of-Features | Multivariate sequence analysis | Learns feature-wise, temporal, and joint 2D attention masks before NBoF |
| Strip Self-Attention | Efficient ViTs | Reduces spatial dimensions of $K,V$ and compresses channel dimensions of $Q,K$ |

A common misconception is that all “S2A” methods denote architectural variants of the Transformer attention kernel. The literature does not support that reading. “System 2 Attention” explicitly leaves “the internal Transformer architecture (including its QKV projections, multi-headed soft attention, feed-forwards, etc.) unchanged,” whereas the vision-oriented variants modify the attention layer itself or its placement in the network [2311.11829].

## 2. Stand-Alone Self-Attention in vision models

“Stand-Alone Self-Attention” investigates whether self-attention can serve as a vision model’s primary spatial operator rather than merely augmenting convolutions [1906.05909]. The layer operates on an input feature map $X \in \mathbb{R}^{H \times W \times C}$, treats each spatial location $(i,j)$ as a token, restricts attention to a local $k \times k$ neighborhood $\mathcal{N}_k(i,j)$, and adds a learned relative positional bias factorized into row and column offsets. Within each head, the attention weights are computed over the local neighborhood rather than globally, and the outputs from all heads are concatenated and optionally projected back to the channel dimension.

This local-window construction is central to the method’s efficiency profile. The paper replaces the $3 \times 3$ spatial convolution inside a ResNet-style bottleneck with a local self-attention layer. In the main experiments, the S2A-ResNet uses $k=7$, $N=8$ heads, and $d_k = C_{\text{mid}}/8$, while spatial downsampling is implemented by $2 \times 2$ average pooling after attention when the original ResNet would have used a strided convolution. The resulting block is: input $\rightarrow 1 \times 1$ conv $\rightarrow$ norm $\rightarrow$ self-attention$(k=7)$ $\rightarrow$ norm $\rightarrow 1 \times 1$ conv $\rightarrow$ residual sum [1906.05909].

Empirically, the full-attention ResNet-50 variant reports $77.4\%$ top-1 accuracy on ImageNet-1K, compared with $76.9\%$ for the ResNet-50 baseline, while using $7.2$B FLOPS instead of $8.2$B and $18.0$M parameters instead of $25.6$M. On COCO 2017 object detection with RetinaNet, the full-attention model reports $\mathrm{mAP}_{[.50:.95]} = 36.6\%$, compared with $36.5\%$ for the baseline, while reducing FLOPS from $182$B to $110$B and parameters from $33.4$M to $22.0$M [1906.05909].

The ablations clarify where the gains arise. Replacing convolutions in later groups is especially effective: “Conv in groups $\{1,2\}$ + Attn in $\{3,4\}$” and “Conv in $\{1\}$ + Attn in $\{2,3,4\}$” both reach $80.7\%$ top-1, while “All Attn” gives $80.2\%$ and “All Conv” gives $79.5\%$. The paper also shows that positional encoding choice is decisive: no positional encoding yields $77.6\%$ top-1, absolute sinusoidal encoding yields $78.2\%$, and relative encoding yields $80.2\%$ [1906.05909]. This directly counters the assumption that the gain comes from content-based interactions alone; the data indicate that 2D relative biases are critical.

## 3. System 2 Attention in large language models

“System 2 Attention” reframes the attention problem in LLMs as one of context selection rather than QKV redesign [2311.11829]. Its starting claim is that soft attention in Transformer-based LLMs can incorporate irrelevant information from context into latent representations, adversely affecting next-token generations. The proposed remedy is a two-step, model-agnostic wrapper around any instruction-tuned LLM.

The first step, “Context Regeneration,” prompts the model to “extract only the unbiased/relevant portions (no opinions or spurious facts), and restate the actual question,” separating the output into “Unbiased text context” and “Question/Query.” The same LLM is called in zero-shot to produce a rewritten context $x'$. The second step prompts the model with “Unbiased context: $\langle x' \rangle$” and “Answer in an unbiased way,” producing the final response. Formally, the paper represents this as a hard-mask-like preprocessing stage: given $x = [x_1,\dots,x_n] \in V^n$, an implicit relevance filter $f_{\text{filter}}$ returns $M = f_{\text{filter}}(x) \in \{0,1\}^n$, and the filtered context is $x' = [x_i : M_i = 1] \in V^k$, after which standard multihead attention proceeds over $x'$ [2311.11829].

Because the Transformer internals are unchanged, System 2 Attention should not be conflated with architectural self-attention variants. Its novelty lies in upstream context filtering via natural-language prompting. The paper also studies several variants: a no-separation prompt, a keep-original-context variant that concatenates $x$ and $x'$ in the second step, a relevance-based prompt for math problems, and ablations that remove the “unbiased” instruction or replace regeneration with instructed prompting alone [2311.11829].

The reported evaluation covers three task types. On factual QA with opinions injected, using modified TriviaQA within SycophancyEval, the baseline LLaMA-2-70B-chat scores $62.8\%$ overall accuracy, the oracle with de-biased prompts scores $82.0\%$, and S2A scores $80.3\%$. The “Suggest Incorrect” subset is especially revealing: baseline $11.0\%$, oracle $82.0\%$, S2A $76.0\%$. On long-form arguments, the baseline has quality $4.7$ and objectivity $2.23$, the oracle has quality $4.6$ and objectivity $3.00$, and S2A has quality $4.6$ and objectivity $3.82$. On GSM-IC math word problems with distractors, S2A reaches $63.0\%$ for random distractors and $57.8\%$ for in-topic distractors, compared with baseline values of $51.7\%$ and $47.5\%$ respectively; all improvements over the baseline are statistically significant with $p<0.01$ under paired bootstrap [2311.11829].

The ablations also delimit the method’s scope. “S2A-Single” reaches $79.1\%$ on TriviaQA, “S2A-NI” reaches $78.2\%$, “S2A-KeepOrig” drops to $74.5\%$, instructed prompting reaches $71.7\%$, and zero-shot CoT reaches $59.2\%$ [2311.11829]. The paper lists three explicit limitations: double inference cost, possible relevance misclassification in long contexts, and incomplete regenerated context for weaker models.

## 4. Self-attention Neural Bag-of-Features

In “Self-Attention Neural Bag-of-Features,” self-attention is adapted to multivariate sequence data by constructing separate feature-wise and temporal attention operators, together with a joint 2D feature-temporal mask [2201.11092]. The input is $X \in \mathbb{R}^{T \times F}$, with $T$ time steps and $F$ features. Two latent projections are introduced: a feature embedding $Z^f = P^f X^\top \in \mathbb{R}^{d \times T}$ and a temporal embedding $Z^t = P^t X \in \mathbb{R}^{d \times F}$.

Feature-wise self-attention forms
$$
Q^f = X^\top W_q^f \in \mathbb{R}^{F \times d}, \qquad
K^f = X^\top W_k^f \in \mathbb{R}^{F \times d},
$$
then computes a raw score matrix $S^f = Q^f (K^f)^\top / \sqrt{d}$ and a row-wise softmax $A^f \in \mathbb{R}^{F \times F}$. Temporal self-attention analogously forms
$$
Q^t = X W_q^t \in \mathbb{R}^{T \times d}, \qquad
K^t = X W_k^t \in \mathbb{R}^{T \times d},
$$
computes $S^t = Q^t (K^t)^\top / \sqrt{d}$, and applies a row-wise softmax to obtain $A^t \in \mathbb{R}^{T \times T}$ [2201.11092].

The joint 2D mechanism learns a single mask $M \in \mathbb{R}^{T \times F}$ over all time-feature cells. The sequence is flattened to $\operatorname{vec}(X) \in \mathbb{R}^{TF}$, projected via $P^{ft} \in \mathbb{R}^{d \times TF}$ to $z^{ft} \in \mathbb{R}^d$, and mapped back through $R \in \mathbb{R}^{TF \times d}$ to a score matrix $S^{ft} \in \mathbb{R}^{T \times F}$. The final mask is either sigmoid-scaled cellwise or normalized by a global 2D softmax over all $TF$ cells. These masks are then applied as
$$
\widehat{X}^f = X(A^f)^\top, \qquad
\widehat{X}^t = A^t X, \qquad
\widehat{X}^{ft} = M \odot X.
$$
Each attended stream is then fed independently into the downstream Neural Bag-of-Features quantization layer, producing fixed-size histograms that can be concatenated or combined for classification [2201.11092].

The architecture can also use multi-head attention, with independent masks per head that are concatenated or averaged. After NBoF quantization and temporal aggregation, the histograms may pass through a residual connection of the form
$$
Z_{\text{out}} = \alpha Z_{\text{hist}} + (1-\alpha)\,\operatorname{LayerNorm}(Z_{\text{hist}}),
$$
or a similar layer-normalized combination, before the final softmax classifier. Training is end-to-end with cross-entropy or binary cross-entropy, Adam, dropout on attention weights, and $L_2$ regularization on projection matrices [2201.11092].

The reported gains are task-specific but consistent. On TUT-UAS2018 acoustic-scene classification, temporal self-attention yields up to $+1.5$ percentage points absolute over standard 2D-Attention, from $56.1\%$ to $57.6\%$ accuracy. On ECG and PCG detection, codeword-temporal joint self-attention and codeword self-attention improve F1 by $0.5$–$1.0$ percentage points over 2D-Attention baselines. Multi-head variants with $h=2$ or $4$ often add another $0.5$–$1.0$ percentage points at the cost of modest extra parameters [2201.11092].

## 5. Strip Self-Attention in efficient Vision Transformers

“Strip Self-Attention” in S2AFormer is an efficiency-oriented reformulation of self-attention for ViTs [2505.22195]. Given $X \in \mathbb{R}^{H \times W \times C}$ and its flattened representation $\hat{X} \in \mathbb{R}^{N \times C}$ with $N = HW$, the method departs from standard MHSA in two ways. First, it applies a spatial reduction to $K$ and $V$ using a depth-wise convolution of kernel size $k \times k$ and stride $k$, producing a reduced feature map $\tilde{X} \in \mathbb{R}^{(H/k) \times (W/k) \times C}$ and $N_s = (H/k)(W/k)$ reduced tokens. Second, it compresses the channel dimensions of $Q$ and $K$ to a narrow width $h \ll C$.

The resulting projections are
$$
Q = \hat{X} W_Q^s \in \mathbb{R}^{N \times h}, \qquad
K = \tilde{\hat{X}} W_K^s \in \mathbb{R}^{N_s \times h}, \qquad
V = \tilde{\hat{X}} W_V \in \mathbb{R}^{N_s \times d},
$$
with $V$ kept at full channel dimension for the final output. Attention is then computed as
$$
A = \operatorname{Softmax}(QK^\top/\sqrt{h}) \in \mathbb{R}^{N \times N_s}, \qquad
Y = AV \in \mathbb{R}^{N \times d}.
$$
Compared with standard self-attention, which forms an $N \times N$ similarity matrix, SSA forms only an $N \times N_s$ matrix, where $N_s = N/k^2$ [2505.22195].

The computational analysis makes the reduction explicit. Standard MHSA has complexity
$$
\mathcal{O}(\mathrm{MHSA}) = 3Nd^2 + 2N^2d.
$$
SSA has complexity
$$
\mathcal{O}(\mathrm{SSA}) =
Ndh + \frac{Ndh}{k^2} + \frac{Nd^2}{k^2} + \frac{N^2h}{k^2} + \frac{N^2d}{k^2}.
$$
When $h \ll d$ and $k>1$, the dominant quadratic term becomes approximately $(N^2 d)/k^2$ rather than $N^2 d$, yielding an approximately $k^2$-fold saving in spatial cost [2505.22195].

The paper’s terminology is also precise. “Strip” refers to both compressed channel “strips,” because $Q$ and $K$ are squeezed from $C$ to $h$, and coarse spatial “strips,” because the depth-wise convolution partitions the feature map into non-overlapping $k \times k$ blocks, each collapsed into one key/value token. In the full Hybrid Perception Block, SSA is interleaved with a depth-wise convolution plus residual, a Local Interaction Module plus residual, and an MLP plus residual [2505.22195].

Representative benchmark results support the efficiency claim. On ImageNet-1K at $224^2$ input, S2AFormer-XS reports $78.9\%$ top-1 with $6.5$M parameters and $0.79$ GMACs, compared with $74.4\%$ for EdgeViT-XXS at $4.1$M parameters and $0.6$ GMACs. On ADE20K semantic segmentation with Semantic FPN, S2AFormer-T reports $38.0\%$ mIoU with $7$M parameters and $25$ GFLOPs, compared with $37.2\%$ mIoU for PoolFormer-S12 at $16$M parameters and $31$ GFLOPs. On COCO with RetinaNet $1\times$, S2AFormer-mini reports $33.4\%$ AP with $11.7$M parameters and $159$ GFLOPs, compared with $31.8\%$ AP for a ResNet-18 backbone with $21$M parameters [2505.22195].

## 6. Attention specialization, analysis, and refinement in audio transformers

Although not itself named “S2A,” the analysis of self-attention in self-supervised audio Transformers provides a useful taxonomy for understanding how attention heads specialize and how they can be ranked, visualized, and refined [2006.03265]. The paper inspects head-level attention maps $A_u^h$ and identifies three prototypical patterns: diagonal, vertical, and global.

Diagonal heads attend primarily to frames near $t \pm \Delta$ and act as local-span or phoneme-aware heads. Vertical heads attend to a small set of fixed key positions independently of the query, appearing as vertical lines in the attention map; these heads tend to focus on or neglect particular phoneme classes such as silence or voiced segments. Global heads display flat, high-entropy, noisy patterns that spread attention broadly [2006.03265].

The classification is defined through three corpus-averaged metrics:
$$
G(h)=\mathbb{E}_u \left[\frac{1}{T}\sum_{q=1}^T H(A_u^h[q,\cdot])\right],
$$
$$
V(h)=\mathbb{E}_u \left[-H\!\left(\frac{1}{T}\sum_{q=1}^T A_u^h[q,\cdot]\right)\right],
$$
$$
D(h)=\mathbb{E}_u \left[-\frac{1}{T^2}\sum_{q,k}|q-k|A_u^h[q,k]\right].
$$
Heads are ranked separately by $G$, $V$, and $D$, and assigned to the category where they obtain their top rank because the three metrics live on different numeric scales [2006.03265].

The functional interpretation is concrete. Diagonal heads capture strictly local context, and block-diagonal variants align almost exactly with phoneme boundaries and can be used for unsupervised phoneme segmentation. Vertical heads ignore temporal locality and instead learn to search for particular phoneme classes; via a phoneme-relation map, some vertical heads consistently focus on or neglect a small subset of phonemes regardless of query, which correlates with speaker-or-noise cues. Global heads are the least useful for phoneme or reconstruction tasks and can be pruned to speed up inference [2006.03265].

The paper also introduces a visualization and pruning toolkit. Heads can be sorted by $G$, $V$, or $D$ and displayed as heat maps, optionally over utterances that maximize each metric. For importance ranking, the authors compare metric-based head pruning with a weight-based baseline $W(h)=\max_{u,q,k} A_u^h[q,k]$. The downstream tasks are spectrogram reconstruction, frame-level phoneme classification, and frame- and utterance-level speaker recognition. Diagonal heads ranked by $D(h)$ are the single most critical group: removing the first approximately $24$ diagonal heads causes a sharp performance drop on all tasks. Vertical heads rank second, particularly for speaker-ID probes. Global heads ranked by $G(h)$ are the least important, and pruning them often improves phoneme classification [2006.03265].

Two inference-time refinements follow directly from this analysis. Head pruning via globalness removes all heads with $G(h)>t$ for a chosen threshold $t$; pruning up to approximately $50\%$ of heads, corresponding to all global heads, does not degrade and sometimes improves phoneme classification, while only mildly harming utterance-level speaker identification. Span-based pruning imposes a fixed radius $r$ by masking all attention outside $|q-k|>r$; as $r$ decreases, phoneme discrimination improves up to a point, though very small $r$ hurts reconstruction [2006.03265]. A plausible implication is that several later “S2A” variants in other domains can be read as structured attempts to bias attention toward relevance, locality, or compressed context rather than treating all tokens as equally eligible recipients of soft attention.

Source: https://www.emergentmind.com/topics/s2a-self-attention