S2A Self-Attention Mechanisms

• S2A Self-Attention is a collective term for several self-attention variants unified by the scaled dot-product mechanism across vision, language, and audio domains.
• It encompasses methods like Stand-Alone Self-Attention, System 2 Attention, Neural Bag-of-Features attention, and Strip Self-Attention, each with distinct architectural interventions.
• Empirical results demonstrate that these approaches can improve efficiency, accuracy, and context filtering, making them valuable for tailored applications in modern ML pipelines.

In contemporary machine learning literature, “S2A self-attention” is not a single standardized mechanism but an overloaded label applied to several distinct constructions. The term appears in at least four technically different settings: “Stand-Alone Self-Attention” in vision backbones (Ramachandran et al., 2019), “System 2 Attention” as a prompting wrapper for instruction-tuned LLMs (Weston et al., 2023), self-attention modules for feature-wise, temporal, and joint 2D weighting in Neural Bag-of-Features pipelines (Chumachenko et al., 2022), and “Strip Self-Attention” in efficient Vision Transformers (Xu et al., 28 May 2025). Across these usages, the common substrate is the standard self-attention operator, which for an input sequence $X \in \mathbb{R}^{T \times d}$ forms $Q = XW^Q$, $K = XW^K$, and $V = XW^V$, computes $A = \operatorname{softmax}(QK^\top/\sqrt{d_k})$ row-wise, and returns $Z = AV$ (Yang et al., 2020).

1. Terminological scope and common formal basis

The shared mathematical core of these methods is scaled dot-product attention, but the intervention point differs substantially across papers. In one line of work, self-attention is the primitive layer itself; in another, it is left untouched internally and is instead preceded by an external relevance-filtering stage; in another, it is specialized to feature and temporal axes before NBoF quantization; and in another, it is compressed spatially and channel-wise for efficient vision inference (Ramachandran et al., 2019, Weston et al., 2023, Chumachenko et al., 2022, Xu et al., 28 May 2025).

Usage of “S2A”	Domain	Defining operation
Stand-Alone Self-Attention	Vision backbones	Replaces $3 \times 3$ spatial convolutions with local self-attention
System 2 Attention	Instruction-tuned LLMs	Regenerates context to keep only relevant or unbiased portions before final generation
Self-Attention Neural Bag-of-Features	Multivariate sequence analysis	Learns feature-wise, temporal, and joint 2D attention masks before NBoF
Strip Self-Attention	Efficient ViTs	Reduces spatial dimensions of $K,V$ and compresses channel dimensions of $Q,K$

A common misconception is that all “S2A” methods denote architectural variants of the Transformer attention kernel. The literature does not support that reading. “System 2 Attention” explicitly leaves “the internal Transformer architecture (including its QKV projections, multi-headed soft attention, feed-forwards, etc.) unchanged,” whereas the vision-oriented variants modify the attention layer itself or its placement in the network (Weston et al., 2023).

2. Stand-Alone Self-Attention in vision models

“Stand-Alone Self-Attention” investigates whether self-attention can serve as a vision model’s primary spatial operator rather than merely augmenting convolutions (Ramachandran et al., 2019). The layer operates on an input feature map $X \in \mathbb{R}^{H \times W \times C}$, treats each spatial location $Q = XW^Q$0 as a token, restricts attention to a local $Q = XW^Q$1 neighborhood $Q = XW^Q$2, and adds a learned relative positional bias factorized into row and column offsets. Within each head, the attention weights are computed over the local neighborhood rather than globally, and the outputs from all heads are concatenated and optionally projected back to the channel dimension.

This local-window construction is central to the method’s efficiency profile. The paper replaces the $Q = XW^Q$3 spatial convolution inside a ResNet-style bottleneck with a local self-attention layer. In the main experiments, the S2A-ResNet uses $Q = XW^Q$4, $Q = XW^Q$5 heads, and $Q = XW^Q$6, while spatial downsampling is implemented by $Q = XW^Q$7 average pooling after attention when the original ResNet would have used a strided convolution. The resulting block is: input $Q = XW^Q$8 conv $Q = XW^Q$9 norm $K = XW^K$0 self-attention$K = XW^K$1 $K = XW^K$2 norm $K = XW^K$3 conv $K = XW^K$4 residual sum (Ramachandran et al., 2019).

Empirically, the full-attention ResNet-50 variant reports $K = XW^K$5 top-1 accuracy on ImageNet-1K, compared with $K = XW^K$6 for the ResNet-50 baseline, while using $K = XW^K$7B FLOPS instead of $K = XW^K$8B and $K = XW^K$9M parameters instead of $V = XW^V$0M. On COCO 2017 object detection with RetinaNet, the full-attention model reports $V = XW^V$1, compared with $V = XW^V$2 for the baseline, while reducing FLOPS from $V = XW^V$3B to $V = XW^V$4B and parameters from $V = XW^V$5M to $V = XW^V$6M (Ramachandran et al., 2019).

The ablations clarify where the gains arise. Replacing convolutions in later groups is especially effective: “Conv in groups $V = XW^V$7 + Attn in $V = XW^V$8” and “Conv in $V = XW^V$9 + Attn in $A = \operatorname{softmax}(QK^\top/\sqrt{d_k})$0” both reach $A = \operatorname{softmax}(QK^\top/\sqrt{d_k})$1 top-1, while “All Attn” gives $A = \operatorname{softmax}(QK^\top/\sqrt{d_k})$2 and “All Conv” gives $A = \operatorname{softmax}(QK^\top/\sqrt{d_k})$3. The paper also shows that positional encoding choice is decisive: no positional encoding yields $A = \operatorname{softmax}(QK^\top/\sqrt{d_k})$4 top-1, absolute sinusoidal encoding yields $A = \operatorname{softmax}(QK^\top/\sqrt{d_k})$5, and relative encoding yields $A = \operatorname{softmax}(QK^\top/\sqrt{d_k})$6 (Ramachandran et al., 2019). This directly counters the assumption that the gain comes from content-based interactions alone; the data indicate that 2D relative biases are critical.

3. System 2 Attention in LLMs

“System 2 Attention” reframes the attention problem in LLMs as one of context selection rather than QKV redesign (Weston et al., 2023). Its starting claim is that soft attention in Transformer-based LLMs can incorporate irrelevant information from context into latent representations, adversely affecting next-token generations. The proposed remedy is a two-step, model-agnostic wrapper around any instruction-tuned LLM.

The first step, “Context Regeneration,” prompts the model to “extract only the unbiased/relevant portions (no opinions or spurious facts), and restate the actual question,” separating the output into “Unbiased text context” and “Question/Query.” The same LLM is called in zero-shot to produce a rewritten context $A = \operatorname{softmax}(QK^\top/\sqrt{d_k})$7. The second step prompts the model with “Unbiased context: $A = \operatorname{softmax}(QK^\top/\sqrt{d_k})$8” and “Answer in an unbiased way,” producing the final response. Formally, the paper represents this as a hard-mask-like preprocessing stage: given $A = \operatorname{softmax}(QK^\top/\sqrt{d_k})$9, an implicit relevance filter $Z = AV$0 returns $Z = AV$1, and the filtered context is $Z = AV$2, after which standard multihead attention proceeds over $Z = AV$3 (Weston et al., 2023).

Because the Transformer internals are unchanged, System 2 Attention should not be conflated with architectural self-attention variants. Its novelty lies in upstream context filtering via natural-language prompting. The paper also studies several variants: a no-separation prompt, a keep-original-context variant that concatenates $Z = AV$4 and $Z = AV$5 in the second step, a relevance-based prompt for math problems, and ablations that remove the “unbiased” instruction or replace regeneration with instructed prompting alone (Weston et al., 2023).

The reported evaluation covers three task types. On factual QA with opinions injected, using modified TriviaQA within SycophancyEval, the baseline LLaMA-2-70B-chat scores $Z = AV$6 overall accuracy, the oracle with de-biased prompts scores $Z = AV$7, and S2A scores $Z = AV$8. The “Suggest Incorrect” subset is especially revealing: baseline $Z = AV$9, oracle $3 \times 3$0, S2A $3 \times 3$1. On long-form arguments, the baseline has quality $3 \times 3$2 and objectivity $3 \times 3$3, the oracle has quality $3 \times 3$4 and objectivity $3 \times 3$5, and S2A has quality $3 \times 3$6 and objectivity $3 \times 3$7. On GSM-IC math word problems with distractors, S2A reaches $3 \times 3$8 for random distractors and $3 \times 3$9 for in-topic distractors, compared with baseline values of $K,V$0 and $K,V$1 respectively; all improvements over the baseline are statistically significant with $K,V$2 under paired bootstrap (Weston et al., 2023).

The ablations also delimit the method’s scope. “S2A-Single” reaches $K,V$3 on TriviaQA, “S2A-NI” reaches $K,V$4, “S2A-KeepOrig” drops to $K,V$5, instructed prompting reaches $K,V$6, and zero-shot CoT reaches $K,V$7 (Weston et al., 2023). The paper lists three explicit limitations: double inference cost, possible relevance misclassification in long contexts, and incomplete regenerated context for weaker models.

4. Self-attention Neural Bag-of-Features

In “Self-Attention Neural Bag-of-Features,” self-attention is adapted to multivariate sequence data by constructing separate feature-wise and temporal attention operators, together with a joint 2D feature-temporal mask (Chumachenko et al., 2022). The input is $K,V$8, with $K,V$9 time steps and $Q,K$0 features. Two latent projections are introduced: a feature embedding $Q,K$1 and a temporal embedding $Q,K$2.

Feature-wise self-attention forms

$Q,K$3

then computes a raw score matrix $Q,K$4 and a row-wise softmax $Q,K$5. Temporal self-attention analogously forms

$Q,K$6

computes $Q,K$7, and applies a row-wise softmax to obtain $Q,K$8 (Chumachenko et al., 2022).

The joint 2D mechanism learns a single mask $Q,K$9 over all time-feature cells. The sequence is flattened to $X \in \mathbb{R}^{H \times W \times C}$0, projected via $X \in \mathbb{R}^{H \times W \times C}$1 to $X \in \mathbb{R}^{H \times W \times C}$2, and mapped back through $X \in \mathbb{R}^{H \times W \times C}$3 to a score matrix $X \in \mathbb{R}^{H \times W \times C}$4. The final mask is either sigmoid-scaled cellwise or normalized by a global 2D softmax over all $X \in \mathbb{R}^{H \times W \times C}$5 cells. These masks are then applied as

$X \in \mathbb{R}^{H \times W \times C}$6

Each attended stream is then fed independently into the downstream Neural Bag-of-Features quantization layer, producing fixed-size histograms that can be concatenated or combined for classification (Chumachenko et al., 2022).

The architecture can also use multi-head attention, with independent masks per head that are concatenated or averaged. After NBoF quantization and temporal aggregation, the histograms may pass through a residual connection of the form

$X \in \mathbb{R}^{H \times W \times C}$7

or a similar layer-normalized combination, before the final softmax classifier. Training is end-to-end with cross-entropy or binary cross-entropy, Adam, dropout on attention weights, and $X \in \mathbb{R}^{H \times W \times C}$8 regularization on projection matrices (Chumachenko et al., 2022).

The reported gains are task-specific but consistent. On TUT-UAS2018 acoustic-scene classification, temporal self-attention yields up to $X \in \mathbb{R}^{H \times W \times C}$9 percentage points absolute over standard 2D-Attention, from $Q = XW^Q$00 to $Q = XW^Q$01 accuracy. On ECG and PCG detection, codeword-temporal joint self-attention and codeword self-attention improve F1 by $Q = XW^Q$02–$Q = XW^Q$03 percentage points over 2D-Attention baselines. Multi-head variants with $Q = XW^Q$04 or $Q = XW^Q$05 often add another $Q = XW^Q$06–$Q = XW^Q$07 percentage points at the cost of modest extra parameters (Chumachenko et al., 2022).

5. Strip Self-Attention in efficient Vision Transformers

“Strip Self-Attention” in S2AFormer is an efficiency-oriented reformulation of self-attention for ViTs (Xu et al., 28 May 2025). Given $Q = XW^Q$08 and its flattened representation $Q = XW^Q$09 with $Q = XW^Q$10, the method departs from standard MHSA in two ways. First, it applies a spatial reduction to $Q = XW^Q$11 and $Q = XW^Q$12 using a depth-wise convolution of kernel size $Q = XW^Q$13 and stride $Q = XW^Q$14, producing a reduced feature map $Q = XW^Q$15 and $Q = XW^Q$16 reduced tokens. Second, it compresses the channel dimensions of $Q = XW^Q$17 and $Q = XW^Q$18 to a narrow width $Q = XW^Q$19.

The resulting projections are

$Q = XW^Q$20

with $Q = XW^Q$21 kept at full channel dimension for the final output. Attention is then computed as

$Q = XW^Q$22

Compared with standard self-attention, which forms an $Q = XW^Q$23 similarity matrix, SSA forms only an $Q = XW^Q$24 matrix, where $Q = XW^Q$25 (Xu et al., 28 May 2025).

The computational analysis makes the reduction explicit. Standard MHSA has complexity

$Q = XW^Q$26

SSA has complexity

$Q = XW^Q$27

When $Q = XW^Q$28 and $Q = XW^Q$29, the dominant quadratic term becomes approximately $Q = XW^Q$30 rather than $Q = XW^Q$31, yielding an approximately $Q = XW^Q$32-fold saving in spatial cost (Xu et al., 28 May 2025).

The paper’s terminology is also precise. “Strip” refers to both compressed channel “strips,” because $Q = XW^Q$33 and $Q = XW^Q$34 are squeezed from $Q = XW^Q$35 to $Q = XW^Q$36, and coarse spatial “strips,” because the depth-wise convolution partitions the feature map into non-overlapping $Q = XW^Q$37 blocks, each collapsed into one key/value token. In the full Hybrid Perception Block, SSA is interleaved with a depth-wise convolution plus residual, a Local Interaction Module plus residual, and an MLP plus residual (Xu et al., 28 May 2025).

Representative benchmark results support the efficiency claim. On ImageNet-1K at $Q = XW^Q$38 input, S2AFormer-XS reports $Q = XW^Q$39 top-1 with $Q = XW^Q$40M parameters and $Q = XW^Q$41 GMACs, compared with $Q = XW^Q$42 for EdgeViT-XXS at $Q = XW^Q$43M parameters and $Q = XW^Q$44 GMACs. On ADE20K semantic segmentation with Semantic FPN, S2AFormer-T reports $Q = XW^Q$45 mIoU with $Q = XW^Q$46M parameters and $Q = XW^Q$47 GFLOPs, compared with $Q = XW^Q$48 mIoU for PoolFormer-S12 at $Q = XW^Q$49M parameters and $Q = XW^Q$50 GFLOPs. On COCO with RetinaNet $Q = XW^Q$51, S2AFormer-mini reports $Q = XW^Q$52 AP with $Q = XW^Q$53M parameters and $Q = XW^Q$54 GFLOPs, compared with $Q = XW^Q$55 AP for a ResNet-18 backbone with $Q = XW^Q$56M parameters (Xu et al., 28 May 2025).

6. Attention specialization, analysis, and refinement in audio transformers

Although not itself named “S2A,” the analysis of self-attention in self-supervised audio Transformers provides a useful taxonomy for understanding how attention heads specialize and how they can be ranked, visualized, and refined (Yang et al., 2020). The paper inspects head-level attention maps $Q = XW^Q$57 and identifies three prototypical patterns: diagonal, vertical, and global.

Diagonal heads attend primarily to frames near $Q = XW^Q$58 and act as local-span or phoneme-aware heads. Vertical heads attend to a small set of fixed key positions independently of the query, appearing as vertical lines in the attention map; these heads tend to focus on or neglect particular phoneme classes such as silence or voiced segments. Global heads display flat, high-entropy, noisy patterns that spread attention broadly (Yang et al., 2020).

The classification is defined through three corpus-averaged metrics:

$Q = XW^Q$59

$Q = XW^Q$60

$Q = XW^Q$61

Heads are ranked separately by $Q = XW^Q$62, $Q = XW^Q$63, and $Q = XW^Q$64, and assigned to the category where they obtain their top rank because the three metrics live on different numeric scales (Yang et al., 2020).

The functional interpretation is concrete. Diagonal heads capture strictly local context, and block-diagonal variants align almost exactly with phoneme boundaries and can be used for unsupervised phoneme segmentation. Vertical heads ignore temporal locality and instead learn to search for particular phoneme classes; via a phoneme-relation map, some vertical heads consistently focus on or neglect a small subset of phonemes regardless of query, which correlates with speaker-or-noise cues. Global heads are the least useful for phoneme or reconstruction tasks and can be pruned to speed up inference (Yang et al., 2020).

The paper also introduces a visualization and pruning toolkit. Heads can be sorted by $Q = XW^Q$65, $Q = XW^Q$66, or $Q = XW^Q$67 and displayed as heat maps, optionally over utterances that maximize each metric. For importance ranking, the authors compare metric-based head pruning with a weight-based baseline $Q = XW^Q$68. The downstream tasks are spectrogram reconstruction, frame-level phoneme classification, and frame- and utterance-level speaker recognition. Diagonal heads ranked by $Q = XW^Q$69 are the single most critical group: removing the first approximately $Q = XW^Q$70 diagonal heads causes a sharp performance drop on all tasks. Vertical heads rank second, particularly for speaker-ID probes. Global heads ranked by $Q = XW^Q$71 are the least important, and pruning them often improves phoneme classification (Yang et al., 2020).

Two inference-time refinements follow directly from this analysis. Head pruning via globalness removes all heads with $Q = XW^Q$72 for a chosen threshold $Q = XW^Q$73; pruning up to approximately $Q = XW^Q$74 of heads, corresponding to all global heads, does not degrade and sometimes improves phoneme classification, while only mildly harming utterance-level speaker identification. Span-based pruning imposes a fixed radius $Q = XW^Q$75 by masking all attention outside $Q = XW^Q$76; as $Q = XW^Q$77 decreases, phoneme discrimination improves up to a point, though very small $Q = XW^Q$78 hurts reconstruction (Yang et al., 2020). A plausible implication is that several later “S2A” variants in other domains can be read as structured attempts to bias attention toward relevance, locality, or compressed context rather than treating all tokens as equally eligible recipients of soft attention.