Slash-Dominant Heads (SDHs) in Transformers

• SDHs are attention heads in decoder-only Transformers that concentrate attention along fixed sub-diagonals, representing consistent token lags.
• They emerge from nearly rank-one query/key matrices and the dominant influence of medium- and high-frequency components in Rotary Position Embedding.
• SDHs contribute to parameter compression, improved long-range context tracking, and mechanistic interpretability in large language models.

Slash-Dominant Heads (SDHs) are attention heads in decoder-only Transformers, particularly LLMs, that exhibit attention score concentration along a fixed sub-diagonal ("slash line") of the attention score matrix. This manifests as persistent patterns where, for a given offset $\Delta$, an attention head focuses disproportionately on positions that are exactly $\Delta$ tokens behind the current token. These phenomena, intrinsic to model architecture and the employment of Rotary Position Embedding (RoPE), have been empirically documented across a broad range of models and elucidated through theoretical analysis (Cheng et al., 13 Jan 2026).

1. Formal Definition and Notation

Let $S(P)\in\mathbb R^{N\times N}$ denote the attention score (after row-wise softmax) for a prompt $P$ of context length $N$ in a causal attention head. For an integer offset $\Delta\ge 0$ and threshold $\kappa\in[0,1]$, the average slash score at lag $\Delta$ is defined by

$$\mathrm{SlashScore}(\Delta) = \mathbb{E}_{P\sim\mathcal D} \left[ \frac{1}{N(P)-\Delta} \sum_{i=\Delta+1}^{N(P)} S_{i,\,i-\Delta}(P) \right].$$

A head is termed \emph{$(\kappa,\Delta)$-slash-dominant} if $\mathrm{SlashScore}(\Delta)\ge\kappa$. Geometrically, such a head concentrates attention along the $\Delta$-th sub-diagonal of $S(P)$, i.e., attends with notable probability to tokens at fixed relative position $-\Delta$.

2. Empirical Properties: Prevalence and OOD Robustness

Empirical analyses reveal that SDHs are ubiquitous in open-source LLMs including Gemma-7B, Llama3-8B, and Qwen2.5-7B. Numerous heads with $\Delta=0,1,2,\ldots$ exhibit $\mathrm{SlashScore}(\Delta)\gtrsim0.1$ for $\Delta<5$, and even at long-range ($\Delta>500$), dozens of SDHs per model can be identified for $\kappa\approx 10^{-3}$. Notably, when inputs are replaced by i.i.d. random tokens (sampled uniformly over the vocabulary), the slash-dominance of the same heads persists with comparable scores. Thus, SDHs are a consequence of model architecture rather than content-driven or semantic effects (Cheng et al., 13 Jan 2026).

3. Architectural Conditions Inducing SDHs

SDHs are closely associated with two structural properties of the model:

(a) Nearly Rank-One Queries and Keys:

Given hidden states $h_1,\ldots,h_N\in\mathbb R^d$ and attention weights $W_Q, W_K$, define $Q=(q_1,\ldots,q_N)^\top$, $K=(k_1,\ldots,k_N)^\top$ with $q_i=h_i^\top W_Q$, $k_j=h_j^\top W_K$. The spectral concentration is quantified by

$$r_1(X) = \frac{\sigma_1^2(X)}{\sum_i \sigma_i^2(X)}, \quad R_{0.95}(X) = \min\left\{ \ell : \sum_{i=1}^\ell r_i(X)\ge0.95 \right\}$$

where $\sigma_i(X)$ denotes singular values. For SDHs, typically $r_1(Q)\gtrsim 0.9$ or $r_1(K)\gtrsim 0.9$, and $R_{0.95}(Q), R_{0.95}(K) \ll d$. Thus, $q_i\approx\alpha_i u$ and $k_j\approx\beta_j v$ for nearly fixed unit vectors $u, v$, with directional and norm variation of at most $\sim$10%.

(b) RoPE Dominated by Medium- and High-Frequency Components:

With RoPE, position $i$ is encoded by $$R_\vartheta(i) = \mathrm{diag}(\rho(i\theta_1),\dots,\rho(i\theta_{d/2}))$$, where $\rho(\phi)$ is a $2\times 2$ block rotation, and $\{\theta_\ell\}$ are monotonically decreasing frequencies. The "slash-dominance frequency condition" requires that for the $d_b/2$ largest frequencies,

$$\left| \sum_{s=1}^{d_b/2} \cos(\theta_s x) - C_1\delta_0(x) - C_2 \right| \le \epsilon,\quad \text{for}\,|x|\le N,$$

with $\epsilon\ll1$. In practice, the active frequencies in SDHs are medium/high, producing a "pulse" frequency response aligned with particular lags $\Delta$. Removal of low-frequency components does not disrupt the slash peak, whereas removal of medium/high frequencies abolishes it.

4. Mathematical Decomposition of SDH Attention

Under the preceding conditions, the pre-softmax attention logit for $i\ge j$ simplifies to

$$\tilde{q}_i^\top\tilde{k}_j = q_i^\top R_\vartheta(i) R_\vartheta(j)^\top k_j \approx u^\top R_\vartheta(i) R_\vartheta(j)^\top v = \sum_{\ell=1}^{d/2} A_\ell \cos(\theta_\ell(i-j) + \varphi_\ell),$$

with amplitudes $A_\ell$ and phases $\varphi_\ell$ dependent on $u$ and $v$. Thus, the logit as a function of the lag $i-j$ becomes a sum of cosines, and constructive interference at integer offsets $\Delta$ gives rise to the observed slash peaks in attention:

$$\mathrm{AttnLogit}(i,j) = \sum_{\ell=1}^{d/2} A_\ell \cos\left(\theta_\ell (i-j) + \varphi_\ell\right).$$

The Fourier-like structure explains the emergent concentration at specific sub-diagonals.

5. Theoretical Results on SDH Emergence

A theoretical treatment considers a shallow, two-layer, single-head, disentangled Transformer with RoPE, trained via gradient descent for in-context linear regression:

• Input: Prompts of the form $P=(x_1,y_1,\dots,x_n,y_n,x_q)$ with embeddings on a cone plus semantic subspace.
• Layer 1: Queries and keys restricted to the cone axis, effectively enforcing rank-one structure; RoPE applied at every layer.
• Training protocol: Two-stage gradient descent. Stage I updates the first-layer queries to focus on the immediate previous token ($\Delta=1$), converging to a 1-slash pattern. Stage II updates second-layer queries to match features between training examples and the query $x_q$.

Key Lemmas:

During Stage I, the difference $A_{i,i-1} - \max_{j\neq i-1} A_{i,j}$ for Layer 1 logits accelerates at $j=i-1$, establishing the slash pattern. In Stage II, attention solidifies on the appropriate feature-matching token.

Main Theorem:

After $\tilde{O}(KN\log N)$ steps (Stage I) and $\tilde{O}(K^2\log K)$ (Stage II), the first layer becomes a $(1-1)$ SDH, the second layer performs accurate feature matching, and the squared loss is $O(1)$. The resulting SDHs are observed to generalize to out-of-distribution (OOD) data (Cheng et al., 13 Jan 2026).

6. Implications and Applications

• Intrinsic Architectural Effect:

SDHs arise systematically from the combination of near rank-one $Q,K$ matrices and the induction structure of high/medium-frequency RoPE, independent of semantic content.

• Parameter Compression:

The near rank-one property enables aggressive low-rank factorization of $Q,K$ without significant loss of model performance.

• Length Generalization:

Modifying or reweighting low-frequency RoPE components can improve model generalization to longer contexts without degrading slash behavior.

• Mechanistic Interpretability:

The structure of SDHs provides insight into how positional encodings in LLMs direct information flow, elucidating the dominant, fixed-lag attention implemented by certain heads.

A plausible implication is that SDHs are essential to reliable long-range context tracking, and interventions on RoPE or the parameterization of queries/keys can be leveraged to manipulate or enhance a model's information propagation capabilities.

7. Summary

Slash-Dominant Heads are a robust and reproducible artifact of attention architectures with Rotary Position Embedding, induced by the near rank-one structure of queries and keys and the frequency response of RoPE. Their identification in modern LLMs, theoretical characterization, and implications for parameter compression, length generalization, and interpretability underscore their significance in transformer research. SDHs emerge independently of semantic data, reflecting architectural and training-driven mechanisms that systematically enforce fixed-lag relative attention (Cheng et al., 13 Jan 2026).