1-Back Attention Heads

• 1-back attention heads are transformer units that preferentially attend to the immediately preceding token or its value-ordered successor.
• The methodology employs sieve-bias scoring and one-tailed statistical tests to quantify significant predecessor bias in models like BERT.
• Empirical results show these heads are multifunctional, often overlapping with local, syntactic, and delimiter roles across different layers.

Searching arXiv for the cited papers and closely related work on 1-back / previous-token attention heads. 1-back attention heads are attention heads whose routing exhibits a systematic predecessor bias. In one line of work on BERT, the term denotes heads that preferentially attend from token position $i$ to the immediately preceding position $i-1$, detected through a unified sieve-bias statistic and one-tailed significance tests across functional roles (Pande et al., 2021). In a distinct mechanistic-interpretability usage, closely related heads are studied as “successor heads,” where the relevant predecessor is not the adjacent token but the previous occurrence of a token whose value is one greater under a natural ordering, such as digits modulo 10 (Gould et al., 2023). Across these usages, 1-back behavior serves as a concrete locus for studying attention-role classification, circuit formation, multifunctionality, and the relationship between attention-head specialization and transformer expressivity.

1. Definitions and terminological scope

The most position-local definition of a 1-back head treats it as a head that, when processing token $t_i$, preferentially attends to $t_{i-1}$. In the sieve-bias framework, this is implemented by the role-specific sieve

$S_{I,1}(t_i)=\{t_{i-1}\}$

for $i>1$, with the first token omitted or skipped if the sieve is empty. Under this definition, the role is fundamentally positional: the object of interest is the immediately preceding token in the same sequence.

A second definition arises in mechanistic studies of autoregressive LLMs. There, each attention head $h$ at token position $t$ defines

$$q_t^{(h)}=W_Q^{(h)}h_t,\qquad k_i^{(h)}=W_K^{(h)}h_i,$$

with attention weights

$$\alpha_{t,i}^{(h)}= \frac{\exp\bigl((q_t^{(h)})^\top k_i^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)} {\sum_{j<t}\exp\bigl((q_t^{(h)})^\top k_j^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)}.$$

In this setting, a “successor head” or 1-back head is defined by concentration of attention on the previous occurrence of the successor token under a natural ordering. For digit tokens, with $i-1$0, the relevant indicator is

$i-1$1

and the head is scored by

$i-1$2

These two definitions are related by a common predecessor motif but are not identical. In the BERT role-classification setting, 1-back is a positional bias. In the successor-head setting, it is a value-ordered retrieval bias. The shared terminology therefore indexes a family resemblance rather than a single invariant operational definition.

2. Sieve-bias detection and statistical testing

The sieve-bias framework introduces a general score for testing whether an attention head is significantly biased toward a hypothesized role $i-1$3 (Pande et al., 2021). For an input sequence $i-1$4, a token $i-1$5, head $i-1$6, and role-specific sieve $i-1$7, the per-token score is

$i-1$8

Because $i-1$9, the denominator is $t_i$0. The statistic therefore measures how many times stronger the head’s attention is to the sieve than to a uniform baseline.

Aggregating over a corpus $t_i$1, the empirical mean role score can be written as

$t_i$2

and, in the notation used for practical averaging,

$t_i$3

For the 1-back specialization, with $t_i$4, the sieve contains only the previous token, so

$t_i$5

Under the null model of no 1-back preference, attention is assumed uniform across the $t_i$6 tokens, yielding population mean

$t_i$7

The hypothesis test asks whether the true mean sieve-bias exceeds a threshold $t_i$8. For a sample $t_i$9, with sample mean $t_{i-1}$0 and sample variance $t_{i-1}$1, the one-tailed statistic is

$t_{i-1}$2

The null hypothesis is that the true mean is at most $t_{i-1}$3, and the head is declared to have role $t_{i-1}$4 if $t_{i-1}$5. In the reported 1-back runs on 1,000 sentences from each of QNLI, QQP, MRPC, and SST-2, the chosen values were $t_{i-1}$6 and $t_{i-1}$7, with $t_{i-1}$8 valid token positions per head.

This framework addresses two stated limitations of earlier role-classification methods: the absence of standard scores across studies or functional roles, and the reliance on average quantities that do not capture statistical significance. Its central contribution is to place role attribution on a common, hypothesis-tested scale.

3. Distribution and multifunctionality in BERT$t_{i-1}$9

Applied to BERT$S_{I,1}(t_i)=\{t_{i-1}\}$0 before fine-tuning on QNLI, the 1-back test identifies 18 significant heads out of 144 total, that is, $S_{I,1}(t_i)=\{t_{i-1}\}$1 (Pande et al., 2021). The detected heads are distributed across all twelve layers:

Layer	Heads	# 1-back heads
0	3, 5, 9	3
1	2, 7	2
2	1, 4, 8	3
3	5, 9	2
4	0	1
5	7, 11	2
6	3	1
7	2	1
8	2, 10	2
9	1	1
10	6	1
11	4	1

Expressed as a fraction of the 12 heads in each layer, layers 0 and 2 each contain $S_{I,1}(t_i)=\{t_{i-1}\}$2 1-back heads; layers 1, 3, 5, and 8 each contain $S_{I,1}(t_i)=\{t_{i-1}\}$3; and layers 4, 6, 7, 9, 10, and 11 each contain $S_{I,1}(t_i)=\{t_{i-1}\}$4. The layer profile is therefore distributed rather than concentrated in a single depth band.

A major empirical result is overlap with other functional roles. Using the same sieve-bias test with $S_{I,1}(t_i)=\{t_{i-1}\}$5 and $S_{I,1}(t_i)=\{t_{i-1}\}$6, the four coarsest comparison roles were local ($S_{I,1}(t_i)=\{t_{i-1}\}$7 window), syntactic (dependency relations nsubj/dobj/amod/advmod), block (same-sentence), and delimiter ($S_{I,1}(t_i)=\{t_{i-1}\}$8). Among the 18 1-back heads, 14 were also significant local attenders, 12 were significant syntactic heads, 4 were block heads, and 16 were delimiter heads.

These overlaps constrain a common simplification according to which a head should realize only one interpretable function. The reported numbers instead support a multifunctional picture: many 1-back heads are simultaneously local, syntactic, and delimiter-biased. In the authors’ wording, many of the 1-back heads are “truly multifunctional.”

4. Successor heads, mod-10 structure, and steerability

In LLMs, the most developed mechanistic account of a 1-back-like circuit is the successor head analysis (Gould et al., 2023). A head is designated a successor head when, for most positions $S_{I,1}(t_i)=\{t_{i-1}\}$9, its attention mass on earlier tokens concentrates almost entirely on exactly the token whose vocabulary value is one greater than the current token’s value. For digit tokens, the operational score is

$i>1$0

When $i>1$1 exceeds a high threshold such as $i>1$2, the head is said to implement the “$i>1$3” function on digit tokens. Empirically, a sharp gap emerges: a small subset of heads have $i>1$4, while the rest lie near random chance, approximately $i>1$5.

The explanatory mechanism is formulated in terms of mod-10 features. A digit token $i>1$6 is represented by

$i>1$7

and a linear probe $i>1$8 is fit on residual stream activations so that

$i>1$9

in mean-squared error. In this basis, key and value projections can be decomposed approximately as sums over the mod-10 features. For canonical successor heads, the value projection satisfies

$h$0

so that the head output at position $h$1 becomes approximately $h$2: the head writes an embedding of the successor digit into the residual stream.

This representation-level account enables direct intervention. To reroute a canonical “$h$3” head into a “$h$4” head, the activation edit

$h$5

is applied before the residual add. The reported effect is to transform behavior so that the head routes attention to the “$h$6” predecessor with almost the same sharpness.

The phenomenon recurs across architectures and scales. The survey covers Pythia from 70M to 12B parameters, GPT-2 from 124M to 1.5B, and Llama-2 from 7B to 70B. Across all scales, roughly $h$7 of all heads qualify as successor heads; smaller models tend to have 5–10 such heads, medium models around 10–15, and the largest up to approximately 25. These heads are not uniformly distributed: they cluster in the middle layers, typically layers 6–12 in a 12-layer model or layers 15–30 in a 64-layer Llama-2, while early layers rarely contain them and very late layers are again sparse.

On natural language data such as The Pile, successor heads remain active on digits but also respond to commas, closing parentheses, and even the word “the,” depending on context. The reported interpretation is polysemanticity: key space appears as a superposition of a digit-detector and a punctuation-detector, while value space remains dominated by successor embeddings $h$8.

5. Developmental emergence and relation to attention sinks

A developmental analysis across three 1B-class models—Pythia 1B, OLMo 1B-0724-hf, and OLMoE 1B-7B-0924—tracks previous-token heads over ten log-spaced revisions per model, for 30 mechanistic-interpretability runs in total (Xu, 1 Jun 2026). Two tools are used. The first is a participation-ratio spectral signal. For each head $h$9 at revision $t$0, outputs at the “second A” query position are collected over a synthetic induction batch of $t$1 sequences, giving an activation matrix $t$2. If $t$3 are its singular values, then

$t$4

High participation ratio indicates a richer, more structured attention pattern.

The second tool is a capability-specific selectivity screen. Previous-token selectivity is defined as

$t$5

In best-class assignment, a head is assigned to the class $t$6 whose selectivity is maximal, provided $t$7. For circuit admission, the threshold is stricter: previous-token heads require $t$8. In the all-head capability-specific screen, every head with $t$9 is admitted regardless of its best-class, with $$q_t^{(h)}=W_Q^{(h)}h_t,\qquad k_i^{(h)}=W_K^{(h)}h_i,$$0.

The timing of emergence differs sharply across models. For the first checkpoint where at least 30 heads exceed the 30× previous-token selectivity threshold, Pythia 1B reaches this point at step 3000, approximately 6.3B tokens and roughly $$q_t^{(h)}=W_Q^{(h)}h_t,\qquad k_i^{(h)}=W_K^{(h)}h_i,$$1 of training; OLMo 1B at step 56000, approximately 117B tokens and roughly $$q_t^{(h)}=W_Q^{(h)}h_t,\qquad k_i^{(h)}=W_K^{(h)}h_i,$$2 of training; and OLMoE 1B-7B at step 5000, approximately 20B tokens and roughly $$q_t^{(h)}=W_Q^{(h)}h_t,\qquad k_i^{(h)}=W_K^{(h)}h_i,$$3 of training.

Layerwise, layers 0 and 1 never produce 1-back heads at any revision: across all 30 runs, $$q_t^{(h)}=W_Q^{(h)}h_t,\qquad k_i^{(h)}=W_K^{(h)}h_i,$$4 in L0–L1. Pythia 1B first shows previous-token heads in mid-layers, beginning at L3; OLMo 1B first shows them at L2, with deeper layers L3–L6 following shortly; and OLMoE 1B-7B first shows them in L7–L10, with later recruitment in L11–L14.

The model-level emergence shapes also differ. Pythia 1B shows a smooth, monotonic ramp from $$q_t^{(h)}=W_Q^{(h)}h_t,\qquad k_i^{(h)}=W_K^{(h)}h_i,$$5 at random initialization to approximately $$q_t^{(h)}=W_Q^{(h)}h_t,\qquad k_i^{(h)}=W_K^{(h)}h_i,$$6 by 6B tokens, with a gradual rise to approximately $$q_t^{(h)}=W_Q^{(h)}h_t,\qquad k_i^{(h)}=W_K^{(h)}h_i,$$7 by the final 300B tokens. OLMo 1B shows a near-zero plateau through approximately 52B tokens, then a sharp jump between adjacent checkpoints, 52B to 117B, from $$q_t^{(h)}=W_Q^{(h)}h_t,\qquad k_i^{(h)}=W_K^{(h)}h_i,$$8 to approximately $$q_t^{(h)}=W_Q^{(h)}h_t,\qquad k_i^{(h)}=W_K^{(h)}h_i,$$9, followed by saturation around $$\alpha_{t,i}^{(h)}= \frac{\exp\bigl((q_t^{(h)})^\top k_i^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)} {\sum_{j<t}\exp\bigl((q_t^{(h)})^\top k_j^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)}.$$0. OLMoE 1B-7B again shows a smooth, gradual ramp, from approximately $$\alpha_{t,i}^{(h)}= \frac{\exp\bigl((q_t^{(h)})^\top k_i^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)} {\sum_{j<t}\exp\bigl((q_t^{(h)})^\top k_j^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)}.$$1 at 20B tokens to approximately $$\alpha_{t,i}^{(h)}= \frac{\exp\bigl((q_t^{(h)})^\top k_i^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)} {\sum_{j<t}\exp\bigl((q_t^{(h)})^\top k_j^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)}.$$2 at 104B, and rising slowly thereafter to approximately $$\alpha_{t,i}^{(h)}= \frac{\exp\bigl((q_t^{(h)})^\top k_i^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)} {\sum_{j<t}\exp\bigl((q_t^{(h)})^\top k_j^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)}.$$3 by 5T tokens.

These trajectories are important because they separate previous-token formation from induction-circuit formation in some but not all settings. In OLMo 1B, induction circuit formation occurs by approximately 23B tokens, whereas 1-back heads do not reach the 30-head, 30× threshold until approximately 117B tokens, a reported gap of about $$\alpha_{t,i}^{(h)}= \frac{\exp\bigl((q_t^{(h)})^\top k_i^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)} {\sum_{j<t}\exp\bigl((q_t^{(h)})^\top k_j^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)}.$$4 in tokens and a contrast in shape: smooth for induction, sharp for the 1-back transition. In Pythia 1B, induction and previous-token heads co-emerge at approximately 6B tokens on the checkpoint grid. In OLMoE 1B-7B, induction also appears by approximately 20B tokens, and the 1-back circuit appears by approximately 20B as well, but with a gradual subsequent ramp.

6. Theoretical context: 1-retrieval and the role of head count

A distinct theoretical literature studies a $$\alpha_{t,i}^{(h)}= \frac{\exp\bigl((q_t^{(h)})^\top k_i^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)} {\sum_{j<t}\exp\bigl((q_t^{(h)})^\top k_j^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)}.$$5 case of generalized retrieval that is often called the “1–back retrieval” problem (Yu et al., 8 Oct 2025). Although this is not a taxonomy of empirical attention-head roles, it provides a formal setting in which the effect of having at least one attention head can be isolated. Sequences are $$\alpha_{t,i}^{(h)}= \frac{\exp\bigl((q_t^{(h)})^\top k_i^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)} {\sum_{j<t}\exp\bigl((q_t^{(h)})^\top k_j^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)}.$$6, with a continuous feature map $$\alpha_{t,i}^{(h)}= \frac{\exp\bigl((q_t^{(h)})^\top k_i^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)} {\sum_{j<t}\exp\bigl((q_t^{(h)})^\top k_j^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)}.$$7, a post-processing map $$\alpha_{t,i}^{(h)}= \frac{\exp\bigl((q_t^{(h)})^\top k_i^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)} {\sum_{j<t}\exp\bigl((q_t^{(h)})^\top k_j^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)}.$$8, and a large index set $$\alpha_{t,i}^{(h)}= \frac{\exp\bigl((q_t^{(h)})^\top k_i^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)} {\sum_{j<t}\exp\bigl((q_t^{(h)})^\top k_j^{(h)} / \sqrt{d_{\mathrm{head}}}\bigr)}.$$9, for example $i-1$00. The target is

$i-1$01

For this $i-1$02 problem, the upper bound states that one head suffices for efficient 1-retrieval. If Assumption (A2) holds for approximating $i-1$03 with exponent $i-1$04 and Assumption (A1) holds for approximating $i-1$05 with exponent $i-1$06, then there is $i-1$07 such that a single-head transformer with per-head dimension $i-1$08 and $i-1$09 total FFN parameters satisfies

$i-1$10

The contrasting lower bound treats the no-head case. Setting $i-1$11, $i-1$12, and $i-1$13 yields

$i-1$14

so any 0-head model that $i-1$15-approximates all 1-retrieval targets of length $i-1$16 must have

$i-1$17

that is, $i-1$18. The stated interpretation is exponential growth in $i-1$19 for fixed $i-1$20.

A further regime allows a single head with embedding dimension $i-1$21. Using the positional basis-vector construction

$i-1$22

softmax attention with trivial $i-1$23 chosen to give uniform weights produces

$i-1$24

so the entire input is memorized up to a global factor $i-1$25. The resulting approximation guarantee is

$i-1$26

where $i-1$27.

The theoretical relevance to 1-back attention heads is indirect but precise. The paper’s stated intuition is that with at least one head specialized to the single retrieval feature, the attention layer can focus on exactly the token(s) in $i-1$28, reducing the burden on the final FFN; without heads, the FFN must encode a $i-1$29-way comparison. This provides a formal complement to empirical studies in which a small number of specialized heads realize sharply defined retrieval-like behaviors.

7. Interpretive themes and recurring misconceptions

Several recurrent themes follow from the cited literature. First, 1-back heads are not uniformly defined across subfields. In statistical role-classification for BERT, they are immediate-previous-position heads. In mechanistic studies of digit circuits, “1-back” can refer to successor heads that retrieve the previous occurrence of a value-ordered successor token. Treating these as identical obscures a genuine terminological split.

Second, 1-back heads are not necessarily isolated monofunctional units. In BERT$i-1$30, most detected 1-back heads overlap with local, syntactic, and delimiter roles. The empirical picture is therefore one of co-location of multiple functional roles in the same attention head rather than strict one-role-per-head decomposition.

Third, the emergence of previous-token heads should not be conflated with the emergence of induction circuits or BOS-attractor heads. In 1B-class models trained on DCLM, induction-circuit formation and attention-sink formation are reported as two transitions, not one, separated by an order of magnitude in tokens and exhibiting different shapes. In other model-and-corpus settings, by contrast, the two phenomena co-emerge in a single smooth ramp.

Fourth, the presence of a 1-back-like head does not imply a purely positional or purely arithmetic computation. Successor heads on natural language data are explicitly described as polysemantic: they can remain aligned with “$i-1$31” behavior on digit features while also participating in punctuation- or word-related subcircuits.

Taken together, these results establish 1-back attention heads as a family of empirically detectable and theoretically informative motifs rather than a single homogeneous object. They can be defined by positional adjacency, by ordered-token successor relations, by capability-specific selectivity during training, or by their role in formal retrieval constructions. Across these settings, their significance lies in making attention-head specialization measurable, comparable, and, in some cases, mechanistically editable.