Previous-Token Heads in Transformers

• Previous-token heads are mechanisms in transformer models that track temporal dependencies by focusing on adjacent tokens using methods like induction and selective attention.
• In speculative decoding, earlier previous-token heads boost throughput by modeling draft sequences where accuracy in initial tokens significantly raises performance metrics.
• Mechanistic studies show these heads aid in ordered retrieval and hidden-state geometry, enabling effective token copying and improved in-context learning alignment.

Previous-Token Heads are a family of mechanisms that assign special status to temporally adjacent or temporally diagnostic tokens in sequence models. Recent work uses the term in at least two distinct senses. In speculative decoding, it denotes the first draft heads in a multi-head draft model, because those heads predict the earliest positions of a candidate continuation and disproportionately affect the number of tokens accepted per verification round. In mechanistic studies of transformers and in-context learning, it denotes attention heads whose attention is concentrated on the immediately preceding token, or closely related induction-style heads that recover the successor of a previous occurrence of the current token. Across these usages, the common theme is temporal dependency tracking: previous-token heads implement or approximate “use the recent or repeated past to predict what comes next” (Li et al., 13 Mar 2025, Yang et al., 24 May 2025, Bajaj et al., 1 Apr 2026).

1. Terminological scope and formal definitions

The expression is not used uniformly across the literature. In speculative decoding it names a position in a draft sequence; in mechanistic interpretability it names a specific attention pattern; and in some papers it overlaps with, while in others it is distinguished from, induction heads.

Usage	Formalization	Source
Speculative decoding	Head $i$ is a previous-token head when $i \le d$, with $P=\{1,\dots,d\}$ and $L=\{d+1,\dots,D\}$	(Li et al., 13 Mar 2025)
In-context learning geometry	A Previous-Token Head is an attention head with $\alpha^h_l(i,i-1)\gg \alpha^h_l(i,j)$ for all $j<i-1$	(Yang et al., 24 May 2025)
Serial-recall mechanism	Heads with the largest induction score $I_h$ are taken to be the previous-token or induction heads	(Bajaj et al., 1 Apr 2026)
Token-form copying	Token induction heads, also called previous-token heads, maximize attention mass on $t-1$ via $$S_{\mathrm{token}(l,h)}=\sum_{t=2}^T a^{(l,h)}_{t,t-1}$$	(Feucht et al., 22 Nov 2025)
Causal-structure selection	A standard induction head attends to the most recent past position $j^*<i$ with the same token and copies its representation	(d'Angelo et al., 9 Sep 2025)

A common misconception is that these are a single, stable object class. The record is narrower and more heterogeneous. In (Bajaj et al., 1 Apr 2026), “previous-token” is used interchangeably with “induction” heads. In (Yang et al., 24 May 2025), by contrast, Previous-Token Heads (PTHs) and Induction Heads (IHs) are separated into different functional roles: PTHs drive separability, whereas IHs and task vectors enhance alignment. This suggests that the term identifies a family resemblance centered on temporal locality or temporal retrieval, rather than one universally agreed circuit.

2. Previous-token heads in speculative decoding

In speculative decoding with draft length $i \le d$0, head $i \le d$1 proposes token $i \le d$2. Gumiho defines the first $i \le d$3 heads as previous-token heads because they handle the first $i \le d$4 draft tokens, and the remaining heads as later-token heads. The key quantity is the expected accepted length per round,

$i \le d$5

where $i \le d$6. Theorem 3.1 states that shifting accuracy from later heads to earlier heads strictly increases $i \le d$7, under the redistribution defined in the paper. In other words, early draft accuracy has higher leverage than later draft accuracy (Li et al., 13 Mar 2025).

Gumiho realizes this by assigning a richer serial architecture to previous-token heads and a lighter parallel architecture to later-token heads. For $i \le d$8, a two-layer mini-Transformer runs serially: $i \le d$9 and the chain continues so that head $P=\{1,\dots,d\}$0 is exposed to head $P=\{1,\dots,d\}$1’s output. For $P=\{1,\dots,d\}$2, multiple two-layer MLP heads with ReLU run in parallel on the shared input $P=\{1,\dots,d\}$3. The model also uses Full Tree Attention (FTA), which “borrows” tokens from longer paths to fill shorter ones at the same depth, increasing average path length without extra KV-cache work (Li et al., 13 Mar 2025).

Empirically, across six tasks and multiple LLMs, Gumiho outperforms Eagle-2. On Vicuna-7B at temperature $P=\{1,\dots,d\}$4, Eagle-2 reaches Speedup $P=\{1,\dots,d\}$5 and $P=\{1,\dots,d\}$6, whereas Gumiho reaches Speedup $P=\{1,\dots,d\}$7 and $P=\{1,\dots,d\}$8; LLaMA-2-70B sees $P=\{1,\dots,d\}$9 speedup. A broader architectural backdrop is that standard draft heads had been sequentially independent, while Hydra introduced sequentially-dependent draft heads and reported throughput improvements of up to $L=\{d+1,\dots,D\}$0 over Medusa decoding and $L=\{d+1,\dots,D\}$1 over autoregressive decoding, indicating that dependence across speculative positions is itself a useful design axis (Ankner et al., 2024, Li et al., 13 Mar 2025).

3. Induction-style previous-token heads and temporal retrieval

Bajaj et al. analyze a related but mechanistically different object: induction heads that implement successor retrieval from repeated tokens. On a doubled sequence $L=\{d+1,\dots,D\}$2, the induction score for head $L=\{d+1,\dots,D\}$3 is

$L=\{d+1,\dots,D\}$4

This measures how much attention, from the second copy of a token, is placed on the token that originally was followed by that same token in the first copy. Heads with the largest $L=\{d+1,\dots,D\}$5 are taken to be the previous-token or induction heads (Bajaj et al., 1 Apr 2026).

The corresponding circuit explanation is a $L=\{d+1,\dots,D\}$6 account of “if $L=\{d+1,\dots,D\}$7, then recall what followed $L=\{d+1,\dots,D\}$8.” Queries and keys are configured so that when token $L=\{d+1,\dots,D\}$9 reappears, the query at the current position matches the previous occurrence of $\alpha^h_l(i,i-1)\gg \alpha^h_l(i,j)$0, while the key projection contains a positional offset that steers retrieval toward the successor position. The attention therefore locks onto the previous occurrence, and the value pathway returns the token that followed it. In the example

$\alpha^h_l(i,i-1)\gg \alpha^h_l(i,j)$1

the second $\alpha^h_l(i,i-1)\gg \alpha^h_l(i,j)$2 retrieves $\alpha^h_l(i,i-1)\gg \alpha^h_l(i,j)$3, the successor of the first $\alpha^h_l(i,i-1)\gg \alpha^h_l(i,j)$4 (Bajaj et al., 1 Apr 2026).

The empirical paradigm adapts free recall. A list of $\alpha^h_l(i,i-1)\gg \alpha^h_l(i,j)$5 randomly shuffled common-word tokens is followed by a $\alpha^h_l(i,i-1)\gg \alpha^h_l(i,j)$6st token that repeats the word at index $\alpha^h_l(i,i-1)\gg \alpha^h_l(i,j)$7, and the model’s next-token distribution is measured over $\alpha^h_l(i,i-1)\gg \alpha^h_l(i,j)$8 independently shuffled lists. Several open-source LLMs show a serial-recall-like pattern with a sharp peak at lag $\alpha^h_l(i,i-1)\gg \alpha^h_l(i,j)$9. Ablating heads with high induction scores substantially reduces this $j<i-1$0 lag bias, whereas ablating random heads does not reproduce the same reduction. The same manipulation also impairs few-shot serial recall more strongly than random-head ablation. Reported effects include, for Gemma-2-9B, a reduction in $j<i-1$1 from $j<i-1$2 to $j<i-1$3 after ablating $j<i-1$4 induction heads, compared with $j<i-1$5 after $j<i-1$6 random-head ablations. In the few-shot recall task, Llama-I drops from $j<i-1$7 with no ablation to $j<i-1$8 after ablating $j<i-1$9 induction heads, versus $I_h$0 after ablating $I_h$1 random heads (Bajaj et al., 1 Apr 2026).

These findings support a mechanistically specific role for induction-style previous-token heads in ordered retrieval. A plausible implication is that temporal context processing in transformers is partly realized by circuits that do not merely repeat local tokens, but reconstruct successor structure from repeated cues.

4. Previous-Token Heads in hidden-state geometry

A distinct line of work studies Previous-Token Heads through the geometry of hidden states in in-context learning. In this formulation, a PTH is an attention head whose attention at the final query position is concentrated on the immediately preceding token: $I_h$2 Heads are ranked by

$I_h$3

averaged over $I_h$4 randomly sampled prompts, and the top $I_h$5 are labeled as PTHs (Yang et al., 24 May 2025).

The paper proposes a two-stage mechanism for in-context learning in classification tasks. Separability emerges in early layers, while alignment develops in later layers. For hidden states $I_h$6 and labels $I_h$7, separability is defined by

$I_h$8

with maximum separability $I_h$9. Theorem 1 shows $t-1$0, with equality when the separating direction aligns with the label-difference unembedding vector $t-1$1. Ablation studies then show that PTHs drive separability, while induction heads and task vectors enhance alignment (Yang et al., 24 May 2025).

The layer-wise pattern is specific. In early layers $t-1$2–$t-1$3, separability $t-1$4 increases sharply under ICL while alignment proxies remain near zero. In middle to late layers $t-1$5, separability plateaus and directional and output alignment surge. PTHs are concentrated in the first stage. In Llama2-70B, removing PTHs causes the separability score at layer $t-1$6 to drop from $t-1$7 to $t-1$8, and averaged over six datasets the final-layer classification accuracy falls from $t-1$9 to $$S_{\mathrm{token}(l,h)}=\sum_{t=2}^T a^{(l,h)}_{t,t-1}$$0. Ablating the same number of random heads changes neither $$S_{\mathrm{token}(l,h)}=\sum_{t=2}^T a^{(l,h)}_{t,t-1}$$1 nor accuracy by more than $$S_{\mathrm{token}(l,h)}=\sum_{t=2}^T a^{(l,h)}_{t,t-1}$$2 percentage point, and the effect is consistent across $$S_{\mathrm{token}(l,h)}=\sum_{t=2}^T a^{(l,h)}_{t,t-1}$$3 models within $$S_{\mathrm{token}(l,h)}=\sum_{t=2}^T a^{(l,h)}_{t,t-1}$$4 percentage points (Yang et al., 24 May 2025).

This usage of PTHs is therefore not primarily about exact token copying. The claim is narrower: by attending to the last demonstration token, PTHs inject label-distinct semantics into the query hidden state and widen the cluster structure needed for linear separability. The paper also states that PTHs are task-agnostic because they simply copy the last token’s embedding, which suggests a mechanism that is structurally simple but geometrically consequential (Yang et al., 24 May 2025).

5. Token induction heads and token-form subspaces

Work on vector arithmetic in hidden-state subspaces treats token induction heads, also called previous-token heads, as circuits that copy exact token form rather than conceptual content. The token-induction score is

$$S_{\mathrm{token}(l,h)}=\sum_{t=2}^T a^{(l,h)}_{t,t-1}$$5

and the top-$$S_{\mathrm{token}(l,h)}=\sum_{t=2}^T a^{(l,h)}_{t,t-1}$$6 heads by this score are collected in $$S_{\mathrm{token}(l,h)}=\sum_{t=2}^T a^{(l,h)}_{t,t-1}$$7. Using the value and output matrices of these heads, the paper defines the token lens

$$S_{\mathrm{token}(l,h)}=\sum_{t=2}^T a^{(l,h)}_{t,t-1}$$8

which projects a hidden state $$S_{\mathrm{token}(l,h)}=\sum_{t=2}^T a^{(l,h)}_{t,t-1}$$9 into a token subspace by $j^*<i$0 (Feucht et al., 22 Nov 2025).

This projection isolates surface-level information. In Llama-2-7b, the paper evaluates parallelogram arithmetic for the present-participle task, exemplified by

$j^*<i$1

Using raw hidden states yields only $j^*<i$2 top-1 nearest-neighbor accuracy on Mikolov’s gram5 task. Using the token lens $j^*<i$3 raises this to $j^*<i$4, whereas the concept lens $j^*<i$5 achieves only $j^*<i$6 on this surface-form task. The paper also gives “writing” $j^*<i$7 “write” $j^*<i$8 “read” $j^*<i$9 “reading” as a concrete example (Feucht et al., 22 Nov 2025).

The interpretation is explicitly orthographic. Token heads route the embedding of the last token into the residual stream so that the model can re-emit literal form information; concept heads instead copy a semantic summary. A plausible implication is that previous-token heads define a usable linear readout of surface-form features that are present in hidden states but partially entangled in the raw residual stream.

6. Selective induction heads and causal-structure selection

Standard induction heads can be formalized as heads that attend to the most recent earlier position $i \le d$00 with the same token and copy its representation,

$i \le d$01

Selective induction heads generalize this idea to settings where the relevant causal lag is not fixed. In the interleaved-Markov-chain task, a lag $i \le d$02 is sampled from $i \le d$03, and the next token depends on $i \le d$04 through a fixed transition matrix $i \le d$05. The Bayes-optimal predictor averages over lags,

$i \le d$06

while asymptotically the posterior concentrates on the maximum-likelihood lag $i \le d$07 (d'Angelo et al., 9 Sep 2025).

The paper provides an explicit $i \le d$08-layer attention-only transformer construction for this behavior. Layer $i \le d$09 extracts in-context transition probabilities with an attention mask that activates only candidate lags in $i \le d$10. Layer $i \le d$11 uses $i \le d$12 heads to aggregate lag-specific scores over time along disjoint diagonals. Layer $i \le d$13 selects the best lag and copies the token at position $i \le d$14. In the hard-attention limit, the resulting output converges to the maximum-likelihood solution. Claim 3.1 and the accompanying ergodic argument establish the asymptotic convergence of the lag-selection statistic, and the paper also gives a two-lag special case via Lemma A.1 (d'Angelo et al., 9 Sep 2025).

Empirically, KL-divergence curves for the constructed model, a disentangled transformer trained from scratch, and a standard transformer trained end-to-end all collapse onto the MLE curve for large sequence length $i \le d$15. Attention-map snapshots on $i \le d$16 show the predicted layer-wise structure: a first layer matching $i \le d$17 plus a lag mask, a second layer with diagonal aggregation patterns, and a third layer with spikes at positions $i \le d$18 colored by the true lag. The same behavior is reported for non-contiguous lag sets such as $i \le d$19 and $i \le d$20 (d'Angelo et al., 9 Sep 2025).

Taken together, these results place previous-token heads within a broader class of temporal-selection circuits. In the simplest case they privilege $i \le d$21; in induction form they retrieve a previous occurrence’s successor; in selective form they choose among several causal lags; and in speculative decoding they designate the early draft positions whose accuracy most strongly determines throughput. The unifying principle is not a single canonical definition, but a recurring computational role: concentrating model capacity on the temporally proximal or temporally diagnostic evidence that most constrains the next prediction.