Induction Head in Transformers

• Induction head is an attention-based circuit in transformers that implements a sequence-completion rule by matching earlier token contexts and copying subsequent tokens.
• It plays a crucial role in in-context learning by facilitating pattern continuation, nearest-neighbor retrieval, and prompt-conditioned prediction.
• Variants include generalized, concept, function, and selective induction heads, which adapt the mechanism to handle diverse data structures and causal lags.

An induction head is an attention-based circuit in a transformer that implements the sequence-completion rule $[A][B]\dots[A]\rightarrow[B]$: when a token or context reappears, the model attends to an earlier matching occurrence and copies what followed it there. In mechanistic interpretability, induction heads are treated as a central substrate for in-context learning, because they enable pattern continuation, nearest-neighbor retrieval over the prompt, and prompt-conditioned prediction without weight updates (Olsson et al., 2022). Subsequent work has broadened the term from a narrow token-copying mechanism to a family of related circuits, including generalized $k$-gram induction on Markov data, concept-level induction over lexical units, function induction over latent transformations such as $c\mapsto c+1$, and selective induction over variable causal lags (Chen et al., 2024).

1. Canonical definition and operational criteria

The canonical induction-head behavior is the mapping $[A][B]\dots[A]\mapsto[B]$: at a later occurrence of $A$, the model predicts the token that followed an earlier occurrence of $A$ (Olsson et al., 2022). In the standard attention formulation for a single head $h$,

$$\text{Attn}_h(x)_t=\sum_{s\le t}\alpha^h_{ts}V_hx_s,\qquad \alpha^h_{ts}=\mathrm{softmax}_s\left(\frac{(Q_hx_t)\cdot(K_hx_s)}{\sqrt{d_k}}\right),$$

an induction head is characterized by a QK circuit that implements prefix matching and an OV circuit that copies the relevant continuation into the residual stream or logits (Olsson et al., 2022).

Two partly distinct definitions coexist in the literature. A mechanistic definition identifies a specific circuit structure in which the head attends from the current position to a past position associated with a matching context and then copies the following token or label (Olsson et al., 2022). A behavioral definition, used in work on repetition, scores heads by high prefix matching probability $p_{\text{match}}$ and strong copy logit boost $\Delta \ell_{\text{copy}}$, labeling as effective induction heads those that maximize both simultaneously (Wang et al., 17 May 2025). This difference is consequential: some papers study the internal QK/OV factorization directly, whereas others diagnose induction via causal influence on copying tasks.

A recurrent misconception is that an induction head is a new architectural module. It is instead a learned pattern implemented by ordinary attention heads or small multi-head circuits (Musat et al., 2 Nov 2025). Another common simplification is to equate induction with literal token copying only. Several later papers preserve the core “match then copy continuation” schema while extending the matched object from single tokens to $k$0-token contexts, lexical units, abstract task functions, or lag-dependent causal parents (Feucht et al., 3 Apr 2025).

2. Circuit structure in transformers

In small two-layer transformers, induction is classically implemented as a two-stage mechanism. A previous-token head in an earlier layer attends from each position $k$1 to $k$2, writing information about the preceding token into the current residual stream; a later induction head then uses that written feature to detect earlier positions whose previous token matches the current token and copies the continuation from there (Olsson et al., 2022). In this picture, the first layer performs positional or previous-token routing, and the second layer performs content-based lookup.

A closely related decomposition appears in controlled synthetic in-context learning tasks. In a minimal item–label setup, layer 1 uses positional information so that each label token attends to its preceding item token, causing the label representation to contain the associated item; layer 2 then uses content matching from the query item to the label representation that stores the same item; a linear readout copies the corresponding label vector (Musat et al., 2 Nov 2025). In the notation of that analysis, the three pseudo-parameters $k$3, $k$4, and $k$5 implement, respectively, first-layer positional attention, second-layer content attention, and output copying, and together realize the induction-head circuit (Musat et al., 2 Nov 2025).

A more functionally explicit decomposition appears in the analysis of two-attention-layer transformers trained on $k$6-gram Markov chain data. There, the limiting mechanism is split into a copier, a selector, and a classifier: the first attention layer copies past tokens within a window to each position; the FFN with normalization selects an informationally relevant subset of parents by generating a feature vector; the second attention layer compares those features to perform a generalized induction-head computation (Chen et al., 2024). This formulation expands the classical previous-token-plus-induction-head story into a full circuit spread across attention, FFN, and normalization.

Theoretical representability results sharpen the architectural picture. One paper proves that a two-layer transformer with one head per layer can represent any conditional $k$7-gram model, hence any $k$8-th order Markov induction-head mechanism, using relative positional encodings, MLP blocks, and layer normalization (Ekbote et al., 10 Aug 2025). Another shows that a two-layer single-head construction is insufficient in one layer but sufficient in two, while a one-layer counterpart cannot solve the task unless it is exponentially large (Ekbote et al., 10 Aug 2025). This suggests that depth, rather than large width alone, is the critical architectural resource for induction-style in-context retrieval in these structured settings.

3. Formal analyses of representation and mechanism

Several papers recast induction heads as explicit estimators over sequence statistics. In Markov settings, the induction circuit is identified with a conditional $k$9-gram estimator: the model finds all earlier positions whose preceding $c\mapsto c+1$0 tokens equal the current $c\mapsto c+1$1-token context and averages the corresponding successor tokens (Ekbote et al., 10 Aug 2025). The same equivalence underlies the definition of a $c\mapsto c+1$2-th order induction head, where attention at the final position is maximized exactly at positions whose preceding $c\mapsto c+1$3 tokens match the final $c\mapsto c+1$4-token suffix (Ekbote et al., 10 Aug 2025).

Approximation-theoretic analyses make the distinction between “lazy” local $c\mapsto c+1$5-gram mechanisms and “rich” induction mechanisms explicit. One paper formalizes a vanilla induction head as

$c\mapsto c+1$6

and then generalizes this to patch-based $c\mapsto c+1$7-gram induction and to arbitrary similarity functions $c\mapsto c+1$8 between context patches (Wang et al., 2024). Its constructive results show that a two-layer single-head transformer without FFNs can approximate the vanilla induction head with error bounded by $c\mapsto c+1$9, that multi-head attention can approximate generalized $[A][B]\dots[A]\mapsto[B]$0-gram induction with error $[A][B]\dots[A]\mapsto[B]$1, and that FFNs become necessary when the similarity function exceeds pure dot-product form (Wang et al., 2024). In that framework, RPE implements memory or shift kernels, dot-product attention implements matching, and FFNs implement richer similarity bases (Wang et al., 2024).

A more symmetry-driven formalization is given in a minimal item–label in-context learning model with isotropic data and a simplified two-layer attention-only transformer. Under zero initialization of the output layer, population loss, and isotropic inputs, gradient descent remains confined to a 19-dimensional parameter subspace even though the full parameter space has dimension $[A][B]\dots[A]\mapsto[B]$2 (Musat et al., 2 Nov 2025). The first-layer matrix $[A][B]\dots[A]\mapsto[B]$3, second-layer matrix $[A][B]\dots[A]\mapsto[B]$4, and readout $[A][B]\dots[A]\mapsto[B]$5 retain a highly constrained block form built only from scalar coefficients multiplying $[A][B]\dots[A]\mapsto[B]$6 and a fixed position-rotation matrix $[A][B]\dots[A]\mapsto[B]$7 (Musat et al., 2 Nov 2025). Empirically, only three of those pseudo-parameters become large, and those three are exactly the ones realizing the induction-head circuit (Musat et al., 2 Nov 2025).

A further generalization concerns causal structure selection. In interleaved Markov chains with multiple possible lags, a three-layer disentangled transformer can implement a selective induction head: earlier layers compute lag-specific evidence scores from normalized transition probabilities, and the final attention layer selects the lag with highest evidence by attending to the token at offset $[A][B]\dots[A]\mapsto[B]$8 (d'Angelo et al., 9 Sep 2025). The constructed predictor asymptotically converges to the maximum-likelihood choice of lag, so induction becomes conditional not merely on content recurrence but on inferred causal structure (d'Angelo et al., 9 Sep 2025).

4. Emergence during training

The training-time emergence of induction heads is one of the defining empirical findings in the area. Across 34 models from 1-layer systems to 40-layer, 13B-parameter transformers, a narrow early-training window exhibits a phase change: in-context learning jumps, induction heads appear abruptly, the training loss curve shows a bump, and a PCA of per-token losses shows a trajectory pivot (Olsson et al., 2022). In the small models emphasized there, 1-layer transformers never develop induction heads, have no phase change, and show weak in-context learning, while models with at least two layers do (Olsson et al., 2022). The paper quantifies in-context learning by

$[A][B]\dots[A]\mapsto[B]$9

with a shift from approximately $A$0 nats before the phase change to approximately $A$1 nats after it (Olsson et al., 2022).

Controlled synthetic studies clarify what must be learned for induction to form. In a two-layer, attention-only transformer trained on Omniglot-based few-shot episodes, induction heads emerge at the same moment as a sharp loss drop from an early plateau near $A$2 to near-zero loss (Singh et al., 2024). That work decomposes the induction circuit into three subcircuits: a layer-1 previous-token circuit, a layer-2 QK match circuit, and a copy circuit (Singh et al., 2024). Using activation clamping throughout training, it shows that each subcircuit individually learns with smooth exponential dynamics when isolated, but their joint co-evolution yields the observed reverse-S loss profile and phase change (Singh et al., 2024). It also finds multiple additive, partly redundant induction heads rather than a single indispensable one (Singh et al., 2024).

The minimal item–label model provides a sharper asymptotic account of emergence time. After restricting the dynamics to the three effective pseudo-parameters corresponding to the induction-head circuit, the analysis proves that the parameters emerge in the order

$A$3

with

$A$4

and total in-context-learning emergence time $A$5 as a function of context length $A$6 (Musat et al., 2 Nov 2025). The same work gives a closed-form loss under whitened inputs and shows empirically that three-parameter training trajectories closely match the full 19-dimensional dynamics (Musat et al., 2 Nov 2025). This provides a formal account of why induction-head formation can appear abrupt even when the underlying dynamics are low-dimensional and smooth.

A complementary dynamics analysis studies a synthetic target mixing a “lazy” 4-gram component and a “rich” induction-head component. In that setting, a two-layer transformer exhibits four phases: rapid partial learning of the 4-gram head, a long plateau for the induction component, an abrupt induction-head emergence phase, and final convergence (Wang et al., 2024). The characteristic induction timescale scales as

$A$7

and the paper attributes the delay to time-scale separation between linear RPE learning and quadratic dot-product parameterization in the induction head (Wang et al., 2024). This suggests that abrupt induction-head formation is not merely an empirical curiosity but a predictable consequence of optimization geometry in simplified settings.

5. Variants and generalizations

The term “induction head” now names a family of circuits rather than a single mechanism. The following variants are explicitly identified in the literature.

Variant	Core matched object	Representative source
Classical induction head	Repeated token or short context, then copy successor token	(Olsson et al., 2022)
Generalized induction head	Informative parent subset in an $A$8-gram Markov process	(Chen et al., 2024)
Concept induction head	Multi-token lexical unit; attends to ends of concepts	(Feucht et al., 3 Apr 2025)
Function induction mechanism	Inferred function such as $A$9 reused at test time	(Ye et al., 14 Jul 2025)
Selective induction head	Lag or causal structure selected in context	(d'Angelo et al., 9 Sep 2025)
Hand-engineered induction head	Fuzzy substring matching within context for next-token prediction	(Kim et al., 2024)

Concept-level induction heads are presented as a distinct route from token-level induction. Using concept copying and token copying scores, one paper argues that concept induction heads copy entire lexical units rather than individual tokens, attend to the last token of multi-token concepts, and are responsible for semantic tasks such as word-level translation, while token induction heads remain essential for verbatim copying of nonsense strings or code (Feucht et al., 3 Apr 2025). Ablating token induction heads causes models to paraphrase where they would otherwise copy verbatim, whereas ablating concept induction heads selectively harms translation, synonyms, and antonyms while leaving nonsense copying largely intact (Feucht et al., 3 Apr 2025). This suggests that the induction mechanism can operate at both sublexical and lexical granularity.

Function induction extends the same schema from token continuation to latent transformation reuse. In off-by-one addition prompts such as $A$0, a set of late-layer heads attends from the final equals sign to earlier answer tokens and carries an induced $A$1 function to the test position (Ye et al., 14 Jul 2025). The identified circuit includes previous-token heads, function-induction heads, and consolidation heads, and it is explicitly described as resembling the classical induction-head structure while operating “one level up the abstraction ladder” (Ye et al., 14 Jul 2025). The same heads are reused in shifted multiple-choice QA, Caesar cipher tasks, and off-by-$A$2 arithmetic (Ye et al., 14 Jul 2025).

Selective induction heads further generalize the mechanism to model selection over latent causal lags. In interleaved Markov chains with lag set $A$3, earlier layers compute lag-specific evidence scores and a final attention layer puts nonzero attention only on columns $A$4 corresponding to candidate lags, with weights $A$5 (d'Angelo et al., 9 Sep 2025). In the hardmax limit, attention concentrates on the maximum-likelihood lag $A$6, so the model chooses which causal offset to copy from based on the prompt itself (d'Angelo et al., 9 Sep 2025).

A different line externalizes induction into an interpretable nonparametric model. “Induction-Gram” adds a hand-engineered induction head to an Infini-Gram $A$7-gram LLM by searching the current context for exact or fuzzy substring matches and aggregating their successor tokens into a next-token distribution (Kim et al., 2024). In that system, the induction head is not a hidden transformer circuit but an explicit mechanism that “attend-then-shift”s in discrete string space using a learned similarity metric $A$8 (Kim et al., 2024). This suggests that induction behavior can be factored out into transparent retrieval modules without abandoning the basic algorithm.

6. Empirical significance, failure modes, and open questions

Induction heads have been repeatedly connected to in-context learning proper. In small attention-only models, ablations show that a small number of heads diagnosed by prefix matching, copying score, and QK eigenvalue tests account for almost all of the increase in in-context learning score across training; removing them largely collapses ICL back toward pre-phase-change levels (Olsson et al., 2022). In more realistic transformers trained on $A$9-gram Markov data, the limiting model is provably a generalized induction head whose copier, selector, and classifier components jointly realize the in-context estimator (Chen et al., 2024). A plausible implication is that “induction head” now functions as a mechanistic bridge between prompt-conditioned generalization and concrete circuit structure.

Induction heads also matter because they can fail pathologically. One paper identifies “induction head toxicity” as the tendency of induction heads to dominate output logits during repetition, thereby excluding other heads from contributing to generation (Wang et al., 17 May 2025). On prompts of the form “Repeat token $h$0 exactly $h$1 times,” tested on Qwen2.5-Instruct 1.5B, 3B, 7B, LLaMA3-Instruct 8B, and Gemma2-it 9B, response length grows roughly linearly for small $h$2 and then suddenly explodes for larger $h$3, often continuing until the context window limit (Wang et al., 17 May 2025). The paper defines a toxicity ratio

$h$4

declares the model toxic when $h$5 with $h$6, and associates toxicity with a positive-feedback loop and entropy collapse (Wang et al., 17 May 2025). In the normal regime $h$7, initial entropy is approximately $h$8 with decay rate approximately $h$9; in the toxic regime $$\text{Attn}_h(x)_t=\sum_{s\le t}\alpha^h_{ts}V_hx_s,\qquad \alpha^h_{ts}=\mathrm{softmax}_s\left(\frac{(Q_hx_t)\cdot(K_hx_s)}{\sqrt{d_k}}\right),$$0, initial entropy is approximately $$\text{Attn}_h(x)_t=\sum_{s\le t}\alpha^h_{ts}V_hx_s,\qquad \alpha^h_{ts}=\mathrm{softmax}_s\left(\frac{(Q_hx_t)\cdot(K_hx_s)}{\sqrt{d_k}}\right),$$1 with decay rate approximately $$\text{Attn}_h(x)_t=\sum_{s\le t}\alpha^h_{ts}V_hx_s,\qquad \alpha^h_{ts}=\mathrm{softmax}_s\left(\frac{(Q_hx_t)\cdot(K_hx_s)}{\sqrt{d_k}}\right),$$2 (Wang et al., 17 May 2025).

The same paper proposes an inference-time mitigation: logarithmic descaling of induction-head outputs by $$\text{Attn}_h(x)_t=\sum_{s\le t}\alpha^h_{ts}V_hx_s,\qquad \alpha^h_{ts}=\mathrm{softmax}_s\left(\frac{(Q_hx_t)\cdot(K_hx_s)}{\sqrt{d_k}}\right),$$3, with $$\text{Attn}_h(x)_t=\sum_{s\le t}\alpha^h_{ts}V_hx_s,\qquad \alpha^h_{ts}=\mathrm{softmax}_s\left(\frac{(Q_hx_t)\cdot(K_hx_s)}{\sqrt{d_k}}\right),$$4 reported as optimal in practice (Wang et al., 17 May 2025). On the repetition-512 task in Qwen2.5-7B-Instruct, logarithmic descaling suppresses repetition loops and restores controlled stopping, whereas constant scaling often remains out-of-distribution and linear scaling $$\text{Attn}_h(x)_t=\sum_{s\le t}\alpha^h_{ts}V_hx_s,\qquad \alpha^h_{ts}=\mathrm{softmax}_s\left(\frac{(Q_hx_t)\cdot(K_hx_s)}{\sqrt{d_k}}\right),$$5 suppresses induction too aggressively (Wang et al., 17 May 2025). Perplexity on diverse generation tasks changes only slightly, from $$\text{Attn}_h(x)_t=\sum_{s\le t}\alpha^h_{ts}V_hx_s,\qquad \alpha^h_{ts}=\mathrm{softmax}_s\left(\frac{(Q_hx_t)\cdot(K_hx_s)}{\sqrt{d_k}}\right),$$6 without descaling to $$\text{Attn}_h(x)_t=\sum_{s\le t}\alpha^h_{ts}V_hx_s,\qquad \alpha^h_{ts}=\mathrm{softmax}_s\left(\frac{(Q_hx_t)\cdot(K_hx_s)}{\sqrt{d_k}}\right),$$7 with logarithmic descaling (Wang et al., 17 May 2025). This suggests that the same circuitry supporting beneficial ICL can also create runaway attractors under adversarially repetitive contexts.

A final development is methodological: induction heads have become a standardized target for circuit-finding pipelines. A spectral-probe recipe ranks heads by a time-integrated participation-ratio signal, screens them by induction-style attention patterns, and then verifies causal necessity by group ablation (Xu, 22 May 2026). Across dense and mixture-of-experts architectures from 51M to 1B-active/7B-total, the method finds 2–6 head induction circuits that are causally necessary in every tested model, with synthetic-induction top-1 dropping by $$\text{Attn}_h(x)_t=\sum_{s\le t}\alpha^h_{ts}V_hx_s,\qquad \alpha^h_{ts}=\mathrm{softmax}_s\left(\frac{(Q_hx_t)\cdot(K_hx_s)}{\sqrt{d_k}}\right),$$8 to $$\text{Attn}_h(x)_t=\sum_{s\le t}\alpha^h_{ts}V_hx_s,\qquad \alpha^h_{ts}=\mathrm{softmax}_s\left(\frac{(Q_hx_t)\cdot(K_hx_s)}{\sqrt{d_k}}\right),$$9 after ablation (Xu, 22 May 2026). The same work reports that the fraction of heads doing identifiable specialized computation is conserved at $p_{\text{match}}$0 to $p_{\text{match}}$1 across the Pythia family, while induction circuits remain 3–11 heads and therefore sublinear in total head count (Xu, 22 May 2026). This indicates that induction heads are both mechanistically specific and empirically stable enough to serve as a benchmark for interpretability methods.

Open questions remain explicit across the literature. Existing proofs often assume population loss, gradient flow, isotropic or orthonormal data, simplified architectures, or synthetic Markov and copying tasks (Musat et al., 2 Nov 2025). Concept-level and function-level variants indicate that induction may operate over abstract representations rather than raw tokens, but the exact relationship between these higher-level circuits and classical token induction remains only partially pinned down (Feucht et al., 3 Apr 2025). Selective induction shows that transformers can choose among causal structures in context, but only for stylized lag-selection tasks (d'Angelo et al., 9 Sep 2025). The accumulated evidence strongly supports induction heads as a core mechanism for in-context learning, while also indicating that the term now spans a hierarchy of related circuits whose interactions in frontier LLMs remain an active subject of study (Olsson et al., 2022).