Successor Heads in Transformer Models

• Successor heads are specialized attention components in Transformer models that map each token to its immediate successor in predefined sequences.
• They are analyzed using linear probing, attribution, and causal editing methods, revealing interpretable mechanisms for modular arithmetic in LLMs.
• Their presence across diverse architectures, including GPT-2, Llama-2, and Pythia, underscores their practical role in sequence generation and error diagnosis.

A successor head is an individual attention head in a Transformer architecture that implements the operation of mapping a token corresponding to an item in a natural sequence (e.g., integers, weekdays, months) to its immediate successor in that ordering. For instance, a successor head maps "Monday" to "Tuesday" or the digit "3" to "4". The mechanism underlying successor heads provides a concrete, interpretable basis for symbolic routines such as incrementation and modular counting. Their interpretability and ubiquity open a path for causal circuit-level editing and analysis in contemporary LLMs spanning millions to tens of billions of parameters (Gould et al., 2023).

1. Mechanistic Definition and Functional Principle

A successor head is defined by its attention distribution: when presented with the embedding of a token $v$ at position $t$, it directs nearly all its attention mass (typically exceeding 95%) toward the positional successor token $v'$ corresponding to $\mathit{succ}(v)$. Formally,

$$\Pr(\text{head attends to } v' \mid x_t = \mathrm{Enc}(v)) = \begin{cases} 1 & \text{if } v' = \mathit{succ}(v), \ 0 & \text{otherwise.} \end{cases}$$

The query and key projections of the head are configured such that the query vector for each ordered item yields a maximal dot product with the key of its successor. As a result, the attention mechanism systematically implements stepwise incrementation across tokens with a predefined sequence, such as numeric digits or days of the week.

2. Interpretability Framework and Analytical Methodology

The mechanistic behavior of successor heads has been elucidated using a combination of probing, attribution, and causal editing methodologies:

• Linear Probing: Lightweight linear probes on the residual stream test whether head outputs encode the successor feature, parameterized by a probe weight matrix $P$ such that $P h_t \approx \mathbf{1}_{v' = \mathit{succ}(v_t)}$.
• Attribution Analysis: Saliency measures, including Attention Norm and Integrated Gradients, establish that attention contributions are localized and attributable to the successor relation.
• Causal Head Editing (Activation Patching): By ablating the activation of a successor head, the model's ability to predict the next item in a sequence is disrupted; inserting the activation into alternative contexts induces successor continuation. This causal relationship is captured by activation patching, where introducing a patch $\Delta_{\mathrm{succ}}$ flips the next-token logit distribution in favor of the successor.

These interpretability tools enable both diagnosis of sequence-related behavior and precise, model-internal modification of successor dynamics (Gould et al., 2023).

3. Modular Feature Structure: The “Mod-10 Features”

In digit-based tasks, successor heads rely on an internal set of learned modular features, termed "mod-10 features." Each feature vector $\mathbf{f}_k \in \mathbb{R}^d$ signals the presence of digit $k$ within the model’s residual stream. The encoding is constructed as

$t$0

with $t$1 a learned one-hot basis. The successor head’s matrix projections implement modular arithmetic, with query/key vectors such that

$t$2

The modular structure extends to vector arithmetic editing: to coerce the head to attend to a non-standard successor, a shift

$t$3

is injected into the residual stream, steering the attention to token $t$4 rather than $t$5. This reveals an explicit, manipulable basis for discrete modular arithmetic embedded within the Transformer architecture.

4. Presence Across Model Families and Scales

Empirical investigations reveal successor heads as a recurrent motif across major LLM architectures:

Model Family	Emergence of Successor Head	Robustness
Pythia	From 70M (head 4.7) upward	Present up to 6.9B
GPT-2	From 124M (layer 3, head 10)	Present up to 1.5B XL
Llama-2	7B, 13B (various layers)	70B: weaker, less sharp
Mistral	7B: weak presence
OPT	13B: nearly absent

This broad architectural presence signals that the underlying mechanisms for successor computation are not narrowly tied to specific training recipes or model depths. Some variance in presence and sharpness (e.g., Mistral and OPT) suggests partial dependence on the training regimen and hyperparameter choices (Gould et al., 2023).

5. Natural Language Manifestations and Polysemanticity

Successor heads are activated by semantically sequential prompts in natural language contexts. For instance, prompting with “Monday, Tuesday, Wednesday, Thursday, …” directs attention from “Thursday” to “Friday,” increasing the log-odds of predicting “Friday” by approximately +10. Ablation of the head disrupts the natural sequence, indicating direct causal responsibility.

An instance of interpretable polysemanticity is observed in Pythia-350M (layer 5, head 2), where a single head attends to both numeric and month-based successors (e.g., “1”→“2” and “March”→“April”). Probing reveals clustering in the head's key space, with separable subspaces for numbers and months. Patching the corresponding feature vector (e.g., $t$6) boosts the attention to the correct successor (“December”), confirming explicit, interpretable multi-domain functionality within the same circuit (Gould et al., 2023).

6. Insights into Internal Numeric and Sequential Representation

The discovery and analysis of successor heads establish that LLMs encode a discrete modular counter within the residual stream. The vectors $t$7 represent a compact subspace for integer arithmetic, which the attention head leverages for increment operations. The same architectural motif extends naturally to “mod-7” (days) and “mod-12” (months) subspaces for other cyclic or sequential domains. This suggests that, at least for certain symbolic processes, LLMs move beyond distributed statistics to implement succinct, human-interpretable mechanisms that can be targeted for scientific discovery and practical manipulation.

7. Significance and Practical Implications

Successor heads provide a directly manipulable and interpretable mechanistic circuit in large-scale LLMs, illuminating the translation from distributed deep representations to concrete symbolic operations. This finding enables:

• Diagnosing mis-sequencing or counting failures,
• Editing successor-related behavior by feature vector patching,
• Steering model generation in structured, sequential tasks.

The presence of interpretable basis vectors in the model underlines the feasibility of circuit-level understanding and control. The study of successor heads forms a blueprint for further mechanistic interpretability research and injects new rigor into the analysis of symbolic competence in foundation models (Gould et al., 2023).