Indirect Object Identification (IOI)

• Indirect Object Identification (IOI) is a next-token prediction benchmark where a language model predicts the indirect object by identifying the non-repeated name in structured prompts.
• The benchmark employs controlled templates and causal intervention methods such as activation and path patching to reveal detailed circuit mechanisms in models like GPT-2 small.
• Analyses of IOI have uncovered a robust, multi-component circuit with specialized attention heads and redundancy, informing both mechanistic interpretability and model performance studies.

Searching arXiv for recent IOI mechanistic-interpretability papers and canonical references. {"query":"Indirect Object Identification GPT-2 small mechanistic interpretability arXiv", "max_results": 10} {"query":"(Wang et al., 2022, Merullo et al., 2023, Franco et al., 2024, Gupta et al., 2024, Zhang et al., 13 Feb 2025, Chhabra et al., 27 Feb 2025, Ensign et al., 2024, Adhikari, 28 Oct 2025, Munigety, 21 May 2026, Nasermoghadasi, 21 May 2026)", "max_results": 10} Indirect Object Identification (IOI) is a next-token prediction benchmark in which a LLM is given a sentence prefix containing two names, one of which is repeated later as the subject of a transfer or speech event, and must predict the other name—the indirect object—at the final position. In the canonical formulation studied in GPT-2 small, examples take forms such as “When Mary and John went to the store, John gave a bottle of milk to …”, where the correct continuation is the name that is not the repeated subject (Wang et al., 2022). IOI became a central benchmark in mechanistic interpretability because it is narrow, behaviorally crisp, linguistically meaningful, and amenable to causal intervention methods that permit circuit-level reverse engineering (Wang et al., 2022, Franco et al., 2024).

1. Task definition and benchmark role

The standard IOI setup distinguishes the correct final name, denoted IO, from the competing incorrect repeated name, denoted S. The canonical notation further separates the first and second occurrences of the subject as S1 and S2, and denotes the final prediction position as END (Wang et al., 2022). The task is typically instantiated with both ABBA and BABA variants, so that the benchmark is not reducible to a fixed positional heuristic (Wang et al., 2022, Franco et al., 2024).

In the original GPT-2 small study, IOI prompts were generated from 15 hand-written templates, using 100 single-token English first names, 20 places, and 20 objects (Wang et al., 2022). On 100,000 examples from this distribution, GPT-2 small achieved a mean logit difference of 3.56, predicted IO over S 99.3% of the time, and assigned a mean IO probability of 49% (Wang et al., 2022). Subsequent work continued to treat IOI as a benchmark precisely because it combines high model performance with a human-readable candidate algorithm: identify the names, detect which name is duplicated, and output the remaining name (Wang et al., 2022).

IOI also functions as a benchmark with unusually favorable intervention structure. Clean and corrupted prompt pairs can differ only in name identity while preserving template, location, and object words, making activation patching, path patching, and related interchange-style interventions technically straightforward (Wang et al., 2022, Munigety, 21 May 2026). This suggests that IOI occupies a distinctive position between natural-language behavior and controlled causal analysis.

2. Canonical circuit in GPT-2 small

The canonical mechanistic account of IOI in GPT-2 small proposes a circuit of 26 attention heads grouped into 7 classes: Name Mover Heads, Negative Name Mover Heads, S-Inhibition Heads, Duplicate Token Heads, Induction Heads, Previous Token Heads, and Backup Name Mover Heads (Wang et al., 2022). These classes instantiate a three-stage functional organization: duplicate detection, inhibition of the duplicated subject, and movement of the surviving name to the output position, with negative and backup components acting as auxiliary structure (Wang et al., 2022).

The decisive late heads are the Name Mover Heads 9.6, 9.9, and 10.0, which operate at END, attend strongly to earlier name tokens, and write in a direction that boosts the logit of the attended name (Wang et al., 2022). Their measured copy score exceeds 95% for all three heads, compared with under 20% for an average head (Wang et al., 2022). Two Negative Name Mover Heads, 10.7 and 11.10, are copy-like in the opposite direction, with a reported negative copy score around 98% (Wang et al., 2022). The paper explicitly includes them because they exert strong direct effects of the opposite sign on the final logits (Wang et al., 2022).

The late-name-copying behavior is controlled upstream by the S-Inhibition Heads 7.3, 7.9, 8.6, and 8.10, identified by path patching to the queries of the name mover heads (Wang et al., 2022). These heads act at END, attend primarily to S2 with average attention 0.51, and alter downstream name-mover queries so that the repeated subject is suppressed (Wang et al., 2022). Their signal has separable token and position components, with position carrying the stronger effect; the reported approximation for logit difference is $2.31 S_{pos} + 0.99 S_{tok}$ (Wang et al., 2022).

Earlier in the circuit, Duplicate Token Heads 0.1 and 3.0 attend from S2 back to S1, while Induction Heads 5.5 and 6.9 attend from S2 to S1+1 (Wang et al., 2022). The induction pathway is supported by Previous Token Heads 2.2 and 4.11, which copy information about a token into the following position (Wang et al., 2022). The full circuit also includes fuzzy supporting heads—such as 0.10 for duplicate detection and 5.8, 5.9 for induction-related support—and a set of 8 Backup Name Mover Heads: 9.0, 9.7, 10.1, 10.2, 10.6, 10.10, 11.2, and 11.9 (Wang et al., 2022). The backup structure matters because ablating all three main name movers reduces performance by only about 5% in logit difference, revealing substantial redundancy (Wang et al., 2022).

3. Interpretability methods and formal evaluation

IOI is tightly associated with a particular methodological toolkit. The original circuit was identified through mean ablation, activation patching, and especially path patching, which isolates direct effects from sender heads to receiver nodes while allowing MLPs to be recomputed (Wang et al., 2022). The paper formalized circuit quality with the average logit-difference functional

$$F(C) = \mathbb{E}_{X \sim p} \left[f(C(X); X) \right],$$

and evaluated a proposed circuit through faithfulness, completeness, and minimality (Wang et al., 2022). For the full 26-head circuit, $F(M)=3.56$ and $|F(M)-F(C)| = 0.46$, corresponding to about 87% of full-model performance (Wang et al., 2022). At the same time, greedy completeness search found subsets with incompleteness as high as 3.09, showing that even the canonical explanation left important gaps (Wang et al., 2022).

Later work refined IOI analysis by moving from head importance to communication content. Sparse Attention Decomposition studies $W_{OV}^{(h)} = W_O^{(h)} W_V^{(h)}$ and $W_{QK}^{(h)} = W_Q^{(h)\top} W_K^{(h)}$, decomposing these maps with $M = U\Sigma V^\top$ and using sparse singular-feature channels to trace residual directions exchanged between heads (Franco et al., 2024). Applied to IOI, this recovers the familiar architecture of duplicate-name detection, suppression, and late name movement, but recasts it as a network of low-dimensional residual features rather than only a list of head labels (Franco et al., 2024).

Other methodological extensions use IOI to probe different explanatory levels. Self-influence analysis extracts sparse circuits with EAP, EAP-IG, and EAP-IG-KL, then computes

$$I_H(x, x) = - \nabla_{\theta} L(x)^\top H^{-1} \nabla_{\theta} L(x)$$

layer by layer to track changing token importance through the circuit (Zhang et al., 13 Feb 2025). In that setting, circuits occupying only about 1–2% of edges preserve at least 85% of model performance, and the dominant qualitative story is a progression from entity identification to relational processing to final disambiguation (Zhang et al., 13 Feb 2025). A further five-stage methodology—probe design, feature extraction, causal validation, robustness testing, and deployment integration—uses IOI to distinguish representation-level selectivity from actual causal necessity, again finding that circuit localization is cleaner than feature-level explanation (Munigety, 21 May 2026).

4. Generalization across tasks and architectures

A major line of work treats IOI not as an isolated benchmark but as a reusable computational motif. In GPT-2 Medium, the Wang et al. circuit largely reproduces, with the same broad classes—duplicate or induction heads, inhibition heads, mover heads, and a negative mover head—active at larger scale (Merullo et al., 2023). On 1000 examples, GPT-2 Medium prefers the IO token to the Subject token in the logits 100% of the time, and the paper argues that the underlying algorithm is not specific to indirect-object syntax but expresses a more general selection-among-alternatives procedure (Merullo et al., 2023). The same study reports about 78% overlap in in-circuit attention heads between IOI and Colored Objects, concretely 25/32 shared heads under its thresholding rule, and shows that adjusting four heads can repair Colored Objects from 49.6% to 93.7% accuracy in an IOI-like manner (Merullo et al., 2023).

IOI has also been exported beyond transformers. In Mamba, the proposed IOI circuit is centered on Layer 39 as a key bottleneck in state-spaces/mamba-370m, a model with about 96% accuracy on the authors’ IOI templates (Ensign et al., 2024). The analysis attributes cross-token routing not to attention heads but to causal convolution and state-space recurrence, finding evidence that Layer 39’s convolution shifts name information one position forward, that name entities are stored linearly in its SSM state, and that Layer 39 writes task-relevant information primarily into the final token position (Ensign et al., 2024). This suggests that IOI’s status as a circuit benchmark extends to recurrent or state-space architectures, even though the interpretable units shift from attention heads to layer-position-state interactions.

Recent reasoning work also uses IOI as a baseline for comparison. An abstract-level example is the study of syllogistic prompts in GPT-2 small, which describes syllogisms as requiring more complex logical reasoning than IOI and explicitly relates newly identified binary mechanisms back to prior IOI analysis (Saraipour et al., 22 Aug 2025). This suggests that IOI increasingly serves as a comparative anchor for broader studies of model reasoning rather than only as a standalone task.

5. Controlled benchmarks and minimal reconstructions

IOI has been turned into a controlled benchmark with known causal structure. InterpBench uses Strict Interchange Intervention Training (SIIT) to instantiate a semi-synthetic transformer that implements a simplified IOI causal model rather than analyzing an unconstrained pretrained model (Gupta et al., 2024). The IOI transformer in that benchmark has 6 layers, 4 attention heads per layer, $d_{\text{model}} = 64$, and $d_{\text{head}} = 16$ (Gupta et al., 2024). Its high-level causal structure contains three named components—Duplicate token head, S-Inhibition head, and Name mover head—with layer-level alignment between the simplified circuit and the low-level model (Gupta et al., 2024). The benchmark trains until models reach 100% IIA and 100% SIIA, then evaluates circuit-discovery methods such as ACDC, Subnetwork Probing, Edgewise SP, EAP, and EAP-IG using ROC AUC against the known ground truth (Gupta et al., 2024). The resulting IOI benchmark is explicitly not a faithful copy of the full GPT-2 small circuit, but a controllable intermediate case between Tracr-style toy programs and real pretrained systems (Gupta et al., 2024).

A complementary direction asks for the minimal mechanism sufficient for symbolic IOI. In an attention-only setting, a one-layer, two-head model trained from scratch on a symbolic IOI task achieves perfect IOI accuracy, whereas a one-layer, one-head model fails and a two-layer, one-head model achieves similar performance through inter-layer composition (Adhikari, 28 Oct 2025). The successful one-layer solution decomposes into an additive head aligned with $$F(C) = \mathbb{E}_{X \sim p} \left[f(C(X); X) \right],$$0 and a contrastive head aligned with $$F(C) = \mathbb{E}_{X \sim p} \left[f(C(X); X) \right],$$1, so their residual contributions sum to amplify the correct token while canceling the incorrect one (Adhikari, 28 Oct 2025). Without positional embeddings, the same one-layer two-head architecture falls to about 70% accuracy and about 67% probability on the correct token (Adhikari, 28 Oct 2025). These results suggest that the elaborate pretrained-model circuit is not the minimal computation required for IOI itself, but a richer realization shaped by pretraining and multitask pressures.

6. Circuit dynamics, feature analyses, and failure modes

IOI has become a laboratory for studying how mechanisms change under training. In GPT-2 small, fine-tuning on a clean IOI dataset amplifies the existing circuit rather than replacing it: average logit difference rises from 6.32 after 1 epoch to 26.83 after 100 epochs, while the discovered circuit remains highly faithful, with faithfulness values from 98.4% to 99.89% and sparsity between 1.92% and 2.68% (Chhabra et al., 27 Feb 2025). The same work identifies Circuit Amplification, localized corruption under poisoned fine-tuning, and neuroplasticity when retraining corrupted models on clean data, with recovered post-reversal circuits reaching 95% and 96% faithfulness in the two corruption settings analyzed (Chhabra et al., 27 Feb 2025).

IOI has also supported token-centric analyses of reasoning trajectories. A self-influence study fine-tunes GPT-2 on a corpus of 1000 sentences and searches circuits over 158 nodes and 32,491 edges, reporting that circuits with about 1–2% of edges can retain at least 85% of model performance (Zhang et al., 13 Feb 2025). The reported layerwise pattern places important nodes in the first layer and the last few layers, and interprets token salience as moving from entity tokens such as “Christina” or “Amy” toward relation tokens such as “gave” before late-layer aggregation (Zhang et al., 13 Feb 2025). The paper presents this as a new lens on IOI rather than a replacement for head-level circuit taxonomies (Zhang et al., 13 Feb 2025).

Feature-level analysis complicates the apparent cleanliness of the benchmark. A sparse-feature audit on 300 IOI prompts finds GPT-2 small at 79.7% accuracy, with 146 Holm-significant SAE features and 105 features exceeding $$F(C) = \mathbb{E}_{X \sim p} \left[f(C(X); X) \right],$$2 (Nasermoghadasi, 21 May 2026). The strongest failure correlate is feature 17,491, with $$F(C) = \mathbb{E}_{X \sim p} \left[f(C(X); X) \right],$$3, but it is essentially silent except when the transferred object is “the keys”; on those 45 prompts GPT-2 small fails 93.3% of the time, compared with 7.5% on the other 255 prompts (Nasermoghadasi, 21 May 2026). Zeroing that feature does not restore performance, changing accuracy on the “keys” subset from 6.7% to 4.4%, so the paper concludes that it is a correlate rather than a sufficient cause (Nasermoghadasi, 21 May 2026).

A related SAE-based causal study recovers highly selective per-name IO features at layer 9 END, with selectivity gaps of 28 to 48 activation units for the top ten features, but finds that these features are only partially causal (Munigety, 21 May 2026). Single-feature ablation drops logit difference by about 0.67 logits on matching prompts, yet ablating the top 15 selective features still leaves the model 98.4% correct and reduces mean logit difference only from +3.71 to +3.33 (Munigety, 21 May 2026). The same paper reports that those fifteen features explain only 30.7% of activation variance, compared with 0.997 for the full SAE reconstruction, and that selectivity ratio anticorrelates with causal drop at $$F(C) = \mathbb{E}_{X \sim p} \left[f(C(X); X) \right],$$4 (Munigety, 21 May 2026). Under prompt reformulation, feature firing remains strong while ablation effects degrade substantially, formalizing a gap between detection robustness and causal robustness (Munigety, 21 May 2026).

7. Limitations, controversies, and continuing significance

The IOI literature is notable for emphasizing its own incompleteness. The canonical GPT-2 small paper explicitly reports strong faithfulness and nontrivial minimality, but also acknowledges worst-case completeness failures and unresolved roles for MLPs, negative name movers, and backup routes (Wang et al., 2022). Subsequent work on sparse subspaces and feature interventions reinforces that caution: residual-stream features remain distributed, singular vectors are basis-dependent approximations, and successful low-dimensional interventions can in principle reflect alternate pathways rather than the hypothesized one (Franco et al., 2024). In controlled benchmarks, the simplifications are equally explicit: InterpBench’s IOI is a simplified version of the manually discovered GPT-2 Small circuit, trained in a single-circuit model with coarse alignment from high-level nodes to entire layers (Gupta et al., 2024).

These limitations have not diminished IOI’s importance. Rather, they have changed its role. IOI now serves simultaneously as a canonical transformer circuit, a benchmark for circuit-discovery methods, a testbed for cross-task reuse, a platform for studying training-induced mechanism change, a source of minimal synthetic reconstructions, and a stress test for the distinction between correlation and cause (Wang et al., 2022, Gupta et al., 2024, Merullo et al., 2023, Chhabra et al., 27 Feb 2025, Munigety, 21 May 2026). A plausible implication is that IOI’s enduring value lies less in being “solved” than in being unusually diagnostic: it exposes redundancy, feature superposition, failure confounds, intervention artifacts, and cross-architecture regularities while remaining concrete enough for detailed causal analysis.