Mechanistic Interpretability Benchmark (MIB)

Updated 4 July 2026

MIB is a standardized framework that evaluates mechanistic interpretability methods by mapping internal neural computations through intervention-based protocols.
It distinguishes between circuit localization and causal variable localization, employing metrics like CPR and CMD to quantify faithfulness.
The benchmark encourages reproducible comparisons while addressing challenges such as non-identifiability and trade-offs in attribution and selection quality.

The Mechanistic Interpretability Benchmark (MIB) is a standardized evaluation framework for mechanistic interpretability methods. In the benchmark and shared-task literature, it is described both as a standardized evaluation suite for mechanistic interpretability methods applied to LLMs and as a benchmark plus codebase for systematically evaluating circuit discovery, while also supporting causal variable localization through intervention-based evaluation on standard tasks, models, and counterfactual inputs (Arad et al., 23 Nov 2025, Nikankin et al., 28 Oct 2025).

1. Conceptual foundations

MIB inherits a demanding conception of mechanistic interpretability. One influential formulation defines mechanistic explanations as valid only if they are Model-level, Ontic, Causal-Mechanistic, and Falsifiable. On this view, the object of explanation is the neural network itself rather than a larger deployed system; the explanatory entities must correspond to real internal variables, features, or circuits; the explanation must specify a continuous causal chain from inputs to outputs; and it must make experimentally refutable claims under ablation, patching, or related interventions (Ayonrinde et al., 1 May 2025).

This framing also motivates explanatory faithfulness as a benchmark target. Faithfulness is not merely behavioral agreement between an explanation and a model; it concerns whether the explanation’s intermediate states match the model’s own internal states across the data distribution. In benchmark terms, this favors intervention-based protocols over purely observational narratives and explains why MIB emphasizes causal localization of circuits and causal variables rather than input-output rationalization alone (Ayonrinde et al., 1 May 2025).

A second conceptual pressure comes from identifiability. In small Boolean-function networks, multiple incompatible explanations can satisfy strong mechanistic criteria simultaneously: multiple circuits can reproduce the same behavior, a single circuit can admit multiple interpretations, and multiple high-level algorithms can align with the same network. The paper on non-identifiability reports, for one XOR MLP, 85 distinct perfect circuits, an average of 535.8 consistent mappings per circuit, and 159 perfect minimal mappings for two candidate algorithms, illustrating that uniqueness cannot be assumed even under strict causal standards (Méloux et al., 28 Feb 2025).

2. Benchmark structure and tracks

In its concrete language-model instantiation, MIB evaluates two distinct but related mechanistic tasks: circuit localization and causal variable localization. Circuit localization asks whether a method can identify causal subgraphs of a model’s computation graph that implement task behavior. Causal variable localization asks whether a method can map hidden activations into interpretable feature spaces and show that interventions on those features reproduce interventions on high-level causal variables (Arad et al., 23 Nov 2025).

MIB supplies standard tasks, models, and counterfactual inputs. The task set includes Indirect Object Identification (IOI), Arithmetic (Addition and Subtraction), Multiple-choice Question Answering (MCQA), AI2 Reasoning Challenge (ARC), and RAVEL for causal variable localization. The model set includes Llama-3.1 8B, Gemma-2 2B, Qwen-2.5 0.5B, and GPT-2 Small, with model-task pairs filtered so that only settings with at least 75% base accuracy are included (Arad et al., 23 Nov 2025).

Track	Object localized	Primary evaluation
Circuit localization	Causal subgraphs or circuits	$\textsf{CPR}$ , $\textsf{CMD}$
Causal variable localization	Featurizer $\mathcal{F}$ , inverse $\mathcal{F}^{-1}$ , and feature indices $\Pi_X$	Interchange-intervention accuracy

For circuit localization, the model is treated as a directed computation graph whose nodes are internal components and whose edges are information-flow relations. A circuit is a subgraph intended to reproduce task-relevant behavior. For causal variable localization, hidden vectors $\mathbf{h} \in \mathbb{R}^d$ are mapped into a feature space $\mathbb{F}^k$ by a featurizer $\mathcal{F}$ , and the benchmark tests whether interventions on selected feature indices $\Pi_X$ behave like interventions on a corresponding high-level causal variable $X$ (Arad et al., 23 Nov 2025).

The BlackboxNLP 2025 shared task adopted a smaller mandatory subset for the circuit-localization track: GPT-2-Small on IOI, Qwen-2.5-0.5B on IOI, and Qwen-2.5-0.5B on MCQA. This subset did not redefine MIB; it operationalized a reproducible public comparison on fixed model-task pairs (Mondorf et al., 8 Oct 2025).

3. Metrics, intervention protocols, and submission format

MIB’s central quantity for circuits is faithfulness. For a circuit $\textsf{CMD}$ 0, a full model $\textsf{CMD}$ 1, and a task-specific scalar score $\textsf{CMD}$ 2, faithfulness is defined as

$\textsf{CMD}$ 3

In the benchmark, $\textsf{CMD}$ 4 is defined from the logit difference between correct answers on base and counterfactual inputs. This turns activation patching into a normalized measure of how much of the model’s task-relevant behavior is recovered by the proposed circuit (Arad et al., 23 Nov 2025).

Two integrated circuit metrics are derived from the faithfulness curve over circuit sizes. Integrated Circuit Performance Ratio ( $\textsf{CMD}$ 5) is the area under the faithfulness curve and rewards circuits that recover performance quickly as edges are added. Integrated Circuit-Model Distance ( $\textsf{CMD}$ 6) is the area between the faithfulness curve and the ideal value $\textsf{CMD}$ 7, so lower values correspond to closer reproduction of full-model behavior, including negative contributions (Nikankin et al., 28 Oct 2025).

MIB also fixes how circuit size is measured. It uses a weighted edge count that treats including a node as equivalent to including all its outgoing edges and including one neuron in a $\textsf{CMD}$ 8-dimensional submodule as including outgoing edges to $\textsf{CMD}$ 9 of the full degree. For leaderboard evaluation, participants provide circuits at maximum size fractions

$\mathcal{F}$ 0

either as binary masks or as importance scores from which top- $\mathcal{F}$ 1 circuits are derived (Arad et al., 23 Nov 2025).

For causal variable localization, the benchmark uses distributed interchange interventions. Let $\mathcal{F}$ 2 be a high-level causal variable and $\mathcal{F}$ 3 the feature indices intended to encode it. The benchmark compares the effect of intervening on $\mathcal{F}$ 4 in a high-level causal model to the effect of intervening on $\mathcal{F}$ 5 in the neural model after mapping activations with $\mathcal{F}$ 6. Faithfulness is the proportion of examples for which these interventions agree in their output effect; reported results usually give mean faithfulness across layers and counterfactual datasets, with the best layer in parentheses (Arad et al., 23 Nov 2025).

4. Shared-task development and method families

The BlackboxNLP 2025 shared task transformed MIB into a community-wide comparison. In the circuit-localization track, three teams submitted eight methods; in the causal-variable-localization track, one team submitted two methods. The shared-task report states that participants achieved notable gains in circuit localization using ensemble and regularization strategies for circuit discovery, and significant gains in causal variable localization using low-dimensional and non-linear projections (Arad et al., 23 Nov 2025).

One methodological line improved the selection stage of circuit discovery while keeping MIB’s attribution framework fixed. The paper on better edge selection uses three additions: bootstrapping to retain sign-stable edges, a Positive-Negative Ratio (PNR) constraint to control the fraction of positively contributing edges, and integer linear programming (ILP) to choose globally optimal connected subgraphs under a size budget. In its experiments on GPT-2 Small, Qwen-2.5 0.5B, and Gemma-2 2B across IOI, MCQA, and ARC-E, these strategies improved or matched greedy MIB baselines on most $\mathcal{F}$ 7 and $\mathcal{F}$ 8 settings, while also making explicit the trade-off between score quality and selection quality (Nikankin et al., 28 Oct 2025).

A second line showed that MIB favors ensembling of circuit-localization methods. Parallel ensembles average edge scores from several attribution methods; sequential ensembles use a fast method such as EAP-IG-inputs to warm-start a slower pruning-based method; and hybrid ensembles combine both. On the shared-task private test set, the reported hybrid-ens method achieved the best overall performance, with the highest average $\mathcal{F}$ 9 of 1.83 and an average $\mathcal{F}^{-1}$ 0 of 0.03, matching the lowest $\mathcal{F}^{-1}$ 1 while improving circuit performance (Mondorf et al., 8 Oct 2025).

The causal-variable track exhibits a different methodological tension: more expressive featurizers can increase interchange-intervention accuracy, but they also raise the concern that the method may inject new information rather than faithfully reveal structure already present in the model. Hypernetwork-based systems such as HyperDAS were developed precisely around this tension, combining automatic localization of token positions with learned low-dimensional concept subspaces and explicit design choices intended to mitigate steering-like failure modes (Sun et al., 13 Mar 2025).

MIB sits within a broader benchmarking ecosystem. InterpBench is a closely related benchmark for transformers with known circuits. It provides 17 semi-synthetic yet realistic transformers, 16 derived from Tracr-compiled RASP programs and one from a simplified IOI circuit, and uses Strict Interchange Intervention Training (SIIT) to enforce that circuit nodes carry the task-relevant computation while non-circuit nodes remain causally inert on the task distribution. Its primary quantitative output is edge-level ROC-AUC for circuit-discovery methods, and it was explicitly proposed as a benchmark for evaluating mechanistic interpretability techniques (Gupta et al., 2024).

Other benchmark-like efforts target different modalities or scales. The IMI dataset for vision models uses a psychophysical 2-AFC paradigm to measure unit-level interpretability and releases more than 130,000 human responses over 767 units across nine models; it was presented as a first-step mechanistic interpretability benchmark for vision and as a basis for training automated predictors of human interpretability (Zimmermann et al., 2023). The MIPS program-synthesis work evaluates mechanistic extraction on a 62-task algorithmic suite for RNNs and treats exact recovery of executable Python programs as the success criterion, effectively creating a mechanistic benchmark for algorithm-learning networks (Michaud et al., 2024).

Several later papers do not define an official MIB, but explicitly present blueprints for one in specific domains. For transformer-based time-series classification, a paper proposes a reusable MIB template built around activation patching, attention saliency, and sparse autoencoders, with a three-level causal probing framework at the layer, head, and timestep levels and a primary causal-effect metric $\mathcal{F}^{-1}$ 2 (Kalnāre et al., 26 Nov 2025). In AI weather models, an open-source mechanistic interpretability tool for GraphCast organizes 83,902,464 latent values across processor steps and nodes, using cosine similarity, PCA, and aligned latent-space directions as candidate feature-discovery primitives for a weather-domain MIB (Tempest et al., 22 Apr 2026). In causal biostatistics, another work outlines how probing, sparse autoencoders, activation patching, ablation, and causal tracing could define benchmark tasks around nuisance-function estimators in TMLE-style pipelines (Conan, 1 May 2025). In jet physics, mechanistic analysis of a Particle Transformer identifies a sparse six-head circuit with a source-relay-readout structure and shows that residual representations are more aligned with the energy-correlator basis than with the $\mathcal{F}^{-1}$ 3-subjettiness basis, suggesting concrete MI benchmark tasks for physics classifiers (Rai et al., 11 May 2026).

These adjacent efforts indicate that “MIB” functions both as a specific language-model benchmark and as a more general design pattern: fixed models, standardized interventions, quantitative faithfulness criteria, and tasks that can expose internal mechanisms rather than merely summarize outputs (Kowalska et al., 24 Nov 2025).

6. Limitations, controversies, and design tensions

A major design tension is non-identifiability. The Boolean-network study argues that current mechanistic criteria can admit many incompatible explanations at once, and concludes that a benchmark should not assume a unique correct circuit or algorithm unless the model is explicitly constructed to make uniqueness plausible. The same paper suggests that, if uniqueness is essential, stricter criteria are needed; if pragmatic goals such as prediction and manipulability are sufficient, then a benchmark may need to accept multiple valid explanations and score them by properties such as faithfulness, parsimony, and robustness rather than exact identity with a single gold mechanism (Méloux et al., 28 Feb 2025).

A second tension is that MIB-like benchmarks are inevitably theory-laden and value-laden. One philosophical analysis argues that any benchmark must choose an abstraction level, a precision metric for matching internal states, and an implicit stance on whether faithful human-understandable explanations exist for the target models. It labels this background assumption a form of Explanatory Optimism and notes that, if important behavior depends on “Alien Concepts,” then a benchmark that requires human-understandable explanations may have unattainable upper bounds (Ayonrinde et al., 1 May 2025).

Operational difficulties also persist. In the shared-task literature, no single circuit-localization method dominates both $\mathcal{F}^{-1}$ 4 and $\mathcal{F}^{-1}$ 5, and blank cells in result tables partly reflect compute constraints rather than purely methodological weakness (Arad et al., 23 Nov 2025). The ILP-based edge-selection work explicitly identifies ILP scalability and the quality of attribution scores as bottlenecks, noting that global selection can improve objective values while only modestly improving faithfulness when edge scores are noisy (Nikankin et al., 28 Oct 2025). Domain-specific blueprint papers add further warnings: manual clean–corrupt pair selection, limited component coverage, non-additivity of multi-patching, dependence on region or reference choice, and the risk that correlation-based latent directions may be only hypothesis-generating unless followed by causal interventions (Kalnāre et al., 26 Nov 2025, Tempest et al., 22 Apr 2026).

Taken together, these tensions imply that MIB is best understood not as a solved scoreboard for mechanistic understanding, but as a structured research program. Its distinctive contribution is to force intervention-based, reproducible comparisons of candidate explanations, while making explicit the unresolved questions about faithfulness, identifiability, scalability, and what should count as mechanistic success.