Pando: Do Interpretability Methods Work When Models Won't Explain Themselves?

Abstract: Mechanistic interpretability is often motivated for alignment auditing, where a model's verbal explanations can be absent, incomplete, or misleading. Yet many evaluations do not control whether black-box prompting alone can recover the target behavior, so apparent gains from white-box tools may reflect elicitation rather than internal signal; we call this the elicitation confounder. We introduce Pando, a model-organism benchmark that breaks this confound via an explanation axis: models are trained to produce either faithful explanations of the true rule, no explanation, or confident but unfaithful explanations of a disjoint distractor rule. Across 720 finetuned models implementing hidden decision-tree rules, agents predict held-out model decisions from $10$ labeled query-response pairs, optionally augmented with one interpretability tool output. When explanations are faithful, black-box elicitation matches or exceeds all white-box methods; when explanations are absent or misleading, gradient-based attribution improves accuracy by 3-5 percentage points, and relevance patching, RelP, gives the largest gains, while logit lens, sparse autoencoders, and circuit tracing provide no reliable benefit. Variance decomposition suggests gradients track decision computation, which fields causally drive the output, whereas other readouts are dominated by task representation, biases toward field identity and value. We release all models, code, and evaluation infrastructure.

• The paper introduces Pando, a benchmark that isolates the utility of mechanistic interpretability in language models not providing faithful explanations.
• Gradient-based methods, especially RelP, consistently boost held-out prediction accuracy by 3–5 percentage points over black-box techniques in challenging regimes.
• The study shows that internal representation tools fail to capture decision-relevant signals, highlighting the need for new model auditing approaches.

Authoritative Review of "Pando: Do Interpretability Methods Work When Models Won't Explain Themselves?" (2604.11061)

Motivation and Benchmark Design

This work introduces Pando, a benchmark designed to isolate the utility of mechanistic (white-box) interpretability methods when LMs are trained not to provide faithful verbal explanations of their decision processes. Standard interpretability evaluation often conflates improvements from better black-box elicitation (prompting for rationales) with the true incremental information identified by internal analysis, a confounder termed in the paper as the elicitation confounder. Pando controls for this by constructing a suite of LMs finetuned on randomly sampled decision-tree rules, with a key explanation axis: models are trained to provide either (a) faithful explanations, (b) no explanation, or (c) systematically misleading explanations based on distractor rules.

Pando adopts a model-organism approach: the ground truth for the decision procedure is known and can be interrogated at the feature level. This enables direct measurement of both end-to-end prediction accuracy (does a method allow an agent to anticipate the LM's label on unseen examples?) and field-level relevance (does a method pinpoint the true decision-relevant fields?). Agents must predict held-out model decisions, given $k=10$ labeled query-response pairs, optionally augmented with an interpretability tool output.

Methodological Framework and Evaluated Techniques

Each Pando problem instance involves a tabular input space with 10 fields (mixed categorical and numeric), and a binary decision label produced by LMs implementing a randomly sampled depth-1 to depth-4 decision tree. Three explanation training regimes are systematically compared:

• Faithful: Explanations directly rationalize the true underlying rule.
• No Explanation: No rationale or text is provided by the LM.
• Unfaithful: The LM produces confident verbalizations of an unrelated distractor rule, ensuring rationalizations are systematically wrong.

Evaluated interpretability methods span:

• Gradient-based: Including per-token embedding gradients (saliency), and Relevance Patching (RelP) [relp2025].
• Representation-based: Logit lens [nostalgebraist2020logitlens], residual activation similarity.
• SAE-based: Using sparse autoencoders for feature-level attribution [cunningham2023sparse, gemmascope2024].
• Circuit-tracing: Attribution-based circuit tracing via feature graphs [ameisen2025circuit].
• Black-box baselines: Standard nearest-neighbor, tree voting, and prompting for a prefilled explanation (verbal rationale continuation).

A fixed LLM-based agent conducts a two-stage process: first, it extracts a natural-language rule from the available evidence (queries and tool outputs); second, it applies this rule to the held-out examples.

Main Empirical Results

White-box Interpretability versus Black-box Elicitation

The main empirical finding is that gradient-based methods, and especially RelP, are the only class of interpretability tools that provide a consistent and measurable gain in held-out decision prediction when the explanation channel is uninformative or misleading.

• When explanations are faithful, simply prompting for verbalization (the prefill agent) achieves or exceeds the accuracy of any white-box method, confirming that under such regimes, true decision features can be “elicited” through prompting alone.
• When explanations are absent or unfaithful, gradient and RelP attribution outperform the sample-only baseline by 3–5 percentage points in held-out accuracy. Other white-box methods (SAE, logit lens, circuit tracing) do not yield reliable benefit over black-box querying, despite internal access.

These dynamics persist across scenarios (car purchase, movie selection, policy violation), tree depths, and input budgets.

Figure 1: Held-out accuracy as a function of query budget across explanation regimes, demonstrating clear advantages for RelP and prefill (in the faithful regime), with RelP maintaining a pronounced lead over sample_only particularly for low sample budgets.

Field-level Attribution and Decision Structure Recovery

Gradient-based attribution, and particularly RelP, yields a much larger improvement in identifying which fields drive the model's decision, as measured by field F1. This effect grows with tree depth and is pronounced at low k:

Figure 2: Field-level F1 versus sample budget in the car purchase scenario, showing that RelP consistently surpasses sample_only across budgets, with the greatest gap under low-sample regimes.

Notably, other interpretability tools produce signals highly correlated with field identity or value—reflecting the LM’s generic task representation—rather than decision relevance. This is quantified through variance decompositions of per-field importance scores, revealing that only gradient-based methods are sensitive to decision-relevance rather than spurious field presence.

Automated Interpretability Agent Optimization

To assess the scope for further improvements via automatic method search, an LLM-powered research agent iteratively optimized interpretability agents on Pando for over 25 hours, exploring 78 agent variants. Starting from a baseline that combined prefill and gradients, only modest improvements were obtained by integrating RelP, candidate rule induction, and more refined prompt engineering.

Figure 3: Progression of automated agent optimization; while incremental improvements from gradient to RelP to the final agent are observed, the overall gain magnitude saturates, indicating limited headroom with existing attribution signals.

Theoretical and Practical Implications

Key numerical results:

• In unfaithful/no-explanation regimes, RelP achieves up to +4.7 percentage points in held-out accuracy over black-box querying, and up to +16.8 points in field F1 (no-explanation setup, intermediate depths).

Contradictory findings:

• Methods based on internal representations—logit lens, sparse autoencoders, circuit tracing—repeatedly fail to improve over the black-box baseline. Empirically, internal signal is only useful if it encodes decision-relevant computation, not merely field presence or descriptive features.

This work underscores a critical lower bound: If interpretability tools do not add value in a controlled context where the ground-truth decision structure is known and the only missing information is whether the LM's computation matches an intractable or misleading explanation, expectations for real-world behavioral auditing should be correspondingly tempered. In particular, circuit-tracing, current SAE feature analysis, and representational readouts are insufficiently sensitive to the mechanisms of interest in policy-driven or safety-critical domains.

Future Directions

The findings motivate (i) development of interpretability tools explicitly discriminating decision computation over spurious feature representation, (ii) research on circuit-tracing and SAE methods that advance past field-identity dominance, and (iii) exploration of adaptive querying or richer query strategies—since all current results are based on fixed, random query budgets. Pando's release—including 720 finetuned models with open code and infrastructure—provides a robust platform for exploring such methodological advances and enables systematic, head-to-head comparisons across interpretability techniques.

Conclusion

Pando offers a rigorous framework for determining when and how mechanistic interpretability can surpass black-box baseline methods at predicting LM behavior under controlled but challenging explanation regimes. Only gradient-based tools, notably RelP, deliver consistent decision-relevant signal beyond what can be elicited through prompting. This finding sets a strong empirical and conceptual baseline for future work in model auditing and interpretability, suggesting a need for fundamentally new approaches when models withhold or obfuscate their internal logic (2604.11061).