Surrogate Fidelity: When Can Open LLMs Explain Closed Ones?

Abstract: Mechanistic interpretability (MI) requires full access to model internals, yet the APIs for most widely deployed LLMs at best expose log-probabilities over output tokens. This creates a surrogate problem: when do measurements made on open models allow us to make claims about a closed model? We evaluate surrogate fidelity at the prediction, attribution, and representation levels. For binary classification tasks, log-odds provide an API-compatible scalar readout of the model's representation space, and leave-one-out attributions provide insight into model behavior. Across eleven models spanning four families (Llama, Qwen, GPT, and Gemini), we find that prediction fidelity substantially overstates attribution fidelity: models that agree on what the answer is often disagree on why. We document an access-validity inversion: white-box signals like attention patterns and perturbation magnitudes are highly stable across models but only weakly predictive of causal attributions, which black-box input ablations capture by design. Mechanistic insight does not automatically transfer to closed targets, and prediction-level agreement is insufficient to warrant such transfer. Code and results are available at https://github.com/facebookresearch/surrogate.

• The paper introduces a surrogate fidelity framework to assess prediction, attribution, representation, and cross-level alignment between open and closed LLMs.
• It demonstrates that high prediction fidelity does not imply reliable causal attribution transfer, especially under API constraints.
• Empirical results across various LLM families reveal significant asymmetries, scaling effects, and the impact of instruction tuning on surrogate explanations.

Surrogate Fidelity in Mechanistic Interpretability: Evaluation and Limits

Motivation and Problem Setting

Interpretability in LLMs typically demands full model access, enabling detailed mechanistic analyses such as circuit discovery and sparse autoencoding. However, the most widely deployed LLMs are accessed via restricted APIs, exposing at best token log-probabilities rather than internal representations. This engenders a surrogate dilemma: to what extent can analyses conducted on open-source models (surrogates) be reliably transferred to explain frontier, closed-source systems (targets)?

The paper formalizes "surrogate fidelity" as the degree to which measurements in accessible surrogate models predict analogous measurements in closed target models. The authors introduce a principled hierarchy of fidelity metrics—prediction, attribution, representation, and cross-level—each probing a successively stricter notion of alignment and transfer. Crucially, some metrics (especially prediction and attribution fidelity) are computable with only top-K log-probabilities, making them viable in common API settings.

Theoretical Framework

The relationship between a transformer's hidden state $z$ and its output probabilities is linear in binary classification tasks: log-odds for tokens $t_+$ and $t_-$ can be directly computed from log-probabilities without access to the normalization constant, and are interpretable as projections onto a readout direction $v = u_+ - u_-$. Attributions—changes in log-odds caused by input perturbations—are dot products $\Delta\ell = \Delta z \cdot v$, decomposable into perturbation norm and directional alignment.

The framework is robust to typical API constraints since log-odds and their responses to ablation interventions require only observable output statistics. Representation fidelity (magnitude and alignment) and cross-level fidelity (prediction of target attributions by surrogate representations) require internal access and are restricted to open-weight models.

Empirical Evaluation

The authors empirically evaluate surrogate fidelity across eleven models spanning four LLM families (Llama, Qwen, GPT, Gemini) and four scales (0.5B–70B parameters) on three binary classification datasets: BoolQ, ANLI, and WinoGrande. Sentence- and word-level ablation and attention analysis are conducted, exploiting open weights where available.

Pairwise prediction and attribution fidelity are visualized via heatmaps; geometric intuition for attribution decomposition is shown with joint distributions over norm, alignment, and LayerNorm-induced rescaling.

Figure 2: Pairwise prediction and attribution heatmaps demonstrate substantial variability across models and families; predictions align well, while attributions do not.

Across all experiments, prediction fidelity is high within families but lower across them. For instance, Llama-70B and Qwen-14B yield $r^2 = 0.684$ prediction fidelity. However, attribution fidelity is consistently much weaker, with median within-family $r^2 = 0.432$ and maximum cross-family $r^2$ only $0.557$. Smaller models (Qwen-0.5B) are outliers, exhibiting low fidelity across all measures.

Attribution Decomposition and Representation Analysis

To explain the failure of attribution-level transfer, the authors analyze how perturbations induced by ablation interventions affect attribution: via their norm, alignment with the readout direction, and LayerNorm-induced scaling. The primary driver of low attribution fidelity is weak alignment between perturbations and decision axes; while models agree on the magnitude of perturbations, the directions along which these perturbations impact predictions are inconsistent.

Figure 4: Perturbations affect attribution through their norm (left), alignment with the log-odds direction $v$ (middle), and indirectly via LayerNorm (right); directional alignment is the limiting factor.

Attention-based signals are highly consistent across models ($t_+$0 for mean-pooled attention), but do not reliably predict causal attribution outcomes. This stability in structural signals (magnitude, attention) is contrasted with the instability in functional signals (alignment, attribution), reinforcing the observation that models can share gross representational structures while diverging in causal evidence pathways.

Figure 1: Fidelity metrics for Qwen 2.5 models exhibit high agreement for attention and perturbation magnitude but low agreement for attributions; attention-based metrics fail as predictors for causal attributions.

Layerwise and Scaling Effects

Layerwise analysis using the logit lens reveals that attribution agreement rises sharply only in later model layers and is largely absent in early depth. This pattern is partially generic, but there is a premium for the actual readout direction used in binary classification.

Scaling studies show that instruction-tuning (rather than mere parameter count) is responsible for substantive separation in log-odds between positive and negative classes, highlighting the impact of posttraining on prediction fidelity.

Figure 3: Instruction-tuned Qwen models exhibit increasing class separation in log-odds with scale, while base models remain indistinguishable.

Figure 8: Log-odds distributions on BoolQ reveal bimodality and class separation for models above 3B parameters; small models exhibit substantial overlap and low fidelity.

Fidelity with Confidence and Asymmetry

Prediction and attribution fidelity are highly dependent on model confidence. Prompts with high mean log-odds yield much higher fidelity; low-confidence prompts are dominated by noise and lack discriminative signal. Directional asymmetry is prominent: small models cannot reliably surrogate large ones, as evidenced by catastrophic NRMSE values ($t_+$1) when using Qwen-0.5B as surrogate for Qwen-14B.

Figure 5: Fidelity versus confidence quantile for Qwen models demonstrates robust dependence on model confidence.

Figure 6: NRMSE heatmaps reveal asymmetry in surrogacy; small models are unreliable surrogates for larger targets.

Multi-Class and Representational Connections

The hierarchy of surrogate fidelity generalizes to multi-class settings using pairwise log-odds and the RV coefficient. These metrics correspond to task-relevant projections for full representational similarity (CKA), establishing theoretical alignment between prediction fidelity and CKA in the projected space.

Figure 7: Prediction fidelity and CKA are perfectly aligned, but attribution fidelity diverges, indicating functional disagreement despite structural similarity.

Figure 13: RV convergence to CKA as projection dimension increases shows rapid approach for task-relevant subspaces, but only random projections reach full CKA.

Practical and Theoretical Implications

Contradictory claim: Prediction-level agreement does not imply attribution-level agreement; models may agree on answers while disagreeing on causal explanations.

API-compatible attribution metrics (leave-one-out ablations) are more causally relevant than privileged white-box signals like attention or perturbation magnitude—this "access–validity inversion" refutes conventional wisdom that mechanistic insights found in surrogates transfer directly to closed targets.

Strong numerical results: On BoolQ, median prediction fidelity between open and closed models is $t_+$2, but attribution fidelity is only $t_+$3. Attention correlations are much higher ($t_+$4), but do not predict attributions. Asymmetry in surrogacy direction is pronounced below 3B parameters; for larger models, surrogacy becomes approximately symmetric and reliable.

Practical applications include screening surrogates based on prediction and attribution fidelity (computable from log-probabilities) before making claims about closed targets. The reliance on ablation-based saliency is justified for API-accessible models, but the authors recommend validating surrogate claims at the level of analysis intended.

Theoretical implications include the need for careful consideration of task-relevant directions, rather than relying on full representational similarity, and the importance of prompt-level interventions for behavioral alignment.

Future Directions

Extending the framework to open-ended text generation and richer interventions (circuit patching, SAE, meaning-preserving perturbations) is essential for broader applicability. Directional surrogacy, voice-transplant susceptibility, and the systematic quantification of prompt-based behavioral transfer promise enhanced diagnostics for surrogate selection. Low attribution-level fidelity observed in SAE features in recent work suggests significant challenges for feature-level generalization.

Conclusion

The surrogate fidelity framework presented rigorously establishes that neither prediction-level nor representation-level agreement guarantees causal agreement in attributions. Practitioners and theorists must validate surrogate model claims at the level of their intended analysis, particularly when making causal or mechanistic assertions about closed frontier models. The evaluation protocol and codebase released with this work provide a foundational basis for future mechanistic interpretability efforts in restricted-access settings.