Circuit Shift Score for Model Monitoring

• Circuit Shift Score (CSS) is a label-free proxy metric that quantifies the rewiring of a model's internal circuits under distribution shift.
• It employs continuous-relaxation circuit discovery and compares in-distribution and shifted circuits using vector or graph-based representations.
• Empirical results show CSS outperforms traditional proxies in predicting performance degradation, offering improved reliability in post-deployment monitoring.

Circuit Shift Score (CSS) is a label-free proxy metric for post-deployment monitoring under distribution shift. It was introduced as part of a circuit-based perspective on generalization in Vision Transformers (ViTs), where a model’s internal “circuits”—the causal pathways by which intermediate representations influence each other—are extracted on in-distribution (ID) data and compared with the corresponding circuits induced by shifted, unlabeled test data. In this formulation, CSS quantifies internal rewiring rather than relying only on model outputs such as confidence or entropy, and is proposed as a predictor of performance degradation when labels are unavailable (Peng et al., 9 Apr 2026).

1. Problem setting and intended use

The motivating setting is real-world deployment, where a model trained on one distribution will inevitably encounter new, shifted data for which labels are unavailable. Two practical scenarios are distinguished: before deployment, model selection on unlabeled target data; and after deployment, monitoring model performance under distribution shift. CSS is tailored to the second scenario, where the central need is a reliable and label-free proxy metric for generalization performance (Peng et al., 9 Apr 2026).

The work introducing CSS argues that standard proxies such as average confidence or entropy tend to break down in post-deployment scenarios because they can over- or under-estimate model reliability. The proposed alternative is to monitor the model’s inner workings rather than only its outputs. In that framing, CSS is not an accuracy estimator derived from predictions alone, but a mechanistic proxy based on how the model’s internal computation changes as the test distribution departs from the ID distribution (Peng et al., 9 Apr 2026).

The same study introduces Dependency Depth Bias for the before-deployment setting and Circuit Shift Score for the after-deployment setting. This division is important because CSS is explicitly defined around comparison with an ID reference circuit and therefore presupposes an already deployed model with access to unlabeled batches from one or more test distributions.

2. Circuit perspective and mechanistic interpretation

In the CSS framework, a ViT is represented by a computational graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ whose nodes are sub-layers, specifically MLP blocks or attention heads, and whose edges indicate data flow. The “circuit” of the model is the subset or weighting of edges identified as most critical for computation on a given dataset. The key claim is that these causal interactions between internal representations undergo systematic rewiring as distribution shift intensifies (Peng et al., 9 Apr 2026).

The mechanistic intuition is straightforward. A circuit is first discovered on the original ID data, and then the same procedure is applied to shifted, unlabeled test data. If many edges change weight or drop out, the model’s internal reasoning has been perturbed, and its output performance is likely to suffer. CSS operationalizes this observation by measuring the discrepancy between the ID circuit and the test-data circuit. External metrics ignore this graph-structured information; CSS leverages it (Peng et al., 9 Apr 2026).

This conception places CSS at the intersection of mechanistic interpretability and robustness evaluation. The metric does not require target labels or ground-truth outputs, and it is designed to be computed directly from the model and unlabeled inputs. A plausible implication is that CSS treats distribution shift as a change in internal causal organization rather than solely as a change in predictive confidence.

3. Formal definition

Let $c_{D_{\mathrm{ID}}} : \mathcal{E} \to \mathbb{R}$ denote the circuit weight mapping on ID data, where each edge’s weight is defined by

$$c_D(e) = \mathbb{E}_{x\sim D}\!\left[ KL\!\left(M_{\setminus\{e\}}(x)\,\|\,M(x)\right)\right],$$

that is, the expected KL divergence in the model’s output when edge $e$ is ablated by mean-replacement of its activation. Let $c_{D_i} : \mathcal{E} \to \mathbb{R}$ be the analogous mapping on the $i$-th test distribution $\mathcal{D}_i$ (Peng et al., 9 Apr 2026).

A representation function

$\mathcal{R}: c \mapsto S$

maps a circuit to some structured space $S$, and a distance $d$ is then chosen on $c_{D_{\mathrm{ID}}} : \mathcal{E} \to \mathbb{R}$0. The Circuit Shift Score is

$c_{D_{\mathrm{ID}}} : \mathcal{E} \to \mathbb{R}$1

The definition is intentionally modular: the circuit representation and the distance are both design choices (Peng et al., 9 Apr 2026).

Two variants are specified.

Variant	Representation and distance
Vector-based	$c_{D_{\mathrm{ID}}} : \mathcal{E} \to \mathbb{R}$2 returns the full edge-weight vector $c_{D_{\mathrm{ID}}} : \mathcal{E} \to \mathbb{R}$3
Graph-based	$c_{D_{\mathrm{ID}}} : \mathcal{E} \to \mathbb{R}$4 returns a weighted graph

In the vector-based variant, the distance may be cosine dissimilarity,

$c_{D_{\mathrm{ID}}} : \mathcal{E} \to \mathbb{R}$5

or the Spearman-rank correlation coefficient (SRCC) dissimilarity,

$c_{D_{\mathrm{ID}}} : \mathcal{E} \to \mathbb{R}$6

In the graph-based variant, the distance may be a spectral distance (Laplacian), NetLSD, or Jaccard-edge-set distance on the top-$c_{D_{\mathrm{ID}}} : \mathcal{E} \to \mathbb{R}$7 edges (Peng et al., 9 Apr 2026).

4. Circuit discovery and computation

CSS depends on a prior circuit discovery stage. The method adopted is a continuous-relaxation approach to circuit discovery, in which each edge $c_{D_{\mathrm{ID}}} : \mathcal{E} \to \mathbb{R}$8 is assigned the real value

$c_{D_{\mathrm{ID}}} : \mathcal{E} \to \mathbb{R}$9

Here, $$c_D(e) = \mathbb{E}_{x\sim D}\!\left[ KL\!\left(M_{\setminus\{e\}}(x)\,\|\,M(x)\right)\right],$$0 denotes the model after ablating edge $$c_D(e) = \mathbb{E}_{x\sim D}\!\left[ KL\!\left(M_{\setminus\{e\}}(x)\,\|\,M(x)\right)\right],$$1 by replacing its activation with its dataset mean. This definition makes the circuit a weighted object, so CSS can compare either the full weight vector or derived graph structure across distributions (Peng et al., 9 Apr 2026).

For efficient computation at ViT scale, several methods were benchmarked on vision tasks, including EAP, EAP-IG, and Causal Tracing. The reported conclusion is that EAP-IG (Edge Attribution Patching with Integrated Gradients) offers the best trade-off between faithfulness and computational cost. Concretely, EAP-IG makes one forward pass to record activations, then integrates gradients over five steps, yielding a per-edge importance estimate. Repeating this for both ID and each test domain yields the two circuits to compare (Peng et al., 9 Apr 2026).

The computation is governed by several explicit assumptions and hyperparameters. The method assumes access to unlabeled batches from each test distribution. Ablation is performed via mean-replacement over the original ID dataset. For EAP-IG, the number of integration steps is fixed at $$c_D(e) = \mathbb{E}_{x\sim D}\!\left[ KL\!\left(M_{\setminus\{e\}}(x)\,\|\,M(x)\right)\right],$$2. For graph-based distances, the circuit is pruned to the top-100 edges by weight, following Hanna et al. 2024. No target labels or ground-truth outputs are used in CSS computation (Peng et al., 9 Apr 2026).

5. Empirical validation

CSS was evaluated on four post-deployment benchmarks: PACS for style shifts, FMoW for temporal and geographic shifts, Camelyon17 for institutional shift, and ImageNet-C/v2/sketch for various corruptions. For each domain $$c_D(e) = \mathbb{E}_{x\sim D}\!\left[ KL\!\left(M_{\setminus\{e\}}(x)\,\|\,M(x)\right)\right],$$3, the study computed the $$c_D(e) = \mathbb{E}_{x\sim D}\!\left[ KL\!\left(M_{\setminus\{e\}}(x)\,\|\,M(x)\right)\right],$$4 variant, that is, vector-based Spearman-rank dissimilarity, and correlated it with true OOD model performance measured by accuracy or $$c_D(e) = \mathbb{E}_{x\sim D}\!\left[ KL\!\left(M_{\setminus\{e\}}(x)\,\|\,M(x)\right)\right],$$5 (Peng et al., 9 Apr 2026).

The reported aggregate result is an average Pearson $$c_D(e) = \mathbb{E}_{x\sim D}\!\left[ KL\!\left(M_{\setminus\{e\}}(x)\,\|\,M(x)\right)\right],$$6 of $$c_D(e) = \mathbb{E}_{x\sim D}\!\left[ KL\!\left(M_{\setminus\{e\}}(x)\,\|\,M(x)\right)\right],$$7 across datasets for $$c_D(e) = \mathbb{E}_{x\sim D}\!\left[ KL\!\left(M_{\setminus\{e\}}(x)\,\|\,M(x)\right)\right],$$8, compared to $$c_D(e) = \mathbb{E}_{x\sim D}\!\left[ KL\!\left(M_{\setminus\{e\}}(x)\,\|\,M(x)\right)\right],$$9 for the best baseline, ATC, corresponding to a relative improvement of $e$0. Dataset-specific values were also reported (Peng et al., 9 Apr 2026).

Benchmark	CSS result	Best baseline
PACS	$e$1	$e$2
FMoW	$e$3	$e$4
Camelyon17	$e$5	$e$6
ImageNet	$e$7	$e$8

Beyond correlation analysis, CSS was also treated as an alarm score. Given a critical performance threshold $e$9 such as $c_{D_i} : \mathcal{E} \to \mathbb{R}$0 accuracy, a CSS threshold $c_{D_i} : \mathcal{E} \to \mathbb{R}$1 was calibrated using synthetic corruptions from CIFAR10-C and stylizations, and the resulting detector was evaluated by the $c_{D_i} : \mathcal{E} \to \mathbb{R}$2 of correctly flagging “silent failures.” In the clinically-relevant range $c_{D_i} : \mathcal{E} \to \mathbb{R}$3, CSS improved alarm-$c_{D_i} : \mathcal{E} \to \mathbb{R}$4 by roughly $c_{D_i} : \mathcal{E} \to \mathbb{R}$5 over confidence-based baselines (Peng et al., 9 Apr 2026).

The same paper also reports that, across various tasks, the two circuit-based metrics introduced there outperform existing proxies by an average of $c_{D_i} : \mathcal{E} \to \mathbb{R}$6 and $c_{D_i} : \mathcal{E} \to \mathbb{R}$7, respectively. Within that pairing, the $c_{D_i} : \mathcal{E} \to \mathbb{R}$8 figure corresponds to CSS in the post-deployment setting.

6. Strengths, limitations, and prospective extensions

CSS is presented as a label-free, mechanistic proxy for performance drift that outperforms purely output-based or feature-based methods. Its formulation is architecture-agnostic so long as one can define a computational graph and ablation, and it adapts seamlessly to new domains without retraining or external supervision (Peng et al., 9 Apr 2026).

Its principal limitation is computational cost. Each EAP-IG pass is reported to take approximately $c_{D_i} : \mathcal{E} \to \mathbb{R}$9 per 32-image batch on an A6000 GPU, roughly $i$0 the cost of a forward pass, which makes sub-second real-time monitoring challenging. Profiling indicates that much of the cost lies in the integrated-gradient steps; replacing EAP-IG with plain EAP, which omits integration, yields a $i$1 speedup with only minor faithfulness loss (Peng et al., 9 Apr 2026).

Several extensions are proposed. These include more efficient circuit estimation, such as zeroth-order gradient approximations or smaller integration budgets; online updating of the ID baseline circuit to handle non-stationary drift; end-to-end training objectives that directly penalize circuit sensitivity to shift; and application to other architectures, including CNNs and diffusion models, via analogous causal-graph definitions (Peng et al., 9 Apr 2026).

Taken together, these properties position CSS as a mechanism for post-deployment monitoring that quantifies internal rewiring. In the terminology of the source paper, it bridges mechanistic interpretability and robust performance evaluation.