CircuitLasso: Sparse Circuit Learning

• CircuitLasso is a circuit-learning framework that leverages lasso regularization to recover sparse dependency graphs in LLMs for mechanistic interpretability.
• It replaces intervention-heavy causal estimation with sparse linear regression over internal model components, emphasizing semantically clear SAE features.
• Empirical evaluations show that CircuitLasso achieves competitive structural accuracy while offering significant runtime improvements over intervention-based methods.

Searching arXiv for the primary paper and closely related circuit-learning baselines to ground the article in current literature. arXiv search query: "CircuitLasso sparse circuit learning interpret LLMs" CircuitLasso is a circuit-learning framework for mechanistic interpretability that replaces intervention-heavy causal estimation with sparse linear regression over internal model components, especially sparse autoencoder (SAE) features. In its principal formulation, a circuit is a sparse directed acyclic subgraph of an LLM’s computation graph whose nodes are neurons or SAE features and whose edges are nonzero regression coefficients oriented by transformer computation order. The method is designed for the regime in which raw neurons are polysemantic, SAE features are more interpretable but very high-dimensional, and existing intervention-based circuit-learning methods become computationally prohibitive. CircuitLasso therefore targets scalable recovery of population-level dependency skeletons that are sparse enough to inspect, faithful enough to ablate, and cheap enough to learn on large feature spaces (Yin et al., 15 Jun 2026).

1. Conceptual basis and problem setting

Circuit learning seeks a small subgraph of the model’s computation graph that explains a behavior in terms of internal computation. In this setting, sparsity is not merely a regularization device: a sparse circuit uses only a small fraction of components and edges, which is taken to improve interpretability and to reduce the inclusion of spurious or weakly relevant dependencies. CircuitLasso adopts this premise directly by defining edges through the support of lasso-regularized regressions and by orienting those edges according to known transformer execution order (Yin et al., 15 Jun 2026).

A central motivation is the mismatch between neuron-level access and semantic interpretability. Individual MLP neurons or attention heads are often polysemantic, so a neuron-based circuit can identify a technically relevant component while remaining semantically ambiguous. CircuitLasso therefore emphasizes SAE features. Given an activation vector $x \in \mathbb{R}^d$, an SAE produces a sparse feature vector $z \in \mathbb{R}^D$, with $D \gg d$, through

$$z = \sigma(W_{\text{enc}} x + b_{\text{enc}}), \quad \hat{x} = W_{\text{dec}} z + b_{\text{dec}},$$

trained with

$$\mathcal{L}_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \alpha\,R_{\text{reg}}(z).$$

The resulting features are described as often monosemantic, which makes feature-level circuits substantially easier to inspect than neuron-level ones. The complication is dimensionality: for GPT-2 small, the reported SAE feature space has $D=32{,}768$, so per-feature or per-edge interventions become prohibitively expensive (Yin et al., 15 Jun 2026).

CircuitLasso addresses that bottleneck by using observational data from ordinary forward passes rather than repeated patching. This shifts the objective from estimating exact intervention effects to recovering a sparse dependency skeleton that approximates how concepts and computations propagate through the network. A plausible implication is that the method is best understood as a structural surrogate for circuit discovery rather than as a replacement for targeted causal verification.

2. Formal definition of circuits and regression objectives

CircuitLasso uses a linear SEM-like surrogate over activations. Let $N$ denote the total number of components across selected model locations, $M$ the number of observations, and $X \in \mathbb{R}^{N \times M}$ the matrix of component activations. The core model is

$X = A^\top X + \varepsilon,$

where $z \in \mathbb{R}^D$0 is a weighted adjacency matrix and $z \in \mathbb{R}^D$1 is a residual interpreted as linearization error. The learning problem is

$z \in \mathbb{R}^D$2

with $z \in \mathbb{R}^D$3 the set of DAGs. The support of $z \in \mathbb{R}^D$4 defines the learned circuit: zero coefficients remove edges, nonzero coefficients instantiate edges, and all components that participate in nonzero coefficients become circuit nodes (Yin et al., 15 Jun 2026).

For transformer neurons, acyclicity is enforced by construction rather than by a generic DAG constraint. Activations are collected from $z \in \mathbb{R}^D$5 ordered locations of width $z \in \mathbb{R}^D$6, flattened into $z \in \mathbb{R}^D$7, and the adjacency matrix is partitioned into $z \in \mathbb{R}^D$8 blocks of size $z \in \mathbb{R}^D$9. CircuitLasso then imposes a block upper-triangular constraint,

$D \gg d$0

which encodes the fact that earlier layers cannot depend on later layers and that, within a layer, attention precedes MLP. This substitutes architecture-induced ordering for NOTEARS-style acyclicity constraints (Yin et al., 15 Jun 2026).

For SAE circuits, the method becomes layer-wise. If $D \gg d$1 and $D \gg d$2 are SAE features at locations $D \gg d$3, CircuitLasso fits

$D \gg d$4

A nonzero entry $D \gg d$5 defines an edge from feature $D \gg d$6 at layer $D \gg d$7 to feature $D \gg d$8 at layer $D \gg d$9. To connect features to a behavior target $$z = \sigma(W_{\text{enc}} x + b_{\text{enc}}), \quad \hat{x} = W_{\text{dec}} z + b_{\text{dec}},$$0, it fits

$$z = \sigma(W_{\text{enc}} x + b_{\text{enc}}), \quad \hat{x} = W_{\text{dec}} z + b_{\text{dec}},$$1

using MSE for regression and cross-entropy for classification. In both cases, the circuit is identified with the support pattern of the lasso solution (Yin et al., 15 Jun 2026).

This formulation makes the term “circuit” precise in a regression-theoretic sense. It is not a symbolic program and not an intervention graph; it is a sparse, layer-respecting dependency graph induced by regularized prediction across model components.

3. Data model, extraction pipeline, and optimization procedure

CircuitLasso is applied in two main regimes. For neuron-level structural recovery, it uses InterpBench, comprising 86 semi-synthetic transformers with ground-truth circuits, including 16 main synthetic cases and one real Indirect Object Identification task. Activations are collected from $$z = \sigma(W_{\text{enc}} x + b_{\text{enc}}), \quad \hat{x} = W_{\text{dec}} z + b_{\text{dec}},$$2 locations, averaged over tokens, and stacked into $$z = \sigma(W_{\text{enc}} x + b_{\text{enc}}), \quad \hat{x} = W_{\text{dec}} z + b_{\text{dec}},$$3 after combining clean and corrupted runs. The target in this setting is self-reconstruction under the ordered block mask (Yin et al., 15 Jun 2026).

For feature-level interpretability, the reported CoLA experiments use GPT-2 small with OpenAI SAEs from Gao et al. (2025), with $$z = \sigma(W_{\text{enc}} x + b_{\text{enc}}), \quad \hat{x} = W_{\text{dec}} z + b_{\text{dec}},$$4 ReLU-linear features and $$z = \sigma(W_{\text{enc}} x + b_{\text{enc}}), \quad \hat{x} = W_{\text{dec}} z + b_{\text{dec}},$$5 sparsity, over 8,551 training sentences from CoLA. Features are extracted from the final outputs of each of 12 transformer blocks, encoded tokenwise, and averaged over tokens to form $$z = \sigma(W_{\text{enc}} x + b_{\text{enc}}), \quad \hat{x} = W_{\text{dec}} z + b_{\text{dec}},$$6. Structural circuits use later-layer features as targets; behavior circuits use the binary grammaticality label. In the Bias-in-Bios setting, the same logic is applied to Pythia-70M, Gemma-2-2B, and Gemma-2-9B using pretrained SAEs, with profession label as target and a single chosen layer per model (Yin et al., 15 Jun 2026).

Optimization is performed with FISTA and backtracking on objectives of the form

$$z = \sigma(W_{\text{enc}} x + b_{\text{enc}}), \quad \hat{x} = W_{\text{dec}} z + b_{\text{dec}},$$7

The implementation uses column standardization, warm-start across both features and the $$z = \sigma(W_{\text{enc}} x + b_{\text{enc}}), \quad \hat{x} = W_{\text{dec}} z + b_{\text{dec}},$$8 path, and a convergence threshold at relative objective decrease $$z = \sigma(W_{\text{enc}} x + b_{\text{enc}}), \quad \hat{x} = W_{\text{dec}} z + b_{\text{dec}},$$9. For neurons, the masked problem decomposes into $$\mathcal{L}_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \alpha\,R_{\text{reg}}(z).$$0 multi-output lasso subproblems. For SAE features, layer-pair regressions solve Eq. (4)-type objectives, while feature-to-target regressions solve Eq. (5)-type objectives. Coefficients are typically thresholded at exact zero from the $$\mathcal{L}_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \alpha\,R_{\text{reg}}(z).$$1 penalty, so no extra threshold is required to extract edges (Yin et al., 15 Jun 2026).

The reported complexity reflects the distinction between neuron-level and SAE-level learning. Under the neuron formulation with $$\mathcal{L}_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \alpha\,R_{\text{reg}}(z).$$2 positions, width $$\mathcal{L}_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \alpha\,R_{\text{reg}}(z).$$3, $$\mathcal{L}_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \alpha\,R_{\text{reg}}(z).$$4, and $$\mathcal{L}_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \alpha\,R_{\text{reg}}(z).$$5 samples, total cost across all subproblems is

$$\mathcal{L}_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \alpha\,R_{\text{reg}}(z).$$6

For SAE layer-to-layer regression, the cost is

$$\mathcal{L}_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \alpha\,R_{\text{reg}}(z).$$7

per layer pair, while feature-to-target regression costs

$$\mathcal{L}_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \alpha\,R_{\text{reg}}(z).$$8

The paper further gives conditions under which CircuitLasso is guaranteed faster than EAP-ig, expressed in terms of $$\mathcal{L}_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \alpha\,R_{\text{reg}}(z).$$9, $D=32{,}768$0, $D=32{,}768$1, $D=32{,}768$2, $D=32{,}768$3, and $D=32{,}768$4, with the general conclusion that for realistic long-sequence, many-block settings the observational lasso objective is cheaper than repeated forward/backward interventions (Yin et al., 15 Jun 2026).

4. Benchmark behavior and comparison with intervention-based methods

CircuitLasso is evaluated against EAP and EAP-ig on InterpBench using Structural Hamming Distance (SHD), defined as the number of edge additions or deletions needed to transform the recovered circuit into the ground-truth circuit. The linear version of CircuitLasso is reported to match the structural accuracy of EAP-ig while being materially faster. A nonlinear variant achieves somewhat lower SHD but at substantially higher runtime, and is treated as a case of diminishing returns rather than the preferred operating point (Yin et al., 15 Jun 2026).

Method	Mean SHD	Mean runtime per case
CircuitLasso-linear	3.16	16.3 s
EAP-ig	2.98	49.1 s
EAP	3.61	33.7 s

The difference between CircuitLasso-linear and EAP-ig is described as statistically indistinguishable, while the runtime comparison implies approximately a $D=32{,}768$5 speedup over EAP-ig and approximately a $D=32{,}768$6 speedup over EAP on a single A100. The nonlinear variant achieves SHD $D=32{,}768$7 but is reported as $D=32{,}768$8 slower, and elsewhere as 3.7$D=32{,}768$9 higher runtime, which the authors interpret as an unfavorable efficiency trade-off for structural discovery (Yin et al., 15 Jun 2026).

For SAE circuits on CoLA, CircuitLasso is compared with shift from Sparse Feature Circuits using faithfulness and completeness under node ablation. The reported result is that CircuitLasso matches shift in faithfulness and completeness without requiring edge-level interventions. It also supports edge ablation by zeroing specific edge weights in the learned linear surrogate, which shift does not. The reported edge-ablation behavior indicates that a small subset of edges can govern behavior, an important distinction between node selection alone and explicit dependency modeling (Yin et al., 15 Jun 2026).

These results define the empirical niche of CircuitLasso. It is not presented as more causally faithful than intervention-based methods on a per-edge basis; rather, it is positioned as a scalable method whose structural accuracy is competitive on benchmark data and whose learned circuits are sufficiently faithful to support downstream analysis.

5. Interpretability of SAE circuits and semantic path structure

A major contribution of CircuitLasso is the ability to learn circuits directly over SAE features rather than over raw neurons. Final-layer feature importance can be ranked either by dataset-level magnitude, $N$0, or by prompt-specific importance, $N$1. Features are then labeled semantically through two inspection procedures: a multi-prompt method that aggregates highly activating tokens across prompts, and a single-prompt perturbation method that tests whether the activation tracks a hypothesized concept (Yin et al., 15 Jun 2026).

The reported examples are unusually concrete. Feature $N$2 activates for tokens such as “himself”, “itself”, and “myself” and is labeled “-self reflexive pronouns”. Feature $N$3 activates for “thirst” and “hungry” and is labeled “hunger/thirst”. Feature $N$4 activates for “tired”, “weary”, and “boring” and is labeled “tired/weary/bored”. Feature $N$5 activates for period, “?”, and “!” and is labeled “ending punctuation”. In the single-prompt test, replacing the sentence-final “.” with “yet” or “,” drives the $N$6 activation to zero, while replacing it with “?” or “!” keeps activation high, which supports the punctuation interpretation (Yin et al., 15 Jun 2026).

Using the layer-to-layer adjacency matrices $N$7, CircuitLasso traces tree-shaped circuits backward from important final-layer features. Three structural patterns are highlighted. First, semantic persistence across layers: the “-self” concept appears in multiple consecutive layers, suggesting stable propagation of a semantic feature. Second, merging and decomposition: later features can merge information from multiple parents or retain only a subset of a parent’s semantics. Third, cause-effect versus spurious correlation: some feature paths appear causally coherent from a human perspective, whereas others reflect dataset-level co-occurrence rather than world-causal structure. The paper’s example is a spuriously correlated path between a final-layer “-self” feature and an earlier “hunger/thirst” feature, plausibly reflecting text co-occurrence such as “He told himself he was hungry” rather than a direct semantic mechanism (Yin et al., 15 Jun 2026).

This suggests that CircuitLasso’s explicit edge structure is useful not only for identifying relevant features but also for exposing how concepts persist, merge, or distort across layers. It also suggests a practical role in selecting candidates for model editing or debiasing, since spurious feature-to-feature dependencies can be located explicitly rather than inferred indirectly.

6. Downstream use, limitations, and terminological context

CircuitLasso is also validated on Bias-in-Bios as a domain-generalization and debiasing workflow. At a selected layer $N$8, the method fits sparse logistic regression

$N$9

ranks features by $M$0, manually identifies top features that encode gender rather than profession, and ablates those features either before using the existing classifier or before retraining a classifier on the ablated representation. The reported outcome is that CircuitLasso and CircuitLasso-retrain are comparable or slightly better than shift and shift-retrain on balanced-test profession accuracy, gender leakage, and worst-group accuracy, with gains more pronounced at larger model scales such as Gemma-2-9B. The text also states that CircuitLasso uses fewer features and less runtime than shift-based baselines, though the exact CircuitLasso counts are not printed in the excerpted runtime table (Yin et al., 15 Jun 2026).

The method’s assumptions are explicitly narrow. The SEM formulation is only SEM-like: activations are deterministic, $M$1 represents linearization error rather than exogenous noise, causal sufficiency does not hold because the residual stream and SAE reconstruction errors introduce unmodeled variables, and the true transformer dependencies are nonlinear. The authors therefore do not claim identifiability of the true causal graph. Instead, the learned structure is a dependency skeleton in a projected linear sense. This is the central interpretive constraint on all recovered edges and paths (Yin et al., 15 Jun 2026).

A second limitation is the dependence on $M$2. The CoLA ablation shows the typical sparsity trade-off: $M$3 yields train accuracy $M$4 and test accuracy $M$5, indicating overfit with no sparsity; $M$6 gives the best reported test accuracy $M$7 with a reasonably sparse circuit; larger $M$8 values degrade both train and test accuracy. Orientation is likewise imposed rather than discovered: earlier-to-later direction follows transformer layer order, so some paths can encode anti-causal patterns relative to ordinary human temporal semantics (Yin et al., 15 Jun 2026).

There is also a separate, informal use of the label “CircuitLasso” in work on learning sparse resistor networks from voltage-current data. In that setting, the phrase refers to learning Laplacian-structured graphical lasso models whose precision matrices correspond to resistor-network conductances, with solver-free spectral graph learning used for scalability. That usage concerns physical network recovery and effective-resistance preservation rather than mechanistic interpretability in LLMs, but it shares the same broad idea of sparse structure recovery under circuit constraints (Zhang et al., 2023).

Taken together, these features place CircuitLasso in a specific methodological niche: it is a sparse-regression framework for learning scalable, population-level circuits over neurons or SAE features; it is empirically competitive with intervention-based baselines on structural recovery; and it is explicitly not an exact causal graph estimator. Its value lies in turning high-dimensional observational activations into sparse, inspectable dependency structure that can then support interpretation, targeted intervention, and downstream model modification (Yin et al., 15 Jun 2026).