Papers
Topics
Authors
Recent
Search
2000 character limit reached

All Circuits Lead to Rome: Rethinking Functional Anisotropy in Circuit and Sheaf Discovery for LLMs

Published 12 May 2026 in cs.CL | (2605.12671v1)

Abstract: In this paper, we present empirical and theoretical evidence against a central but largely implicit assumption in circuit and sheaf discovery (CSD), which we term the Functional Anisotropy Hypothesis: the idea that functions in LLMs are localised to a unique or near-unique internal mechanism. We show that a single LLM task can instead be supported by multiple, structurally distinct circuits or sheaves that are simultaneously faithful, sparse, and complete. To systematically uncover such competing mechanisms, we introduce Overlap-Aware Sheaf Repulsion, a method that augments the CSD objective with an explicit penalty on structural overlap across multiple discovery runs, enabling the discovery of circuits or sheaves with strong task performance but minimal shared structure across a plethora of common CSD benchmarks. We find that this phenomenon becomes increasingly pronounced as the number of discovered sheaves grows and persists robustly across major CSD methods. We further identify an ultra-sparse three-edge sheaf and show that none of its edges is individually indispensable, undermining even weakened notions of canonical or essential components. To explain these findings, we propose a Distributive Dense Circuit Hypothesis and provide a theoretical analysis demonstrating that non-unique, low-overlap circuit explanations arise naturally from high-dimensional superposition under mild assumptions. Together, our results suggest that mechanistic explanations in LLMs are inherently non-canonical and call for a rethinking of how CSD results should be interpreted and evaluated.

Summary

  • The paper challenges the Functional Anisotropy Hypothesis by showing that LLMs utilize multiple distinct circuits instead of a single, unique mechanism.
  • It introduces Overlap-Aware Sheaf Repulsion (OASR) to systematically uncover diverse sparse subnetworks that achieve near-optimal task performance.
  • The findings advocate a shift from canonical minimal explanations to a distributed, redundancy-aware framework for mechanistic interpretability.

Reconsidering Functional Anisotropy in Circuit and Sheaf Discovery for LLMs

Overview

This paper systematically interrogates the Functional Anisotropy Hypothesis in mechanistic interpretability, particularly in the context of Circuit and Sheaf Discovery (CSD) for LLMs. The Functional Anisotropy Hypothesis posits that a given model capability is underpinned by a single, unique, or quasi-unique internal circuit or subnetwork. The authors present both theoretical results and extensive empirical evidence contradicting this assumption. They demonstrate that, for a given task and model, a multitude of structurally disjoint circuits (or sheaves) can be found that are individually sparse, complete, and faithfully reproduce the original model behavior. The paper further develops a novel discovery method—Overlap-Aware Sheaf Repulsion (OASR)—to expose this multiplicity formally and efficiently.

Problem Formulation and Existing Paradigm

Traditionally, CSD approaches identify task-relevant subgraphs (circuits or sheaves) within the model’s computation graph, motivated by the goal of uncovering faithful, minimal, and canonical mechanisms responsible for high-level behaviors. Both heuristic (e.g., ACDC) and optimization-based (e.g., DiscoGP, Edge Pruning) techniques assume that such mechanisms are unique or almost unique; experimental protocols and benchmarks (e.g., Tracr) implicitly reinforce this by evaluating the fidelity of recovered circuits against predefined ground-truth subgraphs and by favoring explanations that are maximally sparse and minimally redundant.

Empirical Evaluation: Multiplicity of Functionally Sufficient Mechanisms

The core empirical finding is that, across a battery of circuit and sheaf discovery methods, tasks, and models, the space of task-faithful subgraphs is rich and highly non-unique. Using OASR, the authors identify multiple sheaves for each task that are both sparse and individually deliver near-optimal performance, yet have minimal edge-level overlap (IoU often below 10%). For the classic IOI task in GPT-2, two discovered sheaves with just 4.1% edge overlap each achieve perfect accuracy according to standard CSD criteria. This phenomenon generalizes: as the number of discovery runs increases (with repulsion penalties), the intersection of all high-quality discovered sheaves approaches zero, while the union expands, highlighting the combinatorial diversity of valid mechanisms.

Moreover, the effect persists under different classes of CSD methods (e.g., ACDC, EAP, EP), against random or systematic exploration, and includes non-trivial tasks such as BLIMP, AGA, ANA, DNA, and code-related benchmarks (Docstring). The instability of identified circuits under minor randomization of traversal order (for ACDC) or innocuous input changes (for EAP) further undermines the notion of a privileged, unique mechanism.

Negation of Canonical or Essential Cores

A possible counter-hypothesis is that, while edge-level explanations may exhibit redundancy, there exists a minimal canonical core—an indispensable subcomponent set that is present in all faithful circuits. The authors rebut this through constructive experiments: the intersection of multiple sparse, high-fidelity sheaves can be reduced to an ultra-sparse, three-edge subgraph for IOI. However, even this intersection fails to be indispensable. Upon removing any one of the three edges, high-performing alternative sheaves can be found, and removing all three does not prevent the recovery of sparse, accurate sheaves for decomposed subtasks (ABBA, BABA variants). Thus, no single edge or component emerges as essential under comprehensive CSD discovery.

Theoretical Analysis: Distributive Dense Circuit Hypothesis

To explain this widespread multiplicity, the authors introduce the Distributive Dense Circuit Hypothesis. They formally prove that, in high-dimensional representation spaces (as characteristic of modern LLMs), computation naturally admits many structurally distinct, low-overlap subgraphs that can yield nearly identical task-level readouts due to combinatorial superposition and linearity. Their theorem, under assumptions of local linearity and sufficient margin stability, establishes the existence of a large family of sparse circuits whose aggregate predictions are within any prescribed tolerance of the original model—despite negligible edge overlap.

This dense circuit regime arises as a direct consequence of the exponential growth in possible computation paths with network depth and width. It is not a rare defect, but a mathematical property of high-dimensional, superposed representation learning.

Implications for Mechanistic Interpretability

These results prompt a critical reassessment of the interpretation, evaluation, and ultimate aims of CSD in LLMs:

  • Non-Canonical Explanations: The identification of a task-faithful circuit or sheaf should not be interpreted as "the mechanism" by which the model implements a capability, but as one of many equally valid computation pathways.
  • Limits of Minimality and Uniqueness: Sparsity and performance do not guarantee that discovered subgraphs reflect necessary or privileged structure. Evaluation protocols must be revised to account for distributed, overlapping, and possibly ephemeral mechanisms.
  • Distributed Computation: Task behavior in LLMs likely emerges from a dense, compositional regime where multiple circuits compete or coexist, rendering reductionist mechanistic explanations insufficient.
  • Redundancy Beyond Backups: The observed multiplicity is not merely a matter of backup components or resuscitated functions upon ablation (as in prior “hydra-effect” studies) but is a persistent, intrinsic feature under standard operation.

Prospective Directions

Practically, this distributed perspective challenges efforts to localize or edit model behavior through mechanistic intervention. It calls for new classes of abstractions that characterize the space of valid mechanisms, their redundancy, and their interactions, shifting emphasis away from canonical minimality towards families of solutions, robustness under intervention, and principled measures of mechanism diversity.

Theoretically, these findings motivate further exploration into the geometric and combinatorial structure of computation in high-dimensional neural systems, as well as new metrics for interpretability that respect the inherent non-uniqueness of internal mechanisms.

Conclusion

The paper provides robust evidence invalidating the Functional Anisotropy Hypothesis, demonstrating both empirically and theoretically that LLM tasks can be faithfully realized by multiple, sparsely overlapping circuits. It fundamentally challenges the canonical-minimalist paradigm in mechanistic interpretability and necessitates a distributed, multiplicity-aware framework for interpreting the internal computations of large-scale neural LLMs. These insights reshape the theoretical and practical landscape for understanding, controlling, and editing LLMs, underscoring the need for fundamentally new approaches in both research and AI development moving forward.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

Overview

This paper looks inside LLMs to see how they actually do a task. Many past studies quietly assumed there’s one special “mechanism” (a small set of connections inside the model) that makes a task work. The authors argue this assumption is wrong. Instead, they show that there are many different “mechanisms” inside the same model that can all solve the same task, even when those mechanisms barely share any parts.

Think of the title “All Circuits Lead to Rome” as the key idea: just like there are many routes to the same city, there are many internal routes an LLM can use to get the right answer.

Key Questions

  • Is there really one unique internal mechanism that an LLM uses for a given skill or task?
  • Can different, mostly non-overlapping parts of the model each do the same job?
  • How can we systematically find these different mechanisms?
  • If multiple mechanisms exist, what does that mean for how we explain and evaluate LLMs?

How They Studied It (Methods, in simple terms)

First, some everyday translations of technical terms:

  • Circuit or sheaf: Imagine the LLM as a huge city of roads (connections) and intersections (components like attention heads and MLPs). A “circuit” or “sheaf” is a small map of roads that’s important for a specific trip (task). A sheaf is stricter: it’s a tiny sub-map that can still get you to the destination even if you close all other roads. A circuit usually needs some help from the rest of the city; a sheaf can work in isolation.
  • Computation graph: This is the full map of the model’s roads and intersections—who sends information to whom.
  • Edge: A “road” between two parts of the model.
  • Overlap (IoU): How much two maps share. If two sub-maps use a lot of the same roads, they have high overlap; if they use very different roads, they have low overlap.
  • Residual stream: A way of seeing a Transformer where each layer adds its own “update” to a shared workspace. Think of many people adding notes to one shared document, one after another.

What they did:

  • They used standard LLM interpretability tools that try to find small sets of edges (roads) that matter for a task. They especially focused on discovering sheaves (small subgraphs that work on their own).
  • Their main new trick is called Overlap-Aware Sheaf Repulsion (OASR). After finding one good sheaf for a task, they add a penalty that discourages re-using the same edges in the next search. This nudges the algorithm to find a different, low-overlap sheaf that still works well. Repeating this gives multiple, distinct sheaves for the same task.
  • They tested this on well-known interpretability tasks, especially the “Indirect Object Identification (IOI)” task, where the model must decide who received an action in a sentence with two names. They also tried other benchmarks to make sure the effect is general.
  • They measured: task accuracy (does the sheaf get the answers right?), sparsity (how small the sheaf is), completeness (does it work in isolation?), and overlap (how similar two sheaves are).
  • They also checked that this phenomenon appears across several popular discovery methods (not just their own), including ACDC, EAP, and Edge Pruning.

Main Findings

  • Multiple different sheaves can solve the same task: They found pairs (and sets of many) sheaves that each achieve high accuracy, yet share very few edges. For example, on IOI, they found two sheaves with perfect accuracy but only about 4% overlap—almost as low as random chance.
  • The more sheaves you find, the less they share: When they discovered 20 high-quality sheaves for the same task, the common intersection (the edges all of them share) became tiny—often close to zero—while each sheaf still worked well.
  • This isn’t just a “backup” effect: Past work sometimes saw “backup” components that only step in if you remove the main one. Here, the different sheaves exist side by side and can work independently during normal operation. They’re not emergency backups; they’re alternative routes the model already has.
  • It shows up across methods: Whether they used DiscoGP (a state-of-the-art sheaf finder), ACDC (a heuristic pruning method), EAP (a gradient-based attribution method), or Edge Pruning, they still saw that multiple low-overlap mechanisms can do the same job.
  • An ultra-small mechanism exists, but it’s not “essential”: They found an extremely tiny sheaf for the IOI task with just three edges that still achieved strong accuracy on its own. That sounds like a “core” everyone must use—but it’s not. None of those three edges is individually necessary: you can remove any one and still find other small sheaves that solve the task. Even when they split IOI into two simpler templates (ABBA and BABA), they could solve them well without any of the three “core” edges. So even this “tiny core” isn’t truly indispensable.
  • Why this happens (theory): They propose the “Distributive Dense Circuit Hypothesis.” In simple terms, LLMs use high-dimensional representations where many patterns are layered on top of each other (like stacking transparent images). In such spaces, there are tons of different small combinations of edges that add up to the same final effect. As models get deeper and wider, the number of valid “routes” explodes. So it’s natural—not surprising—that multiple, different mechanisms can produce the same correct output.

Why This Matters

  • We should stop thinking there’s one “true” mechanism: If many different small subgraphs can all do the same task, then a single discovered circuit shouldn’t be treated as the unique explanation of how the model thinks. It’s one good explanation among many.
  • Rethinking evaluation: Many benchmarks and scoring rules assume there is a single, minimal, canonical circuit to find. The results here suggest those assumptions are too strict. We need evaluation methods that accept multiple valid answers and judge them by faithfulness and usefulness, not uniqueness.
  • A more realistic view of LLMs: LLMs seem to compute in a distributed way—like a city with many parallel routes—rather than funneling everything through one skinny “main road.” This affects how we do safety checks, debugging, and future interpretability tools.
  • Practical takeaway: Circuit and sheaf discovery is still valuable. The found mechanisms are meaningful and causally relevant. But we should interpret them as examples from a family of valid mechanisms, not the one-and-only inner truth.

Big-Picture Impact

This work nudges the field toward a more flexible, realistic picture of how complex AI systems work: multiple overlapping, competing, and partly redundant mechanisms can all get the job done. Understanding and embracing this “many roads to Rome” reality should lead to better interpretability tools, fairer evaluations, and safer, more reliable ways to test and improve LLMs.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge Gaps, Limitations, and Open Questions

Below is a single, actionable list of what remains missing, uncertain, or unexplored in the paper, aimed to guide future research.

  • External validity across models and scales: Test whether non-unique, low-overlap mechanisms persist in larger and more diverse architectures (e.g., Llama-2/3, Mixtral/MoE, encoder–decoder models) and across parameter scales, and quantify how overlap changes with width/depth.
  • Task breadth and realism: Extend beyond IOI, BLIMP, AGA/ANA/DNA variants, and docstrings to complex, multi-step, realistic tasks (e.g., multi-hop QA, program synthesis, translation, long-context reasoning) to assess generality of mechanism multiplicity.
  • Prompt and dataset sensitivity: Systematically quantify how superficial prompt changes (names, phrasing, tokenization, positional placement) alter discovered mechanisms and define robustness benchmarks that enforce task invariances.
  • Accuracy–overlap trade-off: Characterize the Pareto frontier between sheaf fidelity/completeness and structural overlap under OASR; report stability across seeds, datasets, and hyperparameters to understand when repulsion harms performance.
  • In-situ mechanism usage: Develop path-level attribution/measurement tools (e.g., edge-level influence functions, Shapley values, path-integrated gradients) to quantify which discovered sheaves are actually active during unmasked, standard inference.
  • Architectural simplifications: Re-run analyses with full Transformer components (layer norm, biases, residual scaling, RoPE/ALiBi/MQA) rather than abstracted residual graphs to assess whether non-uniqueness is robust to realistic execution details.
  • Edge candidate set design: Investigate how defining the DAG with “edges from all preceding components” inflates the search space; evaluate constrained edge sets (e.g., local residual connections, attention-specific paths) and their impact on non-uniqueness.
  • Statistical baselines for overlap: Provide “overlap expected by chance” baselines and significance tests by comparing to random subgraphs matched on edge density, layer distribution, and node types; quantify when low IoU is statistically meaningful.
  • Theoretical bounds and predictions: Derive quantitative bounds on the number of faithful, low-overlap circuits as functions of width, depth, representation dimension, and decision margin; predict IoU scaling laws and validate empirically.
  • Relation to monosemantic representations: Test whether applying sparse autoencoders (SAEs) or dictionary learning to activations reduces circuit multiplicity; compare CSD before/after SAE transformation to assess alignment with monosemantic features.
  • Training-time causes and control: Study training dynamics that create multiplicity (e.g., redundancy, superposition, implicit ensembling); evaluate whether regularizers (orthogonality, sparsity, diversity suppression) or curricula can enforce functional anisotropy.
  • Evaluation frameworks post-anisotropy: Design multi-mechanism-aware metrics beyond minimality/IoU (e.g., mechanism diversity, coverage of decision boundary, stability under reparameterizations) and reporting standards for ensembles of explanations.
  • Task decomposition methodology: Formalize procedures to automatically decompose aggregate tasks (like IOI) into subtemplates and assess mechanism indispensability per subtask; build detectors of latent subtask structure to avoid misleading “core” claims.
  • Distribution-level fidelity: Move beyond accuracy to match full output distributions (logit vectors, margins) when evaluating sheaves/circuits; test whether non-uniqueness persists under stricter distributional alignment.
  • Complement analysis rigor: Quantify residual task leakage in complements; determine whether complements harbor alternative mechanisms or merely noise; analyze complement performance under OOD shifts and diverse templates.
  • Cross-method alignment: Systematically compare mechanisms found by ACDC, EAP, EP, DiscoGP, and OASR on the same tasks using shared metrics; build a benchmark and protocols to assess cross-method agreement and explain divergences.
  • Dynamic routing and gating: Investigate whether inputs conditionally route computation to different sheaves (e.g., by attention patterns, MLP key/value directions); develop probes that predict mechanism selection and test conditional activation hypotheses.
  • OASR generalization to other objectives: Adapt overlap-aware penalties to EP-style Lagrangian/L0 formulations and to circuit (not only sheaf) discovery; test whether overlap repulsion can be integrated without harming sparse optimization.
  • Minimal sheaf complexity: Establish lower bounds on the number/type of edges needed for near-perfect fidelity across tasks; map brittleness vs robustness of ultra-sparse sheaves (like the three-edge IOI sheaf) under perturbations and data shifts.
  • Reproducibility and compute: Provide full optimization details (learning rates, τ in Gumbel-Sigmoid, overlap penalty weights, seed handling), report run-to-run variance, mode collapse risks, and compute cost scaling for discovering many sheaves (≥100).
  • Theory–practice bridge: Validate the Distributive Dense Circuit Hypothesis on synthetic models with known ground-truth circuits; test whether OASR recovers the predicted low-overlap families and where theory fails (identify necessary assumptions).
  • Impact on intervention and safety: Examine how mechanism multiplicity affects model editing, red-teaming, and trojan removal (e.g., removal of one mechanism leaves others intact); design robust intervention strategies that target mechanism ensembles.
  • Measurement of indispensability: Go beyond single-edge IOI tests to edge-level Average Causal Effect (ACE)/CATE estimates across diverse inputs and tasks; create principled criteria for declaring components indispensable/non-indispensable.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

The following items outline concrete ways practitioners and researchers can use the paper’s findings and OASR method right now. Each bullet names the application, primary sectors, and the enabling tools/workflows, along with key assumptions or dependencies.

  • Mechanism-ensemble interpretability reports for model audits — sectors: software, AI safety, finance, healthcare
    • What: Replace single-circuit explanations with “mechanism portfolios” that enumerate multiple, low-overlap circuits/sheaves for a given capability, report their IoUs, fidelity, completeness, and complement performance.
    • Tools/workflows: Integrate Overlap-Aware Sheaf Repulsion (OASR) with DiscoGP; run multiple discovery seeds; produce a Circuit Coverage Dashboard that shows union/intersection statistics, per-layer distributions, and complement accuracies.
    • Assumptions/dependencies: Access to the residual-stream computation graph and inference-time masking; compute budget for repeated discovery runs; task definitions with clear success metrics.
  • Robustness and safety auditing beyond single-path ablation — sectors: AI safety, policy, enterprise risk
    • What: During red-teaming and safety evaluations, stress-test models by ablating or patching each discovered sheaf and verifying behavior persists or fails; use mechanism coverage to judge whether a “capability removal” is comprehensive.
    • Tools/workflows: OASR-based multi-run discovery; complement evaluation (run masked complement graphs); failure mode cataloging tied to specific sheaves; “denylist” tests that remove all edges in the union of known sheaves.
    • Assumptions/dependencies: Some capabilities may be realized via yet-undiscovered sheaves; coverage is empirical (no completeness guarantees).
  • Regression testing and drift monitoring for model updates — sectors: software, MLOps
    • What: When upgrading models, compare the pre-/post-update mechanism portfolios for key tasks; flag when the model “switches” to qualitatively different sheaves even if accuracy is unchanged.
    • Tools/workflows: CI/CD hook that runs OASR on golden tasks; alerts on IoU shifts and union/intersection drift; “mechanism diff” summaries per layer/head/MLP.
    • Assumptions/dependencies: Stable task datasets; repeatable discovery configs; budget for periodic multi-run discovery.
  • Task-specific sparse inference prototypes — sectors: software, edge/embedded AI, energy
    • What: For narrow tasks (e.g., docstring inference, templated reasoning like IOI), use discovered sheaves as masked subgraphs to prototype lower-latency inference or power savings without weight changes.
    • Tools/workflows: Inference-time edge masking with zero-ablation; task router that triggers masked execution on recognized templates; benchmarking of latency/energy vs. full model.
    • Assumptions/dependencies: Engineering support for partial-graph execution; performance depends on how sparse and faithful the sheaf is for the task distribution; benefits are task- and model-specific.
  • Interpretability benchmarking updates and reproducibility practices — sectors: academia, research tooling
    • What: Report ensembles of circuits/sheaves instead of single solutions; include overlap metrics, complement performance, and sensitivity to traversal order, prompts, and seeds; move beyond minimality-only metrics.
    • Tools/workflows: Extend InterpBench/Tracr-style evaluations with multi-solution scoring; publish mechanism portfolios with code and masks; require “mechanism stability” sections in papers.
    • Assumptions/dependencies: Community adoption; availability of standardized tasks and evaluation harnesses.
  • Safer model editing and feature steering verification — sectors: software, safety
    • What: Before and after a model edit (e.g., SAE steering, ROME-like edits), verify the change across multiple sheaves for the targeted behavior to ensure edits affect the union of realizations rather than just one path.
    • Tools/workflows: OASR-based pre-/post-edit sweeps; “union targeting” checks; mechanism-level regression tests for editing pipelines.
    • Assumptions/dependencies: Access to editing and interpretability stacks (SAEs, editing APIs); compute for repeated discovery.
  • Practitioner guidance on method sensitivity — sectors: software, academia
    • What: Operationalize insights that ACDC (traversal order) and EAP (prompt names) can yield different circuits; standardize robustness checks across seeds, traversal orders, and task-preserving prompt variants.
    • Tools/workflows: Checklists in internal docs; automated sensitivity harnesses that vary traversal/prompt parameters and report circuit variance.
    • Assumptions/dependencies: Awareness and training for teams; minimal extra compute to run small variations.
  • Documentation and user-facing explanation disclaimers — sectors: consumer apps, enterprise
    • What: Clearly communicate that explanations reflect one of many valid mechanisms and are not canonical; reduce overconfidence in single-path causal stories.
    • Tools/workflows: Update model cards and risk statements; add “multiple-mechanism” caveats to explanations presented to end users or regulators.
    • Assumptions/dependencies: Policy/legal review; alignment with organizational transparency goals.

Long-Term Applications

These items build on the paper’s theoretical and empirical results and likely require further research, scaling, or engineering to be production-ready.

  • Mechanism-aware training objectives — sectors: software, AI safety
    • What: Introduce regularizers to either encourage diversity (multiple low-overlap sheaves for resilience) or discourage diversity (concentrate computation for auditability) depending on deployment needs.
    • Tools/products: Training-time overlap penalties or orthogonality constraints; “Mechanism Consolidation” vs. “Mechanism Diversification” regimes; curriculum that decomposes tasks (e.g., ABBA/BABA) to avoid pseudo-indispensability.
    • Assumptions/dependencies: Differentiable surrogates for overlap at scale; stability with large models; possible trade-offs with accuracy and compute.
  • Comprehensive capability removal/containment — sectors: AI safety, policy, defense
    • What: Enumerate and neutralize the union of sheaves that implement a disallowed capability (e.g., jailbreaks, restricted knowledge use), rather than ablating a single “key head.”
    • Tools/products: “Capability Containment” pipelines that (1) discover multiple sheaves, (2) synthesize denylist masks or fine-tuning interventions, and (3) validate with complement tests and adversarial search.
    • Assumptions/dependencies: Discovery must approach sufficient coverage; adversaries may induce new sheaves post-mitigation; governance frameworks to evaluate sufficiency.
  • Mechanism routers and dynamic execution — sectors: software, cloud/edge, energy
    • What: Runtime systems that select among multiple sheaves based on context to optimize latency/energy or robustness; use disjoint sheaves for redundancy in safety-critical settings.
    • Tools/products: “Sheaf Router” that dispatches masked subgraphs; reliability modes that hedge across disjoint sheaves; task detectors for routing.
    • Assumptions/dependencies: Low-overhead masking; detection accuracy for task contexts; graceful fallback when selected sheaf fails.
  • Distillation and compression via sheaves — sectors: edge AI, robotics, mobile
    • What: Distill a task into ultra-sparse submodels derived from discovered sheaves; create task-specialized micro-models with strong fidelity (e.g., for structured reasoning sub-tasks).
    • Tools/products: Sheaf-to-student distillation pipelines; subgraph-to-graph compilers that convert masks into compact architectures; libraries for “function-preserving” pruning.
    • Assumptions/dependencies: Generalization of sheaf fidelity beyond the discovery dataset; hardware-aware compilation; stability under distribution shift.
  • Certification standards for high-stakes deployments — sectors: healthcare, finance, public sector
    • What: Build auditing standards that require multi-mechanism evidence of faithfulness and coverage (not just single-circuit claims), including complement performance and sensitivity analyses.
    • Tools/products: Mechanism coverage metrics in compliance reports; standardized protocols for multi-run discovery; third-party audit platforms offering OASR-based assessments.
    • Assumptions/dependencies: Regulator buy-in; scalable and reproducible discovery methods; privacy/security controls for model internals.
  • Mechanism-level production monitoring — sectors: software, safety-critical systems
    • What: Telemetry that tracks which sheaves are active in production and alerts on shifts (e.g., new or rare sheaves dominating after a data drift).
    • Tools/products: “Mechanism Sentinel” that samples internal activations and matches them to known sheaves; drift dashboards; SLOs defined over mechanism portfolios.
    • Assumptions/dependencies: Efficient online matching of activations to masks; privacy-preserving collection of internals; robust baselines for normal mechanism distributions.
  • Security and backdoor forensics — sectors: cybersecurity
    • What: Search for distributed or redundant trigger pathways; verify whether backdoor removal covers all equivalent sheaves; analyze attacker resilience enabled by non-uniqueness.
    • Tools/products: Multi-sheaf backdoor scanners; “union-of-sheaves” patching with validation on complements; adversarial mechanism synthesis to probe undiscovered paths.
    • Assumptions/dependencies: Ability to induce/observe backdoor activation; scalable discovery under adversarial settings; safeguards to prevent misuse.
  • Architecture design for controllable multiplicity — sectors: AI research, hardware-software co-design
    • What: Develop architectures that either limit superposition (for auditability) or structure it (for redundancy) via modularization, gating, or constrained residual pathways.
    • Tools/products: Modules with explicit sheaf isolation; hardware kernels that accelerate masked residual computation; design rules for interpretable computation graphs.
    • Assumptions/dependencies: Evidence that constraints meaningfully shape mechanism multiplicity without harming capability; co-design with compilers/runtimes.
  • Mechanism-aware education and tooling ecosystems — sectors: academia, edtech
    • What: Curricula and open-source toolkits that teach discovery of multiple mechanisms, overlap-aware methods, and ensemble evaluation; benchmarks that reward coverage over single “ground truth.”
    • Tools/products: OASR-enabled interpretability libraries; “Mechanism Portfolio Explorer” notebooks; next-gen InterpBench with multi-solution scoring.
    • Assumptions/dependencies: Community adoption; maintenance by research groups; accessible compute for students.
  • Resource scheduling based on sparse realizations — sectors: cloud, data centers
    • What: Schedule workloads to sparse subgraphs for known tasks to save energy/costs; consolidate GPU time across tenants using task-sheaf catalogs.
    • Tools/products: Task-to-sheaf registries; schedulers that map requests to masked inference profiles; cost models for sheaf-level execution.
    • Assumptions/dependencies: Predictability of task mix; operational support for heterogeneous masked runs; robust performance guarantees.

Notes on cross-cutting assumptions and limitations:

  • Theoretical claims rely on high-dimensional superposition and local linearity assumptions with margin protections; empirical generality beyond tested models/tasks needs further validation.
  • Discovery quality depends on data curation and fidelity/completeness metrics; poor task specification can yield misleading sheaves.
  • OASR currently augments DiscoGP-style workflows; extending overlap penalties to other paradigms (e.g., EP with hard constraints) may need new relaxations.
  • Enumerating “all” mechanisms is intractable; applications should treat portfolios as coverage-improving, not exhaustive.
View Paper Prompt  View All Prompts 

Glossary

  • ACDC: A heuristic, intervention-based method for circuit discovery that prunes edges based on causal impact on task performance. "ACDC (Conmy et al., 2023) adopts a heuristic, intervention-based approach"
  • Activation interchange: A technique that patches activations from a corrupted input into a model to assess causal roles of edges or components. "activation interchange from corrupted inputs"
  • Attention head: A component of a Transformer’s attention mechanism that computes attention for a subset of the total attention computation. "where ablating a crucial attention head or layer prompts an alternative component"
  • Backup Name-Mover Heads: Attention heads identified as backups that activate to maintain performance when primary heads are ablated. "Backup Name-Mover Heads in GPT-2"
  • Circuit: A causally relevant subgraph of a model’s computation graph that contributes to performing a task. "Circuits capture causally relevant subgraphs within the full model"
  • Circuit and Sheaf Discovery (CSD): The area focused on identifying task-relevant subgraphs (circuits or sheaves) inside LLMs. "circuit and sheaf discovery (CSD; Wang et al., 2022a; Conmy et al., 2023; Syed et al., 2024; Yu et al., 2025, inter alia) have emerged as promising directions"
  • Complement accuracy: Performance of the model when only the edges not selected by a discovered mechanism are retained. "When reporting complement accuracy, we evaluate the com- plementary masked graph obtained by retaining the edges not selected by the discovered mechanism."
  • Computation graph: A directed acyclic graph (DAG) representing components (nodes) and information flow (edges) within the model. "Thus, in the compu- tation graph G = (V, E), V consists of model components"
  • DiscoGP: An optimisation-based method that jointly selects edges to discover standalone, task-sufficient sheaves. "DiscoGP formulates sheaf discovery as a joint optimisation problem"
  • Distributive Dense Circuit Hypothesis: The theoretical claim that multiple distinct, low-overlap mechanisms can faithfully realize the same task due to high-dimensional superposition. "we propose a Distribu- tive Dense Circuit Hypothesis"
  • EAP: A gradient-attribution-based approach that scores edges and selects a subgraph via thresholding. "EAP (Syed et al., 2024) replaces iterative ablations with a gradient-attribution-based formulation"
  • Edge density: The fraction of selected edges among all candidate edges in a discovered mechanism. "We use edge density to denote the fraction of selected edges among all candidate edges"
  • Edge Pruning (EP): A gradient-based optimisation framework that learns binary masks over edges under patched execution to preserve task fidelity. "Edge Pruning (EP; Bhaskar et al., 2024) represents a paradigm shift in circuit discovery by reframing it as a gradient-based optimisation problem."
  • Functional Anisotropy Hypothesis: The assumption that each model function is localized to a unique or nearly unique internal mechanism. "we term the Functional Anisotropy Hypoth- esis: the idea that functions in LLMs are localised to a unique or near- unique internal mechanism."
  • Gumbel-Sigmoid: A continuous relaxation used to enable gradient-based learning over discrete edge selections. "a Gumbel-Sigmoid relaxation with a straight- through estimator"
  • Hydra effect: A phenomenon where alternative components take over functionality after ablation of critical parts. "the hydra effect, where ablating a crucial attention head or layer prompts an alternative component"
  • Indirect Object Identification (IOI): A benchmark task for mechanistic interpretability used to probe model reasoning about coreference-like structures. "indirect ob- ject identification (IOI) task"
  • Intersection-over-union (IoU): A measure of structural overlap between two discovered mechanisms based on edge sets. "IoU(A, B) = 4.1%"
  • Interchange ablation: An ablation regime where activations are patched/interchanged instead of zeroed to test causal roles. "under interchange and zero ablation."
  • KL divergence: A divergence measure used as an objective to align masked model outputs with the full model distribution. "changes in KL diver- gence."
  • Overlap-Aware Sheaf Repulsion (OASR): A procedure that penalizes overlap with previously discovered sheaves to surface alternative, low-overlap mechanisms. "we introduce Overlap-Aware Sheaf Re- pulsion, a method that augments the CSD objec- tive with an explicit penalty on structural over- lap across multiple discovery runs"
  • Residual stream: The central hidden state in Transformer residual architectures that accumulates layerwise updates. "introduced the concept of the residual stream."
  • Reverse topological order: An ordering of nodes used by ACDC to prune edges from later to earlier components in the DAG. "operates in reverse topological order"
  • Sheaf: A subgraph that both causally contributes to the computation and can perform the task in isolation (standalone). "a sheaf, defined as a subgraph that both causally contributes to the computation and can independently sustain task per- formance."
  • Straight-through estimator: A gradient estimator used to backpropagate through discrete sampling decisions. "straight- through estimator"
  • Superposition: The overlap of multiple features/mechanisms in shared high-dimensional representations. "non-unique, low-overlap circuit explanations arise naturally from high-dimensional superposition"
  • Top-k selection: Selecting the k highest-scoring edges (e.g., in EAP) as a thresholding scheme. "as a function of the top-k selection threshold."
  • Tracr: A framework for constructing ground-truth circuits in synthetic tasks to evaluate discovery methods. "for example using Tracr (Lindner et al., 2023)"
  • Zero ablation: An ablation setting where pruned edges contribute zero activation to downstream components. "inactive edges are zero-ablated"
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 437 likes about this paper.

Sign Up for Free