Circuit and Sheaf Discovery (CSD)

• Circuit and Sheaf Discovery (CSD) is a framework that extracts and characterizes sparse subgraphs, known as circuits and sheaves, defining functional mechanisms in logic circuits and LLMs.
• It combines sheaf-theoretic methods with computational algorithms to ensure faithfulness, sparsity, and completeness in identifying critical network substructures.
• Recent findings reveal multiple low-overlap circuits in LLMs, challenging the notion of a unique minimal mechanism and suggesting a distributed representation.

Circuit and Sheaf Discovery (CSD) concerns the extraction, characterization, and interpretation of sparse subgraphs—circuits and sheaves—that implement functional mechanisms in both logic circuits and LLMs. Originating in mathematical formalizations of asynchronous circuit analysis and extended to computational neuroscience and machine learning, CSD provides algorithms and theoretical frameworks to identify substructures that are sufficient or essential for task performance, with particular attention to notions of faithfulness, sparsity, and completeness. Recent advances challenge classical assumptions of functional uniqueness, emphasizing the inherent non-canonical, distributed, and redundant nature of such mechanisms.

1. Formal Definitions and Historical Foundations

In classical circuit theory, a "circuit" is a subgraph of a directed graph (representing gates and wires) whose logical function is isolated and analyzed. The sheaf-theoretic approach introduces a sheaf $S$ over the circuit graph, assigning algebraic data (stalks and restriction maps) to open sets that locally encode logic states, gate behavior, and wire values. The switching sheaf formalism encodes richer information than truth tables yet less than full event-level simulation, capturing intermediate properties such as races and hazards via cohomological invariants (Robinson, 2010).

In the context of LLMs, the transformer architecture is modeled as a directed acyclic computation graph over the residual stream. Each MLP block and attention head corresponds to an edge injecting a vector update. A "circuit" is a sparse subgraph of these edges whose ablation leaves the model's behavior on a specified task essentially unchanged. A "sheaf" strengthens this by requiring that the pruned model, executed in isolation (all other edges zeroed), remains highly faithful to the task (Chen et al., 12 May 2026).

2. Sheaf Formalism and Cohomological Obstructions in Circuits

Given a logic circuit as a directed graph $X$, the edge-star cover $\mathcal{U} = \{U_v\} \cup \{V_e\}$ consists of neighborhoods of gates ($U_v$) and wire interiors ($V_e$) forming a "good cover." The switching sheaf $S$ is constructed using one-hot encoding over $\mathbb{F}_2$, with stalks $S(U_v) = (\mathbb{F}_2^2)^{\otimes m}$ for $m$-input gates and $S(V_e) = \mathbb{F}_2^2$ for wires. Restriction maps are projections or one-hot–lifted gate maps, conforming to the sheaf axioms by compatibility on overlaps.

The Čech cohomology groups $X$0 are computable via sparse Gaussian elimination or incremental Mayer–Vietoris sequences. $X$1 detects consistent global (quiescent) logic states, while $X$2 corresponds to obstructions, such as timing hazards, races, or feedback loops. The dimension of $X$3 increases in the presence of feedback, and a nonzero $X$4 class precisely signals an inability to globally extend local logic assignments due to topological or timing constraints (Robinson, 2010).

3. CSD Methodologies in LLMs

CSD in LLMs is instantiated algorithmically through methods such as ACDC, EAP, Edge Pruning, and DiscoGP. These techniques search for sparse edge-sets (circuits) that satisfy:

• Faithfulness: Retention of high task accuracy under ablation.
• Sparsity: Inclusion of as few edges as possible.
• Completeness (for sheaves): Robust performance when all other edges are ablated (model run in isolation).

Ablation protocols include zero ablation and interchange interventions. Objective functions are composite, typically weighted sums of loss terms measuring fidelity, sparsity, and completeness. Discovery is an optimization, with steps involving iterative pruning or repulsion to explore the space of solutions (Chen et al., 12 May 2026).

4. Non-Uniqueness and the Distributive Dense Circuit Hypothesis

Empirical analyses falsify the Functional Anisotropy Hypothesis, which postulates that each LLM function is localized to a unique or near-unique mechanism. Repeated application of CSD methods—augmented with Overlap-Aware Sheaf Repulsion (OASR)—consistently uncovers multiple, structurally distinct circuits or sheaves supporting the same task. Intersection-over-union (IoU) metrics reveal random or near-random overlap even among highly faithful and sparse solutions (e.g., 4–11% IoU for indirect object identification in GPT-2, with each solution maintaining 100% accuracy and low edge density).

As discovery is repeated for $X$5 runs, the union of discovered edges grows nearly linearly in $X$6, while intersection rapidly decays, sometimes to only a handful of edges (IoU $X$7). This finding generalizes across all tested benchmarks and discovery methods. Discovery of a three-edge ultra-sparse sheaf, none of whose edges is indispensable for task performance, further undermines the search for canonical subgraphs (Chen et al., 12 May 2026).

The theoretical explanation posits that high-dimensional superposition in residual space, combined with local linearity and redundancy, naturally yields many low-overlap circuits that produce $X$8-close logit signatures. The resultant "Distributive Dense Circuit Hypothesis" asserts that for any nontrivial LLM task, multiple faithful, sparse, and structurally distinct circuits or sheaves exist, with no single canonical minimal mechanism.

5. Algorithmic Workflow and Computational Complexity

In circuit analysis (Robinson, 2010), the stepwise CSD workflow is:

1. Construct the directed graph $X$9 of the circuit.
2. Define the edge-star cover $\mathcal{U} = \{U_v\} \cup \{V_e\}$0.
3. Assign switching sheaf stalks and restriction maps.
4. Assemble the sparse Čech coboundary matrix $\mathcal{U} = \{U_v\} \cup \{V_e\}$1.
5. Compute kernel (global sections) and image ranks to obtain $\mathcal{U} = \{U_v\} \cup \{V_e\}$2 and $\mathcal{U} = \{U_v\} \cup \{V_e\}$3 via Gaussian elimination.
6. Use Mayer–Vietoris sequences for efficient incremental updates when adding loops.
7. Interpret the cohomology: nontrivial $\mathcal{U} = \{U_v\} \cup \{V_e\}$4 signals timing obstructions.

Complexity is polynomial in circuit size and exponential only in maximum fan-in. Sparse methods and incremental assembly mitigate computational cost.

For LLMs (Chen et al., 12 May 2026), CSD algorithms optimize loss functions combining fidelity, sparsity, and overlap repulsion. Overlap-aware objectives employ penalties:

$\mathcal{U} = \{U_v\} \cup \{V_e\}$5

where $\mathcal{U} = \{U_v\} \cup \{V_e\}$6 is the base discovery loss (fidelity, sparsity, completeness) and $\mathcal{U} = \{U_v\} \cup \{V_e\}$7 quantifies shared edge support with prior solutions. IoU is used to report structural diversity, and multiple discovery runs quantify the distributional structure of mechanism space.

6. Practical Implications and Interpretative Shifts

The existence of many functionally equivalent, low-overlap circuits and sheaves necessitates reconsideration of mechanistic interpretation in CSD. No single circuit can be identified as "the" explanation for a capability; rather, families of solutions must be characterized statistically. Even ultra