Overlap-Aware Sheaf Repulsion in LLMs

• The paper introduces OASR which uncovers multiple distinct sheaves that maintain high task fidelity while challenging the Functional Anisotropy Hypothesis.
• The method uses gradient-based optimization with an explicit overlap penalty to extract sparse, low-overlap circuits underlying LLM functions.
• Empirical evaluations across benchmarks demonstrate OASR's ability to yield sheaves with overlaps as low as 4–11% while preserving near-perfect accuracy.

Overlap-Aware Sheaf Repulsion (OASR) is a method for systematically uncovering multiple, structurally distinct, yet functionally faithful and sparse mechanistic explanations—termed “sheaves”—for tasks performed by LLMs. OASR augments standard circuit and sheaf discovery (CSD) objectives with an explicit penalty on structural overlap between successively found sheaves, enabling the empirical extraction of numerous low-overlap, high-fidelity circuits. The resulting findings challenge the Functional Anisotropy Hypothesis: the long-standing, often implicit assumption that such tasks are implemented by unique or near-unique internal mechanisms within LLMs (Chen et al., 12 May 2026).

1. Foundations: Definitions and Notations

Within Transformer architectures, let the residual-stream computation graph be denoted as $G=(V,E)$, where $V$ is the set of components—typically attention heads ($h$), MLP blocks ($m$), input, and output nodes—and $E \subseteq V \times V$ are directed edges representing information flow.

• Circuit: A subgraph $C \subseteq E$ functions as a “circuit” for task $T$ if ablating all edges in $E \setminus C$ causes only mild degradation in performance, i.e., the masked model’s fidelity $f_C \approx f_0$.
• Sheaf: A sheaf $S \subseteq E$ is a circuit endowed with standalone faithfulness: when only edges in $V$0 are active, with all others set to zero, task performance remains high.
• Edge Scores/Masks: Each edge $V$1 has a continuous score $V$2 (obtained via a sigmoid of a learnable logit $V$3) and a binary mask $V$4, sampled via the straight-through Gumbel-Sigmoid estimator.
• Structural Overlap: For two edge sets $V$5, overlap is quantified as $V$6. Alternative measures include $V$7 and continuous-score-based $V$8.

2. Overlap-Aware Objective Formulation

The standard objective for sheaf discovery, inspired by DiscoGP, seeks to minimize:

$V$9

where

• $h$0 evaluates the cross-entropy loss of the masked model on task labels.
• $h$1 enforces edge-level sparsity.
• $h$2 penalizes completeness, ensuring the complement does not solve the task.

OASR introduces an overlap penalty:

$h$3

For the $h$4th discovery run, the new objective becomes

$h$5

where $h$6 governs the strength of repulsion from prior sheaves $h$7.

3. Algorithmic Protocol

The OASR methodology operates by iteratively discovering distinct sheaves through gradient-based optimization:

• Initialization: For each run $h$8, initialize $h$9 and Gumbel sampling seeds randomly.
• Optimization: Minimize $m$0 using gradient descent (e.g., Adam), computing hard masks $m$1 via the straight-through Gumbel-Sigmoid.
• Extraction: At convergence, extract $m$2 as the discovered sheaf.
• Coordination: After run $m$3, append $m$4 to the set of prior sheaves for overlap penalization in subsequent runs.

In practice, overlap penalties are efficiently computed using stored binary masks $m$5; penalties take the form $m$6 or via score–mask cross-terms $m$7.

4. Theoretical Insights: Distributive Dense Circuit Hypothesis

Under assumed local linearity and high-dimensional superposition within the model, the class logit $m$8 can be locally linearized:

$m$9

where $E \subseteq V \times V$0 is the contribution of edge $E \subseteq V \times V$1.

Due to the exponentially large number of edge subsets and only polynomially many distinguishable readout vectors (given a margin $E \subseteq V \times V$2), a subset-sum pigeonhole argument guarantees the existence of many structurally distinct edge sets $E \subseteq V \times V$3 such that

$E \subseteq V \times V$4

This result demonstrates that multiple low-overlap sheaves can achieve equivalently high fidelity for the same model and task, providing theoretical support for the empirical findings imposed by OASR.

5. Empirical Characterization

OASR has been extensively evaluated on a range of LLM circuit discovery benchmarks, including IOI (Indirect Object Identification), BLiMP (English grammaticality), AGA/ANA (agreement tasks), DNA variants (Dependency-Number Agreement), and Docstring (code completion).

Benchmark Results

Task	Edge Densities (%)	Completeness (%)	$E \subseteq V \times V$5 (%)
IOI	3.5 vs 4.0	45 vs 46	4–11

Findings include:

• For each task, two sheaves $E \subseteq V \times V$6, $E \subseteq V \times V$7 can be discovered that both attain high accuracy (e.g., IOI: 100% vs 100%), with low edge overlap ($E \subseteq V \times V$8 in the 4–11% range).
• Increasing the number of OASR-driven runs ($E \subseteq V \times V$9) further shrinks the mutual intersection $C \subseteq E$0 to negligible values (tens of edges or less) while maintaining accuracy and sparsity.

Ultra-Sparse Sheaf Discovery

By intersecting 20 OASR runs on IOI, an 11-edge sheaf core retains $C \subseteq E$1 accuracy; exhaustive search within this core yields a 3-edge sheaf $C \subseteq E$2 achieving $C \subseteq E$3 IOI accuracy on its own, with the complement performing at chance.

Indispensability analysis further demonstrates that ablating any single edge $C \subseteq E$4 in $C \subseteq E$5 does not significantly impact total task accuracy (leaving IOI accuracy $C \subseteq E$6 99.9%), and alternative discovery runs excluding $C \subseteq E$7 still yield sparse, high-accuracy sheaves. This strongly undermines arguments for edge-level canonicity or uniqueness.

6. Implications and Reevaluation of Mechanistic Interpretability

The aggregate theoretical and experimental evidence provided by OASR demonstrates a “plethora” of alternative, low-overlap, causally effective sheaves underpinning LLM task performance. These results invalidate any strong or weak form of the Functional Anisotropy Hypothesis—the view that LLM functions are realized through unique or essential internal circuits. As different sheaves can be found that are both sparse and faithful, with little or no overlap and no indispensable components, mechanistic explanations in LLMs are inherently non-canonical.

A plausible implication is the need to fundamentally reconsider how circuit and sheaf discovery results are interpreted: rather than treating discovered explanations as unique or essential, future methodologies must acknowledge the inherent multiplicity and non-uniqueness illuminated by OASR (Chen et al., 12 May 2026).