Distributive Dense Circuit Hypothesis

• Distributive Dense Circuit Hypothesis is a concept positing that LLM computations are supported by many structurally distinct, sparse subnetworks that independently maintain high performance.
• Empirical demonstrations, such as IOI sheaves and ultra-sparse circuits, reveal that these circuits can achieve near-identical task outcomes while sharing minimal structural overlap.
• The hypothesis shifts mechanistic interpretability towards distribution-aware methods, urging revised evaluation metrics that account for multiple redundant computational pathways.

The Distributive Dense Circuit Hypothesis is a foundational concept in mechanistic interpretability for LLMs, articulating that the same computational function can be realized by multiple, structurally distinct, sparse subnetworks—referred to as circuits or sheaves—each faithfully preserving the model’s performance. This hypothesis directly challenges the previously dominant assumption, termed the Functional Anisotropy Hypothesis, which posited that LLM behaviors are localized to essentially unique internal mechanisms. Instead, the Distributive Dense Circuit Hypothesis implies that for any fixed LLM task and evaluation criterion, there exists a combinatorially large space of circuits, any of which can independently support the computation with minimal overlap among their structures (Chen et al., 12 May 2026).

1. Formal Definitions and Hypothesis Statement

Within a decoder-only Transformer, the computation is modeled as a directed acyclic graph over residual-stream states $x^i$, with edges corresponding to additive contributions from attention heads or MLPs. The Distributive Dense Circuit Hypothesis states:

For any LLM task defined by a prompt distribution $\mathcal{D}$ and performance metric $M$, there exist multiple structurally distinct subgraphs $C_1 \neq C_2$ (circuits or sheaves) such that:

• Sparsity: Each circuit comprises only a small fraction of the prunable residual-stream edges.
• $\epsilon$-Faithfulness: Pruning the model to that circuit preserves task performance up to a small tolerance $\epsilon$,

$$M(\mathcal{F}_{C};\, \mathcal{D}) \geq M(\mathcal{F};\, \mathcal{D}) - \epsilon.$$

• Low Overlap: The intersection over union (IoU) ratio is

$$\mathrm{IoU}(C_1, C_2) = \frac{|E(C_1) \cap E(C_2)|}{|E(C_1) \cup E(C_2)|} \leq T$$

for some threshold $T \ll 1$.

Thus, computation is inherently distributed across many alternative, partially redundant subnetworks with little shared structure.

2. Theoretical Foundations and Assumptions

The main existence theorem is established under three general assumptions:

• A.1 Residual-Additive Edge Model: The model’s output logits $g(x^L)$ can be written as the sum over residual-stream edge contributions plus a constant baseline. Pruning edges sets their contributions to zero.
• A.2 Margin Stability: For input distribution $\mathcal{D}$0, a uniform margin $\mathcal{D}$1 exists between the top-1 logit and others. Any pruning-induced logit perturbation is bounded in sup-norm by $\mathcal{D}$2, ensuring classification invariance.
• A.3 Local Linearity: The contribution of each edge to the final output is approximately linear over $\mathcal{D}$3, outside of activation nonlinearity regimes:

$\mathcal{D}$4

The existence theorem asserts that, when the space of possible $\mathcal{D}$5-edge circuits is large, there exist low-overlap pairs $\mathcal{D}$6 both $\mathcal{D}$7-faithful, as $\mathcal{D}$8 (number of $\mathcal{D}$9-edge subsets from $M$0 prunable edges) exceeds the quantized bins in logit space (Chen et al., 12 May 2026).

3. Proof Outline and Mathematical Consequences

The proof leverages a subset-sum reduction and a collision argument:

• Subset-Sum Reduction: Under local linearity, each $M$1-edge circuit’s signature on an eval set is nearly the sum of its individual edge contributions.
• Quantization and Pigeonhole Argument: Quantizing logits to resolution $M$2 gives a polynomial number of bins; since $M$3 grows rapidly, by the pigeonhole principle, two structurally distinct circuits must have similar summed logit signatures, yielding $M$4-faithfulness.
• Margin Condition: With $M$5, the discrete predictions remain invariant.
• Low-Overlap Selection: The combinatorial argument ensures that a colliding pair can be chosen such that their edge overlap is substantially sub-linear ($M$6).

This suggests that redundancy and functional multiplicity are not pathological artifacts, but generic properties in sufficiently overparameterized, superposed high-dimensional models.

4. Empirical Demonstrations and Methodological Variants

Empirical results support the hypothesis:

• Indirect Object Identification (IOI): Using Overlap-Aware Sheaf Repulsion (OASR), two $M$7-sparse IOI sheaves preserving $M$8 accuracy are found to share only $M$9 of edges (IoU = 4.1%). Extending discovery to 20 sheaves yields mutual intersection below $C_1 \neq C_2$0.
• Ultra-Sparse Sheaf Example: A three-edge sheaf achieves $C_1 \neq C_2$1 accuracy on IOI. No edge in this sheaf is individually essential; removing any one allows recovery of a functionally similar sparse alternative.
• Generalization Across Tasks and Methods: Low-overlap faithful circuits arise in BLIMP, agreement/negation, and docstring-completion tasks. Method-specific variation is documented: in ACDC, results depend on head-traversal order; EAP varies with prompt names; optimization-based (Edge Pruning, DiscoGP) methods find different solutions when losses are task-specific (Chen et al., 12 May 2026).

Task/Method	Example Circuit IoU	Notable Behavior
IOI (OASR)	4.1% (2 sheaves)	100% accuracy in both
IOI (20 sheaves)	<0.3%	Multiple distinct, faithful
Ultra-sparse (3-edge)	N/A	No edge is indispensable
BLIMP, docstring	Low IoU observed	Redundancy across methods

5. Implications for Mechanistic Interpretability

The hypothesis has significant consequences for mechanistic interpretability:

• Non-Canonicity: Faithful circuits for LLM capabilities are inherently non-unique and non-canonical.
• Redundancy Landscape: Mechanistic explanations should map out the distributional space of circuits, not converge to a presumed ground-truth or minimal subgraph.
• Evaluation Paradigm Shift: Metrics privileging convergence to a singular circuit (e.g., minimality benchmarks) are unreliable. Evaluations that assume essentiality or minimal sufficiency of discovered circuits are undermined.
• Methodological Guidance: Techniques incorporating explicit overlap-repulsion (e.g., OASR) more robustly surface the diversity of functionally redundant mechanisms.

A plausible implication is that auditing LLMs through circuit discovery will require methods and interpretations that explicitly engage with the distributional structure of possible mechanisms, rather than reporting single "discoveries" as canonical explanations.

6. Generalization and Future Research Directions

The framework and hypothesis apply broadly across tasks, CSD methods, and levels of model sparsity. As empirical evidence accumulates, the distributive view is reinforced by the ease with which overlap-diverse, faithful circuits can be produced, even in ultra-sparse regimes or under varying methodological conditions (Chen et al., 12 May 2026).

Further research is likely to investigate:

• Quantifying the density and diversity of redundant circuits for complex tasks.
• Developing interpretable, distribution-aware benchmarks for CSD.
• Studying the implications for robustness, adversarial editing, and model auditability.

The Distributive Dense Circuit Hypothesis represents a paradigmatic shift from singular, minimal interpretability toward a combinatorial and distributional understanding of internal computation in deep LLMs.