Factual Retrieval in LLMs Is a Redundant, Distributed and Non-Contiguous Process

Abstract: LLMs store and recall factual knowledge, yet the precise mechanism of how entity representations are transformed to enable specific attribute retrieval remains underexplored. In this work, we investigate this mechanism through the lens of an "attribute-computation path"-a sequence of computational steps over the entity representation required to elicit a target attribute. We then propose an iterative patching protocol to identify a minimal subset of layers necessary for this computation. Applying our method to LLaMA 3.1 8B and Qwen3 8B, we find that these paths are non-contiguous, often skipping layers, and that models possess multiple, functionally-equivalent paths for the same entity and fact, highlighting a high degree of redundancy in attribute computation. This implies that knowledge computation is highly distributed, potentially explaining the localization-editing mismatch and suggesting that knowledge storage and retrieval in LLMs is far from being well understood.

• The paper demonstrates that LLMs recall factual attributes via minimal attribute-computation paths using early-to-mid transformer layers.
• It reveals that factual retrieval requires processing across 5–8 non-contiguous layers with substantial layer skipping.
• The study uncovers redundant circuits, showing multiple, functionally equivalent computation paths for the same factual recall.

Attribute Computation Paths Reveal Redundancy and Distribution in Factual Retrieval in LLMs

Introduction

The paper "Factual Retrieval in LLMs Is a Redundant, Distributed and Non-Contiguous Process" (2606.21345) systematically investigates the internal mechanism by which LLMs recall factual attributes associated with entities. Through the novel concept of attribute-computation paths—minimal sequences of computational steps over entity representations that suffice for eliciting target attributes—the study offers granular insight into how LLMs encode and process factual knowledge. The work introduces an iterative activation-patching protocol for identifying these minimal paths and applies this methodology to LLaMA 3.1 8B and Qwen3 8B, yielding quantitative and qualitative findings that challenge prevailing assumptions about knowledge localization and retrieval.

Methodological Framework

The primary methodological advance centers on mapping attribute-computation paths. The investigative protocol involves two key stages: (1) identification of the earliest layer $\ell_{attr}$ at which the entity representation is attribute-sufficient, and (2) iterative construction of minimal, non-contiguous layer sequences necessary for attribute recall, using interventionist activation patching (lock and isolate operations).

The lock operation establishes the earliest attribute-sufficient layer by patching its entity representation to all subsequent layers and evaluating predictive sufficiency. The isolate operation exposes path indispensability by injecting clean entity representations at candidate path layers and counterfactual activations at skipped layers. This greedy search strategy maximizes layer skipping per iteration, ensuring strict minimality and revealing both primary and alternative (redundant) computation paths.

Figure 1: Iterative greedy search process for minimal computation paths, highlighting both failed (invalid) and successful (valid) skips in layer selection.

Empirical Findings

Early-Mid Layer Sufficiency

The empirical results show attribute computation is completed predominantly in early-to-mid transformer layers: average $\ell_{attr}$ is 4.61 for LLaMA 3.1 8B and 7.97 for Qwen3 8B, with some tasks requiring up to layer 20. This supports prior work localizing entity enrichment and factual recall in early-mid MLPs but surfaces more granular evidence for computational sufficiency.

Figure 2: Distribution analysis of $\ell_{attr}$ across prompts, revealing factual attribute computation concludes in early to mid layers.

Multi-Step, Sparse, Non-Contiguous Computation

Path analysis demonstrates that factual recall is not attributable to a single layer but typically requires processing through 5–8 layers. Critically, 33% (LLaMA) and nearly 79% (Qwen3) of paths are non-contiguous, skipping intermediate layers, with substantial average skip sizes. This indicates attribute retrieval does not manifest as sequential accumulation but rather as sparse, distributed computation.

Figure 3: Two functionally equivalent minimal computation paths for “Angela Merkel”, showing compact (early) and distributed (deep) alternatives with non-contiguous layer usage.

Figure 4: Compression ratio analysis evidences significant layer skipping in minimal computation paths, with majority of paths being non-sequential.

Redundant, Distributed Circuitry

Alternative path analysis uncovers redundancy: 80–83% of cases admit longer, deeper alternative computation paths that are functionally equivalent but distinct in layer selection. These paths often end much deeper (mean $\ell_{attr}$: 13.93 for LLaMA, 18.06 for Qwen) and routinely incorporate massive layer jumps, recruiting modules not engaged in the primary path.

Figure 5: Layer usage heatmaps contrasting primary (top row) and alternative (bottom row) computation paths, demonstrating distinct, deeper layer recruitment by alternative paths.

Functional Sufficiency and Necessity

Targeted patching interventions clarify layer roles. Knockout experiments confirm representations at $\ell_{attr}$ are strictly necessary (>94% failure when replaced with counterfactual), yet, for many cases, $\ell_{attr}$ alone is not strictly sufficient; dependent on intermediate representations. The path as a functional unit—rather than a single representation—facilitates recall.

Layer Functionality and Information Propagation

Further experiments dissect whether path layers serve intra-token transformation or inter-token propagation. Longer paths and alternative circuits increasingly depend on attention-mediated propagation across the sequence; up to 80% in alternative paths. Unexpectedly, 15% of all paths require clean propagation as early as layer 0, suggesting early-stage processing is occasionally crucial.

Figure 6: Heatmaps quantifying layer-wise information propagation necessity, including notable dependence on Layer 0 for a subset of paths.

Modular Recombination and Hybrid Circuits

Hybridization analysis—concatenating primary path prefixes with alternative path suffixes—reveals substantial modular flexibility, with 33–35% of prompts admitting multiple successful hybrid paths. This confirms the existence of further redundant and recombinable computation circuits beyond those identified by main search, increasing the lower bound on redundancy.

Per-Relation Analysis and Entity Resolution

Relation-specific analyses reveal heterogeneity in computational cost and layer recruitment, hinting at semantic specificity effects. However, the ranking and layer distribution are inconsistent across models and path types, affirming the non-localized, distributed nature of factual retrieval. Entity resolution is not universally observed along attribute-computation paths; explicit identity formation is a minority occurrence.

Figure 7: Mean computation path length per relation, documenting inter-relation heterogeneity in computational requirements.

Figure 8: Entity resolution detection success rates, confirming limited explicit identity decoding along computation paths.

Effect of Counterfactual Noise and Path Construction

Experiments substituting counterfactual activations with locked layer states confirm path length as an intrinsic task property, but demonstrate that computation depth (termination layer) is sensitive to noise. Removing noise deferred computation to deeper layers, shifting alternative paths’ layer utilization toward the model’s end.

Figure 9: Layer usage heatmaps for primary and alternative paths in noise-free setting, documenting structural shifts in alternative path layer recruitment.

Implications and Theoretical Significance

These findings refine the theoretical understanding of LLM internal mechanics:

• Factual retrieval is realized as distributed, non-contiguous computation, requiring multiple, sparsely selected layers.
• Redundant circuits are pervasive; multiple, non-intersecting paths exist for retrieving the same fact.
• Attribute computation does not correspond to static knowledge localization; facts are not “stored” in isolated model regions but depend on coordinated transformations across the model.
• Practical interventions (editing, patching, modularization) must accommodate the distributed, redundant architecture uncovered, potentially necessitating multi-layer or circuit-level targeting for reliable knowledge attribution or editing.

This nuanced view undermines the simplistic paradigm of knowledge localization and motivates further research into the combinatorial landscape of circuits underlying factual recall.

Conclusion

By introducing attribute-computation paths and applying a novel interventionist protocol within LLMs, the paper establishes factual retrieval as a distributed, non-contiguous, and highly redundant process. The results recalibrate the mechanistic understanding of entity enrichment and factual recall, revealing complex, recombinable circuits underwriting factual memory in transformer architectures. Future directions should include exhaustive circuit identification, model editing strategies attuned to circuit redundancy, and theoretical developments in distributed representation learning in deep LLMs.