A retrieval conditioned rebinding circuit for dynamic entity tracking in large language models

Abstract: To interpret context correctly and retrieve relevant information, LLMs must bind entities to their attributes and update these bindings as state changes. We analyze how LLMs implement this binding process in a dynamic state tracking. Using causal interventions, we identify a retrieval conditioned rebinding mechanism, a compact attention head circuit that encodes swap relevant binding information and reinstates it at readout. Across Gemma and Llama models, this circuit supports rebinding behavior, but the representational signature of the mechanism differs across model families. In Gemma models, the binding signature is clearly expressed in the query/key subspaces of the relevant attention heads, whereas in Llama models, the binding information is carried primarily in key vectors. Overall, our results reveal an interpretable mechanism for context dependent state tracking in LLMs.

• The paper’s core contribution is demonstrating that LLMs can track dynamic entities using a retrieval-time rebinding mechanism instead of costly global state updates.
• Methodologically, causal mediation and path patching pinpoint specialized attention head groups responsible for address routing and payload retrieval.
• Results reveal model-specific Q/K binding strategies that inform architectural design for efficient and interpretable entity tracking.

Retrieval Conditioned Rebinding Circuits for Dynamic Entity Tracking in LLMs

Introduction

Dynamic entity tracking—maintaining associations between entities and their attributes as context evolves—is a core requirement for coherent long-context reasoning in LLMs. The paper "A retrieval conditioned rebinding circuit for dynamic entity tracking in LLMs" (2606.08644) addresses the underlying mechanisms supporting this capacity, focusing on how attention head circuits in transformer models such as Gemma and Llama encode and update binding relations under state-changing operations (e.g., swaps). This work rigorously dissects whether LLMs rely on a global state update akin to mental "simulation" or leverage local, retrieval-time rebinding mediated by dedicated circuit components.

Mechanistic Hypotheses and Task Framework

The authors distinguish two mechanistic accounts for dynamic state tracking:

1. Global State Update: The model reconstructs the entire world state after each update so that answering a query entails straightforward extraction from the updated latent configuration.
2. Retrieval Conditioned Rebinding: The model instead encodes initial bindings and applies swap operations as address-level remappings only during answer retrieval, relying on persistent binding identifiers (binding IDs) rather than propagating explicit object-token modifications throughout context.

Experimental evaluation is conducted on synthetic tasks involving explicit entity and object assignments, with state changes operationalized by swap actions. The core technical innovation is the introduction and localization of a compact, functionally specialized circuit—the retrieval conditioned rebinding circuit—that supports on-demand rebinding at retrieval time without globally updating the contextual state representation.

Causal Mediation and Path Patching Analyses

To validate the retrieval conditioned rebinding hypothesis, the authors employ detailed causal interventions including interchange interventions and path patching to trace token-level and attention-head-level information flow.

The results demonstrate that:

• Box Labels (Reference and Swap Targets): Their influence on the final answer emerges mainly after the swap operation and is propagated via intermediate positions—not directly routed from their initial contextual mentions.
• Target Object: Its representation is routed more directly through later layers, not mediated via box labels.

This supports the thesis that LLMs perform retrieval-time rebinding: swap-relevant information conditions the retrieval circuit, which locates and dereferences the appropriate object via its original context association.

Figure 1: Information flow of crucial tokens using interchange interventions in Gemma-9B, revealing post-swap, retrieval-centric mediation for box labels and direct propagation for object tokens.

The identification of a compact, functional circuit comprising ~3–10% of attention heads is central: these components, subdivided into roles such as answer retriever, dereferencer, position updater, swap transmitter, and binding anchor, are sufficient to reconstitute nearly full-model task performance.

Figure 2: Path patching circuit for retrieval conditioned rebinding, showing directed routing among functional head groups via attention-mediated value and query composition.

Functional Dissection of the Rebinding Circuit

Experimental interventions clarify the functional segregation of head groups:

• Group B ("Dereferencer"/"Pointer" Heads): Causally responsible for retrieval address resolution, as shown by targeted attention pattern patching that selectively increases the logit for the counterfactual object associated with a swap. Group B's effect is consistent with address selection via attention redirection rather than direct object payload transfer.

Figure 3: Content control intervention: object logit remains unchanged when only the attention pattern is patched, verifying address routing role rather than direct content transfer.

• Group A ("Answer Retriever" Heads): Contributes to final extraction of object information, with some models displaying direct payload recovery in these heads.
• Other Groups: Participate in intermediate propagation and position translation necessary for maintaining and updating addressable binding IDs.

  Figure 4: Role specific head patching effects for Gemma-9B, revealing strong address-selection ability of Group B and payload-readout by Group A.

Binding ID Representation in Query/Key Subspaces

The authors rigorously probe whether rebinding is implemented via explicit binding ID matching in the Q/K subspaces of the dereferencer heads. By shifting Q/K representations along empirically derived binding ID direction vectors, they show:

• Query-side Interventions: Strongly redirect attention from the originally encoded address to the counterfactual (swapped) one, inducing corresponding behavioral changes at output.
• Key-side Interventions: Shifts in context-attached keys redirect retrieval, especially when object-token keys are perturbed.
• Q+K Cancellation: Simultaneous shifts in query and key subspaces mitigate redirection effects, indicating that address computation relies on the relative difference between these representations, consistent with a dot-product matching account.

The diagnostic metrics $\Delta R$ (attention ratio shift), $\Delta$logit (answer margin shift), and switch fraction provide quantitative evidence for this mechanism. Notably, Gemma models manifest particularly clear Q/K binding ID matching in dereferencer heads.

Architectural Generalization and Comparative Circuit Analysis

Extending the analysis across Gemma and Llama model families, several robust findings emerge:

• Compactness and Sufficiency: In all models, pruned circuits comprising a minority of attention heads retain the core entity tracking behavior (e.g., Gemma-12B: 83/768 heads; Llama-8B: 40/1024).
• Functional Modularization: Group B consistently mediates address selection/address routing, though the relative contributions of Groups A and B vary slightly across architectures. Llama models tend to implement more key-dominated routing, while Gemma models exhibit Q/K matching dynamics.
• Binding ID Implementation: The representational locus of binding IDs—whether in queries, keys, or both—differs by model. Llama models rely more heavily on key-side information, with less Q/K symmetry compared to Gemma.

  Figure 5: Q/K binding ID interventions quantitatively demonstrate model-dependent differences in Q/K subspace utilization for address matching in the dereferencer heads.

Practical and Theoretical Implications

The explicit circuit-level characterization of retrieval conditioned rebinding clarifies how transformers achieve efficient, compositional state tracking without the overhead of updating the entire contextual state after each operation. This mechanism supports:

• Efficient Memory Use: By avoiding redundant recomputation, models can represent and manipulate dynamic knowledge over long contexts with minimal resource allocation to core components.
• Interpretability: Functional decomposition into address routing and payload recovery provides a basis for reverse engineering and potentially controlling LLM reasoning steps.
• Future Model Improvements: Recognizing family-specific Q/K matching strategies enables targeted interventions during training or architectural modifications to enhance robustness and generalization in entity tracking.
• Broader Applicability: The described mechanisms likely underpin a range of context-sensitive reasoning skills beyond synthetic swap tasks, suggesting that similar circuits could support more complex temporally extended narrative understanding.

Additionally, the circuit-level differences observed between model families raise hypotheses about the interaction between pretraining regimes, scale, and emergent representational strategies—questions suitable for systematic ablation and transfer studies.

Conclusion

This paper delivers a precise mechanistic account of dynamic entity tracking in LLMs, demonstrating that retrieval conditioned rebinding via a specialized attention circuit, rather than global state update, supports context-sensitive reasoning about entities. Although the high-level computational strategy is consistent across Gemma and Llama models, the implementation details—such as distribution and Q/K subspace encoding of binding IDs—exhibit family-specific signatures. Future research should examine the scalability and generality of these circuits in more naturalistic or challenging scenarios, and further exploit mechanistic interpretability to guide model design and training objectives.