Geometric Factual Recall in Transformers

• Geometric factual recall is a paradigm where transformers encode and retrieve facts using the geometry of high-dimensional embedding spaces rather than simple key-value lookups.
• Key mechanisms such as superposition encoding, directional processing, and MLP selector functions allow for efficient relation-specific extraction and logarithmic scaling in capacity.
• Empirical studies reveal that intermediate layers exhibit rotational dynamics with measures like angular divergence, underscoring robust separation and active suppression of incorrect facts.

Geometric factual recall in transformers refers to the class of mechanisms and principles by which transformer architectures encode, store, and retrieve factual associations through the geometry of internal representations, rather than via purely associative memory or brute-force key–value lookups. Emerging research demonstrates that factual knowledge—whether in language, relational, or spatial tasks—is embedded in high-dimensional activation spaces with distinct geometric structure, and that retrieval, verification, and even suppression of facts enact transformations that are fundamentally geometric, with implications for model capacity, interpretability, and robustness.

1. Geometric Factual Recall: Definitions and Foundational Constructs

The geometric factual recall paradigm diverges sharply from classical associative-memory views. Rather than relying on parameter-memory scaling with the number of facts, transformers utilize learned embedding spaces and group attributes such that retrieval is a function of geometric selection (Ravfogel et al., 12 May 2026). Key principles include:

• Superposition Encoding: Subject embeddings encode linear superpositions of per-relation attribute codes, so that a single vector contains all relevant factual associations for a subject.
• Selector Mechanisms: The MLP and, in some settings, attention heads act as relation-conditioned selectors; their function is to extract the relevant block or component from these superpositions—not to store or retrieve individual (subject, relation)→attribute mappings as a table.
• Directional Processing: Correct and incorrect factual continuations are separated not by norm or magnitude but by direction in embedding space (rotational isometry), and detection of truth is accomplished via the angular geometry of state transitions (Marín, 25 Feb 2026).

This recasts factual recall as an interplay between algebraic representational structure, nonlinear selection, and rotation in high-dimensional manifolds.

2. Theoretical and Empirical Mechanisms for Encoder and Decoder-Only Transformers

2.1 Single-Layer Geometric Memorization

In controlled settings, a single-layer transformer can memorize random bijections $g: [N] \times [R] \rightarrow [N]$ with embedding dimension $d = O(R \log N)$. Subject vectors comprise a concatenation of near-orthogonal attribute codes, and the MLP is only required to gate (select) the appropriate slot indexed by the relation (Ravfogel et al., 12 May 2026). The construction demonstrates:

• Logarithmic Scaling: The representational dimension grows logarithmically in the number of subjects and linearly in the number of relations, in contrast to $\Theta(NR)$ scaling for weight-matrix-based associative memory.
• Selector Role of the MLP: For each relation, the MLP acts as a piecewise-linear gate, vanishing outside the desired slot—a pure geometric selector.
• Zero-Shot Transfer: Once trained, the MLP can transfer to entirely new bijections when subject embeddings are reinitialized according to the geometric principle, showing that it implements a generic relation-selection mechanism.

2.2 Multi-Hop and Chain-of-Thought

For $k$-hop queries composed from chains of relations, the embedding dimension must scale exponentially in $k$ for generic no-CoT architectures, but with chain-of-thought (CoT) emission of intermediates, a single layer with $d = \widetilde O(R + k)$ and width $\widetilde O(R)$ suffices—each autoregressive step effectively reduces the recall to a one-hop geometric selection (Ravfogel et al., 12 May 2026).

3. Rotational and Directional Dynamics in Layerwise Processing

Recent work using forced-completion probing has rigorously documented that factual recall and factual rejection in large-scale decoder-only transformers are governed by rotational, not scalar, dynamics (Marín, 25 Feb 2026). Key measured geometric quantities per layer $\ell$ include:

• Trajectory Similarity $\tau(\ell)$: The cosine similarity between hidden states after correct versus incorrect single-token continuation; $\tau$ drops sharply in mid-layers, indicating divergence by direction.
• Displacement Norm Ratio $d = O(R \log N)$0: The norm of the correct and incorrect answer-induced displacement vectors remains near unity at all depths ($d = O(R \log N)$1), implying constant-norm, isometric separation.
• Angular Divergence $d = O(R \log N)$2: The angle between displacement vectors grows rapidly, peaking at intermediate layers ($d = O(R \log N)$3), reaching up to 45°, before partial recovery in later layers.
• Active Suppression: When forced down an incorrect factual path, the logit-lens signal $d = O(R \log N)$4 reverses direction, actively suppressing the correct answer (not merely failing passively), and this suppression only emerges above a critical parameter threshold ($d = O(R \log N)$5B) (Marín, 25 Feb 2026).

The geometric signature of factuality is thus encoded in direction rather than magnitude, with mid-network layers mediating maximal separation and suppression.

4. Layerwise Structure, Superposition, and Manifold Geometry

Intermediate layers in transformer models serve as repositories of superposed attributes, with downstream selection and separation occurring in later layers (Lei et al., 15 Feb 2025). For structured attribute datasets (e.g. periodic table elements):

• Superimposed Subspaces: In mid-network layers, multiple attributes (atomic number, group, period) for a given entity are embedded as linear combinations in activation space; probes trained for each attribute direction exhibit high $d = O(R \log N)$6.
• Geometric Manifolds: Nonlinear but highly structured manifolds—e.g., 3D spirals parameterizing atomic number and group with angular/radial coordinates—are discovered, allowing for interpolation and mapping between factual attributes.
• Transition to Separation: In late layers, representations disentangle so that only the prompted attribute direction is preserved, optimizing output fluency and reducing inadvertent attribute recall.
• Recall without Explicit Prompting: Due to superposition, linear probes can recover unprompted attributes in mid-layers (high $d = O(R \log N)$7 for wrong-attribute prediction), but not in output layers where separation has occurred (Lei et al., 15 Feb 2025).

5. Vector Arithmetic, Task Concept Retrieval, and In-Context Learning

Transformers trained on QA data can provably realize factual recall tasks via vector arithmetic in their residual stream. This mechanism depends on the ability to recover a high-level “task vector” (concept steering vector) and compose it additively with the query (Bu et al., 13 Aug 2025):

• Hierarchical Concept Embeddings: Each task (relation) is embedded as a mutually orthogonal steering vector $d = O(R \log N)$8; facts are encoded as linear combinations of $d = O(R \log N)$9 with task-specific low-level codes $\Theta(NR)$0.
• In-Context Arithmetic: With QA training, the model retrieves the relevant $\Theta(NR)$1 from demonstration, combines it with the query embedding, and decodes via linear read-out—mirroring Word2Vec-like vector algebra for new facts.
• Generalization and Robustness: The construction exhibits strong out-of-domain and dictionary-shift robustness, and predictions on new (or mixed) tasks correspond to Bayesian-model-averaged steering vectors.

ICL-style training with only demonstration data does not yield this clean separation; explicit QA sentences are required for disentangled, arithmetic recall.

6. Geometric Capacity, Rank Bounds, and Storage Limits

The factual storage capacity of a transformer layer is tightly connected to the rank properties of its geometric tensors (Wong, 7 Feb 2025, Vural et al., 16 Mar 2026):

• Tensor Rank Characterization: Knowledge in a database $\Theta(NR)$2 is modeled as a 3-tensor $\Theta(NR)$3, with the attention layer’s induced logit tensor $\Theta(NR)$4. Exact recall requires $\Theta(NR)$5.
• Parameter-Efficient Scaling: The capacity of an attention layer is controlled by the value-output circuit dimension $\Theta(NR)$6; recall grows much more rapidly with this than with query-key circuit size.
• Multiplicative Scaling Law with Non-Orthogonal Embeddings: In realistic settings, with random (non-orthogonal) embeddings, capacity is governed by $\Theta(NR)$7, where $\Theta(NR)$8 is embedding size, $\Theta(NR)$9 sample size, and $k$0 sequence length (Vural et al., 16 Mar 2026). Real transformers thus require multiplicative scaling of embedding dimension and data to suppress interference noise for high-capacity factual recall.

Empirically, thresholded softmax and regularization of value subspaces are effective for maximizing geometric capacity and for controlling hallucination.

7. Implications for Interpretability, Factual Robustness, and Hallucination

The geometric lens on factual recall elucidates interpretability and avenues for factuality intervention:

• Self-Awareness and Linear Separability: Correct and incorrect recall outcomes are linearly separable in high-dimensional activation space at generation time, and this separation is robust to prompt noise and minor perturbations (Tamoyan et al., 27 May 2025).
• Active Suppression as Conflict Resolution: Upon encountering a conflicting (incorrect) factual continuation, transformers do not merely default to chance performance but actively rotate the representation away from the correct answer, indicating built-in conflict modules—a phenomenon that is parameter-threshold-dependent (Marín, 25 Feb 2026).
• Layerwise Probing and Intervention: Because the geometric signal for differentiating correct from incorrect factual recalls peaks in intermediate layers, intervention strategies—such as gating output or supervising subspace allocation—should focus on these depths.
• Transferability and Modular Design: Selector mechanisms in the MLP can be designed as universal modules across fact tables; empirical verification confirms zero-shot transfer when only embeddings are remapped to new bijections, exploiting geometric regularities (Ravfogel et al., 12 May 2026).
• Grounded Spatial Reasoning: Even in purely symbolic tasks with geometric constraints (e.g. recovering positions in a 2D grid), transformer embeddings self-organize into a geometric subspace matching the ground-truth layout—demonstrating that geometric recall extends beyond semantic facts to spatial structures (Hůla et al., 2 Apr 2025).

The geometric perspective reconciles high empirical knowledge capacity with efficient modularity, offering both a mechanistic and a unifying theoretical account of factual recall in modern transformer LLMs.