Relational Rank Geometry in Transformers: Detecting and Steering Hidden-State Relation Frames

Abstract: Transformer hidden states are often interpreted through local or low-order objects: neurons, sparse features, attention heads, residual-stream directions, or activation patches. This paper studies a complementary object: the rank-indexed geometry of relations among token tuples. I use Plucker sign entropy to test whether r-argument relations leave arity-matched orientation signatures in hidden-state space. Across Llama-family 8B, 70B, and 405B checkpoints, true relation tuples show stronger orientation-sign consistency at the expected rank k=r for r=3,...,6 than scrambled tuples under matched random-control audits. Multi-template audits show that the effects survive surface variation, with all tested 405B rows retaining positive expected-rank margins and 8B/70B retaining positive rows with constructor-specific mixed cells. I then ask whether the same relation geometry can be steered. In an edge-grid clean/corrupt intervention assay over 32 prompts, the row/column scaffold and answer format stay fixed while the YES/NO relation map changes, and the corrupt hidden-state relation frame is patched toward clean or placebo targets. In 70B and 405B, clean-targeted relation-frame paths recover clean-answer behavior and residual relation geometry, while centroid-only and equal-norm controls show negligible recovery. Site/order controls further separate marker-site importance from ordered clean-frame geometry: target clean shape and cross-prompt clean shape recover behavior and residual geometry at the marker interface, whereas corrupt-donor transfer, same-site permutation/reflection, wrong-site clean deltas, centroid-only motion, and equal-norm noise fail or remain far below clean-frame paths. The result is a controlled bridge from relation probing to relation-frame intervention: relation rank geometry can be detected, targeted, and behaviorally validated in transformer hidden states.

• The paper presents a diagnostic method using Plücker sign entropy to reveal orientation consistency in relation-bound token tuples.
• It demonstrates controlled interventions, including shape-based and Grassmann steering, to robustly recover clean relational behavior in models like Llama-3.1.
• The findings imply that relational rank geometry captures higher-order interactions, offering a causal handle for modular and precise model interpretability.

Relational Rank Geometry in Transformers: A Formal Examination

Introduction and Motivation

Mechanistic interpretability has primarily focused on localized or low-order structures in transformers, including individual neurons, sparse features, directions, activation patches, and circuit graphs. However, this focus neglects the distributed nature of higher-order relational reasoning, which is essential for tasks involving multi-token configurations where relational structure, not just token identity, determines model behavior. This paper introduces and examines the relation frame—an ordered multivariate configuration of hidden states capturing the geometry of token tuples participating in specific relations. The key contributions are a diagnostic method based on rank-indexed determinant sign statistics (Plücker sign entropy), and a series of controlled interventions demonstrating that these relational geometric objects can be causally manipulated to alter model behavior.

Figure 1: The relation frame is contextualized as a geometric intervention object, distinct from classical interpretability objects like directions, features, or circuit graphs; Plücker sign entropy and steering panels are shown as procedural handles on this object.

Plücker Sign Entropy: Formalism and Diagnostic Power

The central diagnostic—Plücker sign entropy—quantifies orientation consistency among selected token tuples after projection into a fixed analysis subspace. For an $r$-argument relation, the paper tests whether true tuples (i.e., those bound by the intended relation) exhibit consistent determinant sign structure at matched rank $k=r$, compared to scrambled or random control tuples. If consistent orientation is encoded, sign entropy will be lower for true tuples. This diagnostic is sensitive to the token order—a nontrivial requirement in natural language relations.

Figure 2: Conceptual intuition for Plücker sign entropy in the $k=3$ case, illustrating orientation sign changes with tuple reordering and how sign entropy captures structure present in relation-bound tuples.

The empirical finding across Llama-3.1 8B, 70B, and 405B checkpoints is that true relation tuples have significant positive expected-rank enrichment in Plücker sign consistency. This holds across $r=3$ through $6$ and survives multi-template and constructor variations, especially in 405B where all diagonal margins are positive. Control experiments demonstrate that the diagonal expected-rank signal aligns with the arity of the relation, and shifts appropriately with tuple constructor changes (e.g., predicate-plus-argument configurations).

Figure 3: Heatmap displaying $D(r,r)$ diagonal values and multi-template diagonal margins, confirming robust rank-indexed structure, especially in 405B. Mixed cells in lower scale models emphasize constructor specificity.

Edge-Grid Intervention Assay: Bridging Geometry and Causality

A key contribution is the clean/corrupt edge-grid intervention task, operationalized as $8\times8$ scaffolds of fixed token identity but variable YES/NO relation marking. Clean and corrupt relation states serve as targets for hidden-state patching, enabling precise control over which aspects of the relation frame are manipulated.

Figure 4: Schematic of the edge-grid intervention, with shared vocabulary but controlled relation states, providing a tightly localizable patching interface for relation-frame interventions.

Baseline assessment confirms that only Llama-3.1 70B and 405B are consistently behaviorally competent on both clean and corrupt relation states, ensuring that observed behavioral recoveries reflect genuine relational mechanism restoration rather than random task performance.

Figure 5: Clean and corrupt baseline metrics for the edge-grid setting, establishing differential clean/corrupt answer prediction in competent models.

Steering the Relation Frame: Recovery and Controls

The principal intervention involves moving the corrupt relation-marker cloud toward the clean reference via multiple geometrically-motivated paths, including linear patching, shape-only (centroid-invariant) manipulation, centroid-only translation, Procrustes and Grassmann subspace steering, and noise/randomized baselines.

Figure 6: Schematic of the shape-based steering path versus centroid-only control, visualized as low-dim clouds; only shape-aligned interventions robustly bridge the corrupt state to the clean state.

The empirical outcomes are unambiguous: only direct, shape-preserving, and Grassmann paths recover both clean-answer behavior and residual relation geometry to near-complete levels (endpoint behavior recovery $\approx 1.0$, residual geometry $\approx 0.9$ in 70B/405B). Centroid-only and norm-matched noise fail to induce any meaningful recovery. Among sampled paths, smooth relation-frame transitions (linear marker, shape, Grassmann) dominate both endpoint and path-integrated (AUC) coupled behavior-geometry metrics.

Figure 7: Coupled behavior–geometry AUC by method, with direct/shape-based paths dominating performance, and all scale or random controls saturating near zero.

Extensive site-and-order audit establishes that recovery is strictly tied to the ordered, clean-state frame geometry at the correct marker sites. Permuted, reflected, wrong-site, or corrupt donor interventions, even for the correct marker locations, fail to recover behavior or geometry, confirming that both the ordered configuration and clean reference are necessary for effective intervention.

Figure 8: Heatmaps over path fraction, visualizing the recovery dynamics along various steering trajectories; only clean-aligned relation-frame paths provide monotonically increasing recovery.

Figure 9: Endpoint recovery frontier, illustrating method class separation and the isolation of the clean-frame interventions on the upper-right frontier for coupled behavior and geometry recovery.

Implications and Theoretical Scope

This work formally connects representation geometry and causal actionability in transformers via relation-level hidden-state structures. The findings support the thesis that transformers encode higher-arity relational structure in a measurable, orientation-sensitive object whose rank matches relation arity. Contrary to explanations based on mean translation, perturbation energy, or arbitrary subspace deformation, only coherent geometric movement of the structured relation frame steers both behavior and relevant latent geometry.

This suggests a middle ground between atomistic (neuron/feature/direction) and global manifold perspectives, introducing relation frames as modular, patchable objects whose compositional structure supports both analytic diagnostics and causal manipulation. Importantly, relation-frame steering does not claim to localize the complete upstream circuit generating these objects, but isolates a causal handle at the state level—consistent with recent observations about the separability of representation availability and mechanism identification (Basu et al., 18 Mar 2026).

Future work should extend these results to more diverse relational prompts, arbitrary clean/corrupt relation state transfer, and detailed circuit tracing to identify the heads and MLPs constructing and reading out relation frames.

Conclusion

The paper rigorously demonstrates that relational rank geometry is both detectable and causally actionable in large-scale transformers. Plücker sign entropy reliably exposes rank-indexed orientation structure corresponding to relation arity, and shape-aligned intervention on the relation frame decisively recovers both behavior and latent geometry. The empirical and methodological suite sets a new bar for geometric and causal interpretability at the relation level, and opens a new direction for analyzing, modularizing, and intervening on structured multi-token representations in neural sequence models.