Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention

Abstract: The pre-softmax score of an attention head is a bilinear form $score(i,j) = x_i^T M x_j$ in a learned operator $M = W_q^T W_k$. Because M is generally non-symmetric, hence non-normal, it has a complex eigenspectrum and non-orthogonal eigenvectors, the regime where non-Hermitian and random-matrix tools apply. We ask what this spectrum encodes, at three levels for previous-token and induction circuits. Statically, across seven pretrained models spanning three positional schemes, the strongest previous-token heads are spectrally rotational under RoPE and non-rotational, or content-like, where position enters outside QK (learned-absolute and ALiBi); the model-level separation is perfect at every top-k examined (exact permutation $p=0.029$), and zeroing the per-frequency RoPE phase $Im(M_t)$ eliminates induction on a pre-identified previous-token head in all three RoPE models. Dynamically, over public Pythia checkpoints every head originates at the random-matrix (Ginibre) null; the rotational signature emerges with the behavior, not before it, and the population-median suppression that yields the final profile follows circuit formation, so the profile is a consolidated fingerprint, not a precursor. Causally, and at toy scale, no spectral channel is necessary: constrained two-layer training reroutes around every ban with capability intact, albeit at a significant formation delay (four pre-registered contrasts, $q_BH <= 0.016$). The cost structure exposes each scheme's default: imposing symmetry slows learned-absolute models by a factor of 2.9, whereas a RoPE head with a fully symmetric static M still routes directionally via the phase channel, impossible under absolute positions. Within the settings examined, the positional scheme sets the default spectral algebra of an attention head's solution: a fingerprint sculpted after function, not a hard constraint upon it.

• The paper reveals that positional encoding schemes yield distinct spectral fingerprints influencing attention head functions in Transformers.
• It employs non-Hermitian and random-matrix theory to map spectral metrics to specific roles like previous-token and induction circuits.
• Ablation studies show that altering spectral channels delays function formation, highlighting architecture-dependent defaults and flexibility.

Spectral Fingerprints in Attention: A Systematic Dissection of Positional Scheme Effects on QK Spectra

Introduction

This work rigorously dissects how positional encoding schemes—learned-absolute, ALiBi, and RoPE—influence the spectral “algebra” of the attention QK operator $M = W_q^\top W_k$ in Transformer architectures. By adopting tools from non-Hermitian and random-matrix theory, it seeks to clarify: To what extent do complex-valued spectral properties of $M$ carry or constrain behaviorally-causal information for specific attention head functions, especially previous-token and induction circuits? This question is systematically analyzed through static cross-model comparisons, dynamic checkpoint analyses, and dissective interventions in controlled training settings, yielding an architecture-dependent landscape of spectral necessity, default, and functional fingerprinting.

Spectral Decomposition Framework and Null Models

A central analytic contribution is the establishment of matched-rank, random-orientation null models for spectral metrics. $M$ is generally non-symmetric and thus supported on complex eigenspectra with non-orthogonal eigenvectors. Decompositions—SVD, symmetric/antisymmetric splits, real Schur forms, and full complex eigendecompositions—anchor novel and existing metrics: (i) the “directionality metric” $dir = \|M_A\|_F / \|M\|_F$; (ii) a complex directionality score $D_{head}$, the normalized sum over imaginary parts of eigenvalues, and (iii) a “content” score from the symmetric sector. Crucially, these metrics are always referenced to model-specific random orientation nulls, as the spectral baseline for “directionality” is already high in generic low-rank $M$ (e.g., $dir_{\mathrm{null}} \approx 0.707$).

Static Function–Spectrum Mapping Across Positional Schemes

A clear dissociation emerges between attention head function and spectral profile, but one that is conditional on the positional encoding scheme.

• Previous-token heads: Under RoPE, these are robustly spectrally rotational (high $D_{head}$ percentile within the model), while under learned-absolute and ALiBi, they are non-rotational and content-like (low $D_{head}$, symmetric, positive content score). This model-level segregation is perfect at all tested $k$ (Figure 1).

  Figure 2: The content--direction plane (GPT-2). Each point is a head; dashed lines mark the random-orientation null $M$0.

  

  Figure 1: Within-model $M$1 percentiles for each model's top-5 previous-token heads: bottom-quartile for learned-absolute (red) and ALiBi (orange), top-quintile for RoPE (blue).

• Induction heads: Contradicting preregistered expectations, induction heads are generally symmetric, content-like, and not spectrally directional.

The statistical claim is robust: model-level permutation $M$2 for the segregation of $M$3 percentiles. In all RoPE models, the incremental predictive power of $M$4 (complex directionality) over the plain antisymmetric norm $M$5 is positive and statistically significant.

Causal Ablations and Structure-Function Disentanglement

Ablation studies on GPT-2 and RoPE models reveal:

• Symmetric sector dominance: For generic heads, only the symmetric part of $M$6 is load-bearing for per-token cross-entropy, with the antisymmetric part ($M$7) inert except in a discrete subset of heads implementing precise circuits (e.g., canonical induction/prev-token heads).
• Functional specificity: Targeted ablation of $M$8 (the RoPE phase channel) in high $M$9 heads destroys induction behavior (up to $M$0 increase in induction loss) and symmetrizes the relative-position kernel, but has negligible effect elsewhere.

Figure 3: Left: Corpus-averaged relative-position score kernels of top previous-token heads (Pythia-410m) maximize at $M$1 with RoPE-induced periodicity.

Figure 4: Checkpoint natural history, Pythia-410m (blue) and 160m (red); formation window shaded. (a) Previous-token behavior forms sharply and identically in both models. (b) The prev head's rope percentile locks in with behavior formation, not before. (c) Population median $M$2 is suppressed below the Ginibre null post-formation.

Mechanistic Tethering: RoPE Phase and Spectral Signature

Decomposition of $M$3 into per-frequency complex sub-operators $M$4 shows that the imaginary part, $M$5, is the carrier of directionality for previous-token attention under RoPE:

• High correlation ($M$6) between “rotary directional fraction” (rope score) and functional headness (prev-detection score).
• Regressing out per-frequency phase features eliminates the incremental predictive association of $M$7 on prev-token behavior in full-RoPE models, but not in learned-absolute models, confirming the architecture specificity.

Dynamics: Circuit Formation and Fingerprint Consolidation During Training

Tracking Pythia checkpoints, all heads begin at the Ginibre random-matrix null—function and spectral imprint emerge synchronously during the formation window ($M$81–4B tokens), with post-formation suppression of $M$9 in the population but its retention in the principal previous-token head(s). Importantly, spectral “fingerprints” consolidate only after function arrives; they do not prospectively identify circuit formation.

Constrained Training Interventions: Necessity and Default

Directly manipulating the spectral sector during from-scratch training answers questions of necessity versus default:

• No sector is necessary: Models trained under soft constraints blocking antisymmetric or RoPE-phase channels always “reroute” and achieve functional performance, albeit with a quantifiable delay (e.g., $dir = \|M_A\|_F / \|M\|_F$0 slower formation when forced symmetric in APE, $dir = \|M_A\|_F / \|M\|_F$1 with suppressed phase in RoPE).
• Default solution structure is architecture-dependent: Imposing symmetry on APE is much costlier than on RoPE, and vice versa for removing the phase channel.

  Figure 5: Constrained-training interventions: every constraint delays induction formation but all arms reach capability; cost structure isolates scheme defaults.

Implications and Future Directions

This study’s architecture-dependent view of QK spectral structure reconstructs a more granular theoretical and operational taxonomy:

• Spectral structure as fingerprint, not blueprint: The complex eigenstructure is not a “hard constraint”: it arises as the path-of-least-resistance (default) solution imprinted during circuit formation. The “non-Hermitian transformer” framing is only functionally anchored under RoPE, where phase-channel routing is truly utilized.
• Interpretability and circuit analysis: RoPE-phase content (or $dir = \|M_A\|_F / \|M\|_F$2 percentile) robustly flags positional-routing heads post hoc in RoPE models—an efficient triage tool for weight-space analysis.
• Causality versus correlation in interpretability: Post hoc ablation severity is not predictive of developmental necessity; spectral sectors can be fully bypassed at finite search cost.
• Generalization: Points to the importance of scaling such constrained pretraining experiments, analysis of path patching and interaction between multiple heads, and deeper investigation of latent grouping in alternate architectures (e.g., Qwen, Mistral families).

Conclusion

The spectral algebra of the QK attention operator is determined by the model’s positional scheme, providing distinct “fingerprints” for default solutions but imposing no hard “blueprints.” While complex spectral signatures are causally load-bearing in RoPE-based architectures and serve as efficient post hoc markers for crucial circuits, every spectral channel analyzed is ultimately degenerate—circuit formation can reroute with altered search costs when constraints are imposed. Hence, the non-Hermitian perspective on attention is architecture-conditional: essential and causal in RoPE, decorative in learned-absolute, and always subject to developmental economy and trainability constraints. This framework raises important operational and theoretical questions for the field as model architectures and interpretability techniques evolve.