The Geometry of Updates: Fisher Alignment at Vocabulary Scale

Abstract: Training-free source selection for LLM families with shared vocabularies arises in scientific string domains such as SMILES, protein, and genomic sequences, where candidate corpora share a tokenizer but differ in prediction targets. This creates an activation-dark regime: representation-similarity metrics can be uninformative without assumptions about label-conditioned error geometry, while classical update-geometry metrics are computationally prohibitive at vocabulary scale. We show that, in a shared-output head setting, representation metrics (e.g., CKA) are non-identifiable for transfer; models can share identical representations yet have orthogonal head updates. The key identity is that head Fisher alignment is exactly a cosine between kernel mean embeddings in the joint activation-error space, exposing activation, error, and coupling factors rather than requiring a materialized Fisher matrix. FisherSketch estimates this cosine directly in a single streaming pass, making K=128,256 head Fisher alignment practical with a 16 KB task signature (m=4096) and a 192 KB per-task streaming state, small enough to store next to a model hash, but encoding transfer-relevant update structure. Beyond source selection, the same signatures and marginals provide a diagnostic instrument for studying whether LLM task similarity is driven by activations, errors, or their coupling; shared-parameter and internal-layer validations, together with Llama-3.1-8B verbalizer-shift experiments, show that FisherSketch remains informative when activation similarity cannot distinguish tasks.

• The paper demonstrates that activation-only metrics are insufficient, highlighting the importance of label-conditioned error geometry in transfer learning.
• The paper introduces FisherSketch, a one-pass estimator that efficiently compresses update geometry signatures to predict transfer accuracy at vocabulary scale.
• Empirical results show substantial improvements in source selection, validating Fisher alignment as a reliable metric for training-free adaptation in LLMs.

The Geometry of Updates: Fisher Alignment at Vocabulary Scale — Technical Summary

Motivation and Problem Setting

In transfer learning scenarios involving LLM architectures sharing a vocabulary—typified by scientific string domains (SMILES, protein, genomic sequences)—the problem of training-free source selection is acute. Here, candidate corpora share a tokenizer, but label spaces diverge, yielding the “activation-dark” regime: frozen encoder representations cannot distinguish tasks, yet their label-conditioned error geometries, and hence parameter-update directions, may be radically non-aligned. Standard representation similarity metrics (CKA, RSA, SVCCA) operate solely over frozen activations, disregarding the geometry of label-conditioned errors that directly determine the directions of head updates. Further, classical update-geometry metrics (i.e., Fisher alignment) are computationally infeasible at modern vocabulary scales ($K=10^5$).

Representation Similarity versus Update Geometry

The core theoretical result establishes the non-identifiability of Fisher alignment from representation-only metrics. Specifically, Theorem~1 states that even if two models’ representations on a shared probe set are exactly matched ($Z_i=Z_j$), corresponding head-update gradients can be orthogonal, leading to zero head Fisher alignment ($A_{\mathrm{F}}^{\mathrm{head}}(i,j)=0$), regardless of $M(Z_i, Z_j)=M(Z_i, Z_i)$ for any representation-only metric $M$.

Figure 1: When activation similarity is constant across all verbalizer pairs, error geometry alone accounts for the transferability discrepancies between tasks; demonstration from the fixed-prefix regime.

This decoupling is constructively instantiated via a shared-output masked-softmax embedding with disjoint label blocks. Dense-vocabulary regimes (fixed-prefix prompts, e.g., verbalizer shift) also expose this pattern: representations are identical across all verbalizers, while Fisher alignment (and actual transfer) is dictated entirely by error geometry. Consequently, conventional metrics like CKA are fundamentally limited for transfer prediction when label geometries differ.

Figure 2: Spearman correlations for ViT-B/16 (same-dataset task pairs) between Fisher alignment estimators ($\hat S_{ae}$), product-coupling terms, and empirical Fisher alignment; the joint similarity estimator approaches 0.97 at moderate sketch dimensions ($m=4096$).

Theoretical Characterization of Fisher Alignment

Head-layer Fisher alignment, under shared vocabulary, admits an exact three-factor decomposition (Theorem~3): it is the normalized cosine between the kernel mean embeddings constructed in the joint activation–error feature space,

$$A_{\mathrm{F}}^{\mathrm{head}}(i,j) = S_{\mathrm{cov}}(M_{a,i}, M_{a,j})\, S_{\mathrm{cov}}(\Gamma_{e,i}, \Gamma_{e,j})\, \rho_{ij}\ ,$$

where $M_{a,k}=\mathbb{E}[a a^\top]$ and $\Gamma_{e,k}=\mathbb{E}[e e^\top]$ are uncentered activation and error second moments, and $Z_i=Z_j$0 quantifies non-separability (activation–error dependence). Representation-only metrics are blind to both error geometry and their coupling, which cannot, in general, be inferred from activations alone.

FisherSketch: Practical Alignment Estimation at Vocabulary Scale

The main technical tool is FisherSketch, a one-pass, streaming, product-kernel random feature estimator of update geometry. By leveraging the Kronecker structure of head gradients ($Z_i=Z_j$1), FisherSketch estimates the kernel mean embedding in the joint (activation, error) space efficiently, compressing each task into 16 KB signatures for vocabulary sizes $Z_i=Z_j$2 up to $Z_i=Z_j$3 tokens, compared to 61 GB for full error covariance storage. Estimation variance decays as $Z_i=Z_j$4 in the sketch dimension $Z_i=Z_j$5 with only $Z_i=Z_j$6 computation per sample, using factored random Maclaurin features and an SRHT for efficient high-dimensional projections.

Empirical Results and Source Selection

Vocabulary-scale Evaluation

On a 100-domain Llama-3.1-8B benchmark with shared vocabulary ($Z_i=Z_j$7), FisherSketch achieves:

• 45.7% Top-1 source selection accuracy (vs. 4.2% random; Table 1)
• 87.3% Top-3 accuracy
• 98.44% Mean normalized transfer relative to oracle
• Max regret 0.119

Activation-only metrics are competitive under natural domain shifts; however, in “activation-dark” regimes (fixed prompt prefixes), activation similarity is nearly constant, yet transfer varies sharply and FisherSketch remains predictive.

Verbalizer-Shift (Identifiability Stress Test)

In fixed-prefix, label-perturbed tasks, activation similarity is uniformly 1, but LoRA transfer varies by a factor of three across verbalizers. FisherSketch achieves 66.7% top-1 source selection versus 20% for activation-only (random), indicating strong discriminative capacity solely from label-conditioned error geometry.

Figure 3: ViT-B/16 diagnostic of full-network to block-diagonal Fisher gap; the empirical bound is tight for all task pairs, validating the theoretical decomposition in high-profile-cosine regimes.

Figure 4: FisherAtlas UMAP of 22 Pile domains visualized by FisherSketch task signatures, illustrating how clustering is governed by joint activation–error geometry rather than representation similarity alone.

Broader Applicability and Limitations

FisherSketch formalism is applicable to any transformer or deep learning architecture with a shared output (vocabulary, label taxonomy), including convolutional and transformer models with shared-parameter blocks. When output heads are not natively aligned, FisherSketch can be used as an internal-layer diagnostic. Full-network Fisher alignment can be related to the head block via profile cosine and off-diagonal discrepancy diagnostics, with empirical results (ViT-B/16, Llama-3.1-8B/70B) demonstrating tight bounds. The intrinsic label-aware nature constrains portability to architectures or transfer scenarios where label spaces are not comparably aligned.

Theoretical and Practical Implications

This work explicitly delineates settings in which representation similarity metrics are provably insufficient for predicting update compatibility and transferability between tasks. Fisher alignment, and specifically its efficient estimation via FisherSketch, provides a scalable, practical diagnostic for training-free source selection at vocabulary scale. The approach enables not only more reliable automated adaptation and fine-tuning source selection for LLMs and scientific models but also a finer-grained probe of task geometry—disambiguating when task similarity is driven by representations, errors, or their interaction. Diagnostic quantities such as the non-separability coupling $Z_i=Z_j$8 allow practitioners to directly measure structural coupling and its effect on transfer, while the lightweight signatures support dynamic open-set retrieval and compatibility-aware routing.

Notably, strong numerical results demonstrate that FisherSketch delivers robust transfer prediction even when activation similarity collapses (verbalizer shift: 66.7% vs. 20% random), and compresses transfer-relevant update geometry to sub-100 KB per task—feasible for large-scale model indexation and open-set continual learning.

Conclusion

FisherSketch operationalizes the insight that transferability is determined by the geometry of parameter updates, not merely representational similarity. By making Fisher alignment at vocabulary scale computationally feasible (~16 KB/task), it provides a practical, informed basis for training-free, label-aware source selection and transfer diagnostics in multi-domain LLM frameworks. The theoretical decomposition highlights both the necessity and sufficiency of considering activation, error, and their coupling for transfer prediction, expanding the methodological arsenal for adaptation in large neural systems.