FisherSketch for LLM Head Alignment

• FisherSketch is a streaming random-feature method that estimates head empirical Fisher alignment, providing a compact update-geometry signature for shared-vocabulary LLMs.
• It leverages kernel mean embedding in joint activation–error space to reduce computational cost from gigabyte-scale Fisher matrices to kilobyte-scale task signatures.
• The method decomposes task similarity into activation and error geometries using a non-separability ratio, addressing the limitations of representation-only metrics in domains like SMILES, proteins, and genomics.

Searching arXiv for papers on FisherSketch and closely related usages of the term. FisherSketch is a streaming random-feature method for estimating head empirical Fisher alignment between tasks in LLMs with a shared vocabulary / shared output basis, introduced to support training-free source selection in within-family adaptation settings such as SMILES, proteins, and genomics (Sweeney, 25 Jun 2026). In this setting, the central claim is that transfer-relevant similarity is governed not by representation similarity alone, but by the geometry of the parameter updates that tasks induce, especially in the final affine output layer. FisherSketch addresses the resulting vocabulary-scale computational barrier by replacing explicit Fisher matrices with compact task signatures derived from a kernel mean embedding in joint activation–error space, thereby making head Fisher alignment practical at scales such as $K=128{,}256$ with a 16 KB task signature ($m=4096$) and a 192 KB per-task streaming state (Sweeney, 25 Jun 2026).

1. Problem setting and conceptual motivation

FisherSketch was proposed for a specific transfer problem: selecting a source domain for adaptation to a target domain without training, when candidate domains share a tokenizer and output vocabulary but differ in prediction targets (Sweeney, 25 Jun 2026). This is natural in scientific string domains such as SMILES, proteins, and genomics, where corpora may use the same tokenization and next-token prediction head, yet encode different output-side predictive structure.

The method is motivated by what the paper calls the activation-dark regime. In this regime, frozen-representation metrics can be uninformative because similar activations do not imply similar updates. The paper formalizes this through a non-identifiability result for representation-only metrics, defined as metrics depending only on probe representations $Z_i, Z_j$ and not on errors or gradients. It constructs tasks with identical probe representations but different head Fisher alignment, including a witness construction with

$$A_{\mathrm{F}}^{\mathrm{head}(i,j)}=0,\qquad A_{\mathrm{F}}^{\mathrm{head}(i,i)}=1,$$

despite $Z_i=Z_j$ (Sweeney, 25 Jun 2026). The same point is instantiated for dense-vocabulary LLMs through fixed-prefix verbalizer shifts, where answer-position activations are identical by causality but transfer differs because output-token error geometry changes (Sweeney, 25 Jun 2026).

This establishes the central premise: transfer-relevant similarity in shared-output LLM families depends on the joint law of activations and output errors, not on activation similarity alone. A plausible implication is that FisherSketch is best understood not as a generic representation sketch, but as an update-geometry descriptor specialized to settings where output coordinates are aligned across tasks.

2. Fisher alignment and the head-update geometry

The target quantity in FisherSketch is empirical Fisher alignment. For task $k$, with per-example gradient

$$g^{(k)}_\theta(x,y) := \nabla_\theta \ell^{(k)}_\theta(x,y),$$

the paper defines the empirical/data Fisher

$$\mathcal{F}_k(\theta_k) := \mathbb{E}_{(x,y)\sim \mathcal{D}_k} \big[g^{(k)}_{\theta_k}(x,y)\,g^{(k)}_{\theta_k}(x,y)^\top \big].$$

Fisher alignment between tasks $i,j$ is the normalized Frobenius inner product

$$A_{\mathrm{F}(i,j)} := \frac{\langle \mathcal{F}_i, \mathcal{F}_j\rangle_F}{\|\mathcal{F}_i\|_F\,\|\mathcal{F}_j\|_F} = S_{\mathrm{cov}(\mathcal{F}_i,\mathcal{F}_j)}.$$

This is a cosine in Hilbert–Schmidt geometry (Sweeney, 25 Jun 2026).

In the shared-vocabulary LLM setting, the emphasis is on the head Fisher, i.e. Fisher restricted to the final affine output layer. For an affine head, if $m=4096$0 is the uncentered activation entering the head and $m=4096$1 is the gradient of the loss with respect to logits, then the per-example head gradient has exact Kronecker form

$m=4096$2

Equivalently, if $m=4096$3 is the head matrix, then $m=4096$4, and vectorization yields the Kronecker product (Sweeney, 25 Jun 2026).

This exact structure is what makes the method possible. Direct Fisher methods are otherwise prohibitive because the head gradient lives in dimension $m=4096$5 and the head Fisher block is $m=4096$6. Even storing only the output-side error second moment

$m=4096$7

requires a $m=4096$8 matrix, which at $m=4096$9 costs about $Z_i, Z_j$0 GiB per task in float32 (Sweeney, 25 Jun 2026). FisherSketch circumvents this by never materializing either the full Fisher matrix or $Z_i, Z_j$1.

3. Kernel mean embedding formulation

The paper’s key theoretical result is that head Fisher alignment is exactly a cosine between kernel mean embeddings in joint activation–error space (Sweeney, 25 Jun 2026). Starting from two independent draws $Z_i, Z_j$2 and $Z_i, Z_j$3, the Kronecker identity implies

$Z_i, Z_j$4

Therefore

$Z_i, Z_j$5

The paper defines the factor kernels

$Z_i, Z_j$6

and the product kernel

$Z_i, Z_j$7

Using symmetric vectorization,

$Z_i, Z_j$8

so that

$Z_i, Z_j$9

For task $$A_{\mathrm{F}}^{\mathrm{head}(i,j)}=0,\qquad A_{\mathrm{F}}^{\mathrm{head}(i,i)}=1,$$0, define the uncentered kernel mean embedding

$$A_{\mathrm{F}}^{\mathrm{head}(i,j)}=0,\qquad A_{\mathrm{F}}^{\mathrm{head}(i,i)}=1,$$1

Then Theorem 3 gives

$$A_{\mathrm{F}}^{\mathrm{head}(i,j)}=0,\qquad A_{\mathrm{F}}^{\mathrm{head}(i,i)}=1,$$2

and hence

$$A_{\mathrm{F}}^{\mathrm{head}(i,j)}=0,\qquad A_{\mathrm{F}}^{\mathrm{head}(i,i)}=1,$$3

The identity is exact in the paper’s head setting: a common output basis, an affine head, and the usual loss-derived output error $$A_{\mathrm{F}}^{\mathrm{head}(i,j)}=0,\qquad A_{\mathrm{F}}^{\mathrm{head}(i,i)}=1,$$4 (Sweeney, 25 Jun 2026).

This formulation also yields a decomposition into activation geometry, error geometry, and their coupling. The marginal embeddings are

$$A_{\mathrm{F}}^{\mathrm{head}(i,j)}=0,\qquad A_{\mathrm{F}}^{\mathrm{head}(i,i)}=1,$$5

with

$$A_{\mathrm{F}}^{\mathrm{head}(i,j)}=0,\qquad A_{\mathrm{F}}^{\mathrm{head}(i,i)}=1,$$6

If the joint embedding were separable, then head Fisher alignment would factor into activation and error cosines. In general it need not be separable, so the paper defines the non-separability ratio

$$A_{\mathrm{F}}^{\mathrm{head}(i,j)}=0,\qquad A_{\mathrm{F}}^{\mathrm{head}(i,i)}=1,$$7

with the convention $$A_{\mathrm{F}}^{\mathrm{head}(i,j)}=0,\qquad A_{\mathrm{F}}^{\mathrm{head}(i,i)}=1,$$8 when the denominator is zero. This yields the exact decomposition

$$A_{\mathrm{F}}^{\mathrm{head}(i,j)}=0,\qquad A_{\mathrm{F}}^{\mathrm{head}(i,i)}=1,$$9

The paper uses this to interpret task similarity in terms of activation structure, output-error structure, and activation–error coupling (Sweeney, 25 Jun 2026).

4. FisherSketch algorithm and computational structure

FisherSketch approximates the mean embedding $Z_i=Z_j$0 directly in a low-dimensional random-feature space using factored Random Maclaurin features (Sweeney, 25 Jun 2026). For one coordinate, sample independent Rademacher vectors

$Z_i=Z_j$1

Define

$Z_i=Z_j$2

The paper proves

$Z_i=Z_j$3

With $Z_i=Z_j$4 i.i.d. coordinates,

$Z_i=Z_j$5

and

$Z_i=Z_j$6

Thus $Z_i=Z_j$7 is an unbiased random-feature map for the product kernel (Sweeney, 25 Jun 2026).

The per-task streaming estimator is then

$Z_i=Z_j$8

and head Fisher alignment is estimated by cosine: $Z_i=Z_j$9

The factorization is essential for efficiency. A naive sketch on the full $k$0-dimensional head gradient would cost $k$1 per sample. FisherSketch computes activation and error projections separately, reducing this to $k$2 for dense Random Maclaurin. For vocabulary-scale experiments, the paper replaces dense error-side projections with an SRHT backend, padding $k$3 to the next power of two, applying random sign flips and Walsh–Hadamard transforms, and then subsampling. This reduces the error-side cost to

$k$4

and yields total per-sample cost

$k$5

when activations use dense projections (Sweeney, 25 Jun 2026).

The method is explicitly a single-pass streaming estimator. At $k$6, the stored task signature $k$7 in float32 costs

$k$8

and the split-half streaming state used for stable estimation of joint, activation, and error terms costs

$k$9

If $$g^{(k)}_\theta(x,y) := \nabla_\theta \ell^{(k)}_\theta(x,y),$$0 and $$g^{(k)}_\theta(x,y) := \nabla_\theta \ell^{(k)}_\theta(x,y),$$1 are also stored, persistent storage becomes 48 KB per task (Sweeney, 25 Jun 2026).

The paper also uses the term task signature for the $$g^{(k)}_\theta(x,y) := \nabla_\theta \ell^{(k)}_\theta(x,y),$$2-normalized FisherSketch descriptor. In retrieval use, pairwise alignment reduces to an $$g^{(k)}_\theta(x,y) := \nabla_\theta \ell^{(k)}_\theta(x,y),$$3 dot product between signatures (Sweeney, 25 Jun 2026). This suggests that the method functions as a portable per-task update-geometry fingerprint rather than a model-dependent dense statistic.

5. Statistical properties and estimator stabilization

For the unnormalized kernel estimate

$$g^{(k)}_\theta(x,y) := \nabla_\theta \ell^{(k)}_\theta(x,y),$$4

FisherSketch provides an unbiased estimator: $$g^{(k)}_\theta(x,y) := \nabla_\theta \ell^{(k)}_\theta(x,y),$$5 Its variance is bounded by

$$g^{(k)}_\theta(x,y) := \nabla_\theta \ell^{(k)}_\theta(x,y),$$6

Thus sketching error decays as $$g^{(k)}_\theta(x,y) := \nabla_\theta \ell^{(k)}_\theta(x,y),$$7 for fixed sample pairs, while Monte Carlo error in the sample mean embedding decays as $$g^{(k)}_\theta(x,y) := \nabla_\theta \ell^{(k)}_\theta(x,y),$$8 with the number of task samples $$g^{(k)}_\theta(x,y) := \nabla_\theta \ell^{(k)}_\theta(x,y),$$9 (Sweeney, 25 Jun 2026).

The paper stresses, however, that normalized cosines are plug-in ratio estimates and can be noisier than unnormalized inner products. This is presented as one reason why $$\mathcal{F}_k(\theta_k) := \mathbb{E}_{(x,y)\sim \mathcal{D}_k} \big[g^{(k)}_{\theta_k}(x,y)\,g^{(k)}_{\theta_k}(x,y)^\top \big].$$0 is harder to estimate accurately than quantities such as $$\mathcal{F}_k(\theta_k) := \mathbb{E}_{(x,y)\sim \mathcal{D}_k} \big[g^{(k)}_{\theta_k}(x,y)\,g^{(k)}_{\theta_k}(x,y)^\top \big].$$1, $$\mathcal{F}_k(\theta_k) := \mathbb{E}_{(x,y)\sim \mathcal{D}_k} \big[g^{(k)}_{\theta_k}(x,y)\,g^{(k)}_{\theta_k}(x,y)^\top \big].$$2, or $$\mathcal{F}_k(\theta_k) := \mathbb{E}_{(x,y)\sim \mathcal{D}_k} \big[g^{(k)}_{\theta_k}(x,y)\,g^{(k)}_{\theta_k}(x,y)^\top \big].$$3 (Sweeney, 25 Jun 2026).

To stabilize these ratio estimates, the implementation uses split-half log-ratio estimation, U-statistic diagonal correction, positivity clamping, and empirical-Bayes shrinkage on $$\mathcal{F}_k(\theta_k) := \mathbb{E}_{(x,y)\sim \mathcal{D}_k} \big[g^{(k)}_{\theta_k}(x,y)\,g^{(k)}_{\theta_k}(x,y)^\top \big].$$4. The shrinkage rule is

$$\mathcal{F}_k(\theta_k) := \mathbb{E}_{(x,y)\sim \mathcal{D}_k} \big[g^{(k)}_{\theta_k}(x,y)\,g^{(k)}_{\theta_k}(x,y)^\top \big].$$5

$$\mathcal{F}_k(\theta_k) := \mathbb{E}_{(x,y)\sim \mathcal{D}_k} \big[g^{(k)}_{\theta_k}(x,y)\,g^{(k)}_{\theta_k}(x,y)^\top \big].$$6

This shrinks noisy coupling estimates toward $$\mathcal{F}_k(\theta_k) := \mathbb{E}_{(x,y)\sim \mathcal{D}_k} \big[g^{(k)}_{\theta_k}(x,y)\,g^{(k)}_{\theta_k}(x,y)^\top \big].$$7, כלומר the separable proxy, in a conservative way (Sweeney, 25 Jun 2026).

The paper’s guidance on sketch dimension is pragmatic: larger $$\mathcal{F}_k(\theta_k) := \mathbb{E}_{(x,y)\sim \mathcal{D}_k} \big[g^{(k)}_{\theta_k}(x,y)\,g^{(k)}_{\theta_k}(x,y)^\top \big].$$8 improves quality linearly in compute and memory cost, and $$\mathcal{F}_k(\theta_k) := \mathbb{E}_{(x,y)\sim \mathcal{D}_k} \big[g^{(k)}_{\theta_k}(x,y)\,g^{(k)}_{\theta_k}(x,y)^\top \big].$$9 is the main operating point because it preserves rankings well while keeping signatures small (Sweeney, 25 Jun 2026).

6. Empirical validation and operating regimes

The paper validates FisherSketch at several levels (Sweeney, 25 Jun 2026). On moderate-$i,j$0 ViT-B/16 head-Fisher validation, at $i,j$1, Spearman correlations with exact computation are reported as

$i,j$2

and

$i,j$3

The paper attributes the lower $i,j$4 performance to ratio normalization. Timing benchmarks on synthetic tasks show linear scaling and an $i,j$5 speedup over exact $i,j$6 computation at $i,j$7, $i,j$8 (Sweeney, 25 Jun 2026).

For shared-parameter layers, the paper extends the approach to sketch the exact shared-layer Fisher kernel rather than relying on the no-sharing head factorization. At $i,j$9, Spearman rank preservation against exact Fisher is reported as $$A_{\mathrm{F}(i,j)} := \frac{\langle \mathcal{F}_i, \mathcal{F}_j\rangle_F}{\|\mathcal{F}_i\|_F\,\|\mathcal{F}_j\|_F} = S_{\mathrm{cov}(\mathcal{F}_i,\mathcal{F}_j)}.$$0–$$A_{\mathrm{F}(i,j)} := \frac{\langle \mathcal{F}_i, \mathcal{F}_j\rangle_F}{\|\mathcal{F}_i\|_F\,\|\mathcal{F}_j\|_F} = S_{\mathrm{cov}(\mathcal{F}_i,\mathcal{F}_j)}.$$1 for ResNet-18 convolution layers and $$A_{\mathrm{F}(i,j)} := \frac{\langle \mathcal{F}_i, \mathcal{F}_j\rangle_F}{\|\mathcal{F}_i\|_F\,\|\mathcal{F}_j\|_F} = S_{\mathrm{cov}(\mathcal{F}_i,\mathcal{F}_j)}.$$2–$$A_{\mathrm{F}(i,j)} := \frac{\langle \mathcal{F}_i, \mathcal{F}_j\rangle_F}{\|\mathcal{F}_i\|_F\,\|\mathcal{F}_j\|_F} = S_{\mathrm{cov}(\mathcal{F}_i,\mathcal{F}_j)}.$$3 for ViT-B/16 MLP/QKV layers (Sweeney, 25 Jun 2026). This does not redefine FisherSketch, but it shows that the broader sketching program extends beyond the affine-head setting.

The paper’s central application is Llama-3.1-8B with $$A_{\mathrm{F}(i,j)} := \frac{\langle \mathcal{F}_i, \mathcal{F}_j\rangle_F}{\|\mathcal{F}_i\|_F\,\|\mathcal{F}_j\|_F} = S_{\mathrm{cov}(\mathcal{F}_i,\mathcal{F}_j)}.$$4, 100 domains, and 24 source candidates per target for training-free source selection under LoRA adaptation. Using SRHT on the error side, FisherSketch achieves

$$A_{\mathrm{F}(i,j)} := \frac{\langle \mathcal{F}_i, \mathcal{F}_j\rangle_F}{\|\mathcal{F}_i\|_F\,\|\mathcal{F}_j\|_F} = S_{\mathrm{cov}(\mathcal{F}_i,\mathcal{F}_j)}.$$5

with

$$A_{\mathrm{F}(i,j)} := \frac{\langle \mathcal{F}_i, \mathcal{F}_j\rangle_F}{\|\mathcal{F}_i\|_F\,\|\mathcal{F}_j\|_F} = S_{\mathrm{cov}(\mathcal{F}_i,\mathcal{F}_j)}.$$6

Per-target Spearman with normalized transfer is $$A_{\mathrm{F}(i,j)} := \frac{\langle \mathcal{F}_i, \mathcal{F}_j\rangle_F}{\|\mathcal{F}_i\|_F\,\|\mathcal{F}_j\|_F} = S_{\mathrm{cov}(\mathcal{F}_i,\mathcal{F}_j)}.$$7, and sketch stability across SRHT seeds is high, with off-diagonal Spearman $$A_{\mathrm{F}(i,j)} := \frac{\langle \mathcal{F}_i, \mathcal{F}_j\rangle_F}{\|\mathcal{F}_i\|_F\,\|\mathcal{F}_j\|_F} = S_{\mathrm{cov}(\mathcal{F}_i,\mathcal{F}_j)}.$$8 (Sweeney, 25 Jun 2026).

A particularly important comparison concerns activation-only similarity. On these natural domain shifts, activation geometry $$A_{\mathrm{F}(i,j)} := \frac{\langle \mathcal{F}_i, \mathcal{F}_j\rangle_F}{\|\mathcal{F}_i\|_F\,\|\mathcal{F}_j\|_F} = S_{\mathrm{cov}(\mathcal{F}_i,\mathcal{F}_j)}.$$9 is reported as slightly higher in Top-1 at $m=4096$00 with similar regret, but the paper explicitly states that this is not the decisive regime for the method (Sweeney, 25 Jun 2026). The purpose of FisherSketch is to remain informative when activation similarity is misleading or flat.

That regime is provided by the verbalizer-shift stress test. With a fixed prompt prefix and only the final label token changed, the answer-position hidden state is unchanged by causality, so

$m=4096$01

for all verbalizer pairs. Activation-only similarity therefore becomes constant and useless for ranking. In this setting, across 9 runs, FisherSketch obtains

$m=4096$02

while activation-only collapses to the random baseline at

$m=4096$03

The error-only marginal $m=4096$04 is even stronger in this experiment, with

$m=4096$05

(Sweeney, 25 Jun 2026). This is the paper’s most direct empirical demonstration of the non-identifiability theorem.

A proof-of-concept experiment on nine molecular SMILES domains further reports that FisherSketch correlates with cross-domain perplexity reduction at

$m=4096$06

while activation-only is not significant,

$m=4096$07

(Sweeney, 25 Jun 2026).

7. Relation to other “FisherSketch” usages and broader context

The term “FisherSketch” is used literally in the LLM source-selection paper, but several other recent and earlier lines of work are conceptually adjacent.

One distinct line concerns Fisher-information-inspired sketching for cardinality estimation. “Information Theoretic Limits of Cardinality Estimation: Fisher Meets Shannon” defines the Fisher-Shannon number

$m=4096$08

as a measure of bits per unit normalized Fisher information for mergeable sketches, and introduces Fishmonger, a smoothed, entropy-compressed $m=4096$09-PCSA-based sketch with asymptotically unbiased MLE and standard error $m=4096$10 using

$m=4096$11

(Pettie et al., 2020). This work does not define FisherSketch, but it establishes a different sense in which Fisher information can organize sketch design.

A second adjacent line is TopoFisher, which learns topological summaries for simulation-based inference by maximizing a local Gaussian Fisher objective,

$m=4096$12

through a differentiable persistent-homology pipeline (Biagetti et al., 8 May 2026). The method is not a streaming task-signature sketch, but it embodies a broader “FisherSketch” principle: learning compact summaries $m=4096$13 that preserve parameter-relevant Fisher information.

A third related usage appears in randomized linear algebra for Regularized Fisher Discriminant Analysis. “Randomized Iterative Algorithms for Fisher Discriminant Analysis” develops an iterative sketched approximation to the RFDA operator

$m=4096$14

with geometric error contraction under ridge-aware or leverage-based structural conditions (Chowdhury et al., 2018). Again, the term FisherSketch is not used, but the method is a sketching algorithm for a Fisher-derived operator.

These connections indicate that “FisherSketch” has at least three distinct meanings in current research practice: a literal update-geometry sketch for shared-vocabulary LLM transfer (Sweeney, 25 Jun 2026), a Fisher-information accounting framework for streaming sketches (Pettie et al., 2020), and a broader pattern of Fisher-preserving compression in summary learning or discriminant analysis (Biagetti et al., 8 May 2026, Chowdhury et al., 2018). In contemporary usage, however, the capitalized term FisherSketch most specifically denotes the head-Fisher-alignment estimator of (Sweeney, 25 Jun 2026).

8. Limitations, assumptions, and interpretation

FisherSketch is conditional on a shared output basis. The paper makes clear that quantities such as $m=4096$15 are meaningful only when tasks share output coordinates, for example through a common vocabulary or aligned label taxonomy (Sweeney, 25 Jun 2026). If tasks do not share such coordinates, head error geometry is not directly comparable. This restricts the method’s direct applicability.

The main theory is checkpoint-conditional because it uses empirical Fisher at a fixed model state (Sweeney, 25 Jun 2026). Thus FisherSketch measures compatibility of tasks relative to a specific frozen model, not an invariant property of the tasks themselves.

The unnormalized random-feature estimates are unbiased, but the normalized cosine estimator can be noisy at small $m=4096$16 or low $m=4096$17 (Sweeney, 25 Jun 2026). The paper therefore recommends diagnostics, stabilization procedures, and, for new architectures, reporting broader full-network quantities such as profile cosine and off-diagonal discrepancy.

The method is also deliberately specialized to head empirical Fisher alignment. A plausible implication is that its strongest theoretical guarantees hold precisely where the final affine head provides the relevant aligned coordinate system. The shared-parameter extensions in the paper demonstrate broader feasibility, but the exact activation–error product identity is tied to the affine head case (Sweeney, 25 Jun 2026).

Finally, FisherSketch is not intended to replace representation metrics universally. Rather, it is designed for regimes where representation similarity is insufficient because the determinant of transfer lies in output-side error geometry or in activation–error coupling. The verbalizer-shift results are the paper’s clearest evidence for this claim (Sweeney, 25 Jun 2026).

9. Significance

FisherSketch reframes a seemingly intractable second-order transfer criterion as a compact, streaming kernel mean embedding problem (Sweeney, 25 Jun 2026). Its main significance lies in three linked claims. First, it provides a formal negative result against representation-only transfer metrics in shared-output LLM families. Second, it shows that head Fisher alignment can be computed without materializing Fisher matrices, because it is exactly a cosine between joint activation–error mean embeddings. Third, it operationalizes that insight with a sketch whose storage and streaming footprint are small enough to function as a portable task signature.

In that sense, FisherSketch is not merely a compression device. It is a specific answer to the question of what should be compared when two tasks share a model vocabulary but differ in predictive target structure: not just what activations they produce, but what updates they would induce. The method’s defining object is therefore not a representation embedding, but an update-geometry signature (Sweeney, 25 Jun 2026).