Spectral Signature Analysis in Transformers

• Spectral Signature Analysis is a method that represents pretrained transformers using power-law tail indices derived from their weight matrices’ empirical spectral densities.
• It leverages heavy-tailed self-regularization theory to capture intrinsic model properties through scale-invariant, shape-based metrics, enabling robust fingerprinting.
• The approach facilitates efficient unsupervised clustering, lineage tracing, and performance prediction while remaining computationally optimized and data-free.

Spectral Signature Analysis is a model-level method for representing and comparing pretrained LLMs by the shape statistics of their layer-wise weight spectra rather than by prompts, activations, or task-specific benchmarks. In the formulation introduced in "Spectral Signatures of LLMs" (Zhang et al., 3 Jul 2026), the spectral signature is the tensor of power-law tail indices estimated from the empirical spectral densities of transformer weight matrices. The method is data-free, computationally-efficient, and scale-invariant, and is used for model lineage tracing, unsupervised clustering, and broad performance quantification across large open-source model collections (Zhang et al., 3 Jul 2026).

1. Formal definition of the signature

In this framework, a spectral signature is a compact representation constructed from the shape of the empirical spectral density (ESD) of each weight matrix in a pretrained transformer. For a model with $L$ transformer blocks and $N_{\text{mod}}$ module types, the signature is

$Z \in \mathbb{R}^{N_{\text{mod}} \times L},$

with entries

$$Z[i,\ell] = \mathrm{PL}(W_{\text{module}=i,\text{layer}=\ell}),$$

where $\mathrm{PL}$ is the layer-wise power-law tail index estimated from the ESD tail (Zhang et al., 3 Jul 2026).

For standard transformers, $N_{\text{mod}} = 7$, corresponding to the attention projections $W_Q$, $W_K$, $W_V$, $W_O$ and the MLP projections $N_{\text{mod}}$0, $N_{\text{mod}}$1, $N_{\text{mod}}$2. Embeddings are typically excluded, biases are not included, and for mixture-of-experts models the per-expert PL values are averaged to obtain a module-level PL. For some analyses, the tensor is flattened to a vector in $N_{\text{mod}}$3, but the tensor structure is retained in the main pipeline because module-wise organization carries discriminative information.

The method is intended to capture intrinsic properties of pretrained models. Derived variants that share a pretrained backbone, including instruction-tuned and RLHF models, retain highly similar signatures. This makes the representation suitable for unsupervised attribution and provenance analysis without prompts or per-pair matrix alignment. The same representation also supports unsupervised clustering and nearest-neighbor retrieval over model collections.

2. Heavy-tailed spectral theory and shape-based metrics

The theoretical basis is Heavy-Tailed Self-Regularization (HT-SR) theory, which interprets the ESDs of trained neural-network weight matrices as signatures of learned correlations, training quality, and generalization (Zhang et al., 3 Jul 2026). For a rectangular weight matrix $N_{\text{mod}}$4, with $N_{\text{mod}}$5 and eigenvalues $N_{\text{mod}}$6, the ESD is

$N_{\text{mod}}$7

The random-matrix baseline is the Marchenko–Pastur distribution,

$N_{\text{mod}}$8

with $N_{\text{mod}}$9 and $Z \in \mathbb{R}^{N_{\text{mod}} \times L},$0. Departures from this baseline, including extended bulk edges, outliers, and heavy tails, are interpreted as evidence of learned correlations. In HT-SR regimes, the ESD tail follows

$Z \in \mathbb{R}^{N_{\text{mod}} \times L},$1

with tail indices typically in the range $Z \in \mathbb{R}^{N_{\text{mod}} \times L},$2.

The spectral signature uses the Hill estimator on the ESD tail, with ascending-sorted eigenvalues $Z \in \mathbb{R}^{N_{\text{mod}} \times L},$3 and default tail size $Z \in \mathbb{R}^{N_{\text{mod}} \times L},$4:

$Z \in \mathbb{R}^{N_{\text{mod}} \times L},$5

This PL statistic is a shape metric rather than a scale metric. The paper contrasts it with spectral norm, log-scaled norm baselines, and combined scale-shape baselines such as Log_Alpha_Norm. Effective rank is discussed as an optional metric but is not part of the signature itself. The central claim is that the heavy-tailed shape of pretrained weights stabilizes early in pretraining and remains largely unchanged during post-training, so the PL tensor can serve as a persistent model fingerprint.

3. Computation, invariances, and similarity measures

The practical pipeline extracts each 2D module weight matrix from every transformer block, computes the eigenvalues of $Z \in \mathbb{R}^{N_{\text{mod}} \times L},$6 or equivalently the singular values of $Z \in \mathbb{R}^{N_{\text{mod}} \times L},$7, estimates the PL value for each module-layer pair, and aggregates the results into $Z \in \mathbb{R}^{N_{\text{mod}} \times L},$8 (Zhang et al., 3 Jul 2026). Exact SVD has complexity $Z \in \mathbb{R}^{N_{\text{mod}} \times L},$9, but approximate randomized eigendecomposition can be used for larger matrices. The default configuration uses raw PL values rather than normalized ones, though z-score normalization is used for depth-mismatched alignment.

Model similarity is defined as the average Spearman rank correlation over module-aligned layer vectors:

$$Z[i,\ell] = \mathrm{PL}(W_{\text{module}=i,\text{layer}=\ell}),$$0

with prediction distance $$Z[i,\ell] = \mathrm{PL}(W_{\text{module}=i,\text{layer}=\ell}),$$1. Alternative kernels tested for classification ablations include an RBF kernel and cosine similarity, but the main pipeline uses module-wise Spearman correlation.

Two invariance properties are central. First, PL is inherently scale-invariant because it depends on ratios of tail eigenvalues. Second, Spearman correlation depends only on relative ordering across layers, making signature comparison insensitive to affine distortions within module columns. The paper further reports robustness under hidden-unit permutations and output-invariant scalings of consecutive maps. For depth-mismatched models, similarity is computed after order-preserving dynamic programming alignment on z-score normalized module columns.

The computational profile is explicitly favorable. On Llama-2-7B with FP16 on a single NVIDIA L40, the reported signature computation time is 337 seconds, compared with 2,633 seconds for EmbedLLM and 5,393 seconds for LLMDNA. This suggests that the method is designed for corpus-scale organization rather than one-off diagnostic use.

4. Stability under post-training and perturbation

A defining empirical result is that spectral signatures remain stable across post-training regimes, including supervised fine-tuning, instruction tuning, RLHF, and domain-shifted variants (Zhang et al., 3 Jul 2026). For Llama-2-7B variants, the average module-wise Spearman similarity is typically $$Z[i,\ell] = \mathrm{PL}(W_{\text{module}=i,\text{layer}=\ell}),$$2–$$Z[i,\ell] = \mathrm{PL}(W_{\text{module}=i,\text{layer}=\ell}),$$3 across instruction-tuned and domain-shifted models. RL-trained models have reported similarities of approximately $$Z[i,\ell] = \mathrm{PL}(W_{\text{module}=i,\text{layer}=\ell}),$$4–$$Z[i,\ell] = \mathrm{PL}(W_{\text{module}=i,\text{layer}=\ell}),$$5 relative to their base backbones.

The paper gives a concrete example of near-perfect spectral overlap between Llama-3.1 and its Tulu-3 post-trained variant, with matched layers showing $$Z[i,\ell] = \mathrm{PL}(W_{\text{module}=i,\text{layer}=\ell}),$$6, while Llama-2 shows clear deviation. It also states that the signature “locks in” after tens of thousands of pretraining steps and changes minimally during post-training. This suggests that the method primarily tracks pretraining identity rather than downstream specialization.

Robustness extends beyond post-training. Under hidden-unit permutations, PL-based similarity remains $$Z[i,\ell] = \mathrm{PL}(W_{\text{module}=i,\text{layer}=\ell}),$$7, whereas PCS collapses to $$Z[i,\ell] = \mathrm{PL}(W_{\text{module}=i,\text{layer}=\ell}),$$8. Under Gaussian noise injection, PL similarity stays high, including reported values of $$Z[i,\ell] = \mathrm{PL}(W_{\text{module}=i,\text{layer}=\ell}),$$9 at $\mathrm{PL}$0 and $\mathrm{PL}$1 at $\mathrm{PL}$2. The paper also reports that output-invariant scalings of consecutive maps leave PL similarity stable, while PCS degrades sharply. These results are used to argue that the signature measures spectral shape rather than coordinate-dependent weight alignment.

A common misconception is to treat such signatures as substitutes for task evaluation. The paper does not make that claim. Its stronger statement is that the signature is a meaningful proxy for broad performance trends and a robust identifier of model lineage, not a deployment-critical replacement for direct benchmarking.

5. Benchmarking results across lineage, clustering, and prediction

The evaluation uses two corpora: a 128-model set derived from 8 families for lineage, classification, and clustering, and a 499-model open-source corpus covering Llama, Mistral, GPT, and Pythia families with parameter counts from 19M to 70B for performance prediction (Zhang et al., 3 Jul 2026). Baselines include data-aware fingerprints such as REEF and Logits, data-free methods such as PCS and GhostSpec, and scale-metric baselines such as Spectral_Norm, Log_Norm, and Log_Alpha_Norm.

The reported results are summarized below.

Task	Result	Setup
8-way family classification	98.44% accuracy	nearest neighbor by signature similarity
Unsupervised clustering	Silhouette 0.91; Davies–Bouldin 0.22	PL signatures
Runtime per model	337s vs 2,633s vs 5,393s	Ours vs EmbedLLM vs LLMDNA
Prediction significance	100/100 Kendall’s Tau trials	ARC, HellaSwag, MMLU, TruthfulQA

For unsupervised clustering, Spectral Clustering on the signature distance matrix achieves silhouette $\mathrm{PL}$3, while HDBSCAN, BGMM, and K-Means are around $\mathrm{PL}$4. Internal metrics are computed in the original signature space to avoid embedding artifacts. The 2D t-SNE projections show clear family separation, but the paper explicitly relies on silhouette and Davies–Bouldin values rather than visualization alone.

For performance prediction, the method uses distance-weighted $\mathrm{PL}$5-NN with $\mathrm{PL}$6, 5-fold cross-validation, and 100 repeated splits. The reported MAEs are competitive with activation-based baselines: ARC $\mathrm{PL}$7 vs $\mathrm{PL}$8 vs $\mathrm{PL}$9, HellaSwag $N_{\text{mod}} = 7$0 vs $N_{\text{mod}} = 7$1 vs $N_{\text{mod}} = 7$2, MMLU $N_{\text{mod}} = 7$3 vs $N_{\text{mod}} = 7$4 vs $N_{\text{mod}} = 7$5, and TruthfulQA $N_{\text{mod}} = 7$6 vs $N_{\text{mod}} = 7$7 vs $N_{\text{mod}} = 7$8 for Ours, EmbedLLM, and LLMDNA, respectively. The main empirical pattern is that the spectral signature is not uniformly best on every benchmark, but it remains competitive while being much cheaper and entirely data-free.

6. Applications, limitations, and reproducibility

The paper presents three main operational uses of spectral signature analysis (Zhang et al., 3 Jul 2026). For organization of large model collections, one computes $N_{\text{mod}} = 7$9 for each model, builds a pairwise similarity matrix using $W_Q$0, and clusters the models to identify families and sublineages. For lineage verification, one computes the query signature and retrieves the nearest known base by maximizing similarity, optionally using order-preserving dynamic programming alignment for depth-mismatched candidates. For performance anticipation, one interpolates benchmark vectors from nearby signatures using distance-weighted $W_Q$1-NN:

$W_Q$2

The method also comes with explicit limitations. Finite-size and rectangular effects influence the ESD bulk, so comparisons should use consistent module definitions and avoid mixing fundamentally different module types or aspect ratios. The Hill estimator depends on the tail fraction $W_Q$3; the paper uses $W_Q$4 and advises against changing it without strong reasons. Module heterogeneity matters: MLP projections often separate independently trained models more strongly than attention projections, and averaging across all modules can dilute discriminative power. Approximate eigendecomposition is appropriate for large matrices, but numerical stability may require double precision for eigenvalue computation on CPU.

These limitations delimit the scope of the method. It is most appropriate when one needs a compact, prompt-free, scale-invariant representation for organizing, comparing, or roughly ranking many pretrained models. It is less appropriate when the objective is fine-grained capability assessment or deployment-critical evaluation, where direct task benchmarks remain necessary.

Reproducibility is supported by an explicit software release at https://github.com/Ingrid-505/Spectral_Signature. The published defaults are per-module per-layer PL via the Hill estimator, tail fraction $W_Q$5, the seven standard transformer modules, average Spearman similarity across module rows, nearest-neighbor classification, Spectral Clustering or related unsupervised pipelines, and distance-weighted $W_Q$6-NN for benchmark interpolation. Within those defaults, Spectral Signature Analysis is presented as a compact spectral-shape formalism for model-level fingerprinting of LLMs.