Spectral Effective-Rank Entropy

• Spectral effective-rank entropy is defined by normalizing spectral values (e.g., singular values or eigenvalues), computing the Shannon entropy, and exponentiating the result to quantify effective dimensions.
• It unifies the analysis of diverse systems—ranging from Vision Transformer representations to graph Laplacians—by measuring how uniformly spectral mass is distributed across components.
• The approach offers actionable insights into model adaptation and training dynamics by linking spectral dispersion to rank allocation and performance optimization.

Spectral effective-rank entropy denotes a family of spectral-complexity functionals built from a normalized spectrum—typically singular values, covariance eigenvalues, or Gibbs weights—and most often takes the Shannon form $H=-\sum_i p_i\log p_i$, with effective rank given by $e^H$. Across recent work, this quantity is used to measure how uniformly spectral mass is distributed across directions, dimensions, or modes: low values indicate concentration in a few dominant components, whereas high values indicate broader spectral utilization. The same idea appears in analyses of Vision Transformer representations, parameter-efficient fine-tuning, Gaussian covariance geometry, graph Laplacians, and random density matrices, although the precise spectral object and normalization vary by domain (Kapil et al., 17 Jun 2026, Yan et al., 31 Aug 2025, Huang et al., 2024).

1. Core definitions and canonical formulas

A common pattern across the literature is to begin with a nonnegative spectrum $\{w_i\}$, normalize it into a probability distribution $p_i=w_i/\sum_j w_j$, compute Shannon entropy, and then exponentiate that entropy to obtain an “effective number” of active dimensions. In "Transformer Geometry Observatory TGO-I: Spectral Geometry Observatory" (Kapil et al., 17 Jun 2026), the formal definition is

$$p_i=\frac{\lambda_i}{\sum_{j=1}^{D}\lambda_j},\qquad H=-\sum_{i=1}^{D}p_i\log p_i,\qquad r_{\mathrm{eff}}=\exp(H),$$

with $\lambda_i$ the eigenvalues of a layer-wise covariance matrix.

In "ER-LoRA: Effective-Rank Guided Adaptation for Weather-Generalized Depth Estimation" (Yan et al., 31 Aug 2025), the same structure is applied to pretrained weight singular values, but with a power parameter $\gamma$: $$p_i=\frac{\sigma_i^\gamma}{\sum_{j=1}^{K}\sigma_j^\gamma},\qquad H=-\sum_{i=1}^{K}p_i\log p_i,\qquad Rank_{\mathrm{en}}(W)=e^H.$$ The paper explicitly interprets this as an entropy-based effective rank derived from spectral dispersion.

In "FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptation" (Liu et al., 30 Jan 2026), the entropy is computed not from raw singular values but from normalized squared singular values,

$$s_i=\frac{\lambda_i^2}{\sum_j \lambda_j^2},\qquad H(s)=-\sum_{i=1}^{r}s_i\log s_i,$$

and then normalized as

$I(A)=\frac{H(A)}{\log r}\in[0,1],$

with the implementation form

$e^H$0

Setting	Normalized spectral weights	Entropy / effective-rank form
ViT covariance geometry	$e^H$1	$e^H$2
ER-LoRA pretrained weights	$e^H$3	$e^H$4
FlexLoRA LoRA updates	$e^H$5	$e^H$6

These formulas are mathematically close but not identical. The spectrum may be an eigenspectrum or singular-value spectrum, and the weights may be linear, powered, or squared. The papers therefore agree on the entropy mechanism while differing on what exactly is being normalized.

2. Spectral objects and normalization choices

The meaning of spectral effective-rank entropy depends on which operator is being analyzed. In TGO-I, the spectrum is the eigenspectrum of the feature covariance matrix

$e^H$7

so the entropy measures dataset-level allocation of variance across representation dimensions rather than weight geometry or per-sample token structure (Kapil et al., 17 Jun 2026). Because the eigenvalues are normalized by total variance, the resulting entropy and effective rank are scale-invariant with respect to uniform rescaling of the covariance spectrum.

In ER-LoRA, the spectrum is taken from pretrained linear-layer weights inside the vision foundation model backbone, and the entropy-rank is computed from the singular values of the frozen pretrained matrix $e^H$8, not from activations (Yan et al., 31 Aug 2025). The paper separates two spectral objects: the pretrained weight $e^H$9, used to compute $\{w_i\}$0 and $\{w_i\}$1, and the full-fine-tuning residual $\{w_i\}$2, used only for task-direction selection in the pretrained singular basis.

In Spectral-GS, the spectrum is the covariance eigenvalue spectrum of a 3D Gaussian: $\{w_i\}$3 or, in the 3D parameterization used there,

$\{w_i\}$4

The entropy is thus a shape descriptor of Gaussian anisotropy rather than a representational-dimension statistic (Huang et al., 2024).

The graph-thermodynamic formulation in "Spectral Entropy via Random Spanning Forests" (Nicolini, 15 Dec 2025) uses a density matrix

$\{w_i\}$5

with von Neumann entropy

$\{w_i\}$6

The paper does not explicitly define effective rank, but this suggests the standard entropy-rank quantity $\{w_i\}$7 for the Gibbs-weighted Laplacian spectrum.

3. Interpretation and neighboring rank surrogates

The central interpretation is uniform across the entropy-based papers: spectral entropy measures dispersion of spectral mass, and effective rank turns that dispersion into an effective dimensionality. TGO-I states this most directly: if all variance lies in one direction, then $\{w_i\}$8 and $\{w_i\}$9; if variance is perfectly uniform across $p_i=w_i/\sum_j w_j$0 nonzero directions, then $p_i=w_i/\sum_j w_j$1 and $p_i=w_i/\sum_j w_j$2. The paper also emphasizes that Spectral Entropy and Effective Rank contain the same ordering information, since $p_i=w_i/\sum_j w_j$3 (Kapil et al., 17 Jun 2026).

This quantity is not the same as algebraic rank, stable rank, or participation ratio. "Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendation" (Li et al., 22 May 2026) explicitly uses the norm-based stable-rank expression

$p_i=w_i/\sum_j w_j$4

and notes that it does not use a Shannon-entropy effective rank. The paper treats this stable rank as a spectral concentration surrogate aligned with representation diversity, but not identical to entropy-based effective rank.

ER-LoRA introduces a second quantity called stable rank,

$p_i=w_i/\sum_j w_j$5

and explicitly notes that this is not the classical $p_i=w_i/\sum_j w_j$6 formula unless $p_i=w_i/\sum_j w_j$7 is chosen appropriately (Yan et al., 31 Aug 2025). The paper uses entropy-rank for adaptation capacity and stable-rank-like concentration for preservation of dominant pretrained directions.

A further limitation is that entropy-based summaries are permutation invariant over the spectrum. "Ranking power spectra: a proof of concept" (Mei et al., 2019) makes this critique explicit for frequency-domain spectra: standard spectral entropy is invariant under any permutation of power estimates on the frequency grid and therefore erases order structure inherent in the spectrum. The paper proposes rank-order descriptors, $p_i=w_i/\sum_j w_j$8 and $p_i=w_i/\sum_j w_j$9, precisely to recover information that Shannon spectral entropy discards.

4. Representation geometry, anisotropy, and training dynamics

TGO-I uses Spectral Entropy and Effective Rank as longitudinal observables of Vision Transformer representation geometry and reports a consistent increase in both quantities across nearly all monitored layers over 100 epochs (Kapil et al., 17 Jun 2026). The paper’s qualitative pattern is internally coherent: patch embedding and positional embedding remain relatively low-dimensional and stable, deeper Transformer layers exhibit progressively larger Effective Rank, and the final CLS representation shows the strongest increase, the highest effective dimensionality, and the lowest anisotropy.

These entropy increases are paired with decreasing Spectral Anisotropy,

$$p_i=\frac{\lambda_i}{\sum_{j=1}^{D}\lambda_j},\qquad H=-\sum_{i=1}^{D}p_i\log p_i,\qquad r_{\mathrm{eff}}=\exp(H),$$0

increasing Participation Ratio,

$$p_i=\frac{\lambda_i}{\sum_{j=1}^{D}\lambda_j},\qquad H=-\sum_{i=1}^{D}p_i\log p_i,\qquad r_{\mathrm{eff}}=\exp(H),$$1

and progressively flatter eigenspectra. The paper interprets this as progressive redistribution of variance across representational dimensions rather than concentration into a few dominant axes.

In Spectral-GS, the same entropy logic is geometric rather than representational. Low spectral entropy corresponds to covariance spectra concentrated in one or a few directions, which the paper associates with needle-like, highly anisotropic Gaussians and high condition number (Huang et al., 2024). High spectral entropy corresponds to more balanced eigenvalues and more isotropic Gaussian shape. For 2D projected covariance, the paper gives an explicit monotone relation between spectral entropy and condition number: $$p_i=\frac{\lambda_i}{\sum_{j=1}^{D}\lambda_j},\qquad H=-\sum_{i=1}^{D}p_i\log p_i,\qquad r_{\mathrm{eff}}=\exp(H),$$2 with

$$p_i=\frac{\lambda_i}{\sum_{j=1}^{D}\lambda_j},\qquad H=-\sum_{i=1}^{D}p_i\log p_i,\qquad r_{\mathrm{eff}}=\exp(H),$$3

This makes spectral effective-rank entropy a whole-spectrum anisotropy measure rather than merely a rank count.

5. Method-defining uses in adaptation and model design

In ER-LoRA, spectral effective-rank entropy is method-defining rather than diagnostic. The paper’s Selecting–Tuning–Maintaining strategy uses entropy-rank to allocate LoRA capacity layer by layer through

$$p_i=\frac{\lambda_i}{\sum_{j=1}^{D}\lambda_j},\qquad H=-\sum_{i=1}^{D}p_i\log p_i,\qquad r_{\mathrm{eff}}=\exp(H),$$4

so layers with flatter pretrained spectra receive larger adaptation ranks, while layers with sharply decaying spectra receive smaller ranks (Yan et al., 31 Aug 2025). The same stage uses the full-fine-tuned residual projected onto the pretrained singular basis,

$$p_i=\frac{\lambda_i}{\sum_{j=1}^{D}\lambda_j},\qquad H=-\sum_{i=1}^{D}p_i\log p_i,\qquad r_{\mathrm{eff}}=\exp(H),$$5

to select the top-$$p_i=\frac{\lambda_i}{\sum_{j=1}^{D}\lambda_j},\qquad H=-\sum_{i=1}^{D}p_i\log p_i,\qquad r_{\mathrm{eff}}=\exp(H),$$6 task-aware singular directions. In the maintaining stage, the stable-rank-like quantity determines how many dominant pretrained directions should be protected by principal-direction regularization. The ablation reported in the paper isolates three modules—$$p_i=\frac{\lambda_i}{\sum_{j=1}^{D}\lambda_j},\qquad H=-\sum_{i=1}^{D}p_i\log p_i,\qquad r_{\mathrm{eff}}=\exp(H),$$7, $$p_i=\frac{\lambda_i}{\sum_{j=1}^{D}\lambda_j},\qquad H=-\sum_{i=1}^{D}p_i\log p_i,\qquad r_{\mathrm{eff}}=\exp(H),$$8, and $$p_i=\frac{\lambda_i}{\sum_{j=1}^{D}\lambda_j},\qquad H=-\sum_{i=1}^{D}p_i\log p_i,\qquad r_{\mathrm{eff}}=\exp(H),$$9—and shows, on nuScenes-night and nuScenes-rain, an AbsRel progression from $\lambda_i$0 with $\lambda_i$1 alone to $\lambda_i$2 after adding task-aware directions, and to $\lambda_i$3 after adding stable-rank-guided preservation.

FlexLoRA uses spectral energy entropy as a matrix-level importance score for dynamic rank reallocation. The paper ranks matrices by

$\lambda_i$4

then prunes the $\lambda_i$5 least important matrices and expands the $\lambda_i$6 most important ones under a global budget (Liu et al., 30 Jan 2026). Low entropy means energy concentrated in a few singular directions and therefore redundancy; high entropy means more balanced spectral energy and therefore richer structural capacity. Expansion uses zero-impact initialization, with the new singular value initialized to zero.

KRAdapter pushes the same theme from a different angle. "Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product" (Albert et al., 1 Aug 2025) defines effective rank in the appendix as

$\lambda_i$7

and argues that what matters in PEFT is not merely algebraic full rank but flatter singular-value distributions and larger tail singular values. The paper explicitly states that not all full-rank PEFT methods are equally effective because some still have low effective rank, and it reports that the Khatri–Rao product yields consistently higher effective rank and more gradual singular-value decay than the Kronecker product.

6. Extensions, exact settings, and persistent caveats

Several additional lines of work broaden the scope of spectral effective-rank entropy without always naming it directly. "Bures-Hall Ensemble: Spectral Densities and Average Entropies" (Sarkar et al., 2019) gives exact formulas for average von Neumann entropy and purity of random density matrices in the fixed-trace Bures–Hall ensemble. Because entropy-based effective rank is naturally $\lambda_i$8, these results provide an exact spectral-statistical foundation for effective-rank estimates on random quantum states.

"Spectral Homogenization of the Radiative Transfer Equation via Low-Rank Tensor Train Decomposition" (Ju, 12 Feb 2026) does not define entropy-based effective rank, but its central empirical finding is that the spectrally homogenized solution tensor has bounded TT rank as spectral resolution increases. This suggests a bounded intrinsic spectral dimension, though the paper measures TT rank and Tucker spectral rank rather than Shannon entropy of singular values.

"Entropy Rate Bounds via Second-Order Statistics" (Tamir, 2022) likewise does not define effective rank, but its PSD-based bound

$\lambda_i$9

shows that spectral spread can also enter through log-determinant or geometric-mean functionals of a power spectrum. This is a spectral entropy-rate setting rather than an effective-rank definition.

The main caveats are consistent across the literature. First, the same label may hide different normalizations: raw singular values, powered singular values, squared singular values, covariance eigenvalues, or Gibbsized Laplacian eigenvalues are not interchangeable. Second, stable-rank terminology is not uniform: ER-LoRA’s $\gamma$0 differs from the most common classical definition unless $\gamma$1 is chosen appropriately (Yan et al., 31 Aug 2025). Third, some methods use spectral effective-rank entropy directly for optimization or rank allocation, whereas others use it only for analysis. Fourth, entropy-based summaries ignore frequency-order structure, which is why rank-order descriptors such as $\gamma$2 and $\gamma$3 were proposed as complementary, permutation-sensitive quantities (Mei et al., 2019).

Taken together, these works establish spectral effective-rank entropy as a precise but family-dependent concept: an entropy of normalized spectral mass that is often re-expressed as an effective number of active dimensions, directions, or modes. Its value lies in converting spectral shape into a scalar with clear geometric meaning, while its limitations arise from the same abstraction—namely, sensitivity to the chosen spectral object and insensitivity to structural information outside the normalized spectrum itself.