IsoScore: High-Dimensional Isotropy Metric

• IsoScore is a scalar metric that measures how uniformly a point cloud utilizes its vector space by analyzing the distribution of covariance eigenvalues.
• It is rotation- and scale-invariant, outperforming traditional metrics like cosine similarity and partition scores in capturing isotropy.
• IsoScore underpins advances in contextualized language models, clustering tasks, and molecular embeddings, and is integral to differentiable regularization methods such as I-STAR.

IsoScore is a scalar metric that quantifies the degree to which a set of vectors (“point cloud”) uniformly utilizes its ambient vector space, through the spectral properties of its covariance matrix. It underpins recent advances in the quantitative study of isotropy in high-dimensional data representations, outperforming alternative metrics in both theoretical rigor and empirical utility. IsoScore has gained adoption as the standard for isotropy measurement in contextualized LLMs, clustering tasks, and molecular embeddings, and is foundational to differentiable regularization approaches such as I-STAR.

1. Mathematical Definition and Core Properties

For a dataset $X = \{x_1, \ldots, x_m\} \subset \mathbb{R}^n$, the sample covariance matrix is

$$\Sigma = \frac{1}{m}\sum_{i=1}^m (x_i - \bar x)(x_i - \bar x)^\top,$$

with $\bar x$ the empirical mean. Denoting the eigenvalues of $\Sigma$ (principal component variances) as $$\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_n \geq 0$$, IsoScore $\iota(X)$ is derived by measuring the deviation of the normalized eigenvalues from uniformity. The canonical formula is

$$\iota(X) = 1 - \frac{\sum_{i=1}^n \left| \frac{\lambda_i}{\sum_j \lambda_j} - \frac{1}{n} \right|}{2\left( 1 - \frac{1}{n} \right)},$$

where $\iota(X) \in [0, 1]$, with $\iota=1$ for perfectly isotropic (uniform-variance) data, and $\iota=0$ if all variance falls in a single direction (Rudman et al., 2021, Hakim et al., 12 Oct 2025). An equivalent formulation, central to differentiable variants, is based on the “isotropy defect” $$\Sigma = \frac{1}{m}\sum_{i=1}^m (x_i - \bar x)(x_i - \bar x)^\top,$$0: $$\Sigma = \frac{1}{m}\sum_{i=1}^m (x_i - \bar x)(x_i - \bar x)^\top,$$1 and

$$\Sigma = \frac{1}{m}\sum_{i=1}^m (x_i - \bar x)(x_i - \bar x)^\top,$$2

where $$\Sigma = \frac{1}{m}\sum_{i=1}^m (x_i - \bar x)(x_i - \bar x)^\top,$$3 is the vector of eigenvalues (Rudman et al., 2021).

IsoScore is rotation- and scale-invariant, purely a function of the covariance eigenspectrum. If exactly $$\Sigma = \frac{1}{m}\sum_{i=1}^m (x_i - \bar x)(x_i - \bar x)^\top,$$4 out of $$\Sigma = \frac{1}{m}\sum_{i=1}^m (x_i - \bar x)(x_i - \bar x)^\top,$$5 coordinates have nonzero and equal variance, then $$\Sigma = \frac{1}{m}\sum_{i=1}^m (x_i - \bar x)(x_i - \bar x)^\top,$$6; i.e., IsoScore scales linearly with effective dimension utilization.

2. Comparison with Other Isotropy Metrics

Previous isotropy proxies include average random cosine similarity, partition scores, and explained-variance ratios. Empirical and theoretical analysis demonstrates fundamental flaws in these alternatives:

• Average Random Cosine Similarity: Sensitive to data mean; zero-centering can report high isotropy even for data lying on a line (Rudman et al., 2021).
• Partition Score: Admits basis-dependent approximations, is numerically unstable, and fails several invariance properties (Rudman et al., 2021).
• Variance Explored Ratio & Intrinsic Dimension: Require ad hoc parameter choices or mischaracterize variance structure.

In a suite of six validation tests—including mean invariance, scalar invariance, monotonicity for maximally anisotropic cases, and correct scaling with fraction of dimensions used—only IsoScore satisfies all essential axioms of a rotation-invariant global isotropy measure (Rudman et al., 2021).

3. Efficient Computation and Algorithmic Implementation

IsoScore is computed as follows:

1. Center data (subtract mean vector).
2. Compute sample covariance matrix $$\Sigma = \frac{1}{m}\sum_{i=1}^m (x_i - \bar x)(x_i - \bar x)^\top,$$7.
3. Eigendecompose $$\Sigma = \frac{1}{m}\sum_{i=1}^m (x_i - \bar x)(x_i - \bar x)^\top,$$8 to obtain eigenvalues $$\Sigma = \frac{1}{m}\sum_{i=1}^m (x_i - \bar x)(x_i - \bar x)^\top,$$9.
4. Compute normalized variances $\bar x$0.
5. Evaluate IsoScore via the core formula.

The computational complexity is dominated by covariance computation ($\bar x$1 for $\bar x$2 points in $\bar x$3 dimensions) and eigendecomposition ($\bar x$4). For very high $\bar x$5, incremental PCA or randomized SVD are essential (Rudman et al., 2021, Durdy et al., 2023).

Python implementations are provided in the open-source “IsoScore” and “Isotropy_measurements” repositories, offering both clusterwise and global computations, as well as API and CLI support (Durdy et al., 2023).

4. IsoScore Variants: Exponential-Moment and Differentiable Extensions

IsoScore admits several extensions for specialized evaluation and differentiable regularization:

• Exponential-Moment IsoScore (Cluster Isotropy): For a cluster $\bar x$6 in $\bar x$7, define for any unit vector $\bar x$8,

$\bar x$9

with mean radius $\Sigma$0. The isotropy ratio is

$\Sigma$1

Approximations are performed over principal axes or random directions. For sets of clusters, a weighted mean yields global isotropy (Durdy et al., 2023).

• IsoScore⋆ (Differentiable, Shrunk Covariance): Designed for stable estimation and differentiability on minibatches. The shrunk covariance

$\Sigma$2

where $\Sigma$3 is a held-out or accumulated covariance, is used for the standard IsoScore calculation, ensuring numerical stability and enabling backpropagation (Rudman et al., 2023).

IsoScore⋆ is core to the I-STAR regularization method, where it constrains or encourages embedding isotropy as a penalty term during neural network training (Rudman et al., 2023).

5. Empirical Applications and Insights

IsoScore and its variants have been applied to a diverse array of settings:

Protein LLMs

IsoScore reveals that sequence-only models such as ProtBERT and ProtXLNet are highly anisotropic, typically utilizing only 2–14 out of 1024 dimensions. In contrast, multimodal ProteinBERT (sequence + gene ontology) achieves much higher isotropy, using up to 120 of 512 dimensions, suggesting that isotropization via biological supervision can enhance embedding space expressiveness (Hakim et al., 12 Oct 2025).

Clustering Analyses in Materials Science

In “Metrics for quantifying isotropy in high dimensional unsupervised clustering tasks,” IsoScore and related metrics (fractional anisotropy, variance of normalized eigenvalues) quantify the impact of representation and kernel choices in clustering the Inorganic Crystal Structure Database. Post-RBF kernel approximation, dense “magpie” representations show dramatic isotropy increases (e.g., $\Sigma$4 from 0.18 to 0.993), while sparse “compVec” representations are far less affected. For MNIST embeddings, variational autoencoders produce more isotropic clusters than standard autoencoders, despite reduced class separability (Durdy et al., 2023).

Contextualized LLM Embeddings

Application of IsoScore to deep contextualized models (BERT, GPT-2) shows that in practice, most layers use a tiny fraction ($\Sigma$5 IsoScore, often $\Sigma$6 effective dimension) of the available embedding space. Contrary to earlier reports based on flawed cosine statistics, IsoScore shows no systematic trend toward increased anisotropy in deeper layers (Rudman et al., 2021).

Anisotropy Regularization in Neural Networks

I-STAR leverages IsoScore⋆ as a differentiable, mini-batch-stable metric to penalize or reward isotropy in transformer activations. Empirically, encouraging increased anisotropy (negative penalty) often outperforms both naive baselines and older isotropy-encouraging penalties, and aligns with observed reductions in intrinsic dimension of hidden representations (Rudman et al., 2023).

6. Practical Recommendations and Limitations

• IsoScore is agnostic to input mean and scalar scaling, and should be preferred over cosine or partition-based proxies, which can be artificially manipulated by mean-centering or are numerically unstable.
• For stable, interpretable results, sample sizes for covariance estimation should be much larger than ambient dimension (typically, $\Sigma$7). Employ incremental or streaming methods in memory-constrained settings.
• IsoScore can be interpreted heuristically: $\Sigma$8 in $\Sigma$9 implies $$\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_n \geq 0$$0 dimensions are isotropically occupied, guiding model selection and post-processing.
• For small sample settings, always use IsoScore⋆ (with shrinkage), which robustly estimates isotropy and enables integration into end-to-end differentiable objectives (Rudman et al., 2023).
• In real-world data, high anisotropy indicates under-utilization of embedding capacity and potential representational inefficiency; multi-modal or biologically supervised models can partially mitigate this.

7. Availability and Implementation

IsoScore and its clusterwise and global/generalized variants are available as Python libraries (“IsoScore” and “Isotropy_measurements”), with API and CLI tools, plus example notebooks for major empirical benchmarks (ICSD, MNIST, protein models) (Durdy et al., 2023, Rudman et al., 2021). For differentiable regularization, code is included with I-STAR (Rudman et al., 2023).

IsoScore Variant	Setting / Application	Computational Considerations
Original	Global embedding, clustering, NLP	Moderate—requires full eigendecomp.
Exponential-moment	Clustering, materials science	O($$\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_n \geq 0$$1) for Vec; O($$\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_n \geq 0$$2) for Rnd.
IsoScore⋆	Differentiable ML regularization	Adds shrinkage, efficient for minibatch; autodiff compatible

In summary, IsoScore is a rigorously validated, rotation-invariant, and interpretable metric that serves as the de facto standard for measuring isotropy in high-dimensional unsupervised and representation learning tasks (Rudman et al., 2021, Durdy et al., 2023, Hakim et al., 12 Oct 2025, Rudman et al., 2023).