---
title: Intrinsic Dimension of LLM Representations
url: https://www.emergentmind.com/topics/intrinsic-dimension-id-of-llm-representations
type: topic
---

# Intrinsic Dimension of LLM Representations

Intrinsic dimension (ID) quantifies the minimal number of degrees of freedom required to describe the distribution of LLM hidden representations, independent of their extrinsic dimensionality. For modern transformers, ID offers a geometric lens into the internal structure of learned embeddings, characterizing the local or global manifold on which activations lie. Empirical investigations reveal that, while layer hidden states may live in thousands of ambient dimensions, their manifold dimension is consistently much lower (typically O(10)–O(100)), and highly dependent on model architecture, training dynamics, data domain, and cognitive or linguistic task. ID serves as a core diagnostic for probing representational complexity, redundancy, adaptation under fine-tuning or in-context learning, and privacy risk.

## 1. Formal Definition and Mathematical Estimators

The ID of a dataset $\mathcal D \subset \mathbb{R}^D$ is defined as the manifold dimension $d$ satisfying that $\mathcal D$ lies (approximately) on a $d$-dimensional submanifold embedded in $\mathbb{R}^D$ [2511.20315]. Most practical estimators are rooted in local geometry—specifically the scaling of neighborhood distances or angles—under Poisson or isotropy assumptions. Key estimators include:

- **Maximum-Likelihood Estimator (MLE)** (Levina & Bickel): For each point $x$ with $k$ nearest neighbors at distances $(d_1, \ldots, d_k)$,
  \[
  \mathrm{LID}_k(x) = \left[ \frac{1}{k-1} \sum_{i=1}^{k-1} \ln \frac{d_k}{d_i} \right]^{-1}
  \]
  and average over points for global ID [2503.02142].

- **TwoNN Estimator** (Facco et al.): Compute the ratio $\mu_i = d_2/d_1$ for each point and solve $\Pr(\mu \leq t) = 1 - t^{-d}$ via CDF fitting. An ML solution is
  \[
  \hat{d} = \left( \frac{1}{N} \sum_{i=1}^N \ln \mu_i \right)^{-1}
  \]
  This is commonly used due to its minimal hyperparameters ($k=2$) and computational efficiency [2506.09591, 2511.15210, 2412.06245, 2601.03779].

- **Angle-Based ABID**: For $k$-nearest neighbor point clouds, let $c_{ij} = \langle v_i, v_j \rangle$ be cosine similarities of normalized displacement vectors. The empirical second moment,
  \[
  \widehat{m_2}^{(1)} = \frac{1}{k^2} \sum_{i,j=1}^k c_{ij}^2
  \]
  gives the ABID estimator:
  \[
  \hat{d}_{\mathrm{ABID}} = \frac{k^2}{\sum_{i,j} c_{ij}^2}
  \]
  This estimator directly links second-order cosine moments to intrinsic dimension (ABID ≤ affinely spanned dimension) [2006.12880].

- **Spectral/Ritz–Chebyshev**: For large samples, the dimension $d$ required to explain a target variance $t_v$ is estimated by counting leading covariance eigenvalues via Ritz values and Chebyshev projectors without eigendecomposition [2503.09485].

Persistent homology, tight local estimators, and GRIDE (n1, n2)-neighbor generalizations supplement this toolkit depending on geometry and sample regime [2511.15210, 2511.20315].

## 2. Practical Computation on LLM Representations

ID is most often estimated on the following objects:

1. **Embedding matrices** (token/word type): Rows are token vectors from the learned embedding matrix.
2. **Hidden states** (contextualized, per-layer): For fixed-length sequences, extract and aggregate last-token or per-position hidden activations layerwise.

Estimation procedures follow three broad steps:

- **Neighbor search:** For each point in the cloud (token or hidden state), find its $k$ nearest neighbors. For high-dimensional LLM embeddings, FAISS or other approximate methods are recommended [2006.12880, 2503.02142]. For ABID, points are typically pre-normalized for cosine geometry.
- **Estimator computation:** Distances or angles among neighbors are used per the above estimator formulas.
- **Aggregation:** Local ID estimates are averaged (arithmetic or harmonic mean) over all samples, or stratified by layer, data domain, or other metadata.

Computational complexity is $O(n\,k\,D)$ to $O(n\,k^2\,D)$ per point, where $n$ is the number of samples, $k$ is the neighborhood size, and $D$ is representation dimension.

For global estimates (entire embedding layer or layerwise hidden states), $k=5$–$50$ is common, with stability checks for $k$ [2511.20315, 2503.02142]. For robust local ID via ABID, $k\sim 10\times$ expected ID is recommended [2006.12880]. For spectral (variance) estimators, matrix–vector products scale linearly in $ND$ and are highly parallelizable [2503.09485].

## 3. ID in LLM Geometry: Empirical Findings

### Redundancy and Compression

Across embedding layers from Word2Vec (ED=300) to Pythia-12B (ED=5120), observed ID is dramatically lower: e.g., GloVe and Word2Vec have ID ≈ 25 (≈8.3% of ED), and large LLMs with ED ≈ 2000–5000 exhibit ID ≈ 25–120 (redundancy >97%) [2503.02142]. During training, ID rapidly collapses from ED to its final value within the first $10^4$ steps, after which minor refinements occur (“neural collapse”) [2503.02142]. Low ID manifests as an embedding manifold of surprisingly few semantic directions despite nominal parameter scale.

In MCQA and sequence modeling tasks, all tested LLMs display a universal “ID hunchback”: early layers encode with low ID, mid-layers expand and peak (signaling maximum abstraction/complexity), and final layers sharply compress before output [2511.20315, 2601.03779]. Peak ID values are 25–100 versus ambient dimensions of thousands.

### Domain and Text-Type Structure

ID robustly stratifies by textual genre and function: scientific/technical corpus (arXiv, PubMed) manifests low hidden state ID (~8), general Wikipedia and news are intermediate (~9), and creative/narrative domains (stories, opinion) have high ID (~10.5) [2511.15210]. These effects persist across models and ID estimators.

Steering experiments with sparse autoencoders show that formally “scientific” feature directions produce minor reductions in ID, while narrative/personalization directions increase ID, confirming causality [2511.15210].

### Task, Linguistic, and Learning Paradigm Dependence

In controlled syntax experiments, ID scales strongly with formal syntactic complexity (e.g., subordinated > coordinated sentences by ΔID ~5–20), aligning with mid-layer abstraction peaks, but less so with functional or semantic contrasts [2601.03779]. Information-theoretic entropy and ID are complementary: after normalizing for length, they are uncorrelated, as ID measures geometric, not prediction, complexity [2511.15210].

Supervised fine-tuning (SFT) versus in-context learning (ICL): SFT compresses and regularizes representation manifolds (lowers ID), while increasing demonstration count in ICL raises mid- and late-layer ID, plateauing after a few examples [2412.06245]. High ID under ICL suggests richer, more entangled representations, whereas SFT aligns representations more tightly to labels.

## 4. Interpretations and Implications

**Model Efficiency and Compression:** ID quantifies representational redundancy. Selecting LoRA or other low-rank adaptation ranks near ID yields substantial compression without appreciable loss in perplexity [2503.02142]. A low ID signals that most model capacity is “reserve,” only a small subspace is operationally used after training [2503.02142].

**Complexity and Privacy:** ID acts as a suppressor of memorization. High-ID sequences in activation space are less likely to be memorized and reproduced by models, especially in low-duplication regimes. Low-ID (boilerplate or stereotypical) sequences are highly at risk for memorization, especially as model scale increases [2506.09591]. This positions ID as a diagnostic for privacy leak risk.

**Cognitive and Linguistic Probes:** The alignment of ID peaks and complexity markers with formal syntactic operations, and the universality of these signatures across architectures and data conditions, reinforces the utility of ID as a domain-general probe of abstraction and structural computation [2511.20315, 2601.03779].

**Quality Control and Diagnostics:** Extremely low or high ID can signal degenerate (looping) or incoherent generation, respectively, serving as a filter for output audits [2511.15210].

## 5. Limitations, Best Practices, and Future Directions

**Estimator Assumptions:** Distance-based estimators assume locally homogeneous Poisson processes or isotropic geometry, which may break down in highly anisotropic LLM embeddings. Angle-based (ABID) estimators are more robust to ambient noise and cluster boundaries but still presuppose local uniformity [2006.12880]. Piecewise manifold geometries require local or clusterwise estimation.

**Hyperparameter Sensitivity:** Choice of $k$ and neighborhood selection affect local and global ID stability. Recommended practice is to verify ID plateauing as $k$ is varied [2511.20315, 2006.12880]. For persistent homology and spectral methods, sample size and rank thresholds must be set based on data variance and task [2511.15210, 2503.09485].

**Application Scope:** ID is not a universal “difficulty” metric: its proper interpretation is always relative to domain, text type, and downstream metrics, and should be supplemented with entropy and anisotropy [2511.15210]. For short texts ($<$150 tokens), variance in ID estimates is elevated.

**Emergent Directions:** Active research is extending ID analysis to training checkpoints (studying “when” and “where” low-dimensional manifolds form), model scale (beyond 100B parameters and Mixture-of-Experts), cross-linguistic variation, topological characterizations beyond manifold dimension, and the mechanistic connection to attention and circuit-level organization [2601.03779, 2511.20315, 2511.15210]. The relationship between ID plateaus/compression phases and decisiveness under different architectural or learning interventions remains a major focus.

## 6. Summary Table: Representative ID Estimation Methods and Empirical Trends in LLMs

| Estimator                | Input                  | Core Formula / Principle                                           | Typical ID Range (LLMs)  |
|--------------------------|------------------------|---------------------------------------------------------------------|--------------------------|
| MLE (Levina–Bickel)      | Embeddings, hidden     | Local log-ratio of neighbor distances, mean/harmonic mean global    | 20–120 (embedding), 30–100 (hidden) [2503.02142, 2511.20315] |
| TwoNN (Facco)            | Embeddings, hidden     | Pareto of 2nd/1st NN; CDF fit                                      | 8–30 (per-sequence), 20–100 (layerwise) [2506.09591, 2412.06245] |
| ABID (Angle-based)       | Embeddings, hidden     | Second moment of k-NN cosine similarities; $k^2/\sum c_{ij}^2$      | 5–50 (typical local ID for LLM) [2006.12880]                   |
| Persistent Homology/PHD  | Embeddings, hidden     | Scaling of homology count / minimal spanning tree length            | 7–13 (document); stratifies genre [2511.15210]                 |
| Ritz–Chebyshev Spectral  | Embeddings, hidden     | Eigenvalue count for $\geq t_v$ variance by mat–vec products        | O(10)–O(100); high efficiency [2503.09485]                     |

Empirical trends include: (i) ID $\ll$ extrinsic dimension throughout; (ii) high redundancy in large models; (iii) domain and genre stratification; (iv) expansion–compression ID profiles per layer; (v) higher ID during ICL vs SFT; (vi) low ID as a marker of memorization and privacy risk.

## 7. References

- E. Thordsen & E. Schubert, “ABID: Angle Based Intrinsic Dimensionality” [2006.12880]
- V. Kataiwa et al., “Measuring Intrinsic Dimension of Token Embeddings” [2503.02142]
- O. Arnold et al., “Memorization in Language Models through the Lens of Intrinsic Dimension” [2506.09591]
- Pedashenko et al., “Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story” [2511.15210]
- Bozhanova et al., “A Comparative Study of Learning Paradigms in Large Language Models via Intrinsic Dimension” [2412.06245]
- D. Denti et al., “Geometry of Decision Making in Language Models” [2511.20315]
- “Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations” [2601.03779]
- S. Ozturk, “A Novel Approach for Intrinsic Dimension Estimation” [2503.09485]

Source: https://www.emergentmind.com/topics/intrinsic-dimension-id-of-llm-representations