Patnaik-Pearson intrinsic dimension for internal representations of neural networks

Abstract: We define a new measure of intrinsic dimension of a data manifold, which we call the Patnaik-Pearson dimension, and apply this to internal representations of neural networks, in particular transformers. The inspiration for this comes from the HTSR and SETOL work of Martin, Mahoney and Hinrichs, combined with the TwoNN intrinsic dimension estimator of Facco et al. We prove various properties of this intrinsic dimension estimator. Treating weight matrices of neural networks as data manifolds, for weight matrices whose Empirical Spectral Density follows a Pareto (Power Law) distribution, we relate the Patnaik-Pearson dimension to the HTSR and SETOL analysis, and show that critical values of the tail exponent coincide for the two approaches. Using a combination of theoretical and numerical techniques, we study the behaviour of the Patnaik-Pearson dimension of a data manifold under the transformations typical to neural networks. We apply this machinery to the BERT-base and DeepSeek-R1-Distill-Qwen-1 models, to investigate first the Patnaik-Pearson dimension of the initial data manifold of token embeddings, and second the evolution of the Patnaik-Pearson dimension as token embeddings pass through the layers of the model. Code and notebooks used for the numerical results presented here is available at https://github.com/tdhadfield/PatnaikPearson

• The paper introduces the PP intrinsic dimension as a novel estimator that quantifies the global manifold dimensions in neural network representations and is robust against local noise.
• It rigorously analyzes the estimator's behavior under various neural network transformations, demonstrating phase transitions linked to heavy-tailed spectral distributions and layerwise compression.
• Empirical evaluations in models like BERT validate its use for diagnosing overfitting, guiding architectural design, and monitoring training dynamics.

Patnaik-Pearson Intrinsic Dimension for Internal Representations of Neural Networks

Introduction and Context

This paper introduces the Patnaik-Pearson (PP) intrinsic dimension, a novel estimator for the intrinsic dimension of data manifolds, with particular application to analyzing internal representations in neural networks, especially transformer architectures (2606.19268). It builds upon insights from heavy-tailed spectral analysis of weight matrices in deep networks (HTSR, SETOL) and connects foundational ideas with practical numerical estimators such as TwoNN. The author rigorously investigates the mathematical properties of PP dimension, its relationship to spectral densities (particularly Pareto distributions), and presents both theoretical and empirical analyses of its behavior under neural network operations.

Definition and Mathematical Foundation

The PP dimension arises from the moment-matching approach for generalised chi-squared random variables, originally developed by Patnaik and Pearson. Given a matrix $X$ (e.g., token embeddings or weight matrix) with singular values $\lambda_i$, the PP dimension is defined as:

$\PatnaikPearson(X) = \nu(\blambda) = \frac{(\sum_{i=1}^d \lambda_i)^2}{\sum_{i=1}^d \lambda_i^2}$

where $d$ is the embedding dimension. The PP dimension is shown to satisfy normalization and invariance under orthogonal transformations. It is also robust to row/column permutations and scaling, providing a global measurement of dimensionality, distinct from stable rank and local measures such as TwoNN.

Empirical comparisons reveal that PP dimension is typically less sensitive to local geometry and instead captures the global spread of the data manifold. For instance, in $\RR^2$, TwoNN yields 1.942 versus 1.514 for PP, indicating PP's tendency to underestimate local dimension but more reliably track global structure.

Figure 1: 1000 points in $\RR^2$ display a PP dimension of 1.514 vs. TwoNN dimension of 1.942, highlighting the distinction between global and local dimensionality.

Spectral Density, Heavy Tailed Distributions, and Intrinsic Dimension

An extensive theoretical analysis is presented for matrices whose singular values obey distributions commonly encountered in neural networks: uniform, Marchenko-Pastur, and Pareto (power-law). For the latter, which is characteristic of well-trained NN weight matrices (see HTSR/SETOL), the PP dimension is shown to relate directly to the tail exponent $\alpha$:

• For $\alpha > 2$, $\frac{1}{d}\nu(\alpha,d)$ approaches a simple function of $\alpha$ (denoted $\lambda_i$0).
• For $\lambda_i$1, PP dimension sharply decays, converging to zero in the infinite limit. Critical values of $\lambda_i$2 in PP dimension correspond precisely to phase boundaries discovered in HTSR—particularly $\lambda_i$3 delineates optimal training and overfitting.

Figure 2: PP and TwoNN dimension estimates for solid balls of increasing dimension in $\lambda_i$4, demonstrating scalability and robustness of PP dimension under high ambient dimensionality.

Empirical validation is provided for random matrices of large scale, confirming convergence rates and dependence on tail exponents.

Behavior Under Common Neural Network Operations

The paper undertakes comprehensive analysis of the PP dimension under a suite of transformations relevant to neural architectures:

• Matrix multiplication: The "Product Hypothesis" conjectures tight bounds for the dimension of $\lambda_i$5 in terms of the normalized product and minimum of input dimensions, holding under general position.
• Activation (ReLU): Empirically, ReLU reduces normalized PP dimension as a sublinear power-law function, $\lambda_i$6, where $\lambda_i$7 is the normalized input dimension.

Figure 3: ReLU substantially compresses PP dimension, following a sharp nonlinear transformation.

• Addition, Interpolation, Concatenation: Heavy-tailed input dominates the resulting dimension in all such aggregations, with the PP dimension skewed toward the lower input dimension (i.e., more "heavy-tailed").
• Normalisation: Slight increase in PP dimension, especially for smaller input dimension.
• Attention and Softmax: Multihead attention composition is well-modeled by the product hypothesis, and softmax operation behaves as an additive perturbation for low input dimension but is largely invariant for high input dimension.

Application to Transformer Model Embeddings

The PP dimension is empirically evaluated for token embeddings and their layerwise evolution in BERT-base and DeepSeek-R1-Distill-Qwen-1.5B models.

• BERT: For sampled token sets of size between 100 and 10,000, the normalized PP dimension grows from 0.11 to 0.85, indicating increasing manifold occupancy with larger sample sizes.

Figure 4: PP dimension and normalized $\lambda_i$8 for sampled BERT token embeddings show substantial growth with increased sample size.

• Layerwise evolution: For BERT-base, sampled token embeddings (size 512) exhibit monotonic decrease in PP dimension from ~380 to ~120 across model depth, confirming earlier findings of dimension compression in deep architectures.

Figure 5: Progressive layerwise reduction in PP dimension within BERT reflects manifold compression.

• DeepSeek: With substantially larger hidden size and vocabulary, sampled token collections similarly display growth in PP dimension with sample size and marked compression in early layers, followed by partial expansion in later layers.

Figure 6: DeepSeek embeddings undergo sharp PP dimension compression in initial layers, plateau, and subsequent expansion.

Implications and Theoretical Significance

The PP dimension provides a mathematically principled, computationally tractable measure of intrinsic dimension in neural data manifolds, robustly linked to heavy-tailed spectral phenomena ubiquitous in deep learning. Its phase transition properties echo the HTSR/SETOL framework, supporting the notion that heavy-tailedness governs the effective degrees of freedom utilized by neural nets. The global nature of PP dimension, in contrast with local estimators, offers practical advantages for large-scale analysis and tracking of layerwise information compression or expansion.

Applications include:

• Diagnostics of overfitting versus optimal learning via assessment of eigenvalue tail exponents and dimension transitions.
• Architecture-level insights regarding information bottlenecks and redundancy.
• Potential in monitoring training dynamics and transition points (e.g., "grokking").

Further research is warranted in extending PP dimension estimators to more diverse distributions, more general architectures, and integrating with advanced algebraic/topological techniques for manifold exploration.

Conclusion

The Patnaik-Pearson intrinsic dimension is rigorously formalized, shown to possess strong theoretical properties, and empirically validated on transformer-based models. Its close alignment with established heavy-tailed theories situates it as a valuable tool for analyzing information flow and dimensionality in high-capacity neural networks. The observed layerwise decrease in dimension corroborates manifold compression in deep networks, and the product hypothesis provides a foundation for understanding compositional effects. Future developments may include broader application in training dynamics, architectural design, and intrinsic dimension estimation across the spectrum of deep learning models.