IdEst: Assessing Self-Supervised Learning Representations via Intrinsic Dimension

Abstract: Self-supervised learning (SSL) has emerged as a powerful paradigm for learning meaningful representations from unlabeled data. However, the standard protocol for evaluating these representations, linear probing, is computationally expensive, sensitive to hyperparameters, and provides limited insight into the geometric structure of the representation space. In this work, motivated by connections between neural network generalization and intrinsic dimension (ID) we propose IdEst, a method for estimating the ID of SSL representations via the Minimum Spanning Tree dimension estimator ($\mathrm{dim}_\mathrm{MST}$). Across diverse datasets, architectures, and SSL pretraining objectives, we show that IdEst strongly correlates with downstream linear probe performances. Furthermore, we demonstrate that IdEst enables efficient hyperparameter selection, significantly reducing the computational cost compared to supervised alternatives. Our results highlight intrinsic dimensionality as a principled geometric proxy for assessing SSL representations, complementing standard supervised probing protocols.

• The paper introduces IdEst, a novel MST-based intrinsic dimension estimator to assess SSL representations, establishing a link between lower ID and reduced generalization error.
• It demonstrates that MST-based estimation outperforms traditional methods like MLE and TwoNN in stability and computational efficiency across diverse SSL models.
• Empirical studies reveal that as SSL training progresses, MST-ID decreases consistently, proving useful for unsupervised hyperparameter selection and performance evaluation.

Intrinsic Dimension Estimation as an Unsupervised Assessment of Self-Supervised Representations

Motivation and Theoretical Foundations

Self-supervised learning (SSL) has demonstrated efficiency in extracting meaningful representations from unlabeled data by leveraging geometric constraints (alignment, uniformity, variance) rather than explicit class boundaries. Standard evaluation protocols, such as linear probing, are computationally costly, sensitive to hyperparameters, and agnostic to the underlying geometry of the learned space. Motivated by connections between intrinsic dimension (ID) and generalization capacity, this paper introduces IdEst, a method for evaluating SSL representations via the Minimum Spanning Tree (MST) intrinsic dimension estimator, denoted as $\mathrm{dim}_\mathrm{MST}$ (2606.03338).

Theoretical analysis establishes a monotonic relationship between ID and generalization error: for a representation lying on a $d$-dimensional manifold, the classification error converges as $\mathcal{O}(N_D^{-1/d})$ for dataset size $N_D$, as per Lipschitz bounds [konzeffect2024]. Thus, lower ID implies more efficient, compact representations, echoing the manifold hypothesis.

Intrinsic Dimension Estimation: Challenges and Methodology

Estimating ID is nontrivial due to various formalizations—topological, fractal, information-theoretic—and finite-sample limitations. Parametric estimators like TwoNN [facco2017estimating] and MLE [levina2004maximum] assume local isotropy and independence, becoming unstable in high-dimensional, structured or low-sample regimes typical of SSL. The paper rigorously demonstrates their failure even on regular manifolds (e.g., divergence on a 1D helix in $\mathbb{R}^3$) and in presence of dependencies (Figure 1).

Figure 1: MST-based estimation converges to the true intrinsic dimension with increasing sample size, whereas MLE and TwoNN remain unreliable or diverge.

MST-based estimators leverage scaling behavior of minimal spanning trees, relating MST length growth to R\'enyi entropy and providing robustness to noise, density variations, and high ambient dimension [costaDeterminingIntrinsicDimension2006]. The practical algorithm computes the MST length over subsampled representation points and extracts the ID by log-log linear regression, offering provable consistency under weak assumptions.

Comprehensive Evaluation of IdEst

IdEst is applied across 33 foundation models covering ResNet/ViT backbones and 14 SSL objectives (joint-embedding, joint-predictive, vision-language alignment). For each, MST-ID is computed on frozen representations post-standard evaluation (CLS token or averaged patch tokens). Relating MST-ID to linear probe accuracy, extensive empirical studies document robust negative correlation for intra-dataset (ImageNet, iNat-18, iNat-21, SUN397) and inter-dataset settings (ImageNet to CIFAR-10/100), as quantified by Kendall's $\tau \approx -0.6$ and Spearman's $\rho \approx -0.8$.

Figure 2: Correlation between linear probe accuracy and intrinsic dimension across four benchmarks, demonstrating strong negative monotonic relationship.

Furthermore, the predictive signal persists under alternative evaluation protocols (kNN, ImageNet-v2), suggesting that MST-ID captures geometric representation quality agnostic to downstream task or protocol.

Figure 3: Intrinsic dimension remains indicative of model performance when computed on reference datasets or under different evaluation protocols.

Training Dynamics and Model Selection

Tracking MST-ID throughout training reveals that as SSL models improve during offline (periodic checkpoint) or online (joint classifier-head) probing, ID consistently decreases, mirroring gains in linear probe accuracy. This behavior is observed across objectives (VICReg, DINO, I-JEPA) and architectures, evidencing that MST-ID is highly sensitive to the evolving geometric organization of representations.

Figure 4: Evolution of linear probe accuracy and MST-ID for VICReg over training epochs; MST-ID drops as accuracy increases.

The utility of MST-ID extends to unsupervised hyperparameter selection. Empirical comparison with supervised baselines (ImageNet oracle), unsupervised metrics ($\alpha$-ReQ [alpha2022Agrawal], RankMe [garrido2023rankme]), and weakly-supervised LiDAR [thilaklidar] shows that MST-ID reliably selects top-performing configurations across learning rate, regularization, loss-specific parameters, and masking ratios, yielding performance near the supervised upper bound even in the absence of labels.

Computational Efficiency

IdEst incurs significantly lower computational cost compared to linear probing, especially for large-scale models, as MST-ID only requires a single feature extraction pass and MST computation. Empirical timing on H100 GPUs and large ViTs document compute time reductions exceeding order-of-magnitude.

Comparative Analysis with Other Metrics

Rank-based and uniformity metrics (RankMe, $\alpha$-ReQ, uniformity, Wasserstein distance) demonstrate weaker correlation with downstream accuracy, particularly across heterogeneous architectures. RankMe is bounded by output dimension and fails in architecture-comparison contexts. MST-ID, measuring true manifold dimension, proves more robust and generalizable.

Figure 5: RankMe shows weaker correlation with probe accuracy across diverse models compared to MST-ID.

Effective vs. Intrinsic Dimension

The distinction between effective rank (spread across linear dimensions) and intrinsic dimension (minimum degrees of freedom) is articulated via geometric visualization. High-quality SSL representations tend to compress information onto low-ID manifolds while distributing it uniformly (high effective rank), avoiding collapse.

Figure 6: Helix example—embedded in high ambient dimension, but intrinsically a 1D manifold; illustrates gap between effective and intrinsic dimension.

Figure 7: During training, MST-ID decreases while RankMe increases, reflecting compression and uniform spreading, respectively.

Limitations and Future Directions

While MST-ID exhibits strong global ranking capacity, the convergence bound only acts as a performance indicator rather than exact predictor. Difficulty arises when model accuracies cluster or in vision-LLMs prone to modality-induced geometric misalignment. Early training epochs of ViTs lack stable manifold structure, diminishing metric informativeness.

Further research should explore curvature-aware dimensional metrics, combine intrinsic and effective dimension analyses, and integrate MST-ID into SSL objectives for geometry-aware learning. Extensions to dense tasks (segmentation, generation) are feasible given the task-agnostic estimator foundation.

Conclusion

IdEst establishes minimum spanning tree intrinsic dimension as a robust, unsupervised proxy for representation quality in SSL, strongly correlating with linear probe accuracy and enabling efficient hyperparameter selection. Its geometric sensitivity, label-independence, and computational advantage advocate for the adoption of intrinsic dimension metrics as complementary or alternative criteria in mechanistic evaluation and optimization of large-scale self-supervised systems (2606.03338).