Manifold Anchor Dimension: A Local Measure

• The manifold anchor dimension is a defined local measure of intrinsic dataset dimension via tangent space rank at anchor points.
• SVD, CA-PCA, and kNN estimators effectively determine anchor dimension, aiding model selection and data analysis.
• Accurate anchor dimension estimation addresses high-dimensional data challenges, enhancing neural representation insights.

The manifold anchor dimension is a principled local measure of the intrinsic dimension of a high-dimensional dataset, defined through the rank of the tangent space at a fixed anchor point and operationalized via singular value, PCA, or kNN-based estimators. This notion is central both to manifold learning theory and to practical analysis of neural representations, as it captures the number of independent directions along which a system's state or data point can be infinitesimally varied while remaining on or near the data manifold. The accurate determination of local dimension—especially in regimes where the ambient dimension is high, curvature is present, or manifold structure is only locally approximated—is essential for model selection, data compression, and understanding the geometry of learned representations (Zhang et al., 2017, Gilbert et al., 2023, Benkő et al., 2020).

1. Formal Definition and Local Estimation

Given a data set $X \subset \mathbb{R}^D$ lying near a (possibly curved) smooth $d$-dimensional manifold $\mathcal{M} \subset \mathbb{R}^D$, the anchor dimension at a point $x_0 \in X$ is the smallest integer $d$ such that, in a sufficiently small neighborhood of $x_0$, the data is well-approximated by a $d$-dimensional affine subspace (tangent plane or local graph). More concretely, for a set of $n$ points $\{x_i\}_{i=1}^n$ comprising carefully constructed local perturbations or neighborhood samples around $x_0$:

• The anchor dimension $d$ is the minimum $d$ such that $\{x_i\}$ admits a rank-$d$ approximation.
• In SVD-based methods, this is the index of the largest spectral ratio $\sigma_d / \sigma_{d+1}$ in the singular spectrum of the centered neighborhood data matrix, where $\sigma_j$ denote singular values (Zhang et al., 2017).

This geometric concept captures the true degrees of freedom underlying the local data manifold, independent of the ambient space's dimension.

2. Methodologies for Estimating Manifold Anchor Dimension

2.1 SVD/Local PCA-Based Estimation

One operationalizes the notion of anchor dimension using local SVD or PCA. For data $$A = [f_\ell(x_1), ..., f_\ell(x_n)] \in \mathbb{R}^{D \times n}$$ around anchor $x_0$ in a deep neural network layer, the rank is the number of singular values $\sigma_j$ above a specified threshold, typically found as the first sharp ratio drop:

• $\hat{d} = \arg\max_j \frac{\sigma_j}{\sigma_{j+1}}$ (Zhang et al., 2017).

This method assumes that the manifold is locally flat; in practice, eigenvalue magnitudes of the covariance matrix for neighborhood points provide the relevant information.

2.2 Curvature-Adjusted PCA (CA-PCA)

For manifolds with non-negligible curvature, standard local PCA systematically overestimates local dimension as small eigenvalues persist due to curvature-induced variance. CA-PCA corrects for this by calibrating the eigenvalue spectrum against a quadratic embedding rather than a flat ball. Around each anchor:

• The covariance matrix is normalized by the neighborhood radius squared.
• Curvature-adjusted corrections $c(d) \cdot \sum_{j>d} \lambda_j$ are applied, where $\lambda_j$ are ordered eigenvalues and $c(d)$ is a function of candidate $d$.
• The selected $d$ minimizes a joint $\ell_2 + \ell_1$ criterion over the spectrum, making the procedure robust to curvature, noise, and modest neighborhood size (Gilbert et al., 2023).

2.3 Manifold-Adaptive kNN-Based Estimators (FSA)

The FSA methodology provides a closed-form local estimator relying only on kNN distances around the anchor:

• $d_i = \ln 2 / \ln (R_{2k}(x_i) / R_k(x_i))$, where $R_k$ is the $k$-th nearest neighbor distance.
• The sample median of $\{d_i\}$ across the dataset yields a global estimate; this is guaranteed to be asymptotically unbiased under local uniformity (Benkő et al., 2020).
• Exponential corrections for boundary and finite-sample effects improve estimator fidelity further.

3. Properties and Theoretical Advantages

The manifold anchor dimension is sensitive to the true tangent structure at each anchor, distinguishing intrinsic variation from noise and curvature. CA-PCA in particular achieves higher theoretical and empirical accuracy by:

• Allowing larger neighborhood radii without sacrificing dimension estimation accuracy, owing to explicit curvature correction.
• Extending the theoretical validity of calibration from $O(r^2 \Lambda^2)$ (flat) to $O(r^4 \Lambda^4)$ (quadratic), yielding robust estimates when sample size is limited or curvature moderate (Gilbert et al., 2023).

FSA-based estimators are robust to local density fluctuations and, when using the median, shield global estimation from outliers. The maximum-likelihood FSA variant further refines global estimates under i.i.d.~assumptions, while offering closed-form likelihood for the sample.

4. Empirical Evaluation and Practical Insights

Empirical studies underscore the rapid decline in anchor dimension with network depth in CNNs. Across VGG19, measurements for ImageNet classes (Persian Cat, Container Ship, Volcano) show that:

• Conv5 and fully-connected layers shrink the local dimension from $\mathcal{O}(10^5)$ to $\mathcal{O}(10^3)$ (Zhang et al., 2017).
• Manifold dimension is largely category-independent within a given layer, implying class-agnostic compression.
• Deeper pooling layers halve the intrinsic dimension successively.

On synthetic geometric manifolds (e.g., Klein bottle, spheres, $SO(5)$), CA-PCA converges to ground-truth $d$ with fewer samples and greater resistance to overestimation at large neighborhood sizes. In vision manifolds and face-illumination datasets, CA-PCA consistently stabilizes near the correct parameter count, whereas standard PCA overshoots (Gilbert et al., 2023). FSA-based corrected-median estimators achieve competitive or better mean percentage error and error rates versus state-of-the-art alternatives such as Levina–Bickel MLE and DANCo (Benkő et al., 2020).

5. Known Limitations and Corrections

Standard local PCA and SVD-based anchor dimension estimation are biased under curvature and suffer in high noise or boundary conditions. Principal sources of error include:

• Persistent small eigenvalues attributable to curvature, not noise.
• Systematic underestimation due to finite-sample or boundary truncation effects—addressed in FSA via exponential correction fit on synthetic data.

CA-PCA and FSA-corrected approaches explicitly model and subtract curvature- and boundary-induced bias, while maintaining interpretability and computational efficiency.

6. Broader Implications and Applications

Accurate quantification of manifold anchor dimension informs several domains:

• Model selection and capacity control for dimensionality reduction pipelines and generative models.
• Interpreting deep network representations, revealing the contraction of degrees of freedom and the selectivity of internal feature hierarchies.
• Data compression, as the anchor dimension profile exposes redundant directions in high-dimensional representations.
• Analyzing neural dynamics, with lower-dimensional activations correlating to potential causal sources (e.g., seizure onset zones) (Benkő et al., 2020).

The local anchor approach provides a rigorous foundation for probing both theoretical and practical aspects of data geometry across settings where high-dimensional, nonlinear, and curved manifolds arise.

---

Key References:

Approach / Paper	Core Contribution	arXiv ID
SVD/Anchor Methods	Local tangent estimation by spectral drop criterion	(Zhang et al., 2017)
CA-PCA	Curvature-aware, quadratic calibration for dimension estimation	(Gilbert et al., 2023)
FSA (Median-corrected)	kNN-based, sampling-distribution-derived, robust estimators	(Benkő et al., 2020)