Extrinsic Curvature in Neural Manifolds

• Extrinsic curvature is a geometric measure that quantifies how neural manifolds bend in the ambient Euclidean space using the second fundamental form.
• It is computed via methods like PCA-based tangent estimation, quadratic fitting, and stochastic trace estimation, offering insights into network expressivity and regularization.
• Empirical analyses reveal distinct curvature phases across neural network layers, with increased bending near classification layers associated with improved generalization.

Extrinsic curvature in neural manifolds refers to the deviation of a smooth submanifold, defined by neural activations or latent representations, from flatness as measured in the ambient (embedding) Euclidean space. In deep learning, the geometry of such manifolds emerges from the nonlinear transformations performed by neural networks, making extrinsic curvature a quantifiable descriptor of how neural representations are "bent" or "folded" by the network architecture and its training state. This concept is formalized using classical differential geometry, particularly the second fundamental form and the associated principal curvatures, and plays a central role in analyzing network expressivity, regularization, and the relationship between learned representations and generalization.

1. Mathematical Foundations

Let $M \subset \mathbb{R}^D$ be a smooth manifold consisting of neural activations (e.g., the set of latent codes at a given network layer after nonlinearity) with intrinsic dimension $n$. The tangent space $T_x M$ at $x \in M$ is the best linear approximation to $M$ near $x$, while the normal space $N_x M$ is its orthogonal complement in $\mathbb{R}^D$.

The extrinsic curvature at $x$ is encapsulated by the second fundamental form, a symmetric bilinear map

$$\mathrm{II}_x: T_x M \times T_x M \rightarrow N_x M,$$

which measures how the manifold bends in the ambient space. For $u, v \in T_x M$ and a choice of orthonormal normal frame $\{n^\alpha(x)\}$, the scalar components are

$$\mathrm{II}_{ij}^\alpha(x) = \left\langle \partial_i \partial_j \varphi(x) - \Gamma_{ij}^k \partial_k \varphi(x), n^\alpha(x) \right\rangle,$$

where $\Gamma_{ij}^k$ are the Christoffel symbols of the induced metric $g_{ij}(x)$. The shape operator $S_x: T_x M \rightarrow T_x M$ defined by $S_x(u) = -\mathrm{Proj}_{T_x M} (dN_x(u))$ relates to the eigenvalues $\kappa_1, \dots, \kappa_n$—the principal curvatures.

Aggregate curvature measures include:

• Mean curvature: $H(x) = \frac{1}{n} \sum_{i=1}^n \kappa_i(x)$,
• Total curvature-energy: $$\| \mathrm{II}_x \|_F^2 = \sum_{i=1}^n \kappa_i(x)^2$$,
• Average absolute principal curvature: $C(x) = \frac{1}{n} \sum_{i=1}^n |\kappa_i(x)|$.

Coordinate-invariant formulations of extrinsic curvature involve the Dirichlet energy of the Gauss (tangent-space) map on the Grassmannian, as in

$$\mathcal{E}_c(z) = \frac{1}{2} \mathrm{Tr} \left( \sum_{i,j=1}^m (G^{-1})_{ij} \left( \partial_{z^i} T(z) \right)^T \partial_{z^j} T(z) \right),$$

with $G = J_f(z)^T J_f(z)$ (the pullback metric), $J_f(z)$ the Jacobian, and $T(z)$ the tangent-space projector (Lee et al., 2023).

2. Computational Methodologies for Estimating Extrinsic Curvature

Practical estimation of extrinsic curvature in neural manifolds requires methods that operate on finite samples (point clouds) and exploit the Jacobian structure of neural networks.

PCA-based Tangent Space and Quadratic Fitting:

• At a given latent representation $z \in \mathbb{R}^D$, select the $k$ nearest neighbors (usually $k \approx 50$–$100$) from the same data class.
• Center and whiten these neighbors, perform PCA, and retain the top $n$ components to estimate $T_z M$.
• Project residuals onto this tangent plane and fit a quadratic map $r(u) \approx \frac{1}{2} u^T H u$ to estimate the second fundamental form in tangent coordinates.
• Diagonalize $H$ to obtain principal curvatures $\{\kappa_i(z)\}$ (Kaufman et al., 2023).

Stochastic Trace Estimation via Autodiff (Regularization context):

• Avoid computing full Hessians by using Hutchinson’s stochastic trace estimator and efficient Jacobian-vector products (JVPs) and vector-Jacobian products (VJPs), entirely compatible with autodiff frameworks.
• For coordinate-invariant curvature regularization, as in MECAE (Minimum Extrinsic Curvature Autoencoder), the additional computational overhead is $\mathcal{O}(D m)$ per point, with 4–8 JVP/VJPs typically required (Lee et al., 2023).

Curve-based (1D Manifold) Approach:

• For a 1D test manifold (e.g., a circle in input space), pass its parameterization $g(\theta)$ through the network and compute the output curve $h(\theta) = f(g(\theta))$.
• Estimate velocity and acceleration by finite differences and calculate extrinsic curvature via the Frenet–Serret formula: $$\kappa(\theta) = \left( v \cdot v \right)^{-3/2} \sqrt{ (v \cdot v)(a \cdot a) - (v \cdot a)^2 }$$ with $v = \partial_\theta h(\theta)$, $a = \partial^2_\theta h(\theta)$ (Asthana et al., 18 Aug 2025).

Coarse Curvature via Optimal Transport:

• For manifold learning and point clouds, coarse extrinsic curvature can be inferred from the Wasserstein-1 distance $W_1$ between empirical measures on small tubular neighborhoods around $x$ and $x^+$, with normalization against displacement along a chosen tangent: $$\kappa_{\sigma,\epsilon}(x, x^+) = 1 - \frac{W_1( \mu_x^{\sigma, \epsilon}, \mu_{x^+}^{\sigma, \epsilon})}{\|x^+ - x\|}$$ Theoretical expansions connect this quantity to the second fundamental form and mean curvature (Arnaudon et al., 10 Jul 2024).

3. Empirical Profiles and Functional Roles in Deep Networks

Systematic analysis of extrinsic curvature across layers in feedforward convolutional networks reveals a three-phase profile in trained models:

• Initial rise: Curvature increases rapidly from the input to early layers.
• Plateau: Mid-network layers exhibit sustained, high curvature ($\overline{C} \approx 0.06$ for ResNet-50 on CIFAR-10).
• Final upturn: Curvature increases again in the bottleneck and classification head (Kaufman et al., 2023).

In untrained (randomly initialized) networks, curvature monotonically decays towards flatness. Regularization via mixup reduces both overall curvature and the final curvature jump.

Critically, the gap in mean absolute curvature between the last two layers ($\Delta C$) shows strong positive correlation ($\rho \approx 0.76$, $p < 10^{-3}$) with test accuracy in vision tasks: a larger “bend” near the output layer predicts improved generalization.

4. Regularization and Expressivity via Extrinsic Curvature

Extrinsic curvature serves as both a probe and a regularizer of representation quality.

• Regularization: Extrinsic curvature penalties, as in MECAE, flatten the learned manifold in embedding space. Properly weighted, this reduces overfitting on noisy data without collapsing manifold structure. The regularizer is implemented as:

$$L_{\text{MECAE}}(\theta,\phi) = L_{\text{rec}}(\theta,\phi) + \alpha \mathbb{E}_{x}[ \mathcal{E}_c(g_\phi(x); f_\theta) ]$$

and is efficiently differentiated with autodiff (Lee et al., 2023). Comparative studies show both intrinsic (MICAE) and extrinsic (MECAE) curvature regularizers outperform conventional AE/DAE/CAE/IRAE; intrinsic regularization offers slight additional robustness under high noise.

• Expressivity: In zero-shot neural architecture search (NAS), extrinsic curvature of a projected test curve is combined with SVD-based feature map collinearity into a “Dextr” proxy measure. High curvature indicates greater expressivity; high Dextr scores correlate ($>$0.9 Spearman) with downstream test accuracy on NAS benchmarks (Asthana et al., 18 Aug 2025).

5. Invariance, Identifiability, and Algorithmic Considerations

Extrinsic curvature quantification is designed to be invariant under coordinate reparameterizations and (in neuroscientific applications) neuron-index permutations:

• In topologically-aware VAEs, immersion parameterizations are compatible with invariance to latent reparameterizations and ambient axis permutations due to the Euclidean metric (Acosta et al., 2022).
• Estimated curvature magnitudes are robust to architectural choices such as layer width and activation order, provided the embedding remains smooth and sufficiently sampled.

Algorithmic choices involve trade-offs:

• Tangent estimation: PCA-based methods require careful choice of $k$ and local whitening for stability, especially for high dimensions and sparse sampling.
• Quadratic fitting and numerical stability: Ridge-regularization during least-squares estimation of quadratic terms is critical.
• Computational cost: Curvature estimation is feasible for large $D$ due to efficient use of JVP/VJP in modern autodiff libraries, contrasting with the infeasibility of directly forming full Hessians for moderately large $m$ or $D$ (Lee et al., 2023).

6. Relationship to Intrinsic Dimensionality and Point-Cloud Inference

Studies indicate minimal correlation between local intrinsic dimension (as measured via PCA explained variance) and extrinsic curvature: points of high dimension can have low extrinsic bending and vice versa, highlighting their informational independence (Kaufman et al., 2023). This suggests that “flatness” in the embedding space is not a function of representational compactness.

For empirical scientific applications (e.g., neural data analysis and general manifold learning), curvature can be estimated directly from point clouds via covariance expansion (integral invariants) or optimal transport between local empirical measures. These algorithms provide means to infer principal curvatures and mean curvature vectors at scale, given sufficient neighborhood density and sample complexity (Álvarez-Vizoso et al., 2018, Arnaudon et al., 10 Jul 2024).

7. Applications and Implications

Extrinsic curvature analysis has found applications in:

• Deep representation analysis: Revealing characteristic geometric signatures across network depth and connecting geometric events (e.g., last-layer curvature jumps) to generalization performance.
• Regularization of generative models: Encouraging flatter manifolds for denoising and improved latent recovery under noise (Lee et al., 2023).
• Zero-shot architecture search: Label-free proxy evaluation of model expressivity and generalization (Asthana et al., 18 Aug 2025).
• Neural data analysis: Quantitative investigation of the geometry of cognitive representations in high-dimensional neural activity spaces, robust to measurement reordering (Acosta et al., 2022).

A plausible implication is that quantifying and controlling extrinsic curvature provides tools for both interpreting neural network representations and guiding the design of architectures and regularization schemes to optimize expressivity, robustness, and generalization. The uncorrelated nature of intrinsic dimension and extrinsic curvature further suggests the value of combining both types of geometric information for comprehensive manifold analysis in deep learning and neuroscience.