Categorical and geometric methods in statistical, manifold, and machine learning (2505.03862v1)

Abstract: We present and discuss applications of the category of probabilistic morphisms, initially developed in \cite{Le2023}, as well as some geometric methods to several classes of problems in statistical, machine and manifold learning which shall be, along with many other topics, considered in depth in the forthcoming book \cite{LMPT2024}.

• The paper introduces probabilistic morphisms to formalize stochastic relationships in supervised learning and constructs correct loss functions using kernel mean embeddings.
• It proves learnability of overparameterized models by integrating categorical techniques with regularized empirical risk minimization and Lipschitz continuous hypothesis spaces.
• Geometric methods are used to design positive definite kernels for non-Euclidean spaces, enabling analysis of SPD matrices and covariance operators in manifold learning.

This paper explores the application of categorical methods, specifically the category of probabilistic morphisms, and geometric methods to various problems in statistical, machine, and manifold learning. It serves as an overview of topics to be covered in more depth in a forthcoming book [40]. The core idea is that abstract mathematical structures from category theory and differential geometry can provide new theoretical foundations and practical tools for machine learning tasks.

Categorical Methods: Probabilistic Morphisms and Supervised Learning

The paper introduces the category of probabilistic morphisms (denoted Probm), where objects are measurable spaces and morphisms are Markov kernels. A probabilistic morphism $T: X \leadsto Y$ is generated by a measurable map $T: X \to P(Y)$, where $P(Y)$ is the space of probability measures on $Y$. This provides a categorical framework for stochastic relationships between spaces. Key examples include measurable mappings (via the Dirac measure embedding) and regular conditional probability measures.

In statistical learning, particularly supervised learning, the goal is to approximate the relationship between input data $x \in X$ and labels $y \in Y$, often formalized as a conditional probability measure $M_{y|x}: X \to P(Y)$. The paper frames this within the Probm category.

A core concept is the generative model of supervised learning, defined by a tuple $(X, Y, H, R, P_{X \times Y})$, where $H$ is a hypothesis space of potential predictors (often functions from $X$ to $P(Y)$ or a related space), $R$ is a risk/loss function, and $P_{X \times Y}$ is the set of possible joint probability distributions on $X \times Y$. A crucial property is a correct loss function, which has minimizers that correspond precisely to the true regular conditional probability measure $M_{y|x}$ for any distribution $\mu \in P_{X \times Y}$.

The paper demonstrates how probabilistic morphisms and kernel methods can yield correct loss functions. Using the concept of the graph of a probabilistic morphism $h: X \leadsto Y$, denoted $I_h: X \leadsto X \times Y$, which is generated by the map $x \mapsto \delta_x \otimes h(x)$, Theorem 2.6 provides a characterization: $h$ is a regular conditional probability measure for $\mu \in P(X \times Y)$ if and only if $(I_h)_* \mu_X = \mu$, where $\mu_X$ is the marginal measure on $X$ and $(I_h)_*$ is the pushforward operator on measures.

This characterization leads to the construction of a correct loss function using kernel mean embeddings. If $K: (X \times Y) \times (X \times Y) \to \mathbb{R}$ is a measurable positive definite symmetric (PDS) kernel such that the kernel mean embedding map $M_K: P(X \times Y) \to H(K)$ (into the associated RKHS $H(K)$) is an embedding, then the loss function

$R_K(h, \mu) = \|(I_h)_* \mu_X - \mu\|_{H(K)}^2$

is a correct loss function for $h \in \text{Probm}(X, Y)$ and $\mu \in P(X \times Y)$ (Example 2.11). This norm difference measures the distance between the kernel mean embedding of the joint distribution generated by $h$ and the marginal $\mu_X$, and the kernel mean embedding of the true joint distribution $\mu$. Minimizing this loss forces the probabilistic morphism $h$ to match the conditional distribution of $\mu$.

In practice, computing $(I_h)_* \mu_X$ involves integrating $K((x, y), \cdot)$ against $\mu_X$ and $h(x)$, which can be challenging. However, for empirical measures $S_n = \{(x_i, y_i)\}_{i=1}^n \sim \mu$, the empirical loss function using $R_K(h, \mu_{S_n})$ can be minimized. Kernel mean embeddings allow computations involving probability measures to be performed in an RKHS, often relying on Gram matrices, which is computationally feasible.

Learnability of Overparameterized Models

The paper addresses the crucial concept of learnability or generalization ability of a supervised learning model, defined as the existence of a uniformly consistent learning algorithm $A$ that produces hypotheses $A(S_n)$ with estimation error $E_{H,R,\mu}(A(S_n))$ small with high probability for large sample sizes $n$ (Definition 2.14).

A key contribution is proving the learnability of overparameterized supervised learning models by combining categorical methods (probabilistic morphisms, kernel mean embeddings) and geometric properties (Lipschitz spaces, metric spaces). The hypothesis space $H$ is considered a subset of $\text{CLip}(X, P(Y)_{K_2})$, the space of Lipschitz continuous maps from $X$ (equipped with some metric like volume-induced) to $P(Y)$ (equipped with a metric induced by an RKHS norm from kernel $K_2$). The loss function $R_{K_1}$ is the RKHS loss mentioned above, based on a kernel $K_1$ on $X \times Y$.

The proof leverages Vapnik-Stefanyuk's method for solving stochastic ill-posed problems [64], [68], which involves regularized empirical risk minimization (ERM). The idea is that finding the true regular conditional probability measure is an ill-posed problem (small changes in data/empirical measure can lead to large changes in the solution). Regularization helps stabilize the solution. The paper shows that for a specific choice of regularization function $W(h)$ involving norms in function spaces and Lipschitz constants (Lemma 2.21), and under certain compactness conditions on the set of possible data distributions $P_{X \times Y}$ (Condition (L), Proposition 2.18), a regularized ERM algorithm is uniformly consistent (Proposition 2.22). This provides a theoretical basis for why certain function classes (like Lipschitz maps) used in overparameterized models (e.g., neural networks, though not explicitly discussed as such here, the framework applies) can learn effectively under appropriate conditions.

Geometric Methods: Kernels on Non-Euclidean Spaces

The paper explores the challenge and construction of positive definite kernels on spaces that do not have a standard Euclidean vector space or inner product structure, focusing on Symmetric Positive Definite (SPD) matrices $Sym^{++}(n)$ and Positive Definite Hilbert-Schmidt operators $PC_2(H)$. Such kernels are essential for applying kernel methods (like Support Vector Machines, Gaussian Processes, Kernel PCA) to data residing in these spaces, which is common in areas like medical imaging, computer vision, and signal processing where covariance matrices or operators serve as data representations.

The standard Gaussian kernel $K_\gamma(x, y) = \exp(-\gamma d^2(x, y))$ is positive definite on a metric space $(M, d)$ if and only if $d^2$ is negative definite, which in turn happens if and only if $(M, d)$ is isometrically embeddable into a Hilbert space (Theorem 3.2). For Riemannian manifolds, this strong condition means the manifold must be isometric to Euclidean space (Theorem 3.3). This rules out the standard Gaussian kernel based on the geodesic distance for spaces with non-zero curvature.

For SPD matrices $Sym^{++}(n)$, the paper discusses three Riemannian metrics: affine-invariant, Bures-Wasserstein, and Log-Euclidean.

1. Affine-invariant distance $d_{ai}$: Gaussian kernels based on $d_{ai}^2$ are generally not positive definite for $n \ge 2$.
2. Bures-Wasserstein distance $d_{bw}$: Similar to affine-invariant, Gaussian kernels based on $d_{bw}^2$ are generally not positive definite for $n \ge 2$.
3. Log-Euclidean distance $d_{logE}(A, B) = ||\log(A) - \log(B)||_F$. Crucially, $Sym^{++}(n)$ equipped with the Log-Euclidean operations $A \oplus B = \exp(\log(A) + \log(B))$ and $\lambda * A = \exp(\lambda \log(A))$ forms an inner product space with $(A, B)_{logE} = (\log(A), \log(B))_F$. Since the logarithm map $\log: Sym^{++}(n) \to Sym(n)$ is an isometric isomorphism of inner product spaces, positive definite kernels on $Sym^{++}(n)$ can be constructed by applying standard kernel constructions (like Gaussian or polynomial) to the mapped data in $Sym(n)$. For example, $K(A, B) = \exp(-\gamma ||\log(A) - \log(B)||_F^p)$ is positive definite for $\gamma > 0$ and $0 < p \le 2$ (Theorem 3.4). This is highly practical as it translates kernel methods from Euclidean space to $Sym^{++}(n)$ via the matrix logarithm.

The paper generalizes this to Positive Definite Hilbert-Schmidt Operators $PC_2(H)$ on a separable Hilbert space $H$. This requires handling the infinite-dimensional setting where $\log(A)$ may not be bounded and the identity operator is not Hilbert-Schmidt. The solution involves considering unitized operators $A+\gamma I$ and working in the extended Hilbert-Schmidt space $HS_x(H)$ with a modified inner product and norm. $PC_2(H)$ forms a Hilbert space under Log-Hilbert-Schmidt operations $\oplus, *$ and the Log-Hilbert-Schmidt inner product $$((A+\gamma I), (B+\rho I))_{logHS} = (\log(A+\gamma I), \log(B+\rho I))_{HSx}$$ (Theorem 3.6). This again allows constructing positive definite kernels analogous to the Log-Euclidean case (Theorem 3.7).

A particularly relevant practical setting is using RKHS covariance operators as data representations. For a kernel $K$ on a space $X$, a probability measure $\mu$ on $X$ induces a covariance operator $C_\mu \in PC_2(H(K))$. Given empirical data $S_m$, an empirical covariance operator $C_{S_m}$ can be computed (Equation 3.67). The Log-Hilbert-Schmidt distance and inner product between these operators (e.g., $C_{S_{m_1}}+\gamma_1 I$ and $C_{S_{m_2}}+\gamma_2 I$) can be computed efficiently using Gram matrices derived from the kernel $K$ on the original data points (Equation 3.72). This enables the use of kernel methods directly on covariance operator representations. The paper mentions the application in a two-layer kernel machine for image classification, where the first layer computes covariance operators from image features using a kernel $K_1$, and the second layer applies a kernel $K_2$ based on the Log-Hilbert-Schmidt distance/inner product between these operators.

Geometric Manifold Learning Techniques

The paper touches upon geometric manifold learning, focusing on two key theoretical results relevant to practical data analysis:

1. Laplacian Eigenmaps [7, 8]: This technique, used for dimensionality reduction, aims to reveal the intrinsic geometric structure of data points sampled from a low-dimensional manifold embedded in high-dimensional space. The method constructs a graph on the data points, and its Laplacian matrix's eigenvectors are used for embedding. Theorem 4.1 demonstrates that the graph Laplacian (scaled appropriately) converges in probability to the Laplace-Beltrami operator of the underlying manifold. This provides a mathematical justification for using the spectral properties of the data graph to infer the spectral geometry of the manifold, which is then used for dimensionality reduction.
2. Riemannian Manifold Reconstruction [23, 24]: This addresses the fundamental problem of constructing a Riemannian manifold that approximates a given metric space derived from data points, possibly with noise. The work by Fefferman et al. provides theoretical conditions under which such a reconstruction is possible, offering sufficient and necessary criteria for a metric space to be approximated by a Riemannian manifold in the Gromov-Hausdorff or quasi-isometric sense. This theoretical development is crucial for understanding when it is appropriate to model data using a manifold assumption and provides insights into potential algorithms for learning the manifold itself from noisy distance information.

In summary, the paper synthesizes concepts from category theory and differential geometry to provide a deeper theoretical understanding and practical methods for machine learning. Probabilistic morphisms offer a formal language for statistical learning, while geometric methods provide ways to define kernels and analyze data on non-Euclidean structures like SPD matrices and operator spaces, enabling advanced applications in areas like computer vision and signal processing. The theoretical results on learnability connect these abstract mathematical tools to the practical problem of building effective learning algorithms, particularly in the context of potentially overparameterized models and data with complex geometric structure.