RKHS Framework for Metric Learning

• The framework is a rigorous RKHS-based method that learns nonlinear similarity metrics from triplet comparisons using kernel functions.
• It extends classic linear Mahalanobis metric learning into infinite-dimensional Hilbert spaces with Schatten norm regularization to control model complexity.
• Empirical results and KPCA-based optimization demonstrate strong generalization and reduced sample complexity in practical applications such as image retrieval and recommendation systems.

A Reproducing Kernel Hilbert Space (RKHS) framework for metric learning provides a mathematically rigorous and practically tractable approach to learning nonlinear similarity metrics from comparison data, such as triplets. This setup formalizes the task of learning pairwise or triplet-based metrics directly in a Hilbert space induced by a kernel function, leading to strong generalization guarantees and explicit sample complexity results even in infinite-dimensional settings. The framework critically extends the classical linear theory for metric learning in $\mathbb{R}^d$ to general RKHS, thereby encapsulating a wide range of nonlinear methods, including kernel-based approaches and neural network analogs, under a unified theoretical lens.

1. Mathematical Formulation of Metric Learning in RKHS

The RKHS metric learning framework begins by representing each object $x \in \mathbb{R}^d$ as a feature map $\phi(x) \in \mathcal{H}$, where $\mathcal{H}$ is the RKHS associated with a positive definite kernel $$k(x,y) = \langle \phi(x), \phi(y) \rangle_{\mathcal{H}}$$. The core learning objective is to find a bounded linear operator $L : \mathcal{H} \to \mathcal{H}$ such that the induced metric

$$d_L^2(x, y) = \| L \phi(x) - L \phi(y) \|_{\mathcal{H}}^2$$

is compatible with given supervision. The supervision consists of a set of triplets $(x_h, x_i, x_j)$ indicating that "item $h$ is more similar to $i$ than to $j$," which is encoded as

$$\mathrm{sign}\left( d_L^2(x_h, x_i) - d_L^2(x_h, x_j) \right)$$

matching the observed label for each triplet. The operator $L$ generalizes the role of a Mahalanobis matrix $M$ in the linear case, and through the kernel, enables learning highly nonlinear metrics.

The learning algorithm seeks $L$, subject to appropriate norm constraints, to align the induced metric with the observed comparisons, often via empirical risk minimization over a convex surrogate loss.

2. Regularization, Model Classes, and Finite-Dimensional Reduction

Restricting the capacity of $L$ is essential to control overfitting. This is achieved using Schatten $p$-norm constraints:

• Schatten-2 norm (Hilbert–Schmidt/Frobenius): $\|L^\dagger L\|_{S_2} \leq \lambda_F$.
• Schatten-1 norm (Nuclear norm): $\|L^\dagger L\|_{S_1} \leq \lambda_*$.

These constraints induce different regularization effects, controlling rank and effective dimension and directly influencing sample complexity.

Although $L$ in principle acts on an infinite-dimensional $\mathcal{H}$, a central representer theorem result ensures that the solution can be computed by restricting $L$ to the subspace spanned by the kernelized features of the training objects. By leveraging kernel PCA (KPCA), the data are projected into a finite-dimensional space, and metric learning can be recast as optimizing over positive semidefinite matrices $M$ in the KPCA representation.

Distance evaluations for training examples satisfy

$$\|L \phi(x_i) - L \phi(x_j)\|_{\mathcal{H}}^2 = \| \varphi_i - \varphi_j \|^2_M,$$

with $\varphi_i$ the KPCA coordinates.

3. Generalization Guarantees and Sample Complexity

The framework delivers explicit excess risk and sample complexity bounds for the learned metric. Given $|\mathcal{S}|$ observed triplets, a universal bound holds for the excess true risk of the empirical minimizer $\widehat{L}_0$ over the regularized class: $$R(\widehat{L}_0) - R(L^*) \leq 4 \alpha B^2 \lambda_F \sqrt{\frac{6}{|\mathcal{S}|}} + \alpha B^2 \lambda_F \sqrt{\frac{2 \ln(2/\delta)}{|\mathcal{S}|}},$$ where $B$ bounds the RKHS feature norms, $\alpha$ is the Lipschitz constant of the loss, and $\delta$ is the confidence level.

Analogous results are obtained under the nuclear norm constraint; when $L$ has (approximately) low rank $k$, the sample complexity scales as $O(k^2 \ln (k/\delta))$ rather than $O(d^4 \ln(d/\delta))$, matching intuition and prior results from the linear case. The implication is that learning a nonlinear metric is no harder (up to effective dimension) than learning a linear metric.

The theoretical oracle inequalities are validated empirically: with sufficient triplets, training and test error are nearly indistinguishable, and nonlinear kernels (e.g., Gaussian, polynomial) outperform linear kernels when the underlying notion of similarity is nonlinear.

4. Computational Aspects: Practical Optimization via KPCA

Although the initial setup is infinite dimensional, the framework reduces the empirical risk minimization to a convex program in finite dimensions. This program is efficiently solvable using methods for semidefinite programming or projected gradient descent over symmetric positive semidefinite matrices, once the KPCA representations are computed.

The KPCA computation for $n$ objects requires eigendecomposition of the $n \times n$ kernel Gram matrix, with $O(n^3)$ complexity. For larger scale problems, further approximation such as randomized low-rank factorizations (randomly pivoted Cholesky, Nyström methods) can be applied to obtain KPCA maps with reduced computational burden.

In practice, after KPCA, optimization is performed over the metric matrix $M$ in the projected space, with either Frobenius or nuclear norm constraints corresponding to the desired regularization.

5. Empirical Results, Applications, and Implications

The RKHS metric learning framework is supported by simulations and experiments on both synthetic and real data:

• In a synthetic spiral dataset where the true notion of distance is curved (geodesic distance), nonlinear kernel metrics (Gaussian, polynomial, Laplacian) achieve significantly higher accuracy and lower error than any linear (global Mahalanobis) metric, verifying the theoretical advantage of nonlinearity under human-like similarity judgments.
• On the Food–100 dataset with human-annotated triplets, cross-validation shows that kernelized metrics outperform linear ones, and the explicit sample complexity and generalization bounds are empirically realized—once sample size passes a threshold, overfitting is negligible.
• In simulated settings where the ground truth metric is low-rank, sample complexity reductions under nuclear norm regularization are evident.

Practical applications include image retrieval, recommendation systems, and domains such as perceptual similarity or psychophysics, where only relative (triplet) similarity labels are available and the underlying similarity is often nonlinear.

6. Interpretation, Significance, and Limitations

This RKHS-based theory rigorously generalizes classic metric learning from finite-dimensional linear Mahalanobis metrics to nonlinear settings, providing a principled methodology along with finite-sample guarantees. The theory elucidates the benefit of kernels for capturing complex similarity judgments and the importance of norm constraints (Frobenius or nuclear) for controlling effective capacity and sample complexity.

Limitations include:

• The scalability of KPCA for massive datasets, which motivates subsampling or low-rank approximation strategies.
• The assumption of bounded feature norms and Lipschitz loss, which may be violated in highly nonstationary or adversarial settings.
• While neural network–based metric learning methods show strong empirical success, their theoretical analysis remains limited; the RKHS theory serves as a foundation and potential avenue for future theoretical unification.

7. Key Formulas and Summary Table

The critical mathematical structures and guarantees in the framework can be summarized as:

Concept	Formula / Role
Distance in RKHS	$$d_L^2(x_i, x_j) = \|L \phi(x_i) - L \phi(x_j)\|^2_{\mathcal{H}}$$
Triplet response	$$y_t \approx \mathrm{sign}(d_L^2(x_h, x_i) - d_L^2(x_h, x_j))$$
Frobenius norm constraint	$\|L^\dagger L\|_{S_2} \leq \lambda_F$
Nuclear norm constraint	$\|L^\dagger L\|_{S_1} \leq \lambda_*$
Excess risk bound	$$R(\widehat{L}_0) - R(L^*) \leq 4 \alpha B^2 \lambda_F \sqrt{6/|\mathcal{S}|} + \ldots$$
KPCA distance equivalence	$$\|L \phi(x_i) - L \phi(x_j)\|^2_{\mathcal{H}} = \| \varphi_i - \varphi_j \|^2_M$$

These results establish the representational flexibility, generalization properties, and sample efficiency of RKHS-based metric learning and provide a scalable route to implement nonlinear metric learning in modern machine learning applications (Tatli et al., 6 Aug 2025).