Metric Learning Paradigms

• Metric Learning Paradigms are a family of algorithms that learn similarity functions by optimizing embedding spaces, enhancing tasks like classification and clustering.
• They encompass global, local, and nonlinear methods, providing trade-offs between model interpretability, flexibility, and computational efficiency.
• Recent advances integrate regularization, scalable optimization, and hybrid approaches to address overfitting and computational challenges.

Metric learning encompasses a family of algorithms that aim to learn distance or similarity functions from data, typically by optimizing the geometry of an embedding space to improve the performance of downstream tasks such as classification, clustering, verification, retrieval, and semi-supervised learning. The principal goal is to align the induced metric with semantic similarity, often using supervised or weakly supervised signals such as class labels, pairwise similarity/dissimilarity constraints, or triplet relations. Metric learning paradigms vary in parametric structure (global vs. local, linear vs. nonlinear), regularization mechanisms, optimization approaches, and the nature of the supervision. This article surveys the principal paradigms, their mathematical foundations, representative algorithms, and associated trade-offs.

1. Global Mahalanobis Metric Learning

The canonical paradigm in metric learning is global Mahalanobis distance learning, where a single positive semidefinite matrix $M \in S_+^d$ defines a distance $d_M(x,y) = \sqrt{(x-y)^\top M (x-y)}$. The optimization objective typically incorporates constraints to pull similar pairs together and/or push dissimilar pairs apart. Common formulations include:

• Pairwise constraints: Minimizing distances between similar pairs and maximizing those for dissimilar pairs, as in the Xing et al. approach, solved via projected gradient or SDP.
• Triplet constraints: Large-margin frameworks (e.g., LMNN) impose a margin between target neighbors and imposters, solved via convex programming or specialized subgradient methods.
• Information-theoretic approaches: ITML employs the LogDet divergence between the learned metric and a prior, with upper/lower bounds on distances for constrained pairs, solved via Bregman projections.

Pros of global Mahalanobis learning include convexity (ensuring a globally optimal solution), interpretability via linear projections, and amenability to theoretical generalization studies. However, these methods can face overfitting in high dimensions (as $M$ has $O(d^2)$ free parameters), computational bottlenecks from eigen-decompositions or SDPs, and lack the expressivity to capture nonlinear or local structure (Bellet et al., 2013).

2. Local and Parametric Local Metric Learning

To address data heterogeneity, local metric learning endows each region, cluster, or instance with its own metric. Notable approaches include:

• Multiple local Mahalanobis matrices: Assigning a separate $M_k \succeq 0$ to each region, cluster, or point (Bellet et al., 2013).
• Smooth parameterized metric functions: PLML models $M(x) = \sum_{k=1}^m W_k(x) M_{b_k}$, a convex combination of $m$ anchor-point basis metrics, with smoothness enforced by manifold regularization on the weights (Wang et al., 2012).
• Manifold-based or graph-regularized methods: Using Laplacian penalties to couple neighboring metrics along estimated data manifolds.

These techniques provide increased local flexibility and often superior empirical accuracy, especially in multimodal or nonstationary data regimes. However, the price is significantly increased model complexity, nonconvexity, potential overfitting (if not correctly regularized), and greater memory and computation costs. PLML achieves scalability by using a small anchor set and efficient block-coordinate optimization, bypassing the exponential parameter blowup seen in naïve per-instance metrics (Wang et al., 2012).

3. Nonlinear, Kernelized, and Deep Metric Learning

Nonlinear metric learning paradigms generalize beyond linear Mahalanobis structure:

• Kernelized Mahalanobis metric learning: Lifts data into a (possibly infinite-dimensional) RKHS, learning $M$ in feature space, e.g., kernel-LMNN, kernel-ITML (Ghojogh et al., 2022).
• Decision-tree ensembles: Models like GB-LMNN learn nonlinear mappings via boosted regression trees.
• Deep architectures: Siamese or triplet networks train DNNs to produce embeddings where semantic similarity corresponds to Euclidean or cosine closeness, using contrastive or triplet losses, proxy-based losses, and sophisticated mining strategies (Ghojogh et al., 2022).

Nonlinear methods substantially increase representational capacity but introduce nonconvexity, greater risk of overfitting (especially with limited data), and heavy computational demands for training and cross-validation.

4. Regularization and Stability paradigms

A central concern in metric learning is the control of model complexity to ensure generalization and numerical stability. Recent frameworks explicitly constrain metric distortion, margin ratios, or related capacity measures:

• Bounded Distortion Metric Learning (BDML): BDML augments Mahalanobis learning with an explicit condition-number constraint on $M$, i.e., $$\kappa(M) = \lambda_{\max}(M)/\lambda_{\min}(M) \leq \delta$$ for user-specified $d_M(x,y) = \sqrt{(x-y)^\top M (x-y)}$0. This distortion control tightens generalization bounds by limiting embedding eccentricity and improves numerical accuracy in eigen-solvers (Liao et al., 2015).
• Lipschitz-Margin-Ratio (LMR) framework: Maximizes the ratio between the inter-class margin and intra-class dispersion (diameter), leading to generalization bounds via the fat-shattering dimension. This unifies and subsumes paradigms like LMNN and ITML, with practical instantiations based on Mahalanobis distances solved by SDP or ADMM (Dong et al., 2018).

These paradigms provide theoretical guarantees: BDML, for example, directly links distortion bounds to algorithmic stability and empirical risk upper bounds, with explicit constants. Empirically, enforcing moderate distortion (e.g., condition number $d_M(x,y) = \sqrt{(x-y)^\top M (x-y)}$1) optimizes the tradeoff between overfitting and underfitting (Liao et al., 2015).

5. Optimization Strategies and Scalability

Metric learning formulations, especially global Mahalanobis, generate SDPs or convex programs with potentially $d_M(x,y) = \sqrt{(x-y)^\top M (x-y)}$2 variables and $d_M(x,y) = \sqrt{(x-y)^\top M (x-y)}$3 constraints. To address this, several scalable optimization paradigms have emerged:

• Efficient dual approaches: By switching to Frobenius-norm regularization, the primal updates can be written via eigen-decomposition of a small matrix, reducing complexity from $d_M(x,y) = \sqrt{(x-y)^\top M (x-y)}$4 (interior-point SDP) to $d_M(x,y) = \sqrt{(x-y)^\top M (x-y)}$5 per iteration (Shen et al., 2013).
• Multiplicative Weights Update (MWU) solvers: BDML employs MWU, iteratively solving weighted feasibility subproblems and updating constraint weights, achieving provable convergence rates (Liao et al., 2015).
• SVM kernel classification reductions: Doublet and triplet construction with associated degree-2 polynomial kernels converts metric learning into SVM classification, leveraging highly optimized existing solvers for scalability (Wang et al., 2013, Zuo et al., 2015).
• Graph- and manifold-based optimization: PLML and related local-learners use FISTA-project gradient and Laplacian regularization, scaling to tens of thousands of examples (Wang et al., 2012).

Complexity-wise, the closed-form solution of Geometric Mean Metric Learning (GMML) based on SPD manifold geometry allows orders-of-magnitude speedup, as it only requires a single eigen-decomposition per run (Zadeh et al., 2016).

6. Structured, Relational, and Graph-based Paradigms

Metric learning for structured and relational data adapts PSD-matrix-based approaches using side information from data topology or complex advisories:

• Graph-based metric learning: Metrics are trained to optimize class separation in the induced similarity graph, which then drives semi-supervised label propagation. Empirical gains (5–15% accuracy increase) are observed via end-to-end learned graphs versus Euclidean-based graphs (Wauquier et al., 2015).
• Relational constraints: Link-strength functions quantify the similarity of items via common parents or relational context, providing pairwise or triplet constraints that encode both topology and side-attribute similarity. These can be smoothly integrated into standard metric learning solvers (ITML, LSML), systematically boosting k-NN classification accuracy, especially when label-based constraints are weak (Pan et al., 2018).
• Multiple kernel and hierarchical subspace approaches: CLASMK-ML learns class-specific kernel mixtures and low-dimensional subspaces in RKHS, stacking layers to focus on "marginal" points without explicit pairwise constraints, and yielding robust, computationally attractive class separation on multiclass tasks (Yu et al., 2019).

Such paradigms exploit nonvectorial side-information and can be extended for semi-supervised, multi-relational, or heterogeneous data.

7. Hybrid and Emerging Paradigms

Recent research intersects metric learning with other machine learning objectives and structures:

• Metric-learning-based SVM and MKL: SVM_m and MKL_m integrate hyperplane-centric within-class distance measures as convex penalties, leading to joint between/within-class optimization with full kernelizability and improved generalization (Do et al., 2013).
• Learning neighborhoods for metric learning: LNML jointly optimizes both the neighborhood assignment matrix and the metric, adaptively choosing neighborhood size and structure in tandem with metric parameters. This outperforms fixed-neighborhood schemes (e.g., in LMNN), especially in tasks with heterogeneous density (Wang et al., 2012).
• Lifelong metric learning (LML): LML maintains a shared low-rank dictionary across tasks and alternately learns a task-specific Mahalanobis parameter per task, supporting efficient positive knowledge transfer, low memory, and "forward" and "backward" transfer in continual settings (Sun et al., 2017).

These paradigms show that the core metric learning objective can be extended or merged with task-structure, semi-supervised information, or lifelong learning considerations, opening avenues for future expansion and methodological innovation.

---

References

• Bounded-Distortion Metric Learning (Liao et al., 2015)
• Metric Learning for Graph-Based Label Propagation (Wauquier et al., 2015)
• Geometric Mean Metric Learning (Zadeh et al., 2016)
• Learning Neighborhoods for Metric Learning (Wang et al., 2012)
• Lifelong Metric Learning (Sun et al., 2017)
• Spectral, Probabilistic, and Deep Metric Learning: Tutorial and Survey (Ghojogh et al., 2022)
• A Metric-learning Based Framework for SVM and MKL (Do et al., 2013)
• Learning Hierarchical Feature Space Using CLAss-specific Subspace Multiple Kernel—Metric Learning (Yu et al., 2019)
• A Survey on Metric Learning for Feature Vectors and Structured Data (Bellet et al., 2013)
• Parametric Local Metric Learning for NN Classification (Wang et al., 2012)
• Relational Constraints for Metric Learning on Relational Data (Pan et al., 2018)
• A Kernel Classification Framework for Metric Learning (Wang et al., 2013)
• An Efficient Dual Approach to Distance Metric Learning (Shen et al., 2013)
• Iterated Support Vector Machines for Distance Metric Learning (Zuo et al., 2015)
• Metric Learning via Maximizing the Lipschitz Margin Ratio (Dong et al., 2018)