Mahalanobis Distance Regression

• Mahalanobis Distance Regression is a method that uses expert-provided pairwise similarity and dissimilarity constraints to learn a diagonal Mahalanobis metric for guiding feature shrinkage.
• It constructs feature-specific Gaussian or Laplace priors on regression coefficients, thereby generalizing classical regularizers such as ridge and lasso.
• Empirical evaluations demonstrate that with accurate expert input, this approach enhances estimation accuracy and outperforms conventional models like OLS and k-NN.

Mahalanobis Distance Regression is a regression methodology that integrates expert-driven distance metric learning into the regularization of linear models. By eliciting pairwise similarity and dissimilarity judgments from domain experts, a Mahalanobis metric is learned and subsequently used to construct feature-specific priors on regression coefficients. This approach generalizes classical regularization such as ridge and lasso by allowing heterogeneous shrinkage in accordance with expert knowledge, potentially leading to superior performance in high-dimensional settings when expert input is accurate (Mani et al., 2019).

1. Distance Metric Learning via Pairwise Expert Constraints

The Mahalanobis Distance Regression framework begins with eliciting pairwise constraints from a domain expert. The expert provides two sets of ordered index pairs over the training design matrix $X = \{x_i\} \subset \mathbb{R}^n$:

• Similarity pairs: $$S = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be similar}\}$$
• Dissimilarity pairs: $$D = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be dissimilar}\}$$

These constraints correspond to “must-link” and “cannot-link” relations in metric learning.

A Mahalanobis distance metric is learned by solving the following convex optimization problem, restricted to diagonal matrices $A \in \mathbb{R}^{n \times n}$ with $A_{ij}=0$ for $i \neq j$:

$$\begin{aligned} A^\star = \argmin_{A \in \mathbb{R}^{n\times n}} &\sum_{(x_i,x_j)\in S} (x_i-x_j)^\top A (x_i-x_j) \ \text{s.t.}\quad & \sum_{(x_i,x_j)\in D} \sqrt{(x_i-x_j)^\top A (x_i-x_j)} \ge 1, \ & A \succeq 0, \quad A_{ij}=0\;(i\neq j) \end{aligned}$$

This objective contracts distances between similar pairs and ensures an aggregate lower bound among dissimilar pairs to avoid trivial solutions. The diagonality constraint simplifies computational handling and mirrors the eventual application to regression coefficients.

2. Constructing Feature-Specific Priors from Mahalanobis Metrics

Once $A^\star$ is determined, its diagonal elements $A^{\star}_{ii}$ encode the importance of each feature as inferred from expert similarity judgments. These are used to define independent zero-mean Gaussian priors for regression coefficients:

$$\beta_i \sim N(0, A^{\star}_{ii}), \quad i=1, \dots, n$$

or, equivalently, a multivariate normal prior

$$S = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be similar}\}$$0

This establishes feature-specific variance for each coefficient, biasing the regression to reflect the expert’s importance weights.

3. Estimation and Optimization Framework

The Mahalanobis-informed prior integrates into the linear regression likelihood as follows. With $$S = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be similar}\}$$1, the negative log-posterior (MAP) objective is:

$$S = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be similar}\}$$2

If the prior is Gaussian, this yields a closed-form analytic solution:

$$S = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be similar}\}$$3

For a Laplace (double-exponential) prior defined by $$S = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be similar}\}$$4, the penalty becomes weighted $$S = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be similar}\}$$5: $$S = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be similar}\}$$6, and solutions require subgradient or coordinate-descent-based numerical optimisation. Logistic regression can also be regularized with this Mahalanobis prior by adding a penalty term and optimizing via gradient-based methods.

4. Relation to Classical Regularization Methods

The learned Mahalanobis metric provides a natural feature-specific generalization of existing regularizers:

Method	Penalty Matrix	Shrinkage Type
Ridge Regression	$$S = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be similar}\}$$7	Uniform $$S = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be similar}\}$$8
Lasso	$$S = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be similar}\}$$9 (in $$D = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be dissimilar}\}$$0 norm)	Uniform $$D = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be dissimilar}\}$$1
Mahalanobis Reg (DMLreg)	$$D = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be dissimilar}\}$$2	Feature-specific

If $$D = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be dissimilar}\}$$3, Mahalanobis Distance Regression reduces to ordinary ridge regression. With Laplace priors, the approach generalizes lasso to feature-specific $$D = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be dissimilar}\}$$4 penalties, accommodating heterogeneous shrinkage across coefficients (Mani et al., 2019).

5. Empirical Evaluation and Performance Characteristics

Simulation experiments evaluate Mahalanobis Distance Regression (“DMLreg”) against OLS, k-NN, ridge, and lasso. Simulated data with $$D = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be dissimilar}\}$$5 samples and $$D = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be dissimilar}\}$$6 dimensions, and three types of “expert” metrics—perfect, noisy, and incorrect—are used to generate similarity/dissimilarity pairs and re-learn $$D = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be dissimilar}\}$$7.

Key findings:

• DMLreg with perfect or noisy expert knowledge outperforms lasso in validation MSE, without the need for hyperparameter tuning.
• Incorrect expert knowledge degrades DMLreg performance, but it remains superior to OLS and k-NN.
• Even with as few as 25 noisy pairwise comparisons, DMLreg achieves substantial improvements over baselines.
• DMLreg estimates relevant coefficients more accurately and shrinks noise terms more aggressively than lasso.

6. Advantages, Limitations, and Interpretability

Mahalanobis Distance Regression enables:

• Feature-specific regularization, allowing important features (per expert) to experience less shrinkage, and unimportant features to be more aggressively regularized.
• Direct interpretability: the diagonal of $$D = \{(x_i, x_j) \mid (x_i, x_j) \text{ should be dissimilar}\}$$8 quantifies the relative expert-elicited importance of each feature.
• Enhanced regression accuracy in high-dimensional settings when the elicited metric accurately reflects true feature relevances.

Limitations include dependence on a domain expert's ability to supply reliable pairwise judgments, implicit assumption that a diagonal Mahalanobis metric suffices (no cross-feature interactions), and performance sensitivity to the correctness of the inferred metric; incorrect guidance can lead to degraded performance, albeit generally still outperforming OLS or k-NN (Mani et al., 2019).

7. Summary

Mahalanobis Distance Regression (DMLreg) constitutes a principled method for incorporating expert knowledge into high-dimensional regression via a diagonal Mahalanobis distance learned from pairwise similarity/dissimilarity labels. This metric is directly mapped to a prior on regression coefficients, generalizing ridge and lasso by introducing feature-specific shrinkage while offering improved interpretability and empirical performance under accurate expert knowledge (Mani et al., 2019).