Optimal convex $M$-estimation via score matching (2403.16688v2)

Abstract: In the context of linear regression, we construct a data-driven convex loss function with respect to which empirical risk minimisation yields optimal asymptotic variance in the downstream estimation of the regression coefficients. At the population level, the negative derivative of the optimal convex loss is the best decreasing approximation of the derivative of the log-density of the noise distribution. This motivates a fitting process via a nonparametric extension of score matching, corresponding to a log-concave projection of the noise distribution with respect to the Fisher divergence. At the sample level, our semiparametric estimator is computationally efficient, and we prove that it attains the minimal asymptotic covariance among all convex $M$-estimators. As an example of a non-log-concave setting, the optimal convex loss function for Cauchy errors is Huber-like, and our procedure yields asymptotic efficiency greater than $0.87$ relative to the maximum likelihood estimator of the regression coefficients that uses oracle knowledge of this error distribution. In this sense, we provide robustness and facilitate computation without sacrificing much statistical efficiency. Numerical experiments using our accompanying R package 'asm' confirm the practical merits of our proposal.

• The paper introduces a semiparametric M-estimator that achieves minimal asymptotic variance through a novel convex loss function derived from nonparametric score matching.
• The paper develops a data-driven convex loss using antitonic projection of the error score function, ensuring robust and efficient regression estimates.
• The paper demonstrates through numerical experiments that the proposed estimator reduces estimation error and computation time compared to classical methods in non-Gaussian settings.

Optimal convex $M$-estimation via score matching

Introduction

The problem of fitting linear models robustly and efficiently, especially when the error distribution is unknown and potentially non-Gaussian, has been a long-standing area of interest in statistics. In the setting of a linear regression model, we focus on constructing a data-driven convex loss function whereby empirical risk minimization yields an estimator with optimal asymptotic variance for regression coefficients. Our approach extends the classical paradigm by targeting a semiparametric framework, where the error distribution is not specified explicitly. The essence of our contribution lies in identifying an optimal population-level convex loss function derived from a nonparametric extension of score matching, thus offering a balance between computational tractability and statistical efficiency.

Theoretical Framework

Overview of Main Results

We develop a methodology that yields the minimal asymptotic covariance among all convex $M$-estimators for regression coefficients without presuming the error distribution. The procedure involves the identification of a data-driven convex loss function whose empirical risk minimization leads to estimators with desirable asymptotic properties. Specifically, we introduce an optimal convex loss function that is a robust alternative to commonly used losses, such as the squared error and Huber's loss. The foundation of our approach is the antitonic projection of the score function associated with the underlying error distribution, which is computationally efficient and adaptively robust against deviations from model assumptions.

Construction of the Optimal Convex Loss

We construct a semiparametric $M$-estimator through a score matching procedure adapted to the regression setting. For given errors with unknown distribution, the construction involves nonparametrically estimating the error density and its score function, followed by identifying an efficient convex loss function via the antitonic projection of the estimated score. This approach leads to an optimal loss function that attains minimal asymptotic variance for the estimator under the regression model, demonstrating an efficiency advantage over existing methodologies.

Practical Merits and Asymptotic Results

Numerical experiments validate the theoretical advantages of the proposed estimator, showing improved estimation accuracy and robustness across various non-Gaussian error distributions. The empirical findings highlight the superiority of our approach in terms of estimation error and computation time, even for high-dimensional settings. Asymptotically, we prove that the semiparametric estimator derived from our methodology achieves the efficiency bound for convex $M$-estimators, providing a rigorously justified alternative to classical estimators in regression models.

Discussion

The introduction of an optimal convex loss function derived through nonparametric score matching presents a novel avenue in regression analysis that effectively addresses robustness and efficiency challenges. Our contribution lies not only in advancing theoretical understanding but also in offering practical solutions for robust linear model fitting. The proposed estimator's ability to adaptively achieve minimal asymptotic variance, regardless of the error distribution's nature, marks a significant step forward in the quest for reliable and computationally efficient regression techniques. Future developments may explore extensions to more complex models and the integration of our findings into broader statistical learning frameworks.