Minimax Optimal Estimator and Improved Error Rate for the MLE in Logistic Regression with Gaussian Design

Abstract: We study finite-sample parameter estimation in logistic regression with Gaussian design, where the goal is to estimate $\mathbfθ^*\in \mathbb{R}<sup>d$ with $R=|\mathbfθ^*|_2\ge 1$ from i.i.d. samples ${(\mathbf{x}<em>i,y_i)}</em>{i=1}<sup>n,$ $\mathbf{x}_i \sim N(0,\mathbf{I}_d)$, $$y_i\mid \mathbf{x}_i \sim \mathrm{Bernoulli}((1+\exp(-\mathbf{x}_i^\top \mathbfθ<sup>*))<sup>{-1})$$. In this paper, we provide the first minimax optimal estimator, and improve on the best known finite-sample error rate for the maximum likelihood estimator (MLE). These two accomplishments are due to a minimax optimal estimator for the parameter norm $R$. First, we establish the minimax lower bound $Ω(\sqrt{R<sup>3/n})$ for norm estimation. We then improve the best known norm estimation error rate of the MLE, i.e., $O(\sqrt{R<sup>3d/n})$ from Chardon, Lerasle and Mourtada (2024), to $\tilde{O}(\sqrt{R<sup>3/n}+R<sup>2d/n)$. The additional term, $R<sup>2d/n$, appears to be the intrinsic bias of the MLE, as evidenced by the high-dimensional asymptotic theory of Zhao, Sur and Candes (2022) and numerical examples. We show that, however, this additional term is not information-theoretically necessary. To this end, we construct an efficient debiased norm estimator that achieves the error rate $O(\sqrt{R<sup>3/n})$ and is therefore minimax optimal. Combining this with the optimal direction estimator given by the MLE, we establish the minimax optimal rate $Θ(\sqrt{Rd/n}+\sqrt{R<sup>3/n})$ for estimating $\mathbfθ<sup>*$, as well as the improved finite-sample error rate $\tilde{O}(\sqrt{Rd/n}+\sqrt{R<sup>3/n}+R<sup>2d/n)$ for the MLE. Numerical experiments demonstrate that the proposed minimax optimal estimators outperform the MLE.