Finite-Sample Performance of Gradient Descent in Logistic Regression with Gaussian Design

Abstract: We consider the parameter estimation problem in logistic regression with Gaussian design: the estimation of a fixed unknown parameter $θ^*\in \mathbb{R}<sup>d$ ($|θ^*|_2\ge 1$) from $n$ i.i.d. samples ${(x_i,y_i)}_{i=1}<sup>n$, where $x_i\sim N(0,I_d)$ and $$y_i|x_i \sim {\rm Bernoulli}(1/(1+\exp(-x_i^\top θ<sup>*)))$$. Our main aim is to characterize the finite-sample estimation performance and convergence behavior of gradient descent (GD) on the maximum likelihood objective (i.e., the logistic loss). Under small $O(1)$ stepsize and $0$ initialization, we show that GD linearly converges to a small neighborhood of $θ<sup>*$ achieving an $\ell_2$ error of order $O(\sqrt{|θ<sup>*|_2<sup>5d/n})$. This substantially goes beyond existing theoretical results that lack non-asymptotic estimation error rate and exhibit much slower parameter convergence. We also establish a faster local linear convergence to the same statistical error under a large $Θ(|θ<sup>*|_2)$ stepsize. The main technical component is to show that the gradient of the logistic loss satisfies a certain approximate invertibility condition (AIC). To that end, we uniformly control the deviation of the gradient from its population counterpart by covering and peeling arguments, and then show that the population GD is a contraction by a delicate analysis based on the eigenvalues of population Hessian matrices. Finally, we build upon the recent work Matsumoto and Mazumdar (2025) and devise a novel efficient estimator that attains a sharper rate in high dimensions. This indicates that the existing non-asymptotic guarantees exhibit sub-optimal dependence on $|θ<sup>*|_2$, and that in many regimes $Θ(\sqrt{|θ<sup>*|_2d/n})$ is the tight estimation error rate. Numerical examples are provided to corroborate our theoretical results.

• The paper establishes that GD achieves linear convergence to an O(||θ*||√(d/n)) neighborhood after O(||θ*||³ log d) iterations under Gaussian design.
• It shows that using a large stepsize with proper initialization accelerates local linear convergence, capturing the edge of stability in non-separable settings.
• The study introduces a two-stage estimator that separates direction and norm estimation, yielding minimax-optimal error rates outperforming MLE and standard GD in high dimensions.

Finite-Sample Analysis of Gradient Descent in Logistic Regression with Gaussian Design

Problem Statement and Context

The paper addresses the finite-sample performance and convergence behavior of gradient descent (GD) for logistic regression with Gaussian covariates. Given i.i.d. samples $\{(x_i, y_i)\}_{i=1}^n$, where $x_i \sim \mathcal{N}(0, I_d)$ and $$y_i \mid x_i \sim \mathrm{Bernoulli}(1/(1+\exp(-x_i^\top \theta^*)))$$, the objective is to estimate the unknown parameter $\theta^* \in \mathbb{R}^d$ with $\|\theta^*\| \geq 1$. The focus lies both on the statistical estimation accuracy and the optimization dynamics of GD run on the empirical logistic loss.

Previous theoretical work offers sharp asymptotic rates for the MLE and slow, often non-quantitative results for GD, especially under norm-unbounded $\theta^*$ and unbounded Gaussian designs. Most notably, existing bounds for either MLE or GD either have suboptimal dependence in $\|\theta^*\|$, loose rates in $n$, or are restricted to the bounded or separable covariate cases. This paper advances the state of the art by giving sharp non-asymptotic rates for the GD iterates themselves under Gaussian design and by presenting a new, computationally efficient estimator with optimal rates in high dimensions.

Main Contributions

The paper provides three core theoretical results:

1. Finite-Sample Parameter Convergence of GD (Small Stepsize Regime): With stepsize $\eta = O(1)$ and zero initialization, GD achieves a linear convergence rate to an $O(\|\theta^*\|\sqrt{d/n})$ radius neighborhood of $x_i \sim \mathcal{N}(0, I_d)$0 in $x_i \sim \mathcal{N}(0, I_d)$1 norm after $x_i \sim \mathcal{N}(0, I_d)$2 iterations, provided $x_i \sim \mathcal{N}(0, I_d)$3. This improves upon prior GD analyses which exhibit much slower, sublinear rates and establishes, for the first time under Gaussian design, a non-asymptotic parameter error rate for GD matching MLE up to log factors.
2. Local Acceleration of GD (Large Stepsize Regime): When initialized within a $x_i \sim \mathcal{N}(0, I_d)$4 vicinity of $x_i \sim \mathcal{N}(0, I_d)$5 and with a large stepsize $x_i \sim \mathcal{N}(0, I_d)$6, GD achieves a faster local linear contraction with improved per-iteration contractivity, at the same statistical error floor. This result is significant because it directly operationalizes "edge of stability" behavior for non-separable, unbounded designs and quantifies the attainable optimization gain over conservative stepsize regimes.
3. An Efficient Estimator with Sharper Minimax Rates:

The paper designs a two-stage estimator (Algorithm 2): - Compute the direction of $x_i \sim \mathcal{N}(0, I_d)$7 using an iterative method (projected subgradient descent for the ReLU loss, following [23]). - Estimate the norm of $x_i \sim \mathcal{N}(0, I_d)$8 by an efficient projection-based plug-in estimator. This estimator achieves an $x_i \sim \mathcal{N}(0, I_d)$9 error rate that improves on the MLE and GD in high-dimensional settings, attaining the non-asymptotic minimax-optimal error $$y_i \mid x_i \sim \mathrm{Bernoulli}(1/(1+\exp(-x_i^\top \theta^*)))$$0, which is shown to be tight up to log factors whenever $$y_i \mid x_i \sim \mathrm{Bernoulli}(1/(1+\exp(-x_i^\top \theta^*)))$$1.

Theoretical Techniques

Approximate Invertibility Condition (AIC)

The contraction and optimality results for GD hinge upon establishing a novel approximate invertibility condition. The analysis decomposes the GD update into concentration and bias terms:

• The concentration term controls the deviation of the empirical logistic loss gradient from its expectation uniformly over balls around $$y_i \mid x_i \sim \mathrm{Bernoulli}(1/(1+\exp(-x_i^\top \theta^*)))$$2. Precise high-dimensional concentration bounds (using covering arguments, Bernstein-type inequalities, and peeling techniques) are crucial to avoid excessive dependence on $$y_i \mid x_i \sim \mathrm{Bernoulli}(1/(1+\exp(-x_i^\top \theta^*)))$$3 or $$y_i \mid x_i \sim \mathrm{Bernoulli}(1/(1+\exp(-x_i^\top \theta^*)))$$4.
• The bias term is handled using fine spectral properties of the population Hessian of the logistic loss, leveraging Stein's identity to express and analyze the expected gradient in terms of eigenvalue decompositions and contraction mappings.

These tools allow a direct argument that (i) the gradient map is locally/ globally contractive with the right stepsize, and (ii) the stationary point of GD is within $$y_i \mid x_i \sim \mathrm{Bernoulli}(1/(1+\exp(-x_i^\top \theta^*)))$$5 of the true parameter.

Separation of Direction and Norm Estimation

The new estimator leverages the fact that in high SNR ($$y_i \mid x_i \sim \mathrm{Bernoulli}(1/(1+\exp(-x_i^\top \theta^*)))$$6 large) regimes, direction estimation is easier than norm estimation, and conventional maximum likelihood (or GD) simultaneously attempting both is statistically suboptimal. By using sample splitting and specifically designed projections adapted from recent advances in nonlinear regression, the estimator achieves minimax-optimal separation and rates.

Numerical Results

Simulation experiments on moderately large $$y_i \mid x_i \sim \mathrm{Bernoulli}(1/(1+\exp(-x_i^\top \theta^*)))$$7 and varying $$y_i \mid x_i \sim \mathrm{Bernoulli}(1/(1+\exp(-x_i^\top \theta^*)))$$8 and $$y_i \mid x_i \sim \mathrm{Bernoulli}(1/(1+\exp(-x_i^\top \theta^*)))$$9 corroborate the theoretical findings. The empirical curves match the predicted $\theta^* \in \mathbb{R}^d$0 decay and demonstrate the acceleration achieved for large stepsize GD when close to $\theta^* \in \mathbb{R}^d$1. The new estimator outperforms both GD and the empirical MLE in high-dimensional moderate sample regimes, supporting the analytical claims on minimax rates.

Implications and Future Directions

Practical Implications

• The results provide precise sample complexity and iteration complexity guidelines for practitioners using GD in high-dimensional logistic regression under Gaussian covariates. The analyses suggest that with proper stepsize tuning and initialization, GD is both statistically and computationally efficient, even outperforming MLE in certain regimes.
• The direction-norm separation approach offers a pathway to construct theoretically optimal procedures in broader generalized linear models, especially in regimes where norm recovery is information-theoretically impossible or dominates overall error.

Theoretical Implications

• The identification of the sharp phase transition in minimax rates according to $\theta^* \in \mathbb{R}^d$2 redefines the achievable bounds for logistic regression, closing several open questions regarding non-asymptotic behavior under unbounded Gaussian design.
• The approximate invertibility condition and associated proof machinery may be adaptable to other non-linear inverse problems, especially in one-bit/compressed sensing and binary regression.

Open Problems and Directions

• Extension to sparse logistic regression: Incorporating efficient hard-thresholding in the estimator for sparse parameter recovery.
• Analysis of adaptive or stochastic stepsize GD: Developing finite-sample performance guarantees for practical algorithms beyond fixed stepsize, including SGD.
• Extension to non-Gaussian or heavy-tailed designs: Overcoming technical challenges in covering and concentration arguments without rotational invariance.
• Further tightening minimax rates in regimes where $\theta^* \in \mathbb{R}^d$3 remains incompletely characterized.

Conclusion

The paper establishes, for the first time, sharp finite-sample parameter estimation rates and linear convergence bounds for gradient descent in logistic regression under Gaussian design, applicable to both conservative and aggressive learning rates. It also develops a new estimator that provably achieves better rates than MLE and GD in high dimensions, essentially settling the minimax estimation rate question for this problem in the moderate- to high-SNR regime. The analyses advance both the statistical theory and practical understanding of optimization in non-linear models, opening multiple compelling directions for future research.