Risk Comparisons in Linear Regression: Implicit Regularization Dominates Explicit Regularization (2509.17251v1)

Abstract: Existing theory suggests that for linear regression problems categorized by capacity and source conditions, gradient descent (GD) is always minimax optimal, while both ridge regression and online stochastic gradient descent (SGD) are polynomially suboptimal for certain categories of such problems. Moving beyond minimax theory, this work provides instance-wise comparisons of the finite-sample risks for these algorithms on any well-specified linear regression problem. Our analysis yields three key findings. First, GD dominates ridge regression: with comparable regularization, the excess risk of GD is always within a constant factor of ridge, but ridge can be polynomially worse even when tuned optimally. Second, GD is incomparable with SGD. While it is known that for certain problems GD can be polynomially better than SGD, the reverse is also true: we construct problems, inspired by benign overfitting theory, where optimally stopped GD is polynomially worse. Finally, GD dominates SGD for a significant subclass of problems -- those with fast and continuously decaying covariance spectra -- which includes all problems satisfying the standard capacity condition.

• The paper establishes that implicit regularization via gradient descent outperforms explicit ridge regression, achieving lower excess risk in linear regression.
• It introduces novel ridge-type risk bounds that enable instance-wise comparisons between gradient descent, ridge regression, and online SGD.
• The results imply that for problems with fast-decaying covariance spectra, batch gradient descent is minimax optimal and preferable in many settings.

Excess Risk Comparisons in Linear Regression: Implicit Regularization Dominates Explicit Regularization

Overview and Motivation

This paper provides a rigorous, instance-wise comparison of excess risk for three canonical linear regression algorithms: gradient descent (GD), ridge regression, and online stochastic gradient descent (SGD). The analysis is conducted in the random-design setting, focusing on well-specified linear regression problems. The central claim is that implicit regularization via GD consistently dominates explicit regularization via ridge regression, both in terms of constant-factor risk and, for certain problem classes, polynomial rates. The relationship between GD and SGD is more nuanced: GD and SGD are generally incomparable, but GD dominates SGD for problems with fast, continuously decaying covariance spectra.

Formal Problem Setting and Algorithmic Framework

The paper considers linear regression in a separable Hilbert space $\mathcal{H}$, with population risk $$\mathcal{R}(\mathbf{w}) = \mathbb{E}[(\mathbf{x}^\top \mathbf{w} - y)^2]$$ and excess risk $$\mathcal{E}(\mathbf{w}) = \mathcal{R}(\mathbf{w}) - \mathcal{R}(\mathbf{w}^*) = \|\mathbf{w} - \mathbf{w}^*\|^2_{\Sigma}$$, where $\Sigma$ is the covariance operator. The three estimators are:

• Ridge Regression: $$\hat{\mathbf{w}} = (\mathbf{X}^\top \mathbf{X} + n\lambda I)^{-1} \mathbf{X}^\top \mathbf{y}$$, with explicit $\ell_2$ regularization.
• Gradient Descent (GD): Iterative updates with fixed step size $\eta$ and early stopping at $t$ iterations, yielding implicit regularization.
• Stochastic Gradient Descent (SGD): Online updates with exponentially decaying step size, focusing on the last iterate.

The analysis leverages recent tight finite-sample risk bounds for ridge regression and SGD, and introduces new upper and lower bounds for GD.

Main Theoretical Results

GD vs. Ridge Regression: Dominance

The paper proves that for all well-specified linear regression problems, the excess risk of GD is always within a constant factor of ridge regression, when the stopping time for GD is matched to the ridge regularization parameter. More strongly, for a natural subclass of problems (e.g., those with fast-decaying spectra), GD achieves polynomially smaller excess risk than optimally tuned ridge regression.

Figure 1: A summary of results in Section 5, showing the rates for learning $(a,r)$-power law classes and the regimes where each algorithm is minimax optimal.

The key technical contribution is a ridge-type upper bound for GD, matching the structure of the ridge regression risk decomposition. This enables direct comparison and establishes one-sided dominance. The result generalizes prior work that only held under isotropic priors, showing that GD's implicit regularization is robust to anisotropic parameter distributions.

GD vs. SGD: Incomparability and Conditional Dominance

The relationship between GD and SGD is shown to be fundamentally instance-dependent. While previous literature established that SGD can be polynomially worse than GD for certain problems, this paper constructs explicit examples (inspired by benign overfitting theory) where GD is polynomially worse than SGD. This separation is tied to the spectral properties of the covariance operator and the optimization power of batch vs. online methods.

However, for the subclass of problems with fast, continuously decaying spectra (e.g., exponential or polynomial decay), GD dominates SGD: its excess risk is always within a constant factor and can be polynomially better. This subclass includes all problems satisfying the standard capacity condition, which is central in nonparametric regression theory.

Exact Rates under Capacity and Source Conditions

For the $(a,r)$-power law class (covariance eigenvalues $\lambda_i \sim i^{-a}$, source condition $\|\Sigma^{-r} \mathbf{w}^*\|^2_\Sigma \lesssim 1$), the paper computes exact minimax rates for all three algorithms:

• GD is minimax optimal for all $a > 1$ and $r \geq 0$.
• Ridge Regression is minimax optimal for $0 \leq r \leq 1$, but polynomially suboptimal for $r > 1$ (the saturation effect).
• SGD is minimax optimal for $r \geq (a-1)/(2a)$, but polynomially suboptimal otherwise.

These results are summarized in Figure 1 and Table 2 of the paper, and clarify the regimes where implicit regularization is strictly superior.

Technical Innovations

• Instance-wise Risk Comparisons: The paper formalizes statistical dominance in terms of constant-factor and polynomial risk separations, moving beyond minimax theory to instance-wise analysis.
• Novel GD Bounds: The ridge-type and SGD-type upper bounds for GD are new, enabling direct comparison with explicit regularization and online methods.
• Lower Bounds for GD: The construction of hard instances for GD, leveraging benign overfitting, reveals limitations of batch methods in certain high-dimensional, low-noise regimes.
• Spectral Conditions: The identification of fast, continuously decaying spectra as the key condition for GD's dominance over SGD is both theoretically and practically significant.

Implications and Future Directions

Practical Implications

• Algorithm Selection: For practitioners, the results suggest that early-stopped GD should be preferred over ridge regression in most linear regression settings, especially when the covariance spectrum decays rapidly.
• SGD vs. GD: In online or streaming contexts, SGD may outperform GD for certain data distributions, but in batch settings with favorable spectra, GD is preferable.
• Benign Overfitting: The analysis clarifies the role of spectral decay in enabling benign overfitting and the limitations of explicit regularization.

Theoretical Implications

• Implicit vs. Explicit Regularization: The dominance of implicit regularization challenges the conventional wisdom of explicit norm penalties, especially in high-dimensional regimes.
• Separation of Batch and Online Learning: The incomparability of GD and SGD highlights fundamental differences in statistical and optimization properties, motivating further paper of hybrid and multi-epoch algorithms.
• Generalization Beyond Linear Regression: Extending these results to other loss functions (e.g., logistic regression), other regularizers (e.g., LASSO), and more general statistical models is an open direction.

Speculation on Future Developments

• Multi-epoch SGD: The paper conjectures that multi-epoch SGD may dominate both GD and SGD, suggesting a promising avenue for algorithmic development.
• Principal Component Regression (PCR): The role of PCR in random-design settings remains to be fully characterized, especially in relation to implicit regularization.
• Negative Ridge and Oscillatory GD: Allowing negative regularization or larger step sizes in GD may further improve risk, but requires careful analysis of stability and generalization.

Conclusion

This work provides a comprehensive, instance-wise comparison of excess risk for GD, ridge regression, and SGD in linear regression. The main finding is that implicit regularization via GD consistently dominates explicit regularization, both in constant-factor and polynomial regimes, except for certain pathological cases. The nuanced relationship between GD and SGD is clarified, with spectral decay emerging as the key determinant of algorithmic optimality. These results have significant implications for both theory and practice, and open several avenues for future research in statistical learning and optimization.