Gradient Descent: Fundamentals & Applications

• Gradient Descent (GD) is a first-order iterative optimization algorithm that minimizes differentiable cost functions using gradient-based updates in continuous spaces.
• Theoretical analyses of GD detail convergence rates and stability under both convex and nonconvex regimes, emphasizing the impact of step-size and geometry.
• Adaptive variations, including Natural and proximal Gradient Descent, extend standard GD to handle non-differentiable terms and complex manifold structures for faster convergence.

Gradient Descent (GD) is a general-purpose first-order optimization algorithm for continuous cost functions, with central importance in mathematical optimization, data science, and machine learning. For a differentiable objective $f:\mathbb{R}^d\to\mathbb{R}$, GD generates iterates $x_{k+1} = x_k - \eta\nabla f(x_k)$, where the step-size $\eta>0$ is typically tuned for problem geometry. Theoretical analyses and practical enhancements of GD encompass convergence rates, stability, generalization properties, choice of geometry (metric), adaptation to nonconvex regimes, and links to manifold structure and control theory.

1. Classical Formulation and Convergence Properties

GD seeks a minimizer $x^*$ of $f(x)$ via steepest descent under Euclidean geometry. If $f$ is convex and differentiable, and its gradient is globally $L$-Lipschitz,

$$\|\nabla f(x)-\nabla f(y)\|\leq L\|x-y\| \quad \forall x,y\in \mathbb{R}^d,$$

the update

$x_{k+1} = x_k - \frac{1}{L}\nabla f(x_k)$

achieves an $O(1/k)$ convergence rate for objective value: for all $n$,

$f(x_n) - f(x^*) \leq \frac{L\|x_0-x^*\|^2}{2n}$

(Nikolovski et al., 28 Dec 2024). This rate is optimal for general smooth convex problems. Fixed-step sizes require knowledge of $L$.

For strongly convex objectives ($\mu$-PL condition: $\frac12\|\nabla f(x)\|^2 \geq \mu(f(x)-f^*)$), linear convergence is obtained.

2. Extensions: Geometry, Regularization, and Natural Gradient

GD is fundamentally tied to the geometry of the parameter space. In Natural Gradient Descent (NGD), one replaces the Euclidean metric with an adaptive, possibly Riemannian, metric $G(x)$:

$x_{k+1} = x_k - \eta G(x_k)^{-1}\nabla f(x_k)$

(Dong et al., 2022). In statistical learning, $G(x)$ is often the Fisher information matrix. Dong & Le (Dong et al., 2022) generalize NGD to arbitrary metric pullbacks from reference manifolds, enabling tailored geometry for rapid convergence in ill-conditioned or structured problems.

When dealing with regularized objectives, GD is extended to proximal gradient descent, particularly for non-differentiable terms such as $\ell^1$ regularization. The problem

$\min_x\,f(x)+\lambda\|x\|_1$

is solved via the iterative scheme

$$x_{k+1} = \text{prox}_{\lambda\|\cdot\|_1}\big(x_k - \alpha_k\nabla f(x_k)\big)$$

where $$\text{prox}_{\lambda\|\cdot\|_1}(z)_i=\text{sign}(z_i)\max\{|z_i|-\lambda,0\}$$. Recent work adapts the step size based on local rather than global Lipschitz constants, yielding faster and more robust convergence (Nikolovski et al., 28 Dec 2024).

3. GD in Deep and Overparametrized Models

Analyses in modern deep learning often involve overparametrized networks. In such cases, classical global PL and smoothness do not hold, but local versions do (Xu et al., 16 May 2025). For two-layer linear networks, defining weight-dependent “skewing” operators allows tracking per-iterate local PL and descent constants. One can derive adaptive step size rules to obtain linear rates even outside NTK or infinite-width regimes. Depth and skip connections, as shown in (E et al., 2019), stabilize both forward and backward passes, enabling exponential-rate convergence to global zero training error in deep net architectures for suitable kernel-positive-definite initializations.

GD in deep nets exhibits implicit regularization: in the overparametrized, kernel-induced regime, the model output follows a path close to a random-feature model. Generalization error is controlled by early stopping, with the learned predictors remaining in an RKHS ball determined by initialization. The population risk after early stopping scales as $O(1/\sqrt{n})$ when the target lies within the RKHS (E et al., 2019).

4. Adaptive and Accelerated Gradient Dynamics

Acceleration techniques reformulate GD as controlled second-order dynamical systems. Under strong convexity and smoothness, the controlled invariant manifold approach derives the continuous-time ODE

$$\ddot{x} + \beta\nabla^2f(x)\dot{x} + \alpha\dot{x} + \alpha\beta\nabla f(x) = 0$$

(Gunjal et al., 2023). Euler-type discretization recovers optimal momentum schemes: Nesterov’s accelerated GD achieves $O(\sqrt{\kappa}\ln(1/\epsilon))$ complexity, where $\kappa=L/\mu$ is the condition number.

Emerging techniques use terminal attractor and terminal sliding mode theory to design adaptive learning rates ensuring finite-time convergence. Four learning rate schemes—Terminal Attractor (TA), Fast Terminal Attractor (FTA), Placid Terminal Attractor (PTA), Placid Fast Terminal Attractor (PFTA)—adaptively select $\gamma(w)$ so that the energy function satisfies $\dot{E} = -\Omega(E)$ for problem-dependent $\Omega$. These schemes facilitate escape from local minima and stabilization near global minima; the PFTA, in particular, combines speed and boundedness (Zhao et al., 10 Sep 2024).

The “basis function decomposition” analysis shows GD solution trajectories are monotonic when projected onto a post-training conjugate kernel basis, explaining robust convergence in deep nets across optimizers and architectures (Ma et al., 2022).

5. GD in Nonconvex and Non-Lipschitz Regimes

GD is widely used even when the objective lacks global Lipschitz gradients. For differentiable $F(x)$ with only locally Lipschitz gradient, and under a diminishing step size sequence $\{\alpha_k\}$, Patel & Berahas (Patel et al., 2022) prove that:

• If the iterates $\{x_k\}$ remain bounded, then $F(x_k)$ converges to a finite limit and $\|\nabla F(x_k)\|\to 0$.
• The set of limit points is closed, connected, and contains no open subset (either singleton or infinite).
• Pathologies are possible if the iterates diverge, e.g., gradients staying nonzero while $F(x_k)\to\infty$.

This rigorously supports the operational use of diminishing step-size heuristics on nonconvex, locally non-smooth objectives prevalent in deep learning.

6. Strict Saddles, Escaping Saddle Points, and Line-Search Methods

GD on nonconvex $C^2$ functions generically avoids strict saddle points: the probability of convergence to a strict saddle under random initialization is measure zero (Muşat et al., 18 Jul 2025), even for stabilized Armijo backtracking methods with arbitrarily large initial step-size and without global Lipschitz gradient assumption. The backtracking Armijo procedure

x_{k+1} = x_k - \alpha_k \nabla f(x_k)

with stabilization of $\alpha_k$ ensures avoidance of strict saddles for “almost every” choice of initial step size. These guarantees extend to Riemannian manifolds when the retraction is real analytic. However, they are asymptotic—the time to escape a saddle can still be exponential in the worst-case construction (Du et al., 2017).

There exist smooth, bounded, and non-pathological objectives for which GD requires $e^{\Omega(d)}$ iterations to escape $d$-many sequential saddle points, whereas perturbed GD achieves polynomial escape times (injecting random perturbations when the gradient norm is small). The set of iterates converging to a strict saddle is of measure zero, but practical convergence time can be very poor.

7. Minimax Optimality and Fast Rates on Linearly Separable Data

In linearly separable logistic regression, GD with large, risk-adaptive step sizes achieves arbitrarily small risk after at most $O(1/\gamma^2)$ steps ($\gamma$ is the data margin). This is minimax optimal among all first-order batch methods—no batch or online first-order approach can improve on this rate for the worst-case hard datasets constructed (Zhang et al., 5 Apr 2025). The step sizes have the form $\eta_t=\eta(-\ell^{-1})'(R(w_t))$, with $\ell$ the convex margin-based loss. The classical Perceptron algorithm matches this bound on the number of mistakes (updates). Extensions to generic convex losses and certain two-layer networks show analogous optimal rates.

8. Empirical and Diagnostic Insights

Empirical validation is extensive. Experimentally, adaptive learning rate schemes (PFTA, PTA) are competitive with or outperform Adam, RMSProp, and LBFGS in time to solution for toy and real-world tasks, with smoother descent curves and early plateauing of test/train accuracy in deep nets (Zhao et al., 10 Sep 2024). Variable-step proximal GD shows concretely halved iteration counts and wall-clock times relative to fixed-step methods in sparse regression (Nikolovski et al., 28 Dec 2024). Monitoring GD trajectories via basis decomposition provides a practical tool for diagnosing convergence—projection onto the conjugate kernel basis invariably yields monotonic convergence signals, supporting the theoretical framework (Ma et al., 2022).

9. Limitations, Pathologies, and Recommendations

Classical GD is sensitive to conditioning, step-size selection, and saddle-point geometry. Pathologies arise in the absence of global gradient Lipschitzness or when the iterates escape to infinity; convergence proofs always require careful monitoring of boundedness. The escape from strict saddles is generically assured but can be exponentially slow. Adaptive schemes and stochastic perturbations mitigate these issues. In deep learning, generalization relies critically on early stopping; the learned predictor may remain confined to the initial kernel-induced space unless explicit regularization or feature evolution is enforced (E et al., 2019).

10. Outlook and Further Directions

Research continues to generalize GD methods to manifold optimization, infinite-dimensional spaces, structured metrics, and finite-time convergence. Innovations in operator-theoretic, control-theoretic, and geometric analyses provide deeper understanding of convergence and stability phenomena. Future work will explore localized curvature-adaptive step rules, stochastic approximations in large-scale settings, and active regularization schemes to promote representation learning beyond kernel-induced implicit regularization.