Optimal Learning Rate (OLR) Strategies

• Optimal Learning Rate (OLR) is the step-size parameter or schedule in iterative optimization that balances convergence speed, stability, and generalization.
• OLR frameworks leverage mathematical derivations, adaptive algorithms like LQA, LOSSGRAD, and control-theoretic approaches to establish near-optimal training dynamics in neural networks.
• Practical insights include using warmup, decay, and capacity matching strategies to tune learning rates effectively across regimes such as SGD, Adam, LoRA, and sparse networks.

The optimal learning rate (OLR) is the step-size parameter or schedule in iterative optimization, particularly stochastic gradient descent or its variants, that yields the most favorable trade-off between convergence speed, stability, and generalization error in neural network and related models. This concept spans both static values and dynamic schedules, and has been formalized through diverse mathematical frameworks including optimal control, functional scaling laws, statistical learning bounds, and algorithmic search—each with rigorously established criteria and often closed-form prescriptions for optimality.

1. Mathematical Formulations and Phase Behavior

A principal theoretical advance is the derivation of optimal learning rate schedules under the functional scaling law (FSL) framework, in which the loss dynamics of models such as linear regression and LLM pretraining are governed by intrinsic signal and capacity exponents ($s>0$, $\beta>1$). The OLR in these regimes exhibits a sharp phase transition:

• Easy regime ($s \geq 1 - 1/\beta$): The optimal schedule is a power-law annealing of the form $$\eta^*(z) = \eta_{\text{peak}} (1 - z/N)^{2\beta-1}$$, where $$\eta_{\text{peak}} \sim N^{-(1+s\beta-\beta)/(1+s\beta)}$$. This yields minimax-optimal learning rates and final loss scaling as $N^{-s\beta/(1+s\beta)}$.
• Hard regime ($s < 1 - 1/\beta$): The OLR consists of a warmup-stable-decay (WSD) structure: a prolonged period at the largest stable learning rate (capacity-limited), followed by a power-law decay over a vanishing fraction of the training horizon. Here, loss scales as $N^{-s}$, matching the best possible SGD lower bound and favoring stability over aggressive annealing (Li et al., 6 Feb 2026, Bordelon et al., 4 Feb 2026).

For power-law random-feature models (notably, SGD in infinite-width kernel or neural settings), the OLR recapitulates these results, with polynomial-decay (easy phase) and warmup-stable-decay (hard phase) schedules provably outperforming all constant and standard power-law baselines (Bordelon et al., 4 Feb 2026).

2. Algorithmic and Adaptive Approaches to OLR

Multiple algorithms dynamically estimate or search for OLR:

• Local Quadratic Approximation (LQA): Estimates the optimal step $\alpha^* = \|g\|^2/(g^T H g)$ in each update by fitting a local quadratic surrogate; avoids explicit or full Hessian computations by local finite-differences and works per-batch (Zhu et al., 2020).
• LOSSGRAD: Performs a single-step quadratic fit along the gradient direction, adaptively doubling or halving $h$ based on whether $\beta>1$0 or $\beta>1$1, guaranteeing a robust search for locally optimal step-size in SGD and providing strong empirical convergence except where stochasticity or non-quadraticity degrade approximation (Wójcik et al., 2019).
• AdaLRS: Seeks the $\beta>1$2 that maximizes the instantaneous loss descent velocity, relying on the empirical convexity (unimodality) of loss or descent velocity in $\beta>1$3 to identify $\beta>1$4 via sequential, slope-guided upscaling and downscaling (Dong et al., 16 Jun 2025).
• Autonomous Learning Rate Controller (ARC): Implements a supervised meta-controller, trained on real training histories, to select increase/keep/decrease actions for the LR based on recent training and validation loss trajectories, achieving automated adaptation across models, datasets, and optimizers (Dong et al., 2021).
• A Simple Dynamic LR Tuning (AALR): Maintains a scalar $\beta>1$5 (learning rate) and patience $\beta>1$6, applying doubling or halving based on improvement in loss within patience windows, which provably tracks the unknown oracle OLR schedule within a constant factor (Mukherjee et al., 2019).

3. OLR in Specialized Regimes and Optimizers

• Adam-type (sign-gradient) optimizers: Contrary to the linear or $\beta>1$7 scaling in SGD, the OLR as a function of batch size $\beta>1$8 exhibits a "surge" phenomenon: $\beta>1$9 grows with $s \geq 1 - 1/\beta$0 up to $s \geq 1 - 1/\beta$1 and then decreases as $s \geq 1 - 1/\beta$2 increases, with explicit non-monotonic scaling laws derived from Gaussian CLT approximations (Li et al., 2024).
• Low-Rank Adaptation (LoRA): The OLR scaling with adapter rank $s \geq 1 - 1/\beta$3 depends on initialization and the LoRA scaling factor. For typical configurations (Init[A], $s \geq 1 - 1/\beta$4 or Init[B], $s \geq 1 - 1/\beta$5), OLR is nearly rank-invariant, enabling transfer from LoRA experiments to full fine-tuning without the need for per-rank LR sweeping. In other cases, OLR decays as $s \geq 1 - 1/\beta$6, requiring rescaling in grid searches (Chen et al., 5 Feb 2026).
• Pruning and sparse network regimes: The per-cycle OLR maximizing restoration of gradient energy follows an S-shaped profile ("SILO"), with theoretical justification linking the need to increase LR at higher sparsity due to the contraction of activation and gradient norm distributions. Empirically, SILO matches the performance of exhaustive Oracle search at a fraction of the search cost (Liu et al., 2022).

4. Search Procedures and Empirical OLR Tuning

• Parameterization of schedule families: Empirically near-optimal schedule shapes factor peak LR and shape, $s \geq 1 - 1/\beta$7, allowing for robust optimization over shape families (cosine, REX, TPS, etc.) and separation from the instability-constraining $s \geq 1 - 1/\beta$8 (Naganuma et al., 11 Mar 2026).
• Bandit-based search: Lipschitz bandit/Zooming algorithms treat the LR as a 1D arm; adaptive discretization rapidly eliminates suboptimal α and concentrates evaluations around the maximizer, with regret guarantees and empirically superior sample efficiency relative to alternatives (Priyanka et al., 2024).

5. Practical Guidelines and Design Principles

• Schedule shape: Always decouple base rate tuning ($s \geq 1 - 1/\beta$9) from shape tuning (decay law, warmup), and perform log-spaced grid searches over at least 16 values of $$\eta^*(z) = \eta_{\text{peak}} (1 - z/N)^{2\beta-1}$$0 up to instability.
• Warmup and decay: Robust OLR schedules universally feature explicit linear warmup (10-30% of total steps) followed by monotonic decay, with decay starting later under strong regularization or high momentum.
• Capacity/exponent matching: Match the decay exponent (e.g., in power-law, cosine, or linear schedules) to the model's capacity exponent for optimal generalization error scaling. For high-capacity models ($$\eta^*(z) = \eta_{\text{peak}} (1 - z/N)^{2\beta-1}$$1), use decay shapes with sufficient tail exponent ($$\eta^*(z) = \eta_{\text{peak}} (1 - z/N)^{2\beta-1}$$2) to avoid capacity saturation (Li et al., 6 Feb 2026).
• Rank and batch-scale adjustment: For LoRA or large-batch regimes, reference the largest Hessian eigenvalue at initialization ($$\eta^*(z) = \eta_{\text{peak}} (1 - z/N)^{2\beta-1}$$3) or use prescribed invariance/skewing rules (Chen et al., 5 Feb 2026); in Adam-style optimizers, adaptively monitor $$\eta^*(z) = \eta_{\text{peak}} (1 - z/N)^{2\beta-1}$$4 and tune LR to the predicted surge region (Li et al., 2024).
• RL and stochastic approximation: OLR policies for RL combine online error/variance tracking ("PAst Sign Search" at the inner level) with one-step minimax-optimal outer schedules. The convergence can be accelerated to $$\eta^*(z) = \eta_{\text{peak}} (1 - z/N)^{2\beta-1}$$5 with appropriate adaptation (Mounjid et al., 2019).

6. Control-Theoretic and Normative Frameworks

Recent work has cast OLR scheduling as a continuous-time optimal control problem, yielding closed-loop controllers of the form

$$\eta^*(z) = \eta_{\text{peak}} (1 - z/N)^{2\beta-1}$$6

where $$\eta^*(z) = \eta_{\text{peak}} (1 - z/N)^{2\beta-1}$$7 is the current performance and $$\eta^*(z) = \eta_{\text{peak}} (1 - z/N)^{2\beta-1}$$8 is a user- or agent-dependent cost parameter. This closed-loop solution adapts in real time to both observed progress and estimated horizon, generalizing across architectures and providing a normative link from behavior to effort allocation and engagement (Njaradi et al., 12 Jan 2026). Episodic memory of past learning curves enables practical estimation of $$\eta^*(z) = \eta_{\text{peak}} (1 - z/N)^{2\beta-1}$$9 without manual tuning.

7. Impact, Limitations, and Outlook

Empirical studies consistently show that theoretically informed or adaptively estimated OLR policies enable lower final loss, improved accuracy, and more robust convergence compared to static or hand-crafted baseline schedules (Zhu et al., 2020, Mukherjee et al., 2019, Dong et al., 16 Jun 2025, Liu et al., 2022, Naganuma et al., 11 Mar 2026). Limitations are noted for highly non-convex loss landscapes, extreme initializations, and under highly stochastic gradients, where local quadratic or convexity assumptions can fail. Current research directions include robustifying OLR estimation methods, integrating episodic memory and meta-optimization, and extending control-theoretic OLR controllers to deep and reinforcement learning regimes with complex non-stationarity and delayed credit assignment.