Dynamic Learning Rate Scheduler (DLRS)

• Dynamic Learning Rate Scheduler (DLRS) is a mechanism that adjusts the optimizer step size during training using both predetermined schedules and feedback-driven controls.
• It employs diverse strategies including time-driven, loss-driven, and curvature-driven controllers that use signals like epoch count, loss trajectories, and sharpness estimates.
• DLRS integrates with standard optimizers (e.g., SGD, Adam) to modulate learning rates, reducing manual hyperparameter tuning and boosting convergence across various applications.

A dynamic learning rate scheduler (DLRS) is a mechanism that changes the optimizer step size during training rather than fixing a single schedule a priori. In the literature, the term covers both predetermined nonstationary schedules and feedback-driven controllers: local line-search-like schedulers, reinforcement-learned controllers, bandit-based selectors, sharpness- and curvature-aware rules, adversarial-loss regulators, parameter-trajectory decay rules, and loss-change heuristics all fall under this umbrella when they adapt the effective learning rate over time (Iyer et al., 2021, Xu et al., 2019, Dinkel et al., 2024, Naveen, 2024). The common objective is to reduce the manual burden of choosing constant, step, cosine, reciprocal-square-root, or warmup-plus-decay schedules, while matching the nonstationary dynamics of modern optimization more closely.

1. Scope and taxonomy

The contemporary DLRS literature uses “dynamic” in two distinct senses. In the weaker sense, the learning rate is simply time-varying; in the stronger sense, it is state-dependent and reacts online to losses, gradients, curvature proxies, or task-specific control signals. This distinction is explicit in work such as "Cyclical Log Annealing as a Learning Rate Scheduler," which is dynamic only because the rate changes with cycle position, not because it responds to observed optimization feedback, whereas "Learning an Adaptive Learning Rate Schedule" and "MLR-SNet" learn controllers whose outputs depend on training state or history (Naveen, 2024, Xu et al., 2019, Shu et al., 2020). AutoWU occupies an intermediate position: the warmup duration and transition point are determined online from the training-loss trajectory, but the post-warmup phase is a predefined cosine-style decay (Kim et al., 2021).

The resulting taxonomy is best understood by the control signal used to generate $\eta_t$.

Family	Primary control signal	Representative papers
Time-driven nonstationary schedules	Epoch, cycle index, restart interval	CLA (Naveen, 2024), AutoWU (Kim et al., 2021)
Loss/history-driven controllers	Training loss, validation loss, previous LR, recurrent state	RL controller (Xu et al., 2019), MLR-SNet (Shu et al., 2020), GreedyLR (Subramanian et al., 16 Dec 2025), loss-based DLRS (Veerababu et al., 29 Jul 2025)
Geometry- or curvature-driven schedulers	Local quadratic fit, sharpness, Hessian-informed group curvature	LRTuner (Iyer et al., 2021), SALR (Yue et al., 2020), Hi-DLR (Xu et al., 12 Jan 2025)
Domain-structured controllers	Adversarial optimality gap, RL returns, parameter oscillation	Gap-aware scheduler (Hazimeh et al., 2023), LRRL (Donâncio et al., 2024), DLRD (Dinkel et al., 2024)

A recurring conceptual boundary in this literature is between a scheduler and an optimizer. Many DLRS methods leave the update direction $d_t$ to a base optimizer and only modulate the scalar or group-wise step size, whereas adaptive optimizers such as Adam or RMSProp additionally reshape the direction itself. Several papers make this complementarity explicit rather than treating scheduling and adaptive optimization as substitutes (Xu et al., 2019, Dinkel et al., 2024).

2. Core mathematical formulations

Most DLRS work can be expressed through the generic update

$\theta_{t+1} = \theta_t - \eta_t d_t,$

where $d_t$ is the direction produced by SGD, momentum SGD, Adam, AdamW, or another base method. The scheduler’s role is to generate $\eta_t$, or a vector of group-wise learning rates, from local information.

One major formulation treats LR selection as a one-dimensional local optimization problem. LRTuner models the next-step loss along the current search direction as

$\hat L(\epsilon)=k_0+k_1\epsilon+k_2\epsilon^2,$

with $\epsilon$ a perturbation of the current learning rate, and chooses

$\epsilon_{\min} = -\frac{k_1}{2k_2}.$

The key point is that the quadratic is fitted from loss evaluations along the current direction rather than from explicit Hessian computation (Iyer et al., 2021). AutoLRS uses a stage-wise variant of the same idea: every $\tau$ steps it chooses one constant learning rate by approximately solving

$$\min_{\eta} \sum_{x\in D_{val}} L(x;\theta_{t+\tau}(\eta)),$$

with Bayesian optimization and an exponential extrapolation model used to avoid full-horizon evaluation for every candidate (Jin et al., 2021). Hi-DLR generalizes the scalar step size to group-wise rates and, under a diagonal quadratic approximation, uses

$d_t$0

for parameter group $d_t$1 (Xu et al., 12 Jan 2025).

A second formulation uses multiplicative control. The reinforcement-learning scheduler in "Learning an Adaptive Learning Rate Schedule" outputs an action $d_t$2 and updates the global LR by

$d_t$3

with the controller trained by PPO on validation-loss-based reward (Xu et al., 2019). GreedyLR follows an even simpler multiplicative rule: if the current loss improves, divide the previous learning rate by a factor $d_t$4; otherwise multiply by $d_t$5 (Subramanian et al., 16 Dec 2025).

A third family uses direct feedback laws rather than surrogate minimization. SALR computes a sharpness estimate $d_t$6 from normalized ascent and descent probes and sets

$d_t$7

thereby increasing the rate in locally sharper regions and decreasing it in flatter ones (Yue et al., 2020). The gap-aware scheduler for adversarial nets adjusts only the adversary LR so as to keep the adversary loss near a known ideal constant $d_t$8, with multiplicative increase or decrease depending on the sign of $d_t$9 (Hazimeh et al., 2023). DLRD for stochastic variational inference decays the base rate when a signal-to-noise statistic computed from the parameter trajectory falls below a threshold (Dinkel et al., 2024).

3. Control signals and adaptation variables

The most important organizing principle in DLRS research is the choice of state variable. Some controllers use compact, architecture-agnostic histories. The RL scheduler of (Xu et al., 2019) uses current train loss, validation loss, variance of network predictions, variance of network prediction changes, mean and variance of the weight matrix of the final dense layer, and previous-step learning rate. MLR-SNet uses only current minibatch training loss plus an LSTM hidden state, so its recurrence carries the temporal information needed to map loss history to a current LR (Shu et al., 2020).

Loss-based controllers are the simplest and cheapest. GreedyLR reacts to whether $\theta_{t+1} = \theta_t - \eta_t d_t,$0 or not, whereas the 2025 loss-based DLRS computes a normalized within-epoch loss slope

$\theta_{t+1} = \theta_t - \eta_t d_t,$1

from first-batch, last-batch, and mean batch loss, then adjusts the next epoch’s rate through a piecewise rule on $\theta_{t+1} = \theta_t - \eta_t d_t,$2 (Subramanian et al., 16 Dec 2025, Veerababu et al., 29 Jul 2025). AutoWU also uses loss trajectories, but it does not compare raw successive losses; it smooths the observed curve with a Gaussian process and declares that warmup should end once earlier points are very likely to have lower latent loss than the current endpoint (Kim et al., 2021). AutoLRS is likewise loss-driven, but with stage-end validation loss as the decision objective rather than training loss (Jin et al., 2021).

Other methods use richer geometric or dynamical signals. RDBD uses the agreement statistic

$\theta_{t+1} = \theta_t - \eta_t d_t,$3

and checks whether successive correlation signs are consistent; if not, it regrets the previous LR adaptation and rolls it back (Song et al., 2023). SALR measures local sharpness by probing the loss around the current iterate with normalized ascent and descent steps (Yue et al., 2020). DLRD quantifies oscillation versus trend by regressing the parameter trajectory against iteration index and computing an empirical signal-to-noise ratio (Dinkel et al., 2024). LRRL uses cumulative returns and an improvement signal

$\theta_{t+1} = \theta_t - \eta_t d_t,$4

to choose among discrete learning-rate arms in nonstationary deep RL (Donâncio et al., 2024).

The literature also contains domain-structured signals that are not intended to be universal. The gap-aware adversarial scheduler relies on the fact that in several GAN and domain-adversarial objectives the loss of an ideal adversary is known a priori, such as $\theta_{t+1} = \theta_t - \eta_t d_t,$5 for standard GAN/NSGAN, $\theta_{t+1} = \theta_t - \eta_t d_t,$6 for WGAN, and $\theta_{t+1} = \theta_t - \eta_t d_t,$7 for LSGAN (Hazimeh et al., 2023). This shows that DLRS need not be generic; in some settings the strongest controller is tied to equilibrium structure specific to the task.

4. Interaction with optimizers and parameter granularity

A defining feature of many DLRS methods is that they are layered on top of existing optimizers rather than replacing them. LRTuner requires only the current search direction $\theta_{t+1} = \theta_t - \eta_t d_t,$8, so it can wrap SGD with momentum, Adam, or AdamW (Iyer et al., 2021). AutoLRS is explicitly orthogonal to SGD and Adam-family optimizers and chooses one stage-wise global rate on top of them (Jin et al., 2021). DLRD is designed as a scheduler on the base learning rate for SGD, Adam, RMSprop, AdaMax, and Adagrad (Dinkel et al., 2024). AutoWU is likewise a scheduler around AdamP or LAMB in the large-batch setting (Kim et al., 2021). The experimental notes supplied for "Locally Optimal Descent for Dynamic Stepsize Scheduling" also frame the method as a scheduler layer tested with SGD, momentum SGD, and Adam rather than as a standalone optimizer (Yehudai et al., 2023).

Granularity varies substantially. Many methods adjust a single global scalar LR. This includes the RL controller, MLR-SNet, GreedyLR, AutoWU, AutoLRS, CLA, SALR, and the loss-based epoch controller (Xu et al., 2019, Shu et al., 2020, Subramanian et al., 16 Dec 2025). A second group operates at parameter-group level. Hi-DLR assigns one learning rate per user-defined group and is explicitly motivated by differential learning rate, PEFT, LoRA, BitFit, and heterogeneous module curvature (Xu et al., 12 Jan 2025). The CelebA multi-task experiment in Hi-DLR uses 40 group learning rates, one per task-specific output group. A third group reaches per-weight adaptation. Adaptive AutoLR evolves full optimizers with auxiliary state variables that can “fine tune the learning rate for each network weight,” and the evolved ADES update is therefore closer to an optimizer than to a classical scheduler (Carvalho et al., 2021).

This distinction matters because the scheduler literature often overlaps with adaptive optimization without becoming identical to it. Adam and RMSProp change effective per-parameter step sizes through moment statistics; DLRS methods typically control the global or group-wise scalar multiplying those directions. Several papers present this layering as the intended use mode rather than an implementation detail (Xu et al., 2019, Yue et al., 2020).

5. Empirical evidence across tasks and domains

The empirical record for DLRS is broad rather than uniform. Some methods target conventional supervised learning, others adversarial games, large-batch training, deep RL, variational inference, PINNs, or PEFT. Reported gains therefore vary with objective, architecture, and measurement protocol.

Setting	Method	Reported outcome
CIFAR-10 ResNet	RL controller (Xu et al., 2019)	test accuracy $\theta_{t+1} = \theta_t - \eta_t d_t,$9
ImageNet ResNet-50	LRTuner (Iyer et al., 2021)	top-1 $d_t$0; same accuracy in 29% fewer optimization steps
ResNet-50 / Transformer / BERT	AutoLRS (Jin et al., 2021)	speedups of $d_t$1, $d_t$2, and $d_t$3
GANs and DANN	Gap-aware scheduler (Hazimeh et al., 2023)	CelebA tuning budget about one-tenth; up to 27% improvement in FID and 3% in test accuracy
Large-batch CV	AutoWU (Kim et al., 2021)	CIFAR-100 batch 16K: $d_t$4; ImageNet batch 32K: $d_t$5
PINNs and CIFAR-10 CNNs	Loss-based DLRS (Veerababu et al., 29 Jul 2025)	relative error less than 1% for PINNs; VGG-19 $d_t$6, GoogLeNet $d_t$7

Additional evidence broadens rather than simplifies the picture. SALR reports substantial gains across CIFAR, ImageNet, text prediction, and fine-tuning; for example, on CIFAR-10 with ResNet50, SGD-SALR improves test accuracy from $d_t$8 to $d_t$9, and on ImageNet with ResNet152 under matched gradient-call budgets, SGD-SALR reaches $\eta_t$0 compared with $\eta_t$1 for SAM in the longest-budget comparison (Yue et al., 2020). MLR-SNet reports $\eta_t$2 on CIFAR-10 and $\eta_t$3 on CIFAR-100, outperforming the fixed, multistep, exponential, SGDR, Adam, L4, HD, and RTHO baselines listed in that study (Shu et al., 2020). Hi-DLR improves over LoRA with a single LR on 4 of 5 GLUE datasets in the reported table, including CoLA $\eta_t$4 and MRPC $\eta_t$5, while remaining slightly worse on QNLI $\eta_t$6 (Xu et al., 12 Jan 2025). In deep RL, LRRL can substantially improve DQN performance on some Atari tasks, especially with Adam, but is explicitly “not uniformly superior” across all games and optimizer variants (Donâncio et al., 2024).

The literature also contains mixed or negative results, which are integral to any encyclopedia account. CLA is presented as roughly analogous to cosine annealing rather than decisively better, and the paper reports training-loss curves only rather than test accuracy or validation accuracy (Naveen, 2024). The loss-based DLRS for PINNs and image classification shows only a $\eta_t$7 gain for MobileNetV2 on CIFAR-10, despite larger gains for VGG-19 and GoogLeNet (Veerababu et al., 29 Jul 2025). Dynamic AutoLR’s evolved Policy A outperforms the static baseline in a 100-epoch benchmark without early stopping, but under early stopping the same policy exhibits very large variance and underperforms the baseline on average (Carvalho et al., 2021). These results indicate that DLRS performance is strongly mediated by protocol, horizon, and interaction with the underlying training loop.

6. Limitations, ambiguities, and recurring controversies

A first recurring controversy is terminological. Not every nonconstant schedule is adaptive. CLA is explicitly a predefined restart schedule indexed by $\eta_t$8 and $\eta_t$9, and the paper states that in the adaptive DLRS sense it is “not adaptive” (Naveen, 2024). AutoWU is adaptive during warmup discovery but then reverts to a predefined decay (Kim et al., 2021). This matters because papers often compare feedback-driven controllers against time-driven baselines under the same “dynamic” label.

A second issue is computational overhead. The spectrum is wide. GreedyLR changes LR from observed loss with negligible extra machinery; the loss-based epoch controller reports asymptotic overhead $\hat L(\epsilon)=k_0+k_1\epsilon+k_2\epsilon^2,$0 per epoch, “< 0.5% wall-clock overhead per epoch,” and “< 0.1% peak memory overhead” (Subramanian et al., 16 Dec 2025, Veerababu et al., 29 Jul 2025). AutoWU reports less than one second on average for each epoch-end GP fitting and inference block (Kim et al., 2021). LRTuner’s probing overhead is bounded at about $\hat L(\epsilon)=k_0+k_1\epsilon+k_2\epsilon^2,$1 per ImageNet epoch and about $\hat L(\epsilon)=k_0+k_1\epsilon+k_2\epsilon^2,$2 on IWSLT in the cited setups (Iyer et al., 2021). AutoLRS is explicitly more expensive, with total cost described as only slightly above $\hat L(\epsilon)=k_0+k_1\epsilon+k_2\epsilon^2,$3 the model-update time associated with the discovered schedule (Jin et al., 2021). RL-based controllers and evolutionary search add an even larger outer-loop burden because they require repeated full or partial training episodes (Xu et al., 2019, Carvalho et al., 2021).

A third issue is uneven theoretical coverage. Some papers provide convergence analyses, but usually under restricted assumptions: GreedyLR is analyzed for smooth convex SGD with bounded learning rates (Subramanian et al., 16 Dec 2025); RDBD proves a mini-batch convergence theorem under smoothness, bounded stochastic updates, and unbiasedness (Song et al., 2023); SALR’s main theoretical result is local and based on strong convexity in a neighborhood (Yue et al., 2020). By contrast, several practically motivated methods provide no full convergence proof. The gap-aware adversarial scheduler is explicitly empirical rather than theorem-driven (Hazimeh et al., 2023). DLRD notes that it does not prove Robbins–Monro-type properties for its data-dependent decay sequence (Dinkel et al., 2024). CLA provides no formal convergence proof (Naveen, 2024).

A fourth limitation concerns reporting quality and reproducibility. The supplied material for "Locally Optimal Descent for Dynamic Stepsize Scheduling" states that the document fragment is not a full paper but an experimental notes/results sheet without explicit update equations, theorem statements, or pseudocode, even though it compares “greedy + hessian” and “greedy + GNB” against cosine, constant, rsqrt, and other baselines across CIFAR-10, CIFAR-100, ImageNet, Criteo, WikiText, and FastMRI (Yehudai et al., 2023). CLA also contains malformed equations, including its printed logarithmic scheduler formula and softmax expression, so reconstruction requires interpretation (Naveen, 2024). More generally, some papers report validation or test metrics, whereas others emphasize training loss only. This makes direct comparison across DLRS papers structurally difficult.

Finally, the literature repeatedly returns to a common practical conclusion: DLRS reduces but does not eliminate tuning. LRRL still requires choosing the number and values of arms (Donâncio et al., 2024). Hi-DLR depends on externally chosen parameter grouping (Xu et al., 12 Jan 2025). LRTuner still needs an explore duration and an epsilon threshold (Iyer et al., 2021). AutoWU fixes its own confidence, patience, and warmup-range hyperparameters, even though they are reused across experiments (Kim et al., 2021). This suggests that the central research question has shifted from eliminating hyperparameters to choosing control signals and structural priors that make the remaining hyperparameters less task-specific and less burdensome than manually scripting the entire schedule.