---
title: Dynamic Learning Rate Scheduler (DLRS)
url: https://www.emergentmind.com/topics/dynamic-learning-rate-scheduler-dlrs
type: topic
---

# Dynamic Learning Rate Scheduler (DLRS)

A dynamic learning rate scheduler (DLRS) is a mechanism that changes the optimizer step size during training rather than fixing a single schedule a priori. In the literature, the term covers both predetermined nonstationary schedules and feedback-driven controllers: local line-search-like schedulers, reinforcement-learned controllers, bandit-based selectors, sharpness- and curvature-aware rules, adversarial-loss regulators, parameter-trajectory decay rules, and loss-change heuristics all fall under this umbrella when they adapt the effective learning rate over time [2105.14526] [1909.09712] [2412.15745] [2403.14685]. The common objective is to reduce the manual burden of choosing constant, step, cosine, reciprocal-square-root, or warmup-plus-decay schedules, while matching the nonstationary dynamics of modern optimization more closely.

## 1. Scope and taxonomy

The contemporary DLRS literature uses “dynamic” in two distinct senses. In the weaker sense, the learning rate is simply time-varying; in the stronger sense, it is state-dependent and reacts online to losses, gradients, curvature proxies, or task-specific control signals. This distinction is explicit in work such as "Cyclical Log Annealing as a Learning Rate Scheduler," which is dynamic only because the rate changes with cycle position, not because it responds to observed optimization feedback, whereas "Learning an Adaptive Learning Rate Schedule" and "MLR-SNet" learn controllers whose outputs depend on training state or history [2403.14685] [1909.09712] [2007.14546]. AutoWU occupies an intermediate position: the warmup duration and transition point are determined online from the training-loss trajectory, but the post-warmup phase is a predefined cosine-style decay [2107.05855].

The resulting taxonomy is best understood by the control signal used to generate $\eta_t$.

| Family | Primary control signal | Representative papers |
|---|---|---|
| Time-driven nonstationary schedules | Epoch, cycle index, restart interval | CLA [2403.14685], AutoWU [2107.05855] |
| Loss/history-driven controllers | Training loss, validation loss, previous LR, recurrent state | RL controller [1909.09712], MLR-SNet [2007.14546], GreedyLR [2512.14527], loss-based DLRS [2507.21749] |
| Geometry- or curvature-driven schedulers | Local quadratic fit, sharpness, Hessian-informed group curvature | LRTuner [2105.14526], SALR [2011.05348], Hi-DLR [2501.06954] |
| Domain-structured controllers | Adversarial optimality gap, RL returns, parameter oscillation | Gap-aware scheduler [2302.00089], LRRL [2410.12598], DLRD [2412.15745] |

A recurring conceptual boundary in this literature is between a scheduler and an optimizer. Many DLRS methods leave the update direction $d_t$ to a base optimizer and only modulate the scalar or group-wise step size, whereas adaptive optimizers such as Adam or RMSProp additionally reshape the direction itself. Several papers make this complementarity explicit rather than treating scheduling and adaptive optimization as substitutes [1909.09712] [2412.15745].

## 2. Core mathematical formulations

Most DLRS work can be expressed through the generic update
$$
\theta_{t+1} = \theta_t - \eta_t d_t,
$$
where $d_t$ is the direction produced by SGD, momentum SGD, Adam, AdamW, or another base method. The scheduler’s role is to generate $\eta_t$, or a vector of group-wise learning rates, from local information.

One major formulation treats LR selection as a one-dimensional local optimization problem. LRTuner models the next-step loss along the current search direction as
$$
\hat L(\epsilon)=k_0+k_1\epsilon+k_2\epsilon^2,
$$
with $\epsilon$ a perturbation of the current learning rate, and chooses
$$
\epsilon_{\min} = -\frac{k_1}{2k_2}.
$$
The key point is that the quadratic is fitted from loss evaluations along the current direction rather than from explicit Hessian computation [2105.14526]. AutoLRS uses a stage-wise variant of the same idea: every $\tau$ steps it chooses one constant learning rate by approximately solving
$$
\min_{\eta} \sum_{x\in D_{val}} L(x;\theta_{t+\tau}(\eta)),
$$
with Bayesian optimization and an exponential extrapolation model used to avoid full-horizon evaluation for every candidate [2105.10762]. Hi-DLR generalizes the scalar step size to group-wise rates and, under a diagonal quadratic approximation, uses
$$
\eta_k^*=\frac{G_{(k)}^\top g_{(k)}}{g_{(k)}^\top H_{(kk)}g_{(k)}}
$$
for parameter group $k$ [2501.06954].

A second formulation uses multiplicative control. The reinforcement-learning scheduler in "Learning an Adaptive Learning Rate Schedule" outputs an action $a_t$ and updates the global LR by
$$
\eta_t = a_t \eta_{t-1},
$$
with the controller trained by PPO on validation-loss-based reward [1909.09712]. GreedyLR follows an even simpler multiplicative rule: if the current loss improves, divide the previous learning rate by a factor $F\in(0,1)$; otherwise multiply by $F$ [2512.14527].

A third family uses direct feedback laws rather than surrogate minimization. SALR computes a sharpness estimate $\widehat S_k$ from normalized ascent and descent probes and sets
$$
\eta_k= \eta_0 \frac{S}{\operatorname{Median}\left\{\widehat{S}_i\right\}_{i=1}^k},
$$
thereby increasing the rate in locally sharper regions and decreasing it in flatter ones [2011.05348]. The gap-aware scheduler for adversarial nets adjusts only the adversary LR so as to keep the adversary loss near a known ideal constant $V^*$, with multiplicative increase or decrease depending on the sign of $V_d-V^*$ [2302.00089]. DLRD for stochastic variational inference decays the base rate when a signal-to-noise statistic computed from the parameter trajectory falls below a threshold [2412.15745].

## 3. Control signals and adaptation variables

The most important organizing principle in DLRS research is the choice of state variable. Some controllers use compact, architecture-agnostic histories. The RL scheduler of [1909.09712] uses current train loss, validation loss, variance of network predictions, variance of network prediction changes, mean and variance of the weight matrix of the final dense layer, and previous-step learning rate. MLR-SNet uses only current minibatch training loss plus an LSTM hidden state, so its recurrence carries the temporal information needed to map loss history to a current LR [2007.14546].

Loss-based controllers are the simplest and cheapest. GreedyLR reacts to whether $l_t<l_{t-1}$ or not, whereas the 2025 loss-based DLRS computes a normalized within-epoch loss slope
$$
\Delta L_j = \frac{L_j^{(B)} - L_j^{(1)}}{\overline{L}_j}
$$
from first-batch, last-batch, and mean batch loss, then adjusts the next epoch’s rate through a piecewise rule on $\Delta L_j$ [2512.14527] [2507.21749]. AutoWU also uses loss trajectories, but it does not compare raw successive losses; it smooths the observed curve with a Gaussian process and declares that warmup should end once earlier points are very likely to have lower latent loss than the current endpoint [2107.05855]. AutoLRS is likewise loss-driven, but with stage-end validation loss as the decision objective rather than training loss [2105.10762].

Other methods use richer geometric or dynamical signals. RDBD uses the agreement statistic
$$
h_t = \langle g_t,g_{t-1}\rangle
$$
and checks whether successive correlation signs are consistent; if not, it regrets the previous LR adaptation and rolls it back [2310.11291]. SALR measures local sharpness by probing the loss around the current iterate with normalized ascent and descent steps [2011.05348]. DLRD quantifies oscillation versus trend by regressing the parameter trajectory against iteration index and computing an empirical signal-to-noise ratio [2412.15745]. LRRL uses cumulative returns and an improvement signal
$$
f_n' = f_n - \frac{1}{j} \sum_{i=0}^{j-1} f_{n-i}
$$
to choose among discrete learning-rate arms in nonstationary deep RL [2410.12598].

The literature also contains domain-structured signals that are not intended to be universal. The gap-aware adversarial scheduler relies on the fact that in several GAN and domain-adversarial objectives the loss of an ideal adversary is known a priori, such as $V^*=\log(4)$ for standard GAN/NSGAN, $V^*=0$ for WGAN, and $V^*=0.5$ for LSGAN [2302.00089]. This shows that DLRS need not be generic; in some settings the strongest controller is tied to equilibrium structure specific to the task.

## 4. Interaction with optimizers and parameter granularity

A defining feature of many DLRS methods is that they are layered on top of existing optimizers rather than replacing them. LRTuner requires only the current search direction $\vec d$, so it can wrap SGD with momentum, Adam, or AdamW [2105.14526]. AutoLRS is explicitly orthogonal to SGD and Adam-family optimizers and chooses one stage-wise global rate on top of them [2105.10762]. DLRD is designed as a scheduler on the base learning rate for SGD, Adam, RMSprop, AdaMax, and Adagrad [2412.15745]. AutoWU is likewise a scheduler around AdamP or LAMB in the large-batch setting [2107.05855]. The experimental notes supplied for "Locally Optimal Descent for Dynamic Stepsize Scheduling" also frame the method as a scheduler layer tested with SGD, momentum SGD, and Adam rather than as a standalone optimizer [2311.13877].

Granularity varies substantially. Many methods adjust a single global scalar LR. This includes the RL controller, MLR-SNet, GreedyLR, AutoWU, AutoLRS, CLA, SALR, and the loss-based epoch controller [1909.09712] [2007.14546] [2512.14527]. A second group operates at parameter-group level. Hi-DLR assigns one learning rate per user-defined group and is explicitly motivated by differential learning rate, PEFT, LoRA, BitFit, and heterogeneous module curvature [2501.06954]. The CelebA multi-task experiment in Hi-DLR uses 40 group learning rates, one per task-specific output group. A third group reaches per-weight adaptation. Adaptive AutoLR evolves full optimizers with auxiliary state variables that can “fine tune the learning rate for each network weight,” and the evolved ADES update is therefore closer to an optimizer than to a classical scheduler [2103.12623].

This distinction matters because the scheduler literature often overlaps with adaptive optimization without becoming identical to it. Adam and RMSProp change effective per-parameter step sizes through moment statistics; DLRS methods typically control the global or group-wise scalar multiplying those directions. Several papers present this layering as the intended use mode rather than an implementation detail [1909.09712] [2011.05348].

## 5. Empirical evidence across tasks and domains

The empirical record for DLRS is broad rather than uniform. Some methods target conventional supervised learning, others adversarial games, large-batch training, deep RL, variational inference, PINNs, or PEFT. Reported gains therefore vary with objective, architecture, and measurement protocol.

| Setting | Method | Reported outcome |
|---|---|---|
| CIFAR-10 ResNet | RL controller [1909.09712] | test accuracy \(0.7395 \pm 0.0206 \rightarrow 0.8181 \pm 0.0069\) |
| ImageNet ResNet-50 | LRTuner [2105.14526] | top-1 \(75.87 \rightarrow 76.06\); same accuracy in 29% fewer optimization steps |
| ResNet-50 / Transformer / BERT | AutoLRS [2105.10762] | speedups of \(1.22\times\), \(1.43\times\), and \(1.5\times\) |
| GANs and DANN | Gap-aware scheduler [2302.00089] | CelebA tuning budget about one-tenth; up to 27% improvement in FID and 3% in test accuracy |
| Large-batch CV | AutoWU [2107.05855] | CIFAR-100 batch 16K: \(77.62 \rightarrow 81.42\); ImageNet batch 32K: \(74.11 \rightarrow 74.84\) |
| PINNs and CIFAR-10 CNNs | Loss-based DLRS [2507.21749] | relative error less than 1% for PINNs; VGG-19 \(89.28 \rightarrow 91.98\), GoogLeNet \(89.78 \rightarrow 92.90\) |

Additional evidence broadens rather than simplifies the picture. SALR reports substantial gains across CIFAR, ImageNet, text prediction, and fine-tuning; for example, on CIFAR-10 with ResNet50, SGD-SALR improves test accuracy from \(93.25\) to \(94.94\), and on ImageNet with ResNet152 under matched gradient-call budgets, SGD-SALR reaches \(81.70\) compared with \(81.59\) for SAM in the longest-budget comparison [2011.05348]. MLR-SNet reports \(94.80 \pm 0.10\) on CIFAR-10 and \(80.44 \pm 0.17\) on CIFAR-100, outperforming the fixed, multistep, exponential, SGDR, Adam, L4, HD, and RTHO baselines listed in that study [2007.14546]. Hi-DLR improves over LoRA with a single LR on 4 of 5 GLUE datasets in the reported table, including CoLA \(69.13 \rightarrow 81.59\) and MRPC \(75.49 \rightarrow 85.78\), while remaining slightly worse on QNLI \(91.05 \rightarrow 90.48\) [2501.06954]. In deep RL, LRRL can substantially improve DQN performance on some Atari tasks, especially with Adam, but is explicitly “not uniformly superior” across all games and optimizer variants [2410.12598].

The literature also contains mixed or negative results, which are integral to any encyclopedia account. CLA is presented as roughly analogous to cosine annealing rather than decisively better, and the paper reports training-loss curves only rather than test accuracy or validation accuracy [2403.14685]. The loss-based DLRS for PINNs and image classification shows only a \(+0.07\) gain for MobileNetV2 on CIFAR-10, despite larger gains for VGG-19 and GoogLeNet [2507.21749]. Dynamic AutoLR’s evolved Policy A outperforms the static baseline in a 100-epoch benchmark without early stopping, but under early stopping the same policy exhibits very large variance and underperforms the baseline on average [2103.12623]. These results indicate that DLRS performance is strongly mediated by protocol, horizon, and interaction with the underlying training loop.

## 6. Limitations, ambiguities, and recurring controversies

A first recurring controversy is terminological. Not every nonconstant schedule is adaptive. CLA is explicitly a predefined restart schedule indexed by $T_{cur}$ and $T_i$, and the paper states that in the adaptive DLRS sense it is “not adaptive” [2403.14685]. AutoWU is adaptive during warmup discovery but then reverts to a predefined decay [2107.05855]. This matters because papers often compare feedback-driven controllers against time-driven baselines under the same “dynamic” label.

A second issue is computational overhead. The spectrum is wide. GreedyLR changes LR from observed loss with negligible extra machinery; the loss-based epoch controller reports asymptotic overhead \(O(1)\) per epoch, “< 0.5% wall-clock overhead per epoch,” and “< 0.1% peak memory overhead” [2512.14527] [2507.21749]. AutoWU reports less than one second on average for each epoch-end GP fitting and inference block [2107.05855]. LRTuner’s probing overhead is bounded at about \(1000/15000 \approx 6.66\%\) per ImageNet epoch and about \(500/3300 \approx 15\%\) on IWSLT in the cited setups [2105.14526]. AutoLRS is explicitly more expensive, with total cost described as only slightly above \(2\times\) the model-update time associated with the discovered schedule [2105.10762]. RL-based controllers and evolutionary search add an even larger outer-loop burden because they require repeated full or partial training episodes [1909.09712] [2103.12623].

A third issue is uneven theoretical coverage. Some papers provide convergence analyses, but usually under restricted assumptions: GreedyLR is analyzed for smooth convex SGD with bounded learning rates [2512.14527]; RDBD proves a mini-batch convergence theorem under smoothness, bounded stochastic updates, and unbiasedness [2310.11291]; SALR’s main theoretical result is local and based on strong convexity in a neighborhood [2011.05348]. By contrast, several practically motivated methods provide no full convergence proof. The gap-aware adversarial scheduler is explicitly empirical rather than theorem-driven [2302.00089]. DLRD notes that it does not prove Robbins–Monro-type properties for its data-dependent decay sequence [2412.15745]. CLA provides no formal convergence proof [2403.14685].

A fourth limitation concerns reporting quality and reproducibility. The supplied material for "Locally Optimal Descent for Dynamic Stepsize Scheduling" states that the document fragment is not a full paper but an experimental notes/results sheet without explicit update equations, theorem statements, or pseudocode, even though it compares “greedy + hessian” and “greedy + GNB” against cosine, constant, rsqrt, and other baselines across CIFAR-10, CIFAR-100, ImageNet, Criteo, WikiText, and FastMRI [2311.13877]. CLA also contains malformed equations, including its printed logarithmic scheduler formula and softmax expression, so reconstruction requires interpretation [2403.14685]. More generally, some papers report validation or test metrics, whereas others emphasize training loss only. This makes direct comparison across DLRS papers structurally difficult.

Finally, the literature repeatedly returns to a common practical conclusion: DLRS reduces but does not eliminate tuning. LRRL still requires choosing the number and values of arms [2410.12598]. Hi-DLR depends on externally chosen parameter grouping [2501.06954]. LRTuner still needs an explore duration and an epsilon threshold [2105.14526]. AutoWU fixes its own confidence, patience, and warmup-range hyperparameters, even though they are reused across experiments [2107.05855]. This suggests that the central research question has shifted from eliminating hyperparameters to choosing control signals and structural priors that make the remaining hyperparameters less task-specific and less burdensome than manually scripting the entire schedule.

Source: https://www.emergentmind.com/topics/dynamic-learning-rate-scheduler-dlrs