Effective Learning Rate (ELR) in Optimization

• Effective Learning Rate (ELR) are characteristics of step size used in optimization algorithms under various transformations, including normalization, optimizer changes, and gradient statistics; making it crucial for understanding true parameter updates in different training scenarios, especially for SGD variants and adaptive optimizers.
• To accurately track and understand ELR, researchers and practitioners must consider nominal learning rates, schedule policies, gradient statistics, optimizer transformations, parameter norms, normalization effects, noise, curvature, and training stage.
• Key applications include learning rate scheduling, adaptive control, scale-invariant optimization, continual learning, and large language model (LLM) scaling, each requiring a nuanced understanding of ELR to optimize performance and maintain stability during training.
• Key applications
• Key applications

Effective learning rate (ELR) is not a universally standardized quantity. In optimization research, it generally denotes the step size actually governing parameter or functionally relevant motion after accounting for gradients, optimizer transformations, normalization, parameter norms, stochasticity, and scheduling. Depending on context, ELR may mean a learned scalar multiplier in SGD, a normalized-space displacement in scale-invariant networks, a layer-wise gradient-to-weight ratio, a coordinate-wise realized rate, or an information-update rate in a bipartite stochastic system. This terminological plurality is central: a nominal learning rate supplied to an optimizer is not generally equivalent to the effective update experienced by the model.

1. Conceptual distinctions and formal meanings

For ordinary SGD, the parameter update is

$\theta_{t+1}=\theta_t-\eta_t\nabla L(\theta_t),$

so the displacement is

$\Delta\theta_t=-\eta_t\nabla L(\theta_t).$

The nominal rate $\eta_t$ is only the scalar multiplier. The update norm is

$\|\Delta\theta_t\|=\eta_t\|\nabla L(\theta_t)\|,$

and therefore varies with the gradient even when $\eta_t$ is constant. A displacement-based scalar ELR can be interpreted as

$$\eta_{\mathrm{eff},t}=\frac{\|\Delta\theta_t\|}{\|\nabla L(\theta_t)\|},$$

when the denominator is nonzero, or through a directional projection such as

$$\eta_{\mathrm{eff},t}^{\mathrm{dir}} = \frac{\langle-\Delta\theta_t,\nabla L(\theta_t)\rangle} {\|\nabla L(\theta_t)\|^2}.$$

These expressions are interpretations rather than a universal convention.

For momentum and adaptive optimizers, the distinction is stronger. A generic update can be written

$\theta_{t+1}=\theta_t-\eta_tu_t,$

where $u_t$ is a momentum-accumulated or preconditioned direction. In Adam, for example,

$$u_t=\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$

so that

$\Delta\theta_t=-\eta_t\nabla L(\theta_t).$0

The actual update depends on gradient history, moment estimates, coordinate-wise normalization, weight decay, parameter scale, and optimizer state. Consequently, equal nominal learning rates under SGD, Adam, AdamW, RMSprop, or Adagrad do not imply equal effective updates.

Several research uses of ELR can be distinguished:

Context	Effective quantity
Plain or learned SGD	Scalar multiplying the stochastic gradient
Scale-invariant networks	Motion in normalized weight space
Layer-wise normalized networks	Gradient-to-weight norm ratio
Stochastic optimization	Relaxation coefficient governing noisy projections
Learning-rate perturbation	Realized coordinate-wise learning-rate vector
Information thermodynamics	Rate of information renewal caused by downstream transitions

The term should therefore be qualified by its operational definition, such as normalized-space ELR, layer-wise ELR, or optimizer-level effective step size.

2. Schedule-dependent and learned scalar rates

A learning-rate policy is a trajectory rather than a single number. In a general bounded schedule,

$\Delta\theta_t=-\eta_t\nabla L(\theta_t).$1

where $\Delta\theta_t=-\eta_t\nabla L(\theta_t).$2 and $\Delta\theta_t=-\eta_t\nabla L(\theta_t).$3 define rate boundaries and $\Delta\theta_t=-\eta_t\nabla L(\theta_t).$4 determines interpolation. The resulting optimization behavior depends on the visited range, update timing, decay or growth factors, cycle length, terminal rate, model, dataset, optimizer, and training stage. “Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks” (Wu et al., 2019) studies 13 fixed, decaying, and cyclic functions and implements them in LRBench, which evaluates classification accuracy, confidence, variance, cost, and robustness.

This policy-level view explains why the same initial rate can produce different outcomes under fixed, step-decay, polynomial, exponential, inverse-time, triangular, sinusoidal, cosine, or composite schedules. Excessive decay can make the terminal rate effectively zero, while cyclic policies can alternate exploratory and refining phases. In the reported CIFAR-10 experiments, useful ranges differed substantially between CNN3 and ResNet-32, demonstrating that nominal ranges do not transfer unchanged across architectures (Wu et al., 2019).

Several methods replace hand-designed schedules with controllers. “Reinforcement Learning for Learning Rate Control” (Xu et al., 2017) uses an actor-critic system in which an LSTM actor produces a scalar action

$\Delta\theta_t=-\eta_t\nabla L(\theta_t).$5

and a critic estimates its long-term value. The primary learner remains plain SGD:

$\Delta\theta_t=-\eta_t\nabla L(\theta_t).$6

The reward is one-step loss reduction,

$\Delta\theta_t=-\eta_t\nabla L(\theta_t).$7

while the critic estimates discounted future loss decrease. The resulting ELR is scalar, time-varying, and potentially mini-batch dependent, but not layer-wise or coordinate-wise. The method uses a compact state, experimentally the average loss on the current mini-batch, and uses separate samples for critic and actor updates to expose gradient disagreement.

The Autonomous Learning Rate Controller (ARC) (Dong et al., 2021) instead classifies loss histories into increase, constant, or decrease actions. Its multiplicative control rule is

$\Delta\theta_t=-\eta_t\nabla L(\theta_t).$8

ARC observes training loss, validation loss, and nominal learning-rate histories, but not gradient norms, update norms, optimizer states, or parameter movement. It therefore influences ELR indirectly. With SGD, nominal and effective scalar rates are close; with momentum or adaptive optimizers, the optimizer transformation remains part of the realized step.

Evolutionary methods provide another form of scalar schedule control. AutoLR (Carvalho et al., 2020) evolves executable scheduler programs using Structured Grammatical Evolution. The scheduler returns one scalar rate shared by all dimensions and is invoked once per epoch. Its policies can be constant or conditional and may depend on epoch number or the previous rate. The method specializes schedules to a fixed architecture, but does not measure update norms, gradient norms, optimizer-normalized rates, or parameter-relative displacement.

LREnsemble (Jin et al., 2024) treats learning-rate tuning as both optimization and model generation. Different policies produce different parameter trajectories and error patterns; rather than retaining only the best trial, LREnsemble selects complementary models for an ensemble. Its effective-rate notion is empirical and policy-level, encompassing warm-up, decay, cyclic phases, and the resulting optimization path. It does not define ELR as a normalized update magnitude, and its theoretical variance argument assumes simplified independence between learning-rate and gradient variables.

3. ELR in scale-invariant and normalized networks

In scale-invariant networks, the function satisfies

$\Delta\theta_t=-\eta_t\nabla L(\theta_t).$9

Only the direction $\eta_t$0 affects the represented function. Differentiation yields

$\eta_t$1

The gradient is therefore tangent to the radial sphere, while its magnitude decreases as the parameter norm increases.

For a scale-invariant parameter group with norm $\eta_t$2, the normalized-space effective learning rate is

$\eta_t$3

For the full parameter vector,

$\eta_t$4

“Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes” (Kodryan et al., 2022) studies this quantity directly by fixing the norm and optimizing on a sphere. Its projected-gradient formulation isolates intrinsic dynamics from norm growth. The corresponding effective step size is $\eta_t$5, where $\eta_t$6 is the gradient measured in normalized coordinates.

With several scale-invariant groups,

$\eta_t$7

Fixing a global ELR therefore does not fix individual layer ELRs. Norm is redistributed among groups, creating feedback: a group with a large effective step tends to lose relative norm and subsequently receives a smaller ELR, whereas a group with a small effective step gains relative norm and receives a larger ELR.

The fixed-ELR spherical dynamics exhibit three empirical regimes:

1. Convergence: low ELR permits traversal of sharp regions and convergence to low-loss minima.
2. Chaotic equilibrium: intermediate ELR produces sustained oscillations, stabilized effective-gradient norms, and a nonzero loss level.
3. Divergence: very high ELR produces decorrelated, random-walk-like motion and loss approaching random-guess behavior.

For the toy scale-invariant function

$\eta_t$8

the theoretical convergence condition is

$\eta_t$9

For multiple groups, the corresponding condition is

$\|\Delta\theta_t\|=\eta_t\|\nabla L(\theta_t)\|,$0

The transition boundaries in realistic neural networks are empirical and depend on architecture, width, depth, stochasticity, augmentation, label noise, and model capacity. Higher ELR within the convergence regime can accelerate convergence and favor flatter solutions, but excessive ELR can trap training in chaotic equilibrium or cause divergence.

“On the Weight Dynamics of Deep Normalized Networks” (Mehmeti-Göpel et al., 2023) defines a layer-wise ELR through

$\|\Delta\theta_t\|=\eta_t\|\nabla L(\theta_t)\|,$1

Under its model,

$\|\Delta\theta_t\|=\eta_t\|\nabla L(\theta_t)\|,$2

where $\|\Delta\theta_t\|=\eta_t\|\nabla L(\theta_t)\|,$3 is the layer’s base gradient magnitude. The actual relative update is approximately $\|\Delta\theta_t\|=\eta_t\|\nabla L(\theta_t)\|,$4, with $\|\Delta\theta_t\|=\eta_t\|\nabla L(\theta_t)\|,$5 the global learning rate.

Normalization creates an auto-rate-tuning effect. Orthogonal updates increase weight norms, and the resulting norm growth suppresses subsequent gradients. For two layers, the ELR ratio

$\|\Delta\theta_t\|=\eta_t\|\nabla L(\theta_t)\|,$6

converges to one under the paper’s constant-base-gradient model. However, a sufficiently large initial learning rate can exceed the current critical rate

$\|\Delta\theta_t\|=\eta_t\|\nabla L(\theta_t)\|,$7

causing an ELR-order flip and widening the layer-wise spread. The network-wide relative logarithmic spread is

$\|\Delta\theta_t\|=\eta_t\|\nabla L(\theta_t)\|,$8

The proposed subcritical warm-up computes current layer ELRs and chooses a global rate from the two largest ELRs, rapidly reducing spread without requiring a manually chosen warm-up length.

“Normalization and effective learning rates in reinforcement learning” (Lyle et al., 2024) extends this analysis to continual learning and RL. For ordinary gradient updates, it defines

$\|\Delta\theta_t\|=\eta_t\|\nabla L(\theta_t)\|,$9

while normalized-gradient updates yield

$\eta_t$0

Parameter-norm growth therefore creates an implicit ELR decay even under constant nominal learning rate. This can be useful in stationary tasks but harmful in nonstationary RL, where plasticity may be lost before later tasks are learned.

Normalize-and-Project (NaP) addresses this coupling by normalizing activations and projecting weights back to fixed-norm surfaces. The update–project procedure preserves directional learning while removing uncontrolled norm growth, making the explicit learning-rate schedule more closely aligned with ELR. The paper reports that NaP improves robustness in continual CIFAR-10 memorization and sequential Atari experiments, while preserving or slightly improving stationary vision and language performance (Lyle et al., 2024).

A thermodynamic formulation in “Can Training Dynamics of Scale-Invariant Neural Networks Be Explained by the Thermodynamics of an Ideal Gas?” (Sadrtdinov et al., 10 Nov 2025) uses the same intrinsic definition,

$\eta_t$1

For isotropic tangent noise with variance $\eta_t$2, the effective temperature is

$\eta_t$3

Under fixed nominal learning rate, the stationary norm is determined jointly by learning rate, weight decay, noise covariance, and dimension. Thus the stationary ELR is not simply the optimizer’s nominal rate. It depends on the self-consistent norm produced by radial contraction and stochastic outward motion.

4. Stochasticity, perturbation, and geometry-dependent rates

For stochastic gradient descent, the learning rate controls both deterministic motion and the scale of stochasticity. “On Learning Rates and Schrödinger Operators” (Shi et al., 2020) approximates SGD with the learning-rate-dependent SDE

$\eta_t$4

or, for mini-batch noise variance $\eta_t$5,

$\eta_t$6

The effective stochastic scale is therefore of order $\eta_t$7. In strongly convex problems, the continuous-time convergence rate is independent of $\eta_t$8, while the iteration-scale rate is $\eta_t$9. In nonconvex landscapes, the spectral rate behaves as

$$\eta_{\mathrm{eff},t}=\frac{\|\Delta\theta_t\|}{\|\nabla L(\theta_t)\|},$$0

where $$\eta_{\mathrm{eff},t}=\frac{\|\Delta\theta_t\|}{\|\nabla L(\theta_t)\|},$$1 is a saddle-barrier height. Small learning rates reduce stationary error but can make inter-basin transitions exponentially slow. This gives a theoretical interpretation of learning-rate decay as a transition from exploration to concentration.

The paper distinguishes SGD from gradient descent and SGLD. The limiting ODE for gradient descent does not retain learning-rate-dependent diffusion, while standard SGLD has a diffusion coefficient independent of the discretization step. In SGD, the learning rate remains in the diffusion coefficient and therefore changes the effective temperature of the dynamics (Shi et al., 2020).

LEAP, introduced in “Learning Rate Perturbation: A Generic Plugin of Learning Rate Schedule towards Flatter Local Minima” (Liu et al., 2022), augments a nominal scalar schedule $$\eta_{\mathrm{eff},t}=\frac{\|\Delta\theta_t\|}{\|\nabla L(\theta_t)\|},$$2 with a coordinate-wise random vector:

$$\eta_{\mathrm{eff},t}=\frac{\|\Delta\theta_t\|}{\|\nabla L(\theta_t)\|},$$3

Equivalently,

$$\eta_{\mathrm{eff},t}=\frac{\|\Delta\theta_t\|}{\|\nabla L(\theta_t)\|},$$4

The update becomes

$$\eta_{\mathrm{eff},t}=\frac{\|\Delta\theta_t\|}{\|\nabla L(\theta_t)\|},$$5

The perturbation is multiplicative, coordinate-dependent, and gradient-scaled. Conditional on $$\eta_{\mathrm{eff},t}=\frac{\|\Delta\theta_t\|}{\|\nabla L(\theta_t)\|},$$6, its covariance is

$$\eta_{\mathrm{eff},t}=\frac{\|\Delta\theta_t\|}{\|\nabla L(\theta_t)\|},$$7

LEAP therefore modifies the distribution of realized steps while preserving the nominal schedule in expectation. Its diffusion analysis predicts longer residence times in flatter minima, but the experiments do not directly measure flatness or establish that flat-minimum selection is the sole cause of improved test error. Generic regularization and optimizer interactions remain alternative explanations.

ELRA—Exponential Learning Rate Adaption—adapts a scalar rate from the angle between consecutive gradients (Kleinsorge et al., 2023). If

$$\eta_{\mathrm{eff},t}=\frac{\|\Delta\theta_t\|}{\|\nabla L(\theta_t)\|},$$8

then

$$\eta_{\mathrm{eff},t}=\frac{\|\Delta\theta_t\|}{\|\nabla L(\theta_t)\|},$$9

The c2min rule is

$$\eta_{\mathrm{eff},t}^{\mathrm{dir}} = \frac{\langle-\Delta\theta_t,\nabla L(\theta_t)\rangle} {\|\nabla L(\theta_t)\|^2}.$$0

Positive cosine indicates an insufficient step and increases $$\eta_{\mathrm{eff},t}^{\mathrm{dir}} = \frac{\langle-\Delta\theta_t,\nabla L(\theta_t)\rangle} {\|\nabla L(\theta_t)\|^2}.$$1; negative cosine indicates overshoot or oscillation and decreases it. The p2min rule estimates a local quadratic minimum:

$$\eta_{\mathrm{eff},t}^{\mathrm{dir}} = \frac{\langle-\Delta\theta_t,\nabla L(\theta_t)\rangle} {\|\nabla L(\theta_t)\|^2}.$$2

These methods are rotation invariant because they use Euclidean norms and inner products rather than coordinate-wise second moments. They are not, however, free of all hyperparameters: c2min uses momentum coefficients, and p2min requires clipping and restart safeguards. Neither variant has a general convergence theorem for stochastic nonconvex neural-network training.

A related scalar mechanism appears in “Explainable Learning Rate Regimes for Stochastic Optimization” (Yang, 19 Aug 2025). It estimates

$$\eta_{\mathrm{eff},t}^{\mathrm{dir}} = \frac{\langle-\Delta\theta_t,\nabla L(\theta_t)\rangle} {\|\nabla L(\theta_t)\|^2}.$$3

and a trial-point variation

$$\eta_{\mathrm{eff},t}^{\mathrm{dir}} = \frac{\langle-\Delta\theta_t,\nabla L(\theta_t)\rangle} {\|\nabla L(\theta_t)\|^2}.$$4

Its proposed rate is

$$\eta_{\mathrm{eff},t}^{\mathrm{dir}} = \frac{\langle-\Delta\theta_t,\nabla L(\theta_t)\rangle} {\|\nabla L(\theta_t)\|^2}.$$5

The intended rule is that the rate increases when stochastic-gradient norms decrease and decreases when they increase. The method is scalar and uses an additional stochastic-gradient evaluation; its stated formulation does not provide denominator regularization, clipping, or a general nonconvex convergence theorem.

For noisy randomized Kaczmarz, the learning rate is a relaxation coefficient:

$$\eta_{\mathrm{eff},t}^{\mathrm{dir}} = \frac{\langle-\Delta\theta_t,\nabla L(\theta_t)\rangle} {\|\nabla L(\theta_t)\|^2}.$$6

“An optimal scheduled learning rate for a randomized Kaczmarz algorithm” (Marshall et al., 2022) derives

$$\eta_{\mathrm{eff},t}^{\mathrm{dir}} = \frac{\langle-\Delta\theta_t,\nabla L(\theta_t)\rangle} {\|\nabla L(\theta_t)\|^2}.$$7

The schedule begins near one when signal error dominates noise and decreases as noise becomes relatively important. Its asymptotic behavior is

$$\eta_{\mathrm{eff},t}^{\mathrm{dir}} = \frac{\langle-\Delta\theta_t,\nabla L(\theta_t)\rangle} {\|\nabla L(\theta_t)\|^2}.$$8

Here ELR is directly the fraction of the noisy projection applied, rather than an optimizer-normalized parameter displacement.

5. Component-wise and architecture-specific effective rates

A single global rate can impose an inappropriate compromise on heterogeneous parameter groups. “Decoupled Relative Learning Rate Schedules” (Ludziejewski et al., 4 Jul 2025) introduces RLRS, which assigns separate schedules to Transformer components relative to a common base AdamW schedule. The groups are embedding, attention, unembedding, feed-forward layers, routers, and experts.

For component $$\eta_{\mathrm{eff},t}^{\mathrm{dir}} = \frac{\langle-\Delta\theta_t,\nabla L(\theta_t)\rangle} {\|\nabla L(\theta_t)\|^2}.$$9,

$\theta_{t+1}=\theta_t-\eta_tu_t,$0

The endpoints are

$\theta_{t+1}=\theta_t-\eta_tu_t,$1

and

$\theta_{t+1}=\theta_t-\eta_tu_t,$2

The method controls explicit group-wise AdamW coefficients rather than directly measuring

$\theta_{t+1}=\theta_t-\eta_tu_t,$3

Because AdamW applies moment normalization and decoupled weight decay, actual group motion depends on optimizer state and parameter scale in addition to $\theta_{t+1}=\theta_t-\eta_tu_t,$4.

RLRS is particularly motivated by MoE models, in which router stability and expert learning can require different schedules. Early reduction of router and expert rates can delay premature routing stabilization and expert specialization, while later increases can support fine-tuning. The reported experiments use relative schedules tuned on small models and reused on models up to 27 times larger, with speedups of up to 23% in the reported MoE configuration (Ludziejewski et al., 4 Jul 2025). The transferable object is the relative component schedule, not the absolute base learning rate.

The same principle appears in large-language-model scaling. “On the Nonlinearity of Learning Rate Scaling for LLM Training” (Yang et al., 28 Jun 2026) defines ELR as the normalized weight-space displacement,

$\theta_{t+1}=\theta_t-\eta_tu_t,$5

For approximately transverse AdamW updates,

$\theta_{t+1}=\theta_t-\eta_tu_t,$6

The paper averages this quantity over selected tensors and training time. It finds that optimal nominal learning rates develop upward curvature when extrapolated across model size and data scale, whereas optimal ELR is closer to a log-linear scaling law. Data-axis extrapolation is more reliable than model-axis extrapolation, and AdamH—an optimizer that fixes weight norm and normalizes the Adam update—produces a substantially cleaner ELR scaling relationship.

This result emphasizes that nominal-rate scaling can be distorted by slowly evolving norms. Under AdamW, the mapping from $\theta_{t+1}=\theta_t-\eta_tu_t,$7 to normalized-space motion changes between pre-equilibrium and equilibrium regimes. A rate that appears optimal at one training scale may therefore be systematically miscalibrated at another even when the underlying desired directional step is similar.

6. Control, optimization objectives, and interpretation

Learning-rate control can also be formulated as a normative resource-allocation problem. “Optimal Learning Rate Schedule for Balancing Effort and Performance” (Njaradi et al., 12 Jan 2026) treats learning speed $\theta_{t+1}=\theta_t-\eta_tu_t,$8 as a costly control variable in continuous-time gradient flow:

$\theta_{t+1}=\theta_t-\eta_tu_t,$9

With reward rate

$u_t$0

and quadratic effort cost

$u_t$1

the undiscounted optimal closed-loop controller is

$u_t$2

The rate is high when expected future performance gains are large and falls as performance approaches its expected terminal level. This is an effort-optimal learning-speed controller, not necessarily the same as a parameter-update ELR. It controls the rate of gradient-flow dynamics while the realized displacement still depends on the performance gradient.

The distinction between learning speed and ELR is therefore important. A controller may optimize cumulative performance, validation loss, final accuracy, or effort-adjusted reward without directly controlling update norms. Conversely, a method may precisely control normalized-space motion while making no claim about final task performance. The two objectives can conflict: rapid update turnover may increase an information-update rate or stochastic exploration while reducing instantaneous accuracy.

The information-theoretic learning rate in “What We Learn from the Learning Rate” (Brittain et al., 2017) illustrates this distinction outside neural optimization. For a bipartite Markov chain with upstream subsystem $u_t$3 and downstream subsystem $u_t$4, the learning rate is

$u_t$5

It is the contribution to the mutual-information derivative caused specifically by transitions of $u_t$6:

$u_t$7

Thus $u_t$8 is an information-update or information-turnover rate, not the amount of information currently stored. In steady state,

$u_t$9

and it can also be interpreted as the rate at which information held by the current sensor becomes obsolete as the upstream signal evolves. The thermodynamic bound

$$u_t=\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$0

connects information renewal to subsystem entropy production, but does not establish that maximizing $$u_t=\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$1 universally maximizes sensing quality.

The paper’s models show that mutual information and learning rate can behave similarly in a simple feed-forward $$u_t=\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$2 sensor, yet diverge in feedback systems and multistate oscillating signals. A sensor can become more informative while its learning rate decreases, or its learning rate can peak at an intermediate response speed even while instantaneous and delayed mutual information increase monotonically. The same conceptual caution applies to machine-learning ELR: a larger effective update, larger update turnover, or larger stochasticity does not necessarily mean better tracking, lower loss, or better generalization.

7. Measurement, applications, and limitations

ELR measurement should begin by identifying the relevant invariant or objective.

For ordinary SGD, useful diagnostics include

$$u_t=\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$3

For momentum or adaptive optimizers, the optimizer-transformed direction $$u_t=\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$4 should also be recorded. For normalized networks, parameter norms and normalized directional displacement should be tracked. For layer-wise analysis, the ratio

$$u_t=\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$5

and the spread

$$u_t=\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$6

are relevant. For schedule evaluation, nominal rate trajectories should be considered alongside accuracy, confidence, loss, convergence cost, robustness, and terminal-rate behavior.

The principal applications include:

• Learning-rate scheduling: selecting fixed, decaying, cyclic, warm-restart, or learned trajectories.
• Adaptive control: using loss histories, gradient geometry, or predicted future performance.
• Scale-invariant optimization: controlling normalized-space motion independently of parameter-norm growth.
• Deep-network trainability: reducing layer-wise ELR disparities during warm-up.
• Continual learning and RL: preventing normalization-induced ELR decay from eliminating plasticity.
• LLM scaling: transferring normalized-space step sizes rather than nominal AdamW rates.
• MoE optimization: assigning distinct schedules to routers, experts, attention, embeddings, and feed-forward components.
• Stochastic exploration: modifying the diffusion or basin-escape behavior of SGD through learning-rate scale or perturbation.
• Information processing: quantifying the turnover of correlations in dynamical sensor systems.

Important limitations recur across the literature. Many proposed methods control only a nominal scalar coefficient and do not measure the actual update after momentum, adaptive preconditioning, weight decay, normalization, or parameter scaling. Scalar methods cannot directly correct coordinate-specific curvature or layer-specific gradient scales. Gradient-based controllers can be unstable when gradients are noisy, nearly zero, or inconsistent with a local line-search model. Gaussian learning-rate perturbations can generate negative or excessively large realized rates unless clipped. Schedule search can be computationally expensive, and early stopping or low-fidelity evaluation can select policies that appear effective initially but destabilize long training.

Theoretical results are usually conditional. The Schrödinger-operator analysis assumes smooth confining objectives and gives sharp barrier-dependent behavior for specific nonconvex classes (Shi et al., 2020). The Kaczmarz schedule optimizes an error bound and is exactly optimal for the algorithm only under additional sharpness conditions (Marshall et al., 2022). Scale-invariant ELR analyses rely on exact or approximate scale invariance, orthogonality, simplified SGD, and assumptions about gradient or noise dynamics (Kodryan et al., 2022, Mehmeti-Göpel et al., 2023, Sadrtdinov et al., 10 Nov 2025). The normative control formulation assumes smooth continuous-time dynamics, an effort cost, and an estimate of final performance (Njaradi et al., 12 Jan 2026).

The most general conclusion is that ELR is a state-dependent dynamical quantity, not merely a number printed in an optimizer configuration. Its value can be determined by

$$u_t=\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$7

Accordingly, ELR should be reported with its definition, measurement scale, aggregation procedure, optimizer, parameter grouping, and schedule. A nominal learning rate may be useful for reproducing an optimizer configuration, but ELR is the more appropriate object when the scientific question concerns actual parameter motion, normalized functional change, layer-wise trainability, stochastic exploration, or the cost of maintaining information.