Mpemba Effect in Valley–River Model

• The paper establishes that a higher initial learning rate can accelerate convergence by eliminating the dominant slow relaxation mode in the valley–river model.
• It employs stochastic dynamics, Fokker–Planck theory, and spectral analysis to quantify the Mpemba effect and rationalize WSD learning-rate protocols in deep learning.
• The framework connects metastable state kinetics with optimization dynamics, providing practical guidelines for tuning hyperparameters to improve loss reduction efficiency.

The Mpemba effect refers to the counterintuitive phenomenon where a system initially at a higher “temperature” cools or relaxes more rapidly than an identical system starting at a lower temperature, when both are quenched to the same cold bath. In the context of machine learning, particularly LLM training, this effect manifests in the rate at which optimization dynamics equilibrate under changes to the learning rate. The valley–river model provides a unifying minimal landscape for understanding the emergence and quantification of the Mpemba effect, connecting stochastic thermodynamics, metastable state kinetics, and the mechanistic justification for the widely used warm-up, plateau, decay (“WSD”) learning-rate schedule in deep learning.

1. Valley–River Model Foundations

The valley–river model describes a loss landscape as a composite of sharp (“valley”) and flat (“river”) directions, with the loss function parameterized as

$L(x, y) = c(y) + \tfrac{1}{2} a(y) x^2.$

Here, $x$ represents sharp directions with large positive curvature $a(y)$, while $y$ parametrizes flatter regions that control global drift. The stochastic dynamics under isotropic noise of strength $\eta$ (interpreted as the learning rate or temperature) are given by coupled Langevin equations,

$$\dot{x} = -\partial_x L + \sqrt{2\eta}~\xi_x(t), \quad \dot{y} = -\partial_y L + \sqrt{2\eta}~\xi_y(t),$$

with $\xi_{x,y}$ independent standard white noises. There is a pronounced timescale separation: the $x$ valley direction equilibrates rapidly ($\tau_x \ll \tau_y$), allowing for a quasi-equilibrium treatment in $x$ while $y$ remains far from equilibrium.

Upon integrating out the fast $x$ direction, the effective “free energy” landscape for $y$ is

$F_\eta(y) = c(y) + \frac{\eta}{2} \ln a(y).$

The associated Fokker–Planck dynamics for the probability density $p(y, t)$ follow

$$\partial_t p = \mathcal{L}_\eta p, \qquad \mathcal{L}_\eta = \partial_y[\partial_y F_\eta(y) + \eta \partial_y].$$

This reduction establishes the core of the valley–river analogy with metastable stochastic systems (Liu et al., 6 Jul 2025).

2. Thermodynamic Quench and the Mpemba Effect

A thermodynamic “quench” is operationalized as an abrupt drop in effective temperature (learning rate) from a higher plateau value $\eta_p$ to a lower “bath” value $\eta_b < \eta_p$. The evolution of $p(y, t)$ post-quench admits the expansion

$$p(y, t) = \pi_{\eta_b}(y) + \sum_{n\geq2} a_n(\eta_p) u_n(y) e^{-\lambda_n(\eta_b)t},$$

where $\pi_{\eta_b}$ is the stationary distribution at temperature $\eta_b$, $u_n$ are eigenfunctions of $\mathcal{L}_{\eta_b}$, and $\lambda_n$ are the corresponding eigenvalues. The slowest nontrivial mode ($u_2$, $\lambda_2$) controls late-time convergence. The crucial amplitude

$a_2(\eta_p) = \int u_2(y) \, \pi_{\eta_p}(y) \,dy$

encodes the initial overlap of the pre-quench stationary state with the dominant slow mode.

The Mpemba effect is observed whenever $|a_2(\eta_h)| < |a_2(\eta_l)|$ for two candidate plateau learning rates $\eta_h > \eta_l > \eta_b$, i.e., the hotter initialization yields faster convergence post-quench—direct generalization of the original effect to stochastic gradient descent in valley–river models (Liu et al., 6 Jul 2025, Walker et al., 2022).

3. Analytical Conditions and Classification

The amplitude $a_2(\eta)$ is, in general, a nonmonotonic function of $\eta$. The strong Mpemba effect occurs at a “strong Mpemba point” $\eta^*$, defined by

$$\int u_2 \, \pi_{\eta^*} = 0, \quad \eta^* \neq \eta_b,$$

which eliminates the slowest relaxation mode entirely. The next-slowest mode ($\lambda_3$) then governs convergence, yielding an exponential speed-up. The existence and location of $\eta^*$ are determined by the sign change in

$$\frac{d a_2}{d\eta} = \frac{1}{\eta^2} \mathrm{Cov}_{\pi_\eta}[F_\eta, u_2] = \left(\text{const} / \eta^2\right) \mathrm{Cov}_{\pi_\eta}[\ln a(y), u_2(y)].$$

A nonmonotonic dependence of $|a_2(\eta)|$ (implying a zero crossing) is thus dictated by the structure of the covariance between $\ln a(y)$ and $u_2(y)$ under $\pi_\eta$, mirroring the general condition for strong- and weak-type Mpemba phenomena in classical metastable systems (Liu et al., 6 Jul 2025, Walker et al., 2022, Chétrite et al., 2021).

4. Quench Protocols: WSD Schedules in Learning

The warm-up, plateau, decay (WSD) learning-rate protocol aligns precisely with the thermodynamic two-stage quench paradigm:

• Warm-up (“pre-heating”): $\eta$ is ramped up to $\eta_p$ to avoid destabilizing sharp directions.
• Plateau phase: $\eta$ is held at $\eta_p$ until the valley direction equilibrates ($t_\text{stable}\gtrsim\tau_x$), fixing the $y$ distribution in a nonequilibrium state best prepared for subsequent decay.
• Decay (quench): $\eta$ is rapidly reduced to $\eta_b$, inducing a nonequilibrium relaxation dominated by the slowest river mode.

This mapping renders the late-time dynamics,

$$p(y, t; \eta_0 \to \eta_b) \approx \pi_{\eta_b}(y) + a_2(\eta_0) u_2(y) e^{-\lambda_2(\eta_b)t},$$

explicit, and motivates the selection of $\eta_0 = \eta^*$ to eliminate the slowest mode and achieve fastest loss decrease during the decay (Liu et al., 6 Jul 2025).

5. Metric Connections and Mean First Passage Times

The spectral-expansion approach in Fokker–Planck theory relates direct relaxation rates and amplitudes to mean first passage times (MFPTs) in the double-well landscape. For a one-dimensional valley–river potential $U(x)$, occupation probabilities of each well, $\Pi_L(T)$ and $\Pi_R(T)$, and associated MFPTs between wells, $\langle\tau_{L,R}\rangle_{L,R,T}$, admit explicit expressions in the small-diffusion (Kramers) limit (Walker et al., 2022):

• Strong Mpemba criterion: $\Pi_L(T^*) = \Pi_L(T_b)$ — the initial distribution matches bath equilibrium populations, yielding $a_2(T^*)=0$.
• Weak Mpemba criterion: $a_2(T)$ is extremal (non-monotonic in $T$), so that a hotter system can relax faster than a warm one even if $a_2$ does not vanish.

This quantitative description links thermodynamic observables (e.g., extractable work, $$W_{\mathrm{max}}(T_{\text{initial}}) \propto D_{\mathrm{KL}}[\rho_{\mathrm{leq}} \| g_{T_b}]$$) directly to the non-monotonicity required for the Mpemba effect (Chétrite et al., 2021).

6. Practical Implications for Optimization Schedules

The valley–river plus Mpemba analysis systematizes the rationale for WSD-style learning-rate schedules in deep learning:

• The warm-up phase avoids destabilizing fast directions by slow ramping to plateau.
• The plateau is not just a region of minimal risk, but acts as a preheating step—best tuned to $\eta_p \approx \eta^*$, the strong Mpemba point, which can be sought by minimizing the late-time loss decline slope or via Hessian curvature diagnostics.
• The decay should be tuned to maintain $x$ (valley) equilibrium while still producing a sharp quench for $y$; suitable decay laws satisfy $|\dot{\eta}|/ \eta \ll k \eta$ for parameter $k$ controlling $y$ timescale separation.
• Examples include exponential decay ($\dot{\eta} = -a \eta$) or shallow power-law decay ($\dot{\eta} = -k \eta^2$), with $m \lesssim a$ in generalized decay $\dot{\eta} = -m \eta^p$, $p\in[1,2)$ to enforce valley equilibration (Liu et al., 6 Jul 2025).

These insights provide a principled framework for minimizing heuristic hyperparameter searches and justify the empirical outperformance of warm-up/high plateau/decay schedules relative to simple monotonic decay.

7. Generalization and Theoretical Context

The metastable Mpemba effect in the valley–river model is underpinned by fundamental properties of coarse-grained, multi-well energy landscapes. The effect’s emergence requires only appropriate asymmetries in well depths, barriers, and curvatures—no fine tuning or exotic physics. Both analytic theory and experiment confirm that the non-monotonic dependence of extractable work or domain occupancies on initial temperature is equivalent to a non-monotonic relaxation time, i.e., to the Mpemba effect (Chétrite et al., 2021, Walker et al., 2022). This theoretical apparatus extends to general stochastic optimization systems exhibiting timescale separation and metastability, providing a robust bridge between nonequilibrium thermodynamics and modern machine learning training schedules.