BELAY: Two-Way Averaging for NN Training

• BELAY is a training algorithm for neural networks that generalizes EMA via a damped harmonic oscillator, enabling two-way weight coupling.
• It replaces EMA’s one-way, passive averaging with a dynamic, bidirectional feedback mechanism using a spring-damper analogy.
• Empirical results indicate that BELAY can reduce lag and enhance convergence stability, particularly under high learning rates and noisy conditions.

BELAY is a training algorithm for neural network optimization that generalizes exponential moving average (EMA) of model weights by treating the online model and the averaged model as two coupled masses in a damped spring system. Introduced in "Exponential weight averaging as damped harmonic motion" (Patsenker et al., 2023), it replaces EMA’s one-way, passive averaging with a two-way coupling in which the smoothed trajectory can also pull back on the live model during training. In the paper’s formulation, standard weight EMA is recovered as a limiting case in which the online model has infinite mass and is therefore unaffected by the averaged trajectory.

1. Problem setting and motivation

BELAY is motivated by a mismatch between the ubiquity of weight EMA and the limited understanding of why EMA improves training-time and inference-time stability. In the paper’s convention, standard EMA is written as

$$\mathbf{w}^{\mathrm{EMA}}(t+1)=\alpha \mathbf{w}^*(t)+(1-\alpha)\mathbf{w}^{\mathrm{EMA}}(t), \qquad \mathbf{w}^*(t)=\mathbf{w}(t)+g(\mathbf{w}(t)),$$

where $g(\mathbf{w}(t))$ is the optimizer update and $\alpha \in [0,1]$ controls averaging (Patsenker et al., 2023).

The paper identifies three practical limitations of this scheme. First, EMA is one-way coupled: the averaged weights are pulled toward the online weights, but the online weights receive no stabilizing feedback from the averaged trajectory. Second, EMA is sensitive to $\alpha$: if $\alpha$ is too large in the paper’s convention, the average lags badly behind training, whereas if $\alpha$ is too small, the average tracks too closely and loses much of its variance-reduction effect. Third, good choices of $\alpha$ depend on the total training horizon $T$, which makes transfer across runs awkward.

BELAY is designed to preserve EMA’s smoothing while reducing its passivity. The central objective is not merely to maintain a better evaluation-time model, but to use the smoothed trajectory to stabilize the optimization path itself. The paper summarizes this as using “the strong model stability properties of EMA to guide parameter updates during training” (Patsenker et al., 2023).

2. EMA as a damped two-body system

The paper’s core conceptual move is to interpret EMA as a discretized damped harmonic oscillator with two particles at positions $\mathbf{w}_1(t)$ and $\mathbf{w}_2(t)$, masses $g(\mathbf{w}(t))$0 and $g(\mathbf{w}(t))$1, spring constant $g(\mathbf{w}(t))$2, and damping coefficients $g(\mathbf{w}(t))$3. Particle 1 is subject to the optimizer’s external force; particle 2 is not. The continuous-time equations are

$g(\mathbf{w}(t))$4

$g(\mathbf{w}(t))$5

Using a forward Euler-style discretization with step $g(\mathbf{w}(t))$6, and then choosing

$g(\mathbf{w}(t))$7

the velocity terms cancel between steps. Defining

$g(\mathbf{w}(t))$8

the paper obtains

$g(\mathbf{w}(t))$9

$\alpha \in [0,1]$0

Identifying $\alpha \in [0,1]$1 with the online weights $\alpha \in [0,1]$2 and $\alpha \in [0,1]$3 with $\alpha \in [0,1]$4, standard EMA is recovered in the limit $\alpha \in [0,1]$5, since then $\alpha \in [0,1]$6 and the online model ceases to feel the spring (Patsenker et al., 2023).

This interpretation is the paper’s main theoretical contribution. The online weights become one particle under an optimizer-induced external force; the averaged weights become a second particle; the coupling between them is a spring; and standard EMA corresponds to the special case in which only the averaged particle is meaningfully pulled. In that sense, BELAY does not discard EMA, but embeds it in a broader dynamical family.

3. Algorithmic form of BELAY

BELAY introduces finite online mass $\alpha \in [0,1]$7, so that the averaged trajectory influences the online model as well. In the paper’s Algorithm 1, the parameter set is $\alpha \in [0,1]$8, together with optimizer update function $\alpha \in [0,1]$9 and learning rate $\alpha$0 (Patsenker et al., 2023).

Symbol	Role
$\alpha$1	spring constant controlling attraction between the two weight vectors
$\alpha$2	mass of the online model weights
$\alpha$3	mass of the EMA weights
$\alpha$4	damping coefficients
$\alpha$5	external force from the optimizer
$\alpha$6	learning rate

With $\alpha$7, the algorithm first computes an optimizer step

$\alpha$8

It then defines momentum remnants

$\alpha$9

and mixing coefficients

$\alpha$0

The position updates are

$\alpha$1

$\alpha$2

The paper emphasizes a simplified practical regime in which damping eliminates carried-over velocity each step:

$\alpha$3

Then $\alpha$4, and BELAY reduces to a two-way averaging scheme:

$\alpha$5

$\alpha$6

This is the form actually used in the experiments. Relative to EMA, the difference is structural: BELAY still computes an optimizer step for the online model, but then mixes online and averaged weights in both directions. Standard EMA is recovered when $\alpha$7, equivalently when $\alpha$8 in the online update (Patsenker et al., 2023).

4. Analytical interpretation and tuning heuristics

The paper’s clearest analytical statement is that EMA is a special case of BELAY. In the infinite-mass limit for the online particle, the spring cannot move $\alpha$9, so the averaged model passively trails the online model exactly as in ordinary EMA. BELAY therefore enlarges the design space rather than replacing the EMA intuition outright (Patsenker et al., 2023).

A second claim is that BELAY should reduce lag while increasing responsiveness. Standard EMA reduces variance but necessarily trails the online trajectory because coupling is one-way. BELAY’s two-way coupling partially anchors the online weights to a smoother path. The intended effect is a trade: less destructive separation between live and averaged trajectories, with some of EMA’s smoothing transferred back into the training dynamics.

A third claim concerns damping. Setting

$\alpha$0

fully dampens the system, so velocity is reset between time steps. Deviating from this introduces a momentum-like term. BELAY therefore unifies two ingredients that are usually discussed separately: EMA-like averaging and momentum-like inertial effects. The paper treats the damping coefficients as physically interpretable controls over stability.

The paper also sketches a relationship to momentum optimizers. For SGD with momentum,

$\alpha$1

so the velocity is an exponentially weighted sum of past gradients. For a linear loss, the derivation shows that such averaging can be interpreted as evaluating gradients at an EMA-smoothed iterate. The paper presents this as an interpretive bridge rather than a formal theorem.

To reduce EMA’s dependence on total training length, the paper derives a heuristic scaling law for the spring constant:

$\alpha$2

with $\alpha$3 in the reported experiments. This is explicitly a heuristic design rule, not a formal guarantee.

Equally important are the limits of the analysis. The paper does not prove faster asymptotic convergence than EMA, lower statistical bias than EMA, Lyapunov stability guarantees, or nonconvex convergence theorems. Its theoretical advantages are interpretive and structural: an exact mechanical analogy, EMA as a limiting case, interpretable control of coupling and damping, and a plausible reduction in lag with improved robustness under aggressive optimization settings (Patsenker et al., 2023).

5. Empirical evaluation

The empirical study has two parts: ill-conditioned two-dimensional optimization and score-based diffusion models. In the 2D tests, the compared methods are BELAY + Adam, BELAY + SGD, EMA + Adam, EMA + SGD, Adam, and SGD, evaluated on Rosenbrock, Beale, $\alpha$4, and $\alpha$5. The paper reports that BELAY and EMA improve stability relative to raw Adam and SGD; in the high learning rate regime, only BELAY + Adam and EMA + Adam are reported to converge without serious instability; and BELAY is often among the fastest to converge in the trajectory plots. For these tests, BELAY + Adam / SGD uses $\alpha$6, with $\alpha$7 chosen to zero out velocity terms, and the EMA baseline uses $\alpha$8 (Patsenker et al., 2023).

The generative-model experiments compare BELAY and EMA on score-based diffusion models. The strongest numerical win is on MNIST; CIFAR-10 is essentially parity.

Dataset	BELAY	EMA
MNIST	Train Loss 0.042, Test Loss 0.040, FID 15.2	Train Loss 0.056, Test Loss 0.061, FID 18.1
CIFAR-10	Train Loss 0.176, Test Loss 0.170, FID 1.99	Train Loss 0.175, Test Loss 0.168, FID 1.98

For MNIST, the model is the score-based diffusion model of Song & Ermon (2020), with BELAY settings $\alpha$9, $\alpha$0, $\alpha$1, damping set to zero out velocity, and Adam learning rate $\alpha$2. For CIFAR-10, the model is the diffusion model of Karras et al. (2022), with $\alpha$3, $\alpha$4, $\alpha$5, again with damping chosen to zero out velocity. The appendix also varies $\alpha$6 over

$\alpha$7

which the paper uses to illustrate that BELAY helps most when $\alpha$8 is finite but not so small that the online model becomes overconstrained (Patsenker et al., 2023).

The empirical picture is therefore asymmetric. On MNIST, BELAY clearly outperforms EMA on train loss, test loss, and FID. On CIFAR-10, it remains competitive but is not numerically superior on the reported summary metrics. The paper’s own practical conclusion is correspondingly measured: BELAY can improve robustness and sometimes convergence speed, especially in noisy or high-learning-rate regimes, but EMA remains simpler and more established.

6. Interpretation, scope, and disambiguation

BELAY is best understood as EMA plus feedback: an EMA-like averaged model is retained, but it is no longer merely retrospective. The smoothed trajectory becomes a stabilizing reference for the live model. This makes BELAY most natural in settings where EMA already helps, but where one also wants the smoothing effect to shape the optimization path itself. The principal additional burden is parameterization: BELAY adds $\alpha$9, whereas standard EMA needs only the averaging coefficient. The paper therefore presents BELAY not as a universal replacement for EMA, but as a more expressive coupled dynamical system (Patsenker et al., 2023).

The acronym should also be distinguished from several unrelated or only conceptually adjacent uses in the arXiv literature. "Recursive Belief Vision LLM" introduces RB-VLA, a belief-centric VLA architecture rather than a method named BELAY (Bagaria et al., 24 Feb 2026). "Belief Engine: Configurable and Inspectable Stance Dynamics in Multi-Agent LLM Deliberation" introduces BE, not BELAY (Yang et al., 14 May 2026). "ABBEL: LLM Agents Acting through Belief Bottlenecks Expressed in Language" introduces ABBEL (Lidayan et al., 23 Dec 2025), and "Agent-BRACE: Decoupling Beliefs from Actions in Long-Horizon Tasks via Verbalized State Uncertainty" introduces Agent-BRACE (Singh et al., 12 May 2026). These works share an interest in explicit state or belief representations, but they address long-horizon control or deliberation rather than weight averaging. Lower-case belay in climbing mechanics is likewise unrelated to the optimizer described here (Harutyunyan et al., 2016).