Bias-Corrected EMA (BEMA)

• BEMA is a bias-corrected averaging method that removes the initialization-induced bias in standard EMA, thereby reducing optimization lag.
• It employs a time-varying smoothing coefficient and a power-law correction to balance variance reduction with rapid convergence.
• Empirical results show that BEMA accelerates training in language-model fine-tuning and enhances performance in stochastic minimax tasks.

Searching arXiv for the specified BEMA paper and closely related work. Bias-Corrected EMA (BEMA) denotes a bias-corrected iterate-averaging scheme designed to preserve the variance-reduction effect of exponential moving averages while removing the finite-time lag induced by dependence on early iterates. In language-model fine-tuning, BEMA was introduced as a minimal augmentation of standard EMA: it corrects the residual pull toward the initialization $\theta_0$, which can otherwise slow optimization despite smoothing stochastic oscillations (Block et al., 31 Jul 2025). The acronym also appears in a distinct line of work on stochastic minimax optimization, where bias-corrected momentum recursions use Hessian–vector products to control momentum drift and improve iteration complexity (Cai et al., 2024). In the literature represented here, the unifying theme is bias correction of moving-average or momentum estimators so that stabilization does not come at the cost of persistent optimization error.

1. Core update rule

For model weights $\theta_t \in \mathbb{R}^d$, the fixed-$\beta$ exponential moving average is defined by

$$m_0 = \theta_0, \qquad m_t = \beta m_{t-1} + (1-\beta)\theta_t, \qquad 0<\beta<1.$$

Unrolling gives

$$m_t = (1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i + \beta^t \theta_0.$$

The term $\beta^t \theta_0$ is the source of the finite-time bias: the average remains pulled toward the initial point $\theta_0$, especially when $\beta \approx 1$ and $t$ is not large (Block et al., 31 Jul 2025).

The corresponding exact bias-corrected estimator is

$$\hat m_t =\frac{m_t-\beta^t\theta_0}{1-\beta^t} =\frac{(1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i}{1-\beta^t}.$$

An equivalent form is

$\theta_t \in \mathbb{R}^d$0

The practical BEMA update uses a time-varying smoothing coefficient and a power-law correction: $\theta_t \in \mathbb{R}^d$1

$\theta_t \in \mathbb{R}^d$2

with $\theta_t \in \mathbb{R}^d$3. In this formulation, $\theta_t \in \mathbb{R}^d$4 replaces the exact factor $\theta_t \in \mathbb{R}^d$5 by a practical schedule. When $\theta_t \in \mathbb{R}^d$6, so that $\theta_t \in \mathbb{R}^d$7, the method reduces to pure EMA; when $\theta_t \in \mathbb{R}^d$8, it recovers the classical exact debias approximately when $\theta_t \in \mathbb{R}^d$9 (Block et al., 31 Jul 2025).

This construction is intentionally minimal. The update leaves the optimizer unchanged and augments only the averaging mechanism. A plausible implication is that BEMA is best understood not as a new optimizer, but as a stabilizer layered on top of existing fine-tuning pipelines.

2. Bias, lag, and the finite-time correction

The central motivation for BEMA is the observation that standard EMA trades variance reduction for a finite-time bias toward old iterates. In the language-model fine-tuning setting described in the paper, stochasticity caused by small batch sizes can produce large oscillations in generation quality. EMA smooths these fluctuations, but the weight on $\beta$0 and earlier iterates can make the averaged model trail the trajectory of the underlying optimizer, creating optimization lag (Block et al., 31 Jul 2025).

The bias-correction mechanism explicitly subtracts the initialization-induced component and renormalizes the remaining weights. In exact form, this removes the contribution of $\beta$1. In practical form, the additive term $\beta$2 approximates the required correction using only the current iterate, the initialization, and a scalar schedule.

Two limiting behaviors are emphasized in the exposition. With weak or absent correction, BEMA collapses back to EMA and therefore reintroduces lag. With overly aggressive correction, the stabilizing role of averaging can be diminished or reversed. This is reflected empirically in the sensitivity of the parameter $\beta$3: larger $\beta$4 corresponds to weaker correction and recovers EMA-like lag, whereas too small $\beta$5 can destabilize training (Block et al., 31 Jul 2025).

A common misconception is that any moving average improving smoothness must also improve optimization progress. The finite-time analysis here contradicts that simplification: smoothing and unbiasedness are distinct properties, and BEMA is specifically intended to combine them.

3. Theoretical formulation under stochastic quadratic dynamics

The theoretical analysis in the language-model paper studies stochastic quadratic optimization through the Ornstein–Uhlenbeck stochastic differential equation

$\beta$6

where $\beta$7 and $\beta$8 is the minimizer. In this framework, an online stabilizer is treated as an estimator $\beta$9 derived from the trajectory $$m_0 = \theta_0, \qquad m_t = \beta m_{t-1} + (1-\beta)\theta_t, \qquad 0<\beta<1.$$0 (Block et al., 31 Jul 2025).

Three reference estimators are compared.

The last iterate of vanilla SGD satisfies

$$m_0 = \theta_0, \qquad m_t = \beta m_{t-1} + (1-\beta)\theta_t, \qquad 0<\beta<1.$$1

so it retains a non-vanishing variance floor $$m_0 = \theta_0, \qquad m_t = \beta m_{t-1} + (1-\beta)\theta_t, \qquad 0<\beta<1.$$2.

The flat average, identified with Polyak–Ruppert averaging,

$$m_0 = \theta_0, \qquad m_t = \beta m_{t-1} + (1-\beta)\theta_t, \qquad 0<\beta<1.$$3

obeys

$$m_0 = \theta_0, \qquad m_t = \beta m_{t-1} + (1-\beta)\theta_t, \qquad 0<\beta<1.$$4

Standard EMA achieves the same $$m_0 = \theta_0, \qquad m_t = \beta m_{t-1} + (1-\beta)\theta_t, \qquad 0<\beta<1.$$5 variance rate but incurs an additional bias term

$$m_0 = \theta_0, \qquad m_t = \beta m_{t-1} + (1-\beta)\theta_t, \qquad 0<\beta<1.$$6

When $$m_0 = \theta_0, \qquad m_t = \beta m_{t-1} + (1-\beta)\theta_t, \qquad 0<\beta<1.$$7, that bias dominates, producing the lag discussed in the empirical sections.

The paper then derives a maximum-likelihood estimator via Girsanov’s theorem: $$m_0 = \theta_0, \qquad m_t = \beta m_{t-1} + (1-\beta)\theta_t, \qquad 0<\beta<1.$$8 This estimator is unbiased and satisfies

$$m_0 = \theta_0, \qquad m_t = \beta m_{t-1} + (1-\beta)\theta_t, \qquad 0<\beta<1.$$9

which matches the Cramér–Rao lower bound. BEMA is then interpreted as a practical surrogate for this optimal finite-time estimator: it approximates the $$m_t = (1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i + \beta^t \theta_0.$$0 term by a scalar correction $$m_t = (1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i + \beta^t \theta_0.$$1, while replacing the flat integral by an EMA (Block et al., 31 Jul 2025).

The stated consequence is that BEMA removes the $$m_t = (1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i + \beta^t \theta_0.$$2 initial bias while retaining $$m_t = (1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i + \beta^t \theta_0.$$3 variance, thereby accelerating convergence over both standard EMA and vanilla training in the finite-time regime. This suggests that the main contribution is not asymptotic variance reduction alone, but the elimination of the transient bias term that matters most in realistic training horizons.

4. Algorithmic realization and practical configuration

The implementation is presented as a two-line modification of standard EMA. The algorithm stores $$m_t = (1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i + \beta^t \theta_0.$$4, maintains the EMA state $$m_t = (1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i + \beta^t \theta_0.$$5, and returns the corrected average $$m_t = (1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i + \beta^t \theta_0.$$6: $$\hat m_t =\frac{m_t-\beta^t\theta_0}{1-\beta^t} =\frac{(1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i}{1-\beta^t}.$$7

The overhead is also explicit. Relative to standard EMA, the method requires one additional full-size vector $$m_t = (1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i + \beta^t \theta_0.$$7 and one EMA copy $$m_t = (1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i + \beta^t \theta_0.$$8. The extra computation consists of one vector difference $$m_t = (1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i + \beta^t \theta_0.$$9, one elementwise scaling by $\beta^t \theta_0$0, and one addition (Block et al., 31 Jul 2025).

The recommended defaults are:

• $\beta^t \theta_0$1, $\beta^t \theta_0$2
• $\beta^t \theta_0$3, $\beta^t \theta_0$4
• burn-in $\beta^t \theta_0$5
• update frequency $\beta^t \theta_0$6 steps

The accompanying implementation notes characterize the hyperparameters functionally. Smaller $\beta^t \theta_0$7 yields stronger bias correction; $\beta^t \theta_0$8 is suggested as a starting range. The smoothing exponent $\beta^t \theta_0$9 governs averaging strength, with $\theta_0$0 near $\theta_0$1 often best and $\theta_0$2 recovering flat averaging. The update frequency $\theta_0$3 trades compute against staleness, and $\theta_0$4 is described as a good balance. If GPU memory is limited, $\theta_0$5 can be stored on CPU. The method is also described as a drop-in replacement for EMA that does not require changes to the optimizer or learning-rate schedule; learning-rate decay is optional (Block et al., 31 Jul 2025).

These details place BEMA in the class of low-overhead training stabilizers rather than optimizer redesigns. The practical emphasis is on preserving existing training recipes while altering only the averaging operator.

5. Empirical behavior in language-model fine-tuning

The empirical study evaluates BEMA on Qwen2.5-1.5B, Gemma3-1B, and Llama3.2-1B. The main fine-tuning setup uses Tulu-3-SFT with approximately $\theta_0$6 sequences of fewer than 4096 tokens, for 2 epochs, batch size 256, learning rate $\theta_0$7, and AdamW. The compared stabilizers are vanilla training, EMA with $\theta_0$8, OUEMA, and BEMA. Evaluation comprises train and test cross-entropy together with closed-loop generation on BoolQ, GSM8K, and MMLU-HS; generation uses 50 samples per prompt at $\theta_0$9, with average accuracy reported (Block et al., 31 Jul 2025).

The reported findings have three main parts.

First, in convergence behavior, BEMA test-loss converges faster than EMA and suppresses stochastic oscillations without introducing lag. Second, on BoolQ and GSM8K, BEMA reaches a peak approximately $\beta \approx 1$0 points higher in accuracy than EMA and more than 20 points above vanilla in early training steps. Third, cross-model experiments on Gemma3-1B and Llama3.2-1B show similar qualitative behavior, namely accelerated and smoother training loss under BEMA (Block et al., 31 Jul 2025).

Ablation results further delimit the operating regime. The default $\beta \approx 1$1 is reported as robust across tasks. Larger $\beta \approx 1$2, corresponding to weaker correction, recovers EMA lag; smaller $\beta \approx 1$3, corresponding to stronger correction, can destabilize. Among EMA schedules, the best power is reported around $\beta \approx 1$4. The study also reports that updating every $\beta \approx 1$5 steps works well, burn-in $\beta \approx 1$6 is effective, and the lag parameter $\beta \approx 1$7 has minimal effect.

Additional comparisons position BEMA relative to related averaging schemes. OUEMA improves over EMA but remains inferior to BEMA. Double EMA mitigates some lag but trails BEMA in both convergence speed and final accuracy. The paper’s summary statement is that BEMA significantly accelerates convergence and improves final performance on large-scale language-model fine-tuning (Block et al., 31 Jul 2025).

This empirical pattern suggests that the main benefit appears in the early and intermediate finite-time regime, where training instability and estimator lag are both large enough to matter.

6. Related usage: bias-corrected momentum for stochastic minimax optimization

A separate usage of the acronym BEMA appears in stochastic minimax optimization, where it denotes bias-corrected momentum algorithms rather than weight averaging. In that setting, the objective is

$\beta \approx 1$8

and the momentum variables $\beta \approx 1$9 are updated using stochastic gradients together with Hessian–vector products that approximate the ideal correction $t$0 (Cai et al., 2024).

The recursions are

$t$1

After clipping or normalization, the iterates are updated by

$t$2

Two variants are given. HCMM-1 uses gradient clipping and is analyzed under a nonconvex–strongly-concave structure; HCMM-2 uses momentum normalization and is analyzed under a nonconvex–PL structure. The assumptions include $t$3-Lipschitz gradient, $t$4-Lipschitz Hessian, unbiased bounded-variance first- and second-order stochastic oracles, and, for HCMM-1, a bounded gradient norm. Under these conditions, both variants achieve $t$5 iteration complexity for finding an $t$6-stationary point, improving over the $t$7 benchmark associated here with standard SGD or SGDA (Cai et al., 2024).

The role of Hessian–vector products is central. The paper states that, under $t$8-Lipschitz Hessian, the residual in the bias correction is $t$9, the variance of the momentum estimator becomes $$\hat m_t =\frac{m_t-\beta^t\theta_0}{1-\beta^t} =\frac{(1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i}{1-\beta^t}.$$0, and this permits $$\hat m_t =\frac{m_t-\beta^t\theta_0}{1-\beta^t} =\frac{(1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i}{1-\beta^t}.$$1. Empirically, the methods are evaluated on five LIBSVM datasets—“mushrooms,” “phishing,” “ijcnn1,” “a9a,” and “w8a”—for a distributionally robust logistic-regression problem with $$\hat m_t =\frac{m_t-\beta^t\theta_0}{1-\beta^t} =\frac{(1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i}{1-\beta^t}.$$2, $$\hat m_t =\frac{m_t-\beta^t\theta_0}{1-\beta^t} =\frac{(1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i}{1-\beta^t}.$$3, and $$\hat m_t =\frac{m_t-\beta^t\theta_0}{1-\beta^t} =\frac{(1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i}{1-\beta^t}.$$4. HCMM-1 and HCMM-2 are reported to converge significantly faster than STORM-GDA and SAGDA, with HCMM-2 usually exhibiting the fastest finite-time decay and HCMM-1 showing slightly stabler curves on certain datasets.

Although this minimax line of work differs substantially from the language-model averaging scheme, the shared label “bias-corrected momentum” or “bias-corrected EMA” reflects a common design principle: the moving estimator is adjusted so that stabilization does not accumulate a systematic drift term.

7. Position within iterate averaging and momentum methods

Within the language-model setting, BEMA is explicitly contrasted with vanilla last-iterate training, Polyak–Ruppert averaging, standard EMA, OUEMA, and Double EMA. Its defining claim is not merely that averaging helps, but that the specific initial-point bias of EMA can be removed without sacrificing the $$\hat m_t =\frac{m_t-\beta^t\theta_0}{1-\beta^t} =\frac{(1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i}{1-\beta^t}.$$5 variance behavior associated with averaging (Block et al., 31 Jul 2025).

Within the minimax setting, the related BEMA momentum schemes are contrasted with STORM-GDA and SAGDA, and the claimed improvement arises from correcting the drift of momentum by means of stochastic Hessian–vector information rather than from averaging model weights (Cai et al., 2024).

Taken together, these usages indicate that “Bias-Corrected EMA” is not a single monolithic method but a family resemblance across methods that modify EMA- or momentum-style recursions to suppress finite-time bias. In the weight-averaging formulation, the bias is the residual dependence on $$\hat m_t =\frac{m_t-\beta^t\theta_0}{1-\beta^t} =\frac{(1-\beta)\sum_{i=1}^t \beta^{t-i}\theta_i}{1-\beta^t}.$$6; in the minimax momentum formulation, the bias is the drift between stale momentum and current gradients. The common methodological objective is therefore precise: retain smoothing or variance reduction while eliminating the error term that makes stabilization lag behind optimization.