Bias-Corrected EMA (BEMA)

• BEMA is an advanced smoothing technique that corrects for initialization and lag bias inherent in the standard EMA using multiplicative and additive corrections.
• It accelerates convergence and reduces variance in stochastic, non-stationary scenarios, making it valuable for SGD training and adaptive time series analysis.
• BEMA integrates dynamic decay scheduling and learnable correction mechanisms to boost performance in deep neural network optimization and transformer architectures.

A Bias-Corrected Exponential Moving Average (BEMA) is an enhancement of the classical exponential moving average (EMA) technique that introduces explicit mechanisms to remove the bias or “lag” characteristic of standard EMA schemes. BEMA aims to retain the variance-reduction and smoothing properties of EMA, while accelerating convergence and improving downstream performance, especially in stochastic, non-stationary, or online learning scenarios such as deep neural network optimization. The methodology and theoretical foundations of BEMA have been developed and analyzed in the context of stochastic process modeling, quantile estimation, adaptive time series analysis, stochastic gradient descent (SGD) training, and transformer architectures.

1. Fundamental Principles of Exponential Moving Average and Its Bias

The standard EMA for a sequence of iterates or observations $\{\theta_t\}$ proceeds recursively as

$$\theta^{\text{EMA}}_t = (1-\beta_t) \cdot \theta^{\text{EMA}}_{t-1} + \beta_t \cdot \theta_t$$

with $\beta_t \in (0,1)$. The EMA weights recent values more strongly than earlier values, which reduces short-term variance and stabilizes noisy sequences. However, this recursive structure introduces two forms of bias:

• Initialization bias: Early EMA values are predominantly influenced by the initial parameter $\theta_0$, especially when $t$ is small and $\beta_t$ is small.
• Lag bias: EMA “lags” behind the current parameter trajectory, effectively introducing a smoothing delay. This is particularly problematic in settings where the parameter vector $\theta_t$ evolves rapidly or in the high-variance regime at the start of training (Block et al., 31 Jul 2025, Li et al., 5 Nov 2024, Morales-Brotons et al., 27 Nov 2024, Duda, 2020, Köhne et al., 15 May 2025).

This inherent bias has practical implications: it can slow convergence, reduce responsiveness in non-stationary regimes, and degrade the fidelity of parameter tracking in applications such as online prediction or adaptive optimization.

2. Formal Bias Correction Schemes (BEMA): Theory and Algorithms

The BEMA modifies the classical EMA to eliminate these sources of bias, mainly by introducing a bias correction term or a dynamic weighting schedule.

2.1 Multiplicative Bias Correction

A canonical approach is to normalize the EMA by the cumulative discount factor, which yields an unbiased average under stationarity: $$\text{BEMA}_t = \frac{\theta^{\text{EMA}}_t}{1 - \alpha^t}$$ where $\alpha$ is the fixed decay factor (Morales-Brotons et al., 27 Nov 2024).

2.2 Additive/Linear Correction

For online learning and SGD, a linear bias correction is used. One calculates the EMA as usual, then augments with a correction proportional to the change from the initial parameter: $$\begin{aligned} \theta^{\text{EMA}}_t &= (1-\beta_t)\theta^{\text{EMA}}_{t-1} + \beta_t\theta_t \ \alpha_t &= (\beta_t)^\lambda \quad \text{(with } 0 < \lambda < 1)\ \theta^{\text{BEMA}}_t &= \theta^{\text{EMA}}_t + \alpha_t\cdot (\theta_t-\theta_0) \end{aligned}$$ This scheme “re-centers” the smoothed average toward the current trajectory, counteracting lag while preserving variance reduction (Block et al., 31 Jul 2025).

2.3 Structural Bias Correction via Weight Decay Schedule

A fundamentally different approach replaces the fixed $\beta$ in EMA with a time-dependent schedule—e.g., $p$-EMA:

$$\gamma_n = 1 - \frac{1}{(n+1)^p}, \quad p \in (1/2,1]$$

and

$$\tau_{n+1} = \gamma_n \tau_n + (1-\gamma_n)\tilde{\tau}_{n+1}$$

such that the weight on recent data decays to zero subharmonically, ensuring that the averaged estimate's variance diminishes with time (i.e., eventual unbiasedness and strong stochastic convergence) (Köhne et al., 15 May 2025).

2.4 Bias Correction in Structured Models

In advanced architectures, such as predictor-corrector transformer layers or state space sequence models, EMA-style weighting coefficients are set as exponential functions of a learnable decay parameter $\gamma$: $$P_{t+1} = y_t + \sum_{i=1}^n [\gamma \cdot (1-\gamma)^{n-i}\cdot \varphi_i]$$ where $\varphi_i$ are intermediate residuals analogous to Runge-Kutta stages. Here, BEMA arises by optimizing $\gamma$ via gradient descent, implicitly shifting more weight to lower-bias, higher-order terms—thus correcting bias in both a data-driven and model-driven manner (Li et al., 5 Nov 2024).

3. Theoretical Analysis and Convergence Guarantees

The introduction of explicit bias correction, either multiplicatively, additively, or by decay scheduling, impacts both bias and variance properties of the averaging process:

• General setting: In the continuous-time limit for SGD on quadratic objectives (modeled as an Ornstein–Uhlenbeck process), the “optimal” estimator of the minimum,

$$\hat{\theta}^\text{MLE}_T = \frac{1}{T} \int_0^T \theta_t dt + \frac{A^{-1}}{T}(\theta_T - \theta_0)$$

is unbiased and attains the Cramér–Rao lower bound for mean squared error (Block et al., 31 Jul 2025). The practical BEMA algorithm approximates this with a polynomial-weighted blend of the plain EMA and the latest increment, achieving accelerated decay of both bias and variance relative to standard EMA.

• $p$-EMA convergence: When the decay on the most recent weight follows $1/(n+1)^p$ for $p\in(1/2,1]$, almost sure convergence of the average to the true mean is achieved under mild mixing conditions, overcoming limitations of the constant-weight EMA where error plateaus (Köhne et al., 15 May 2025).
• Blocking Effects of Initialization: Bias correction is most important early in training or in online estimation, when available data is limited. Both analytic corrections (multiplicative or additive) and dynamic decay schedules quickly reduce the dominating influence of the initial state (Morales-Brotons et al., 27 Nov 2024, Pandey, 2022).

4. Implementation and Empirical Impact

4.1 Algorithmic Integration

A typical BEMA update in deep learning is achieved with minimal code change. Empirically, implementations use a two-stage update:

param_EMA.data = (1 - beta_t) * param_EMA.data + beta_t * param.data alpha_t = beta_t ** 0.4 param_BEMA.data = alpha_t * (param.data - param_0.data) + param_EMA.data

with $\lambda = 0.4$ as a practical hyperparameter for bias removal. The correction term’s weight may be annealed or adapted per problem (Block et al., 31 Jul 2025).

4.2 Performance in Modern Applications

Empirical results in LLM fine-tuning, stochastic process denoising, and online adaptation show that BEMA:

• Accelerates convergence of training and test loss, exhibiting reduced lag in loss minimization (Block et al., 31 Jul 2025).
• Leads to higher final accuracy on LLM generation and question-answering benchmarks (e.g., BoolQ, MMLU-HS, GSM8K).
• Reduces oscillations and output format errors in closed-loop (autoregressive) evaluation.
• Outperforms standard EMA and traditional iterate averaging in terms of both convergence rate and final evaluation metrics, substantiating the theoretical analysis (Block et al., 31 Jul 2025).
• In structured sequence models, learnable EMA-based weights in predictor-corrector frameworks lower local truncation error and improve translation BLEU scores and LLMing perplexity (Li et al., 5 Nov 2024).

A table summarizing key mechanisms:

BEMA Variant	Correction Mechanism	Domain
Analytical	EMA / (1 – $\alpha^t$)	Deep RL, Adam
Additive	$\alpha_t$·(iter – init) + EMA	LM fine-tuning
p-EMA	Decay $\propto$ $1/(n+1)^p$	SGD, SDEs
Dynamic learned	Learnable EMA decay $\gamma$	Transform./RK

5. Connections to Related Adaptive Estimation Methods

BEMA is conceptually related but distinct from several existing bias correction and smoothing methods:

• Bias-corrected moment estimates in optimizers (e.g., Adam): These divide the moving averages by factors such as $(1-\beta^t)$ to maintain unbiasedness when $t$ is small (Morales-Brotons et al., 27 Nov 2024, Pandey, 2022).
• Moving exponential average (MEA): Restricts computation to a window of length $N$, with bias controlled by appropriate tuning of $\alpha$ and window length (Klinker, 2020).
• p-EMA: Embeds bias correction into the decay schedule, providing stronger convergence than EMA but distinct from post-hoc multiplicative corrections (Köhne et al., 15 May 2025).
• Structural/Model-driven Correction: In system identification or ODE-based neural architectures, BEMA-style learning enables the model to dynamically calibrate the averaging schedule for minimum bias and variance (Li et al., 5 Nov 2024, Abuduweili et al., 2019).

6. Limitations, Hyperparameters, and Open Questions

The effectiveness of BEMA depends on several factors:

• Choice and adaptation of correction strength ($\alpha_t$, $\lambda$, decay schedule): Overcorrecting may undermine the desired implicit regularization, while undercorrecting leads to lag (Block et al., 31 Jul 2025, Morales-Brotons et al., 27 Nov 2024).
• Early vs. late training dynamics: BEMA’s utility is highest when the learning process is nonstationary or high-variance; as the system approaches stationarity, all averages converge (Block et al., 31 Jul 2025, Köhne et al., 15 May 2025).
• Context-dependence: In online filtering (e.g., MEKF with EMA in Kalman filtering (Abuduweili et al., 2019)), bias correction in the classical Adam/BEMA sense is less prominent—but EMA smoothing of both step and covariance matrices (EMA-V, EMA-P) improves robustness and convergence, acting as a de facto bias dampener.
• Analytical vs. learnable correction: Data-driven adaptation (e.g., learned $\gamma$ decay in transformers) can implicitly calibrate bias correction, but lacks the interpretability of analytical normalization (Li et al., 5 Nov 2024).

7. Broader Applications and Theoretical Significance

• In stochastic optimization, BEMA provides unbiased or accelerated estimators of underlying parameter trajectories, which is crucial for transfer learning, self-supervised teacher-student training, and adaptive control (Block et al., 31 Jul 2025, Morales-Brotons et al., 27 Nov 2024, Abuduweili et al., 2019).
• In time series analysis and adaptive parameter estimation (e.g., EPD fitting (Duda, 2020)), EMA or BEMA-style recursion enables tracking of local regime shifts with near-optimal statistical efficiency.
• In sequence modeling and neural ODEs, exponentially weighted predictor-corrector schemes expand model expressivity while mitigating error accumulation and initialization bias (Li et al., 5 Nov 2024).
• Theoretical models based on stochastic calculus (OU processes) reveal that BEMA-style estimators can achieve minimum-variance unbiasedness, meeting statistical lower bounds (Cramér–Rao) for noise-laden estimation tasks (Block et al., 31 Jul 2025).

---

In summary, Bias-Corrected Exponential Moving Average represents a theoretically grounded advancement over standard EMA, offering both provable statistical advantages and practical benefits in complex, nonstationary, and stochastic learning systems. Its numerous variants adapt to a range of modeling scenarios, balancing variance reduction, bias elimination, and computational tractability.