Dynamic EMA Proxy Variant

• Dynamic EMA proxy variants are adaptive strategies that integrate exponential moving averages within iterative training loops to reduce noise and variance in updates.
• They are applied in domains like stochastic games, neural rendering, and dynamic data pruning to enhance convergence and robustness.
• Implementing these variants involves tuning decay parameters and synchronizing EMA updates with model evolution for computational efficiency.

A dynamic EMA proxy variant refers to a family of algorithmic strategies that leverage the exponential moving average (EMA) construct as a surrogate or "proxy"—often dynamically adapted or coupled—to enhance learning, regularization, or robustness in stochastic optimization, adaptive data curation, and high-variance model training. Unlike static EMA practices, where EMA is used for post-hoc smoothing or test-time ensembling, dynamic EMA proxy variants systematically integrate the proxy during optimization, decision, or model evaluation cycles, enabling improved stability, adaptivity, and computational efficiency across diverse machine learning settings.

1. Principle and Definition

Dynamic EMA proxy variants employ EMA recursions as live, algorithmically coupled proxies within iterative training or selection loops. These proxies serve various roles:

• Variance reduction in updates: Replacing high-variance quantities (e.g., previous stochastic gradient, per-sample loss) with their exponentially smoothed EMA proxies.
• Consistency or regularization surrogates: Substituting entire models, parameter sets, or outputs by EMA averages within architectural or optimization constraints, as in dual-model systems.
• Adaptive smoothing: Dynamically adjusting EMA coefficients ($\beta$, $\alpha$, or analogues) as functions of recent stochasticity, trajectory history, or problem structure.

This dynamic integration stands in contrast to standard EMA, which usually maintains a passive, trailing average and is only deployed at inference or for model averaging at the end of training.

2. Algorithmic Instantiations

Several classes of dynamic EMA proxy variants have been proposed, each tuned to its problem structure:

• Dynamic EMA Correction for Stochastic Games: "Omega" introduces an EMA of historic gradients as a correction term within the optimistic gradient descent-ascent framework. The EMA proxy $\tilde F_t$ aggregates all past stochastic gradients, and replaces the previous-iteration sample in the update:

$$w_{t+1} = w_t - \eta \big[ (1+\alpha) F_{\xi_t}(w_t) - \alpha \tilde F_{t-1} \big]$$

$$\tilde F_t = (1-\beta) F_{\xi_t}(w_t) + \beta \tilde F_{t-1}$$

This structure reduces variance in min-max games, enabling robust and efficient (one-gradient-call) convergence unattainable by static alternatives (Ramirez et al., 2023).

• EMA Proxy for Consistency in Neural Rendering: In the "Asymmetric Dual 3DGS" system for 3D Gaussian Splatting, a full dual-model regime is replaced by a dynamic EMA proxy. Only the primary model $G_1$ is optimized, while its EMA copy acts as a regularized surrogate for enforcing consistency losses:

$$G_{\mathrm{EMA}}^t = \beta G_{\mathrm{EMA}}^{t-1} + (1-\beta) G_1^t$$

Densification and dynamic topology (cloning, splitting, pruning) necessitate synchronized proxy maintenance, with empirically tuned decay ($\beta=0.8$). This design achieves nearly the artifact suppression of the dual-model regime, at substantially reduced computational cost (Li et al., 4 Jun 2025).

• Dynamic EMA for Per-Sample Score Estimation: The Batch Loss Score for dynamic data pruning computes, per sample, a persistent importance score via an EMA of observed batch loss—a proxy for the unavailable per-sample loss signal. This EMA,

$s_i(t) = \alpha s_i(t-1) + (1-\alpha) L(B_t, t)$

is shown to function as a first-order IIR low-pass filter, attenuating high-frequency batch-composition noise while tracking persistent loss contribution, yielding empirically robust pruning without per-sample loss computation (Zhou et al., 6 Apr 2026).

• Adaptive EMA in Optimizer Moments: OptEMA generalizes Adam-like optimizer EMAs via closed-loop, step-size-adaptive decay parameters ($\alpha_t$, $\alpha$0) that react in real time to the gradient trajectory. These dynamic proxies support both first and second moment adaptive filtering, yielding convergence guarantees that are tight in both stochastic and zero-noise regimes—unlike constant-$\alpha$1 baselines (Yuan, 10 Mar 2026).
• Dynamic Switching/Anchoring: Switch EMA (SEMA) executes a dynamic periodic reset from EMA proxy to the live model weights (e.g., at each epoch), uniting flatness and sharp descent. Iterates alternate between EMA-proxied and gradient descent steps in a scheduled manner, lowering both oscillatory noise and generalization gap, and empirically outperforming both static EMA and standard stochastic averaging (Li et al., 2024).

3. Theoretical Guarantees and Analysis

The impact of dynamic EMA proxies is fundamentally determined by their filtering properties and their coupling to the optimization dynamics or noise structure:

• Variance Reduction and Stability: In the Omega algorithm, the EMA proxy replaces a single-sample (high-variance) historical gradient with a convex combination of all previous gradients, reducing the variance of the correction term without increasing the number of stochastic gradient evaluations (Ramirez et al., 2023).
• Frequency Separation and Smoothing: Theoretical analysis of dynamic EMA as in BLS shows the proxy acts as a low-pass filter, smoothing high-frequency noise associated with batch composition in per-sample losses, while preserving slow-varying, informative signal contributions (Zhou et al., 6 Apr 2026).
• Strong Law and Stochastic Convergence: The $\alpha$2-EMA adaptation introduces decay-to-zero assignment to the most recent sample, ensuring strong almost sure convergence properties for random dynamical systems, under mild mixing/autocorrelation decay, an improvement over standard EMA where averaging noise persists indefinitely (Köhne et al., 15 May 2025).
• Zero-Noise Optimality: OptEMA's closed-loop decays guarantee that, in the deterministic (noise-free) regime, convergence matches the optimal deterministic rate, whereas standard EMA only attains suboptimal stochastic rates (Yuan, 10 Mar 2026).
• Momentum and Physical Analogy: The BELAY scheme generalizes EMA as a physically coupled damped harmonic oscillator, offering design principles for tuning the inertial and coupling parameters to interpolate between overdamped (robust/stable) and underdamped (agile, fast-tracking) proxy dynamics (Patsenker et al., 2023).

4. Implementation Practices and Hyperparameter Guidance

Practical implementation of dynamic EMA proxies differs by application and requires hyperparameter tuning:

• Decay Parameter Selection: Recommended default decay values are $\alpha$3 for 3DGS EMA proxy (Li et al., 4 Jun 2025), $\alpha$4 for Omega (Ramirez et al., 2023), and $\alpha$5 for BLS (Zhou et al., 6 Apr 2026). Lowering $\alpha$6/$\alpha$7 increases adaptivity but also increases sensitivity to noise.
• Structural Synchronization: When the modeled object undergoes dynamic structural changes (e.g., dynamical cloning/splitting/pruning of Gaussian sets), the EMA proxy must duplicate, reinitialize, or remove entries in step to maintain synchronization (Li et al., 4 Jun 2025). For scalar or vector-valued proxies, storage scales linearly with sample/model size.
• Masking and Alternation: In the EMA-3DGS regime, effective suppression of confirmation bias and error modes relies not only on the EMA proxy but also on the deterministic alternation of complementary masking schemes (e.g., adaptive hard mask $\alpha$8 vs. learned soft mask $\alpha$9), which injects structural asymmetry otherwise lost in single-model+proxy training (Li et al., 4 Jun 2025).
• Usage Frequency: Proxy updates typically occur at each iteration, immediately after the primary model update, though variants such as SEMA perform synchronous switching/reset at longer intervals (e.g., per epoch) (Li et al., 2024).
• One-line or Multi-line Integration: Several variants are designed for seamless code integration—e.g., BLS offers a one-line proxy injection, and SEMA requires only a single additional lines for switch logic (Li et al., 2024, Zhou et al., 6 Apr 2026).

5. Empirical Performance and Comparative Evaluation

Empirical studies demonstrate that dynamic EMA proxy variants provide tangible benefits:

• Stochastic Saddle-Point Games: On 2D and 100D stochastic bilinear/quadratic games, Omega shows consistently faster and more robust convergence than independent-sample OG (ISOG) or SGD, with a $\tilde F_t$0 proxy smoothing out instability that otherwise leads to divergence (Ramirez et al., 2023).
• 3D Neural Rendering: The Dynamic EMA Proxy in Asymmetric Dual 3DGS achieves near-parity reconstruction quality (PSNR, SSIM, LPIPS) relative to full dual-model regimes while reducing training time by 33–45% and halving memory consumption. Ablation confirms that masks and alternating update strategies are critical for this performance (Li et al., 4 Jun 2025).
• Dynamic Data Pruning: Substituting BLS for true per-sample losses retains $\tilde F_t$1 accuracy deviation across 14 vision and multi-modal datasets, including image classification, segmentation, and large-scale vision-language tasks, while avoiding the need for intrusive per-sample loss computation (Zhou et al., 6 Apr 2026).
• Generalization and Flatness/Sharpness: SEMA demonstrably achieves improved test accuracy, faster convergence, and lower final variance than static EMA or SWA, across classification, generative, video, regression, and NLP benchmarks, as well as under challenging optimizers (AdamW, LAMB, SAM) (Li et al., 2024).

Variant/Domain	Key Proxy Role	Empirical Benefit
Omega (Stochastic Games)	EMA of gradients in correction	Robust/faster convergence over ISOG
3DGS EMA Proxy	EMA as model surrogate	0.05–0.3dB drop, 33–45% faster
BLS in Pruning	EMA of batch loss per sample	$\tilde F_t$2 acc delta, minimal code
OptEMA	Decay-adaptive moment EMA	Optimal/noiseless convergence
SEMA	Epochic switch to EMA	Best generalization/convergence

6. Connections and Theoretical Distinctions

The notion of a dynamic EMA proxy bridges and extends several theoretical and algorithmic paradigms:

• Contrast with Constant-EMA: Where classical EMA (e.g., standard Adam) operates with fixed, open-loop decay and serves only as a passive reference, dynamic proxies feature closed-loop, data- or trajectory-adaptive control, and active algorithmic integration within the learning loop (Yuan, 10 Mar 2026, Köhne et al., 15 May 2025).
• Physical System Analogies: The damped harmonic oscillator analogy provides a geometric and dynamic intuition for how moving-average proxies track, stabilize, and modulate model parameter evolution, and motivates novel "spring-coupled" update regimes (e.g., BELAY) that interpolate between classical and proxy-driven learning (Patsenker et al., 2023).
• Strong Law and Adaptive Filtering: Subharmonic decay variants ($\tilde F_t$3-EMA) ensure almost sure convergence and vanishing tracking noise even when the underlying stochastic process has persistent autocorrelation, further distinguishing dynamic EMA proxies from classical approaches (Köhne et al., 15 May 2025).
• Proxy-as-Model, not Statistic: In modern regimes (e.g., 3DGS, SEMA), EMA proxies are not just statistical summaries but actively determine, constrain, or anchor the trajectory of the primary model—a fundamental methodological shift.

7. Practical Implications and Future Directions

Dynamic EMA proxy variants offer significant advantages in computational efficiency, robustness, and adaptability:

• Scalability: Avoiding dual-model updates (e.g., EMA-3DGS) enables training on larger scenes or higher-dimensional parameterizations (Li et al., 4 Jun 2025).
• Plug-and-Play Robustness: Proxy structures with one-line or minimal-code integration support rapid adoption, as demonstrated in dynamic pruning and switching strategies (Zhou et al., 6 Apr 2026, Li et al., 2024).
• Hyperparameter Transferability: Adaptive decays (OptEMA, $\tilde F_t$4-EMA) minimize the need for retuning across stochastic and deterministic regimes (Yuan, 10 Mar 2026, Köhne et al., 15 May 2025).
• Architectural Generality: Demonstrated efficacy spans CNNs, transformers, diffusion models, semi-supervised pipelines, and geometric/multi-view vision (Li et al., 2024, Li et al., 4 Jun 2025).
• Theoretical Foundations: Ongoing developments seek to unify adaptive EMA proxy regimes with strong convergence results, frequency-domain analyses, and physical/dynamical analogies.

A plausible implication is that dynamic EMA proxy variants will remain central to the design of scalable and robust algorithms in areas requiring stochastic optimization, architectural regularization, or sample-efficient adaptive selection. Their structure enables fine-grained control over variance, noise, and bias, while accommodating the resource and integration constraints of modern deep learning systems.