EMA ProtoUp: Exponential Prototype Update Strategy

• EMA ProtoUp is a prototype updating mechanism that employs exponential moving average to incrementally blend new features into robust and stable representations.
• It balances adaptation and stability by using a decay parameter to control the influence of recent observations while preserving historical information.
• It is widely applied in multimodal, federated, and self-supervised learning, supported by rigorous theoretical analysis of bias, variance, and convergence.

The Exponential Prototype Update Strategy, frequently denoted as EMA ProtoUp, is a prototype updating mechanism that employs an exponential moving average (EMA) to incrementally blend new feature or model information into prototype representations. This approach is designed to stabilize prototype learning in dynamic or stochastic environments, maintaining robust, smoothly evolving representations that minimize the negative impact of abrupt changes, noise, or non-stationarity, while supporting adaptation to complex distributions (e.g., multimodal, heterogeneous, or non-IID data). EMA ProtoUp has emerged as a foundational component in a variety of machine learning domains, including multimodal representation learning, online model adaptation, self-supervised and federated learning, and length generalization in sequence models. Its design, theoretical properties, and empirical validation have been detailed and refined across multiple recent works.

1. Fundamental Principles and Update Rules

The core mechanism of EMA ProtoUp is the weighted blending of incoming (typically batch-aggregated) feature information into an existing prototype, using a decay parameter λ to control the contribution of the historical prototype:

$$P_{\text{new}} = \lambda \cdot P_{\text{old}} + (1 - \lambda) \cdot f_{\text{new}}$$

Here, $P_{\text{old}}$ is the preceding prototype (encoding previous features or knowledge), $f_{\text{new}}$ is the new feature or model output to be incorporated, and $\lambda \in (0, 1)$ is the decay or momentum parameter. A lower λ emphasizes rapid adaptation to new data, while a higher λ emphasizes stability. Typical settings restrict λ to $(0, 0.5)$ to strike a balance between prototype reactivity and historical continuity (Jiang et al., 7 Oct 2025).

Recent work has introduced parameter scaling (e.g., λ or β adjustment with batch size scaling (Busbridge et al., 2023)), time-dependent decay (e.g., p-EMA with λ or γ_n converging to 1 as n increases (Köhne et al., 15 May 2025)), or explicit bias correction (e.g., BEMA (Block et al., 31 Jul 2025)) to govern the trade-off between bias, variance, and responsiveness in prototype updates.

2. Theoretical Analysis: Bias, Variance, and Convergence

The statistical behavior of EMA ProtoUp is characterized by a distinctive exponential decay of both effective bias and variance errors in each eigenspace of the data covariance matrix. For prototype or parameter sequences $\{w_t\}$ updated via SGD with constant learning rate $\delta$ and EMA parameter $\alpha$, the aggregated prototype

$$\overline{w}_N = \alpha^N w_0 + (1-\alpha) \sum_{t=0}^{N-1} \alpha^{N-1-t} w_t$$

achieves excess risk

$$\mathbb{E}[L(\overline{w}_N)] - L(w^*) \leq \text{EffectiveBias} + \text{EffectiveVar}$$

where, in each data eigenspace, the bias decays exponentially as $\max\{\alpha^N, (1-\delta\lambda_i)^N\}$, and the variance is strictly reduced compared to unaveraged or flat average SGD (Li et al., 19 Feb 2025).

The choice of λ or α is central, directly controlling the bias–variance trade-off. Making λ close to one yields low variance but induces a lag (bias); lower values deliver faster adaptation but higher variance. p-EMA adaptations, where the decay factor is λ_n = 1 - 1/(n+1)^p (p ∈ (½, 1]), ensure that the influence of the latest observations vanishes subharmonically with n, achieving almost-sure convergence even for autocorrelated or dependent samples (Köhne et al., 15 May 2025).

Bias correction schemes such as BEMA provide theoretical acceleration, compensating for the delayed response of EMA by injecting a correction proportional to the current–initial state difference (Block et al., 31 Jul 2025, Zsámboki et al., 9 Oct 2025). This ensures that the variance-reduction properties of EMA are preserved without introducing optimization lag, matching the Cramér–Rao lower bound in models such as the Ornstein–Uhlenbeck process.

3. Algorithmic Realizations and Modifications

Several algorithmic extensions of the base EMA ProtoUp have been proposed and validated:

• Wandering prototypes: To address intraclass heterogeneity, prototypes not only represent central modes but also edge or moderately distant observations ("wandering prototypes"), which are updated using the same EMA formula. This maintains robustness to non-homogeneous classes and captures edge-case structure (Jiang et al., 7 Oct 2025).
• Dynamic decay scaling: When mini-batch size is altered by a factor κ, maintaining consistent training dynamics requires updating the EMA decay to $\rho_{\text{new}} = \rho^\kappa$ (Busbridge et al., 2023).
• Switch EMA / SEMA: Combining standard EMA with periodic resetting of active parameters to the EMA values (for example, every epoch) yields improved flatness–sharpness trade-offs and faster convergence (Li et al., 14 Feb 2024).
• Bias-corrected EMA (BEMA): In online or sequence modeling regimes, the bias-corrected variant

$$\theta^{\text{BEMA}}_n = \alpha_n (\theta_n - \theta_0) + \theta^{\text{EMA}}_n$$

where α_n decays to zero, accelerates convergence and stability, particularly in fine-tuning and generalization tasks (Block et al., 31 Jul 2025, Zsámboki et al., 9 Oct 2025).

• Strong convergence p-EMA: For settings with heavy noise or temporally correlated samples, employing p-EMA ensures the noise component vanishes asymptotically and the prototype estimate converges almost surely to the steady-state value (Köhne et al., 15 May 2025).

4. Applications Across Modalities and Paradigms

EMA ProtoUp has been deployed in various settings, including but not limited to:

• Multimodal representation learning: In cancer survival analysis, integrating whole slide images and genomics via a shared prototype space updated with EMA ProtoUp ensures stable and interpretable cross-modal relationships, with added wandering prototypes improving sensitivity to heterogeneity. This yields state-of-the-art predictive and interpretable results across multiple medical datasets (Jiang et al., 7 Oct 2025).
• Federated and semi-supervised learning: EMA-based teacher–student frameworks (e.g., FedSwitch) employ local or global EMA updates for pseudo-label generation, adapting prototypes or teacher models in a communication-efficient and privacy-preserving manner in federated environments (Zhao et al., 2023).
• Online model adaptation: EMA ProtoUp is used within EKF and curriculum learning strategies for robust, stable parameter tracking in nonstationary or real-time data streams, where dynamic multi-epoch updates are gated by EMA-filtered error thresholds (Abuduweili et al., 2019).
• Self-supervised and pseudo-labeling methods: EMA ProtoUp stabilizes teacher models in self-supervised learning, dictating the accuracy and effectiveness of pseudo-labels in approaches such as BYOL and DINO, and scaling EMA parameters according to optimizer batch size is necessary to preserve learning dynamics (Busbridge et al., 2023).
• Gradient surgery and bilevel optimization: In constrained multi-objective settings, EMA is used to smooth the estimate of the primary objective gradient, ensuring that constraints (e.g., orthogonality in gradient projection) are preserved reliably under stochasticity (Hsieh et al., 5 Feb 2024).
• Sequence length generalization: EMA and its bias-corrected variant BEMA are shown to improve generalization across longer sequences in transformers, helping to stabilize logit displacement in attention-only models and compensate for softmax-induced dilution effects during length extrapolation (Zsámboki et al., 9 Oct 2025).

5. Empirical Performance and Comparative Analysis

Extensive empirical studies demonstrate several consistent advantages of EMA ProtoUp versus static prototype updates, tail or flat averaging, or direct last-iterate usage:

• Improved generalization: Exponential smoothing enables more robust prototype or parameter trajectories, supporting generalization on heterogeneous or shifted distributions, and across a wide range of tasks, from medical prediction (Jiang et al., 7 Oct 2025), federated image classification (Zhao et al., 2023), to transformer length generalization (Zsámboki et al., 9 Oct 2025).
• Stability and convergence: Empirical results consistently show reduced variance, damped oscillations, and accelerated convergence relative to standard SGD or Adam methods without EMA (Ahn et al., 28 May 2024, Li et al., 14 Feb 2024, Block et al., 31 Jul 2025).
• Interpretability and traceability: The incremental nature of EMA ProtoUp ensures each prototype’s evolution can be visualized, linked to specific data points or features, and does not suffer from abrupt re-initializations that would hinder interpretability (Jiang et al., 7 Oct 2025).
• Adaptive and scalable design: When combined with scaling rules (e.g., for large-batch training (Busbridge et al., 2023)) or curriculum-like sample reweighting (Abuduweili et al., 2019), EMA ProtoUp enables adaptive learning rates and batch sizes without sacrificing performance.
• Limitations: The main drawback is the introduction of a lag bias, which in standard (uncorrected) EMA can be non-negligible for large λ or in short-horizon settings. Bias correction, dynamic decay, or p-EMA variants mitigate this at the cost of additional computation or tuning (Block et al., 31 Jul 2025, Köhne et al., 15 May 2025).

6. Extensions, Theoretical Connections, and Design Considerations

Theoretical investigations connect EMA ProtoUp to damped harmonic oscillator dynamics, where the prototype or parameter update mimics a mass–spring–damper system. This analogy provides conceptual tools for controlling the balance between responsiveness and stability—interpreted as tuning the “spring constant” and damping coefficients of the update process (Patsenker et al., 2023). Recent work such as BELAY exploits this connection to introduce feedback from prototypes to the underlying parameter updates, enhancing control and robustness.

Moreover, the choice of decay scheduling (fixed vs dynamic λ), inclusion of bias correction, or p-EMA weighting, and the integration of additional curriculum or curriculum-like strategies (e.g., dynamic multi-epoch update policies) should be guided by the statistical properties of the data (stationarity, noise structure), application requirements (adaptation speed, interpretability), and the scale of the system (batch size, modality count). Consistency of training dynamics across batch sizes—or hardware platforms—often necessitates explicit scaling of EMA parameters (Busbridge et al., 2023).

In summary, EMA ProtoUp offers a theoretically grounded, empirically validated, and broadly applicable strategy for prototype and parameter updating, with clear guidelines for balancing stability, adaptation, and interpretability. Its extensibility to adaptive decay, bias correction, and dynamic scheduling—validated across a diverse range of applications and theoretical settings—establishes it as a core tool in modern prototype-based and online learning systems.