---
title: 'EMA-Sink: Exponential Moving Average for Streaming Data'
url: https://www.emergentmind.com/topics/exponential-moving-average-sink-ema-sink
type: topic
---

# EMA-Sink: Exponential Moving Average for Streaming Data

The Exponential Moving Average Sink (EMA-Sink) is a constant-memory streaming mechanism for maintaining exponentially weighted averages under sequential data ingestion. EMA-Sink generalizes exponential smoothing to both fixed- and variable-interval sampling regimes, and has recently seen critical application in sequence modeling, notably in autoregressive diffusion processes for streaming video generation [1502.03465], [2512.04678]. In parallel, strong stochastic convergence properties for adaptive EMA-Sink variants have been established in the ergodic theory and stochastic gradient descent literature [2505.10605], distinguishing EMA-Sink from classical constant-decay EMA.

## 1. Definitions and Update Recursions

EMA-Sink maintains a running estimate of an exponentially weighted mean over time-indexed observations $\{X_k\}$, optionally adapting to non-uniform arrival times.

**Variable-interval (Version 1)**  
For $t_1 \leq t_2 \leq \dots$, define time constant $\tau>0$, $\Delta_k = t_k - t_{k-1}$, and $\alpha_k = \exp(-\Delta_k/\tau)$.  
The EMA-Sink maintains numerators $\tilde{X}_k$ and denominators $\tilde{w}_k$:
\[
\tilde{X}_k = X_k + \alpha_k \tilde{X}_{k-1}, \quad \tilde{X}_1 = X_1
\]
\[
\tilde{w}_k = 1 + \alpha_k \tilde{w}_{k-1}, \quad \tilde{w}_1 = 1
\]
Yielding a bias-corrected average:
\[
\hat{X}_k = \tilde{X}_k / \tilde{w}_k
\]

**Fixed-interval (Version 2, Single-State)**  
For fixed step $\Delta$, define constant $\alpha = \exp(-\Delta/\tau)$, and recursively update:
\[
B_k = (1-\alpha) X_k + \alpha B_{k-1}, \quad B_1 = X_1
\]
$B_k$ converges asymptotically to the exponentially smoothed mean as the normalizing denominator $\tilde{w}_k \to 1$.

In streaming attention architectures, EMA-Sink operates by maintaining fixed-size "sink" vectors (projections of evicted tokens) updated with exponentials upon window slide:
\[
S^i = \alpha S^{i-1} + (1-\alpha) K^{i-w}
\]
for keys, and analogously for values [2512.04678].

## 2. Effective Memory, Bias, and Hyperparameters

EMA-Sink's design provides direct control over the effective averaging window and bias.

- **Effective Sample Count**: For fixed $\alpha$, the variance of the smoothed sequence for i.i.d. noise $\sigma^2$ is $\sigma^2 (1-\alpha)/(1+\alpha)$. Equating this to $\sigma^2 / n$ yields the "effective sample count":
  \[
  n = \frac{1+\alpha}{1-\alpha}, \quad \alpha = \frac{n-1}{n+1}
  \]
- **Effective Time Window**: In the small $\Delta$ regime, the real-time window $T$ relates to $\tau$ as $T \approx 2\tau$, or equivalently set $\tau = T/2$. For discrete time:
  \[
  \alpha = e^{-\Delta/\tau}, \quad e^{-\Delta/\tau} = \frac{T/\Delta - 1}{T/\Delta + 1}
  \]
  When $\Delta/T \ll 1$, $\tau \approx T/2$ is justified.
- **Bias and Initialization**: Two-state EMA-Sink (Version 1) avoids startup bias, while the single-state variant (Version 2) incurs slight initial bias, vanishing after several $\tau$ periods. In sequence attention, a static sink (no EMA update) leads to strong bias toward initial tokens ("frame copying"), while EMA-Sink yields a dynamic compromise [2512.04678].

Hyperparameter tuning chiefly involves $\alpha$:
- $\alpha \to 1$: Long horizon memory, slow decay of history, increased lag.
- $\alpha \to 0$: Rapid adaptation, minimal effective memory, high variance.
Empirical ablations in streaming video show increased dynamic scores with $\alpha \in [0.9, 0.99]$, with optimal drift/motion tradeoffs depending on the application [2512.04678].

## 3. Theoretical Guarantees and Convergence Properties

Extensions of EMA-Sink employing time-dependent decay ($p$-EMA with $\alpha_n = C/(n+1)^p$, $p\in(\tfrac{1}{2}, 1]$) achieve almost sure convergence under mild mixing/autocorrelation assumptions [2505.10605].

- **Weighted SLLN**: General strong law of large numbers for weighted averages applies when variance and covariance decay meet summability requirements:
  \[
  \Var\left(\sum_{k=1}^n b_kX_k\right) = O(A_n^2/\psi(A_n)), \quad b_n \leq A_n/\psi(A_n)
  \]
  With $b_n/\Lambda_n \to 0$ and suitably slow decay, $S_n/A_n \to \mathbb{E}[X]$ almost surely.
- **Fixed-$\alpha$ EMA Limitation**: For constant decay, the weight on new samples remains bounded away from zero, preventing the averaged variance from vanishing. $p$-EMA resolves this via subharmonic decay rates.
A plausible implication is that in adaptive learning rate applications, $p$-EMA estimators reliably track variance and mean without excess lag, and converge to stationary expectations of the underlying Markov chain [2505.10605].

## 4. Implementation, Complexity, and Streaming Workflow

EMA-Sink is designed for efficient streaming with O(1) time and space.

- **Streaming Pseudocode (Version 1)**:  
  ```
  state: last_time ← t0
         X_acc    ← 0
         w_acc    ← 0
  on_receive(X_new at t_new):
      Δ ← t_new − last_time
      α ← exp(−Δ/τ)
      X_acc ← α·X_acc + X_new
      w_acc ← α·w_acc + 1
      last_time ← t_new
      ĤX ← X_acc / w_acc
      output ĤX
  ```
  For attention-based window compression in autoregressive models:
  ```
  sink_K = α * sink_K + (1−α) * K_evict
  sink_V = α * sink_V + (1−α) * V_evict
  ```
- **Complexity**:  
  - Update: O(1) exponentiation, multiplies, and adds per step.
  - Memory: O(1) per state accumulator.
  - No need for historical data storage.

This design enables real-time streaming without cache growth or unbounded memory requirements, fitting applications where the time horizon N can be arbitrarily large [1502.03465], [2512.04678].

## 5. Applications in Sequence Modeling and Adaptive Optimization

EMA-Sink has broad applicability in time-series analysis, deep learning, and compression of long-horizon dependencies:

- **Streaming Video Generation**: In diffusion-based streaming architectures, EMA-Sink fuses evicted sliding-window tokens to produce sink key/value vectors. This preserves global context and recent dynamics, eliminating catastrophic drift and initial-frame copying observed in static sinks. Models employing EMA-Sink outperform both window-only and static-sink baselines, especially in dynamic score and drift metrics [2512.04678].
- **Adaptive Step Size in SGD**: $p$-EMA is directly used to average gradient norms and variances for adaptive learning rate schemes. By decaying the weighting factor $\alpha_n$ to zero, the estimator's noise is suppressed and the step size is reliably adjusted, preventing explosion or stagnation [2505.10605].
- **General Streaming Statistics**: EMA-Sink's constant-memory, bias-corrected updates make it broadly applicable to on-the-fly computation of means and higher moments in both fixed- and variable-rate settings [1502.03465].

## 6. Comparative Metrics, Ablation, and Limitations

Empirical results substantiate EMA-Sink's role in both maintaining coherence and capturing motion:

| Model Variant                 | Dynamic Score | Drift   |
|-------------------------------|---------------|---------|
| EMA-Sink + Re-DMD (full)      | 64.06         | 2.51    |
| –w/o EMA (static sink)        | 35.15         | 2.65    |
| –w/o any sink (window only)   | 51.56         | 5.08    |

Qualitative examples distinguish drifting (window only), frame copying (static sink), and natural transitions (EMA-Sink) [2512.04678].

Limitations identified include dependence on $\alpha$ selection (single value may not fit heterogeneous dynamics), loss of fine-grained details in coarse aggregation, and requirement that EMA-Sink be paired with semantically aware distillation (such as Re-DMD) to preserve fidelity [2512.04678].

A plausible implication is that adaptive or multi-sink formulations, or an $\alpha$ schedule responsive to content complexity, may mitigate loss of detail while retaining the long-horizon benefits.

## 7. Concluding Remarks and Outlook

EMA-Sink establishes a mathematically principled and empirically validated approach for streaming average estimation and memory compression under exponential weighting. It achieves constant-time and memory operation, direct control over memory dynamics via decay parameters, and strong convergence guarantees when using time-adaptive $p$-EMA schedules. Recent advances in streaming generative modeling demonstrate its practical utility, while theoretical studies explicate its advantages over classical EMAs [1502.03465], [2512.04678], [2505.10605].

Future research involves refinement of context-adaptive sinks, integration with more granular semantic measures, and extension to multi-resolution and multi-modal streaming scenarios. EMA-Sink remains central to scalable, bias-resilient long-horizon estimation in both statistical and deep learning models.

Source: https://www.emergentmind.com/topics/exponential-moving-average-sink-ema-sink