Role of trainable decay rates in EMA-based memory

Investigate whether learning element-wise decay rates in an Exponential Moving Average (EMA) memory enables the separation of fast and slow memory features that accounts for the observed performance gains on Mem‑RPE.

Background

The authors evaluate EMA memory with fixed versus trainable decay factors and find that trainable decay improves Mem‑RPE performance.

They hypothesize that element-wise trainable decay allows the model to represent multiple timescales via fast and slow features, explaining the improvement.