---
title: Cumulative Memory Recurrent Unit (CMRU)
url: https://www.emergentmind.com/topics/cumulative-memory-recurrent-unit-cmru
type: topic
---

# Cumulative Memory Recurrent Unit (CMRU)

The **Cumulative Memory Recurrent Unit (CMRU)** is a parallelizable recurrent cell introduced to preserve the **persistent, discrete memory** and **hardware-friendliness** of the Bistable Memory Recurrent Unit (BMRU) while restoring **good gradient flow** and improving **learning stability** and **performance** to be competitive with modern parallelizable RNNs such as LRUs, minGRUs, and SSM-based models [2605.11855]. Its defining change is a **cumulative update formulation** that replaces BMRU’s destructive overwrite step with an update that adds a quantized contribution on top of a scaled previous state, thereby creating skip-connections through time while retaining event-driven, hysteretic memory dynamics. The same work also introduces **$\alpha$CMRU**, a relaxed variant with input-dependent update scale that trades strict quantization for higher expressivity [2605.11855].

## 1. Historical and conceptual setting

CMRU emerged from the line of work on **Memory Recurrent Units (MRUs)**, which were proposed to combine **persistent memory via multistability** with **parallelizable computation over sequence length** [2601.09495]. In that framework, the BMRU was introduced as a proof-of-concept recurrent cell whose hidden state is bistable, quantized, and compatible with the parallel scan algorithm. BMRU addressed two issues simultaneously: **persistent memory** through multistability with two quantized attractors $\pm \alpha$ per dimension, and **analog hardware mapping** through direct implementability with Schmitt triggers in CMOS, enabling **sub-$\mu$W** power, including **100 nW keyword spotting** [2605.11855].

The motivation for CMRU is tied to the limitations of existing sequence models. Modern parallelizable RNNs and Transformers are described through **linear or linearizable state dynamics** of the form
\[
h_t = A h_{t-1} + B x_t,
\]
with eigenvalues of $A$ strictly inside the unit circle for stability; this implies **monostability**, a unique attracting fixed point, and **fading memory**, meaning that information stored in $h_t$ decays exponentially with elapsed time [2605.11855]. Parallelizable gated RNNs such as minGRU avoid nonlinear recurrent dependence and remain scan-compatible, but because their input-only gate satisfies $z_t \in (0,1)$, they have no strict retain or overwrite mode and therefore no true persistent memory [2605.11855].

Against that backdrop, BMRU supplied exact retain behavior and hysteretic, discrete state transitions, but its performance lagged behind LRUs and minGRUs on complex sequential problems. The central diagnosis leading to CMRU is that **gradient blocking during state updates** is the main reason BMRU underperforms modern parallelizable RNNs on demanding tasks [2605.11855]. A related but distinct tradition appears in HiPPO, where a “cumulative memory” unit can be understood as a memory core that stores an optimal polynomial projection of the full input history, especially in the HiPPO-LegS construction; this suggests that the term *cumulative memory* has both a quantized hysteretic realization in CMRU and a polynomial-projection interpretation in HiPPO-style recurrent memory [2008.07669].

## 2. From BMRU to the cumulative update rule

In the original BMRU, the hidden state $h_t \in \mathbb{R}^d$ is **quantized** so that each component is constrained to two values $\pm \alpha_i$, with learned $\alpha \in \mathbb{R}^d_{>0}$. Given input $x_t \in \mathbb{R}^m$, BMRU defines
\[
\hat{h}_t = W_x x_t + b_x,
\]
\[
\beta_t = \left|W_\beta x_t + b_\beta\right|,
\]
\[
z_t = \HOp\left( |\hat{h}_t| - \beta_t \right) \in \{0,1\}^d,
\]
and the state update
\[
h_t = z_t \odot \SOp(\hat{h}_t) \odot \alpha + (1 - z_t)\odot h_{t-1}.
\]
Here, $\HOp$ is the componentwise Heaviside step function, $\SOp$ is the sign function, and $\odot$ denotes the Hadamard product [2605.11855].

This yields two operating modes per dimension. If $|\hat{h}_{t,i}| < \beta_{t,i}$, then $z_{t,i}=0$ and the cell is in **retain mode**:
\[
h_{t,i} = h_{t-1,i}.
\]
If $|\hat{h}_{t,i}| \ge \beta_{t,i}$, then $z_{t,i}=1$ and the cell is in **update mode**:
\[
h_{t,i} = \alpha_i\, \mathrm{sign}(\hat{h}_{t,i}) \in \{-\alpha_i,+\alpha_i\}.
\]
The result is a **bistable** hidden state with hysteretic thresholds and exact persistence whenever threshold crossings do not occur [2605.11855].

CMRU changes only the update part of this mechanism. The candidate, threshold, and gate remain unchanged:
\[
\hat{h}_t = W_x x_t + b_x,\qquad
\beta_t = |W_\beta x_t + b_\beta|,\qquad
z_t = \HOp(|\hat{h}_t|-\beta_t).
\]
The new state equation is
\[
h_t =
z_t \odot \left[\SOp(\hat{h}_t) \odot \alpha + \varepsilon h_{t-1}\right]
+ (1-z_t)\odot h_{t-1},
\]
with scalar hyperparameter $\varepsilon \in [-1,1]$, restricted to $[0,1]$ in most experiments [2605.11855].

Per dimension, retain mode is unchanged:
\[
h_{t,i}=h_{t-1,i}.
\]
Update mode becomes
\[
h_{t,i}=\alpha_i\,\mathrm{sign}(\hat{h}_{t,i})+\varepsilon h_{t-1,i}.
\]
BMRU therefore performs **full replacement** during updates, whereas CMRU performs a **cumulative** update, similar to a residual connection through time [2605.11855].

## 3. Gradient flow, memory regimes, and state-space interpretation

The principal theoretical advantage of CMRU is the restoration of gradient flow during updates. For BMRU, treating $z_t$ as constant in the local Jacobian gives
\[
\frac{\partial h_t}{\partial h_{t-1}} = \DiagOp(1-z_t).
\]
Hence, in retain mode,
\[
\left.\frac{\partial h_t}{\partial h_{t-1}}\right|_{z_t=0}=I,
\]
but in update mode,
\[
\left.\frac{\partial h_t}{\partial h_{t-1}}\right|_{z_t=1}=0.
\]
Any update event therefore cuts the gradient path from $h_t$ to $h_{t-1}$, which the paper links to poor convergence, high sensitivity to initialization, and brittle learning on long sequences [2605.11855].

For CMRU, the recurrence can be written as
\[
h_t = \DiagOp(1-z_t+\varepsilon z_t)\, h_{t-1}
      + z_t \odot \SOp(\hat{h}_t)\odot\alpha,
\]
so the Jacobian becomes
\[
\frac{\partial h_t}{\partial h_{t-1}} = \DiagOp(1-z_t+\varepsilon z_t).
\]
In update mode,
\[
\left.\frac{\partial h_t}{\partial h_{t-1}}\right|_{z_t=1}=\varepsilon I.
\]
Thus, whenever $0<\varepsilon\le 1$, updates no longer block gradients, and when $\varepsilon=1$ the gradient magnitude is exactly $1$ in both retain and update modes, giving **skip-connection-like behavior through time** [2605.11855].

The paper interprets $\varepsilon$ as selecting among three regimes. When $\varepsilon=0$, CMRU reduces to the original BMRU, with bistable states $\pm\alpha$ and gradient blocking preserved. When $0<\varepsilon<1$, the model implements **event-driven fading memory**: memory decays with the number of update events rather than with elapsed time, while retain intervals remain perfectly persistent. When $\varepsilon=1$, the cell becomes a **quantized cumulative integrator**:
\[
h_t = h_{t-1} + \SOp(\hat{h}_t)\odot\alpha
\]
in update mode, with identity Jacobian in both modes [2605.11855].

This modification also changes the reachable state structure. In BMRU, each dimension can only take $\pm \alpha_i$, so the overall state space is limited to at most $2^d$ points. In CMRU with $\varepsilon=1$,
\[
h_{t,i}=h_{0,i}+k_i\alpha_i,\qquad k_i\in\mathbb{Z},
\]
so reachable states form a **lattice** of integer multiples of $\alpha$, yielding countably infinite discrete attractors [2605.11855]. The paper presents this as a substantial increase in representational capacity while preserving discrete spacing desirable for analog circuits.

CMRU also admits a **selective state-space representation**
\[
h_t = A(x_t)h_{t-1}+B(x_t)x_t,
\]
with
\[
A(x_t)=\DiagOp(1-z_t+\varepsilon z_t).
\]
In retain mode, $A(x_t)=I$ and all eigenvalues are $1$, directly providing persistent memory. In update mode, $A(x_t)=\mathrm{diag}(\varepsilon)$, so all eigenvalues equal $\varepsilon$; for $0<\varepsilon<1$ they lie inside the unit circle, and for $\varepsilon=1$ they remain $1$ [2605.11855]. This places CMRU at an explicit interface between persistent-memory dynamics and the state-space vocabulary used for linear RNNs and SSMs.

## 4. $\alpha$CMRU and representational trade-offs

The **$\alpha$CMRU** variant relaxes the fixed quantization scale of CMRU by introducing an input-dependent scale
\[
\alpha_t = W_\alpha x_t + b_\alpha \in \mathbb{R}^d,
\]
and update
\[
h_t =
z_t \odot \left[\SOp(\hat{h}_t)\odot\alpha_t + \varepsilon h_{t-1}\right]
+ (1-z_t)\odot h_{t-1}.
\]
The sign pattern remains discrete through $\SOp(\hat{h}_t)$, but the magnitude can vary continuously with the input [2605.11855].

This changes the geometry of the memory state. In CMRU, hidden states remain tied to fixed multiples of a global $\alpha$; in $\alpha$CMRU, hidden states are no longer restricted to fixed multiples of a global $\alpha$ and can densely cover $\mathbb{R}^d$ [2605.11855]. The trade-off is explicit: strict quantization and explicit bistability at fixed levels are lost, but the model gains **smooth, high-precision continuous memory** while preserving event-driven accumulation and exact persistence in retain mode.

The paper positions the two variants differently. CMRU retains **quantized states**, **persistent memory**, and **noise-resilient dynamics** essential for analog implementation. $\alpha$CMRU is described as better suited for **large-scale digital ML** where quantization is not a hardware constraint [2605.11855]. A common misconception is therefore to treat the two cells as interchangeable: the reported results indicate that they share the cumulative-update principle, but only CMRU preserves the strict discrete-state structure and explicit analog-hardware mapping of the BMRU lineage.

A second misconception is that the gains of CMRU arise simply from larger state expressivity. On sequential MNIST, the paper concludes instead that **gradient flow from cumulative updates, not increased representational capacity, is the crucial improvement** [2605.11855]. This point matters especially when comparing CMRU to $\alpha$CMRU, because the fixed-scale CMRU still improves dramatically relative to BMRU once cumulative updates are enabled.

## 5. Training setup, architecture, and computational characteristics

The experimental architecture uses a common **backbone** into which all recurrent cells are plugged. The model dimension is $m=256$, the recurrent cell has state dimension $d$ varied across experiments from $4$ to $256$, and the number of recurrent blocks is $r\in\{1,3,6\}$ depending on the task [2605.11855]. Each block contains a recurrent sublayer producing
\[
y_{1:T} = \NormOp(h_{1:T}) \odot \sigma(\LinearOp(x_{1:T})),
\]
followed by a pointwise MLP with GLU,
\[
\mathrm{MLP}(x)=\LinearOp_2(\GLUOp(\LinearOp_1(x))),
\]
with hidden size $4m$, and skip connections with learned gain
\[
\mathrm{output}=\upsilon\odot x+\SubLayerOp(x),\qquad \upsilon \approx 1.
\]

Pooling depends on the task. **Last pooling** is used for tasks explicitly testing long-range memory, namely sMNIST, copy-first, and parity, whereas **mean pooling** is used for LRA tasks and audio tasks [2605.11855]. Standard sinusoidal positional encodings are concatenated to inputs and linearly projected.

Training uses **AdamW** with $\beta_1=0.9$, $\beta_2=0.99$, $\epsilon=10^{-8}$, and weight decay $10^{-4}$. The learning rate is $10^{-3}$ with $1\%$ linear warmup and cosine decay to $10^{-5}$. Gradient norm is clipped to $1$, batch size is $64$, no task-specific hyperparameter tuning is performed, and each configuration is run with $5$ seeds with mean $\pm$ min–max reported [2605.11855].

Because the Heaviside and sign operators are non-differentiable, training uses **surrogate gradients**:
\[
\frac{d\HOp(x)}{dx} \approx \frac{1}{1 + (\pi \alpha_\text{surr} x)^2},
\qquad \alpha_\text{surr}=1,
\]
so the forward dynamics remain exactly discrete while the backward pass is smoothed [2605.11855]. Computationally, CMRU, $\alpha$CMRU, BMRU, LRU, and minGRU are all **affine in $h_{t-1}$**, which means they support **associative scan** over sequence length and can therefore be parallelized on GPUs and TPUs [2605.11855]. This inherits the scan-compatibility emphasized in the earlier MRU formulation, where recurrences of the form $h_t=a_t\odot h_{t-1}+b_t$ admit a Blelloch-style scan implementation [2601.09495].

For models intended to be deployed with $\varepsilon=0$ on existing Schmitt-trigger BMRU hardware, the paper proposes **$\varepsilon$-annealing**: start at $\varepsilon=1$, keep it for $5\%$ of steps, linearly decay to $0$ over the next $70\%$, and hold at $0$ for the final $25\%$; only checkpoints after $\varepsilon=0$ are saved and evaluated [2605.11855]. This is presented as a way to preserve the deployment dynamics of BMRU while benefiting from the gradient flow of CMRU during training.

## 6. Empirical performance and benchmark profile

On **sequential MNIST** with last pooling, $d=32$, and one layer, the effect of $\varepsilon$ is direct. For **CMRU, $d=32$**, $\varepsilon=0$ yields **$\sim 30\%$ accuracy, high variance**, whereas $\varepsilon=1$ yields **$\sim 96\%$ accuracy, low variance**. For **$\alpha$CMRU, $d=32$**, $\varepsilon=0$ yields **$\sim 54\%$ accuracy**, and $\varepsilon=1$ yields **$\sim 97\%$ accuracy** [2605.11855]. The paper reports that increasing $\varepsilon$ monotonically improves both mean accuracy and cross-seed stability, with the largest gains on the harder sMNIST relative to pMNIST.

At small state dimension, the reported scaling behavior is notable. With $\varepsilon=1$ and a single layer on sMNIST, at **$d=4$**, CMRU reaches **$\sim 84\%$** and $\alpha$CMRU **$\sim 88\%$**, while LRU and minGRU are described as significantly worse and more variable. At **$d=256$**, all methods are **$\sim 96$–$97\%$**, but the CMRU variants still show lower variance [2605.11855]. This indicates that the cumulative memory mechanism is especially advantageous in the small-model regime emphasized by ultra-low-power applications.

The strongest specialization appears on **copy-first** tasks. In the **discrete 15-class copy-first, noise-free** setting with $d=4$ and last pooling, **CMRU ($\varepsilon=1$) achieves $100\%$ accuracy up to sequences of length $L=10{,}000$**, while **LRU/minGRU collapse to near random for large $L$** [2605.11855]. In the **continuous, noise-free** version, CMRU with $d=4$ achieves MAE near the theoretical quantization bound $\mathcal{E}^*_{\text{MAE}}$, and $\alpha$CMRU achieves essentially zero MAE. In the **continuous with heavy noise** variant, with $x_0 \sim \mathcal{U}(-1,1)$ and later inputs i.i.d. from the same distribution, $\alpha$CMRU with $\varepsilon=1$ remains robust across all $L$ and noise with MAE $\sim 0$, whereas LRU and minGRU degrade rapidly with $L$ and noise and at $L=10k$ essentially fail, with MAE near the $0.5$ baseline [2605.11855].

On larger-scale noisy copy-first with **$r=6$, $d=256$, $L=5000$**, the paper reports that **CMRU and $\alpha$CMRU solve the task robustly for all seeds**, while **LRU and minGRU still fail entirely** [2605.11855]. The interpretation given is architectural rather than merely parametric: the advantage comes from persistent-memory structure rather than only from small-model efficiency.

On the **Long Range Arena**-style tasks, the performance profile is mixed. On **IMDb sentiment**, LRU has the best mean at **$\sim 67\%$**, minGRU is slightly below, and CMRU/$\alpha$CMRU are **$\sim 65$–$66\%$**, within **$\sim 2\%$**. On **sCIFAR10**, LRU and minGRU are best at **$\sim 60$–$64\%$**, especially at larger depth, while CMRU and $\alpha$CMRU are slightly behind but close, with the gap shrinking with depth [2605.11855]. By contrast, on **ListOps**, at **$r=3,6$**, CMRU and $\alpha$CMRU reach **$\sim 61\%$ accuracy**, LRU is **$\sim 40\%$** and largely fails, and minGRU is intermediate at **$\sim 55$–$60\%$**. On **Pathfinder**, CMRU reaches **$\sim 90\%$** and $\alpha$CMRU **$\sim 92\%$**, while LRU and minGRU are **$\sim 50\%$**, which is chance at this small capacity of **$3\times 32$** [2605.11855]. The paper interprets these results as evidence that persistent-memory cells have a clear advantage on discrete and long-range compositional tasks.

For **character-level language modeling on Shakespeare**, the reported cross-entropy losses are: **$\alpha$CMRU: 1.443**, **CMRU: 1.455**, **LRU: 1.504**, **minGRU: 1.453**, **Hybrid minGRU+CMRU: 1.442**, and **Hybrid minGRU+$\alpha$CMRU: 1.441** [2605.11855]. This supports the claim that CMRU and $\alpha$CMRU are competitive on smooth statistical tasks and that **persistent + fading memory** hybrids are complementary.

In the **audio keyword spotting** regime with very small state dimensions, the differences are small. On **KWS Digits (11 classes)**, all models are **$\sim 95$–$96\%$ accuracy even at $d=4$**; CMRU at **$d=16$** reaches **$95.38\%$**, while LRU at **$d=16$** reaches **$96.34\%$**. On **KWS All (35 classes)**, at **$d=4$**, CMRU is **$\sim 89\%$** and LRU/minGRU are **$\sim 90$–$90.5\%$**; at **$d=16$**, CMRU is **$89.76\%$**, LRU **$91.82\%$**, and minGRU **$91.13\%$** [2605.11855]. The important point emphasized is that a single-layer CMRU with $d=16$ still fits existing hardware assumptions while maintaining competitive accuracy.

## 7. Analog implementation, model landscape, and related cumulative-memory frameworks

CMRU is explicitly designed for **always-on sensors, implants, and related settings requiring sub-$\mu$W operation for years** [2605.11855]. In BMRU, quantized bistability and hysteresis map directly to **Schmitt trigger circuits**, where states are stored as discrete logic levels and do not accumulate analog errors [2605.11855; 2601.09495]. CMRU with $\varepsilon=1$ extends this hardware picture to a **hysteretic charge accumulator**: cumulative updates correspond to adding discrete charges or currents to an integrator, and the paper proposes a native analog circuit based on a cascade of BMRU cells interleaved with Differential Pair Integrators, with staircase states at multiples of $\alpha$ [2605.11855]. The reported Cadence simulations show event-driven increments or decrements of output by $\alpha$ every $\tau$ ms in update mode, perfect retain in retain mode, and instantaneous power in the deep nW regime, including **$\sim 40$ nW in the example**.

Within the broader model landscape, the paper contrasts CMRU and $\alpha$CMRU with LRUs, SSMs, and minGRU. LRUs and SSMs are strong on tasks such as sCIFAR and IMDb, but are limited to fading memory and typically rely on digital implementations or capacitor-based analog realizations [2605.11855]. minGRU is fully parallelizable and strong on sCIFAR, IMDb, and language modeling, but its gates remain in $(0,1)$ and therefore do not provide true retain or overwrite modes [2605.11855]. CMRU and $\alpha$CMRU, by contrast, supply **explicit persistent memory via multistability and exact retain mode**, and at $\varepsilon=1$ they provide perfect gradient flow through time while remaining affine in the previous state [2605.11855].

A useful clarification concerns the relation between CMRU and **HiPPO**. HiPPO-LegS defines a memory state as the coefficient vector of the optimal projection of the entire history onto a polynomial basis under the measure
\[
\mu^{(t)}(x)=\frac{1}{t}\mathbf{1}_{[0,t]}(x),
\]
leading to the ODE
\[
\frac{d}{dt} c(t) = -\frac{1}{t} A c(t) + \frac{1}{t} B f(t),
\]
and discrete recurrence
\[
c_{k+1} = \left(I-\frac{1}{k}A\right)c_k + \frac{1}{k}B f_k
\]
for full-history cumulative memory [2008.07669]. This is a different construction from the quantized, hysteretic CMRU introduced in 2026, but the juxtaposition is informative: HiPPO provides a cumulative-memory mechanism through optimal polynomial projection, whereas CMRU provides cumulative memory through **event-driven discrete accumulation with hysteresis**. This suggests two distinct technical meanings of cumulative recurrent memory: one based on analytically derived compression of all history, the other on quantized multistable state transitions.

The main contributions claimed for CMRU are therefore fourfold: diagnosis of **gradient blocking** in BMRU as the primary training bottleneck; introduction of **cumulative updates** that restore gradient flow while preserving persistent memory; a **selective SSM representation** in which $\varepsilon$ directly controls eigenvalues; and an explicit path toward **ultra-low-power analog RNNs** that combine quantized, noise-resilient persistent memory with gradient-friendly training [2605.11855]. A plausible implication is that CMRU occupies a specific niche rather than replacing fading-memory models wholesale: it is strongest where discrete, long-range retention and hardware-constrained efficiency are both first-order design criteria.

Source: https://www.emergentmind.com/topics/cumulative-memory-recurrent-unit-cmru