Cumulative Memory Recurrent Unit (CMRU)

• CMRU is a recurrent unit that implements a cumulative update mechanism, preserving hysteretic, discrete memory and enabling parallelizable, hardware-friendly computation.
• Its design replaces destructive state overwrites with additive quantized contributions, thereby creating skip-connections through time that improve gradient flow.
• CMRU bridges persistent multistability with modern RNN architectures, offering robust long-range dependency retention and efficient low-power analog implementation.

The Cumulative Memory Recurrent Unit (CMRU) is a parallelizable recurrent cell introduced to preserve the persistent, discrete memory and hardware-friendliness of the Bistable Memory Recurrent Unit (BMRU) while restoring good gradient flow and improving learning stability and performance to be competitive with modern parallelizable RNNs such as LRUs, minGRUs, and SSM-based models (Brandoit et al., 12 May 2026). Its defining change is a cumulative update formulation that replaces BMRU’s destructive overwrite step with an update that adds a quantized contribution on top of a scaled previous state, thereby creating skip-connections through time while retaining event-driven, hysteretic memory dynamics. The same work also introduces $\alpha$CMRU, a relaxed variant with input-dependent update scale that trades strict quantization for higher expressivity (Brandoit et al., 12 May 2026).

1. Historical and conceptual setting

CMRU emerged from the line of work on Memory Recurrent Units (MRUs), which were proposed to combine persistent memory via multistability with parallelizable computation over sequence length (Geeter et al., 14 Jan 2026). In that framework, the BMRU was introduced as a proof-of-concept recurrent cell whose hidden state is bistable, quantized, and compatible with the parallel scan algorithm. BMRU addressed two issues simultaneously: persistent memory through multistability with two quantized attractors $\pm \alpha$ per dimension, and analog hardware mapping through direct implementability with Schmitt triggers in CMOS, enabling sub-$\mu$W power, including 100 nW keyword spotting (Brandoit et al., 12 May 2026).

The motivation for CMRU is tied to the limitations of existing sequence models. Modern parallelizable RNNs and Transformers are described through linear or linearizable state dynamics of the form

$h_t = A h_{t-1} + B x_t,$

with eigenvalues of $A$ strictly inside the unit circle for stability; this implies monostability, a unique attracting fixed point, and fading memory, meaning that information stored in $h_t$ decays exponentially with elapsed time (Brandoit et al., 12 May 2026). Parallelizable gated RNNs such as minGRU avoid nonlinear recurrent dependence and remain scan-compatible, but because their input-only gate satisfies $z_t \in (0,1)$, they have no strict retain or overwrite mode and therefore no true persistent memory (Brandoit et al., 12 May 2026).

Against that backdrop, BMRU supplied exact retain behavior and hysteretic, discrete state transitions, but its performance lagged behind LRUs and minGRUs on complex sequential problems. The central diagnosis leading to CMRU is that gradient blocking during state updates is the main reason BMRU underperforms modern parallelizable RNNs on demanding tasks (Brandoit et al., 12 May 2026). A related but distinct tradition appears in HiPPO, where a “cumulative memory” unit can be understood as a memory core that stores an optimal polynomial projection of the full input history, especially in the HiPPO-LegS construction; this suggests that the term cumulative memory has both a quantized hysteretic realization in CMRU and a polynomial-projection interpretation in HiPPO-style recurrent memory (Gu et al., 2020).

2. From BMRU to the cumulative update rule

In the original BMRU, the hidden state $h_t \in \mathbb{R}^d$ is quantized so that each component is constrained to two values $\pm \alpha_i$, with learned $\alpha \in \mathbb{R}^d_{>0}$. Given input $\pm \alpha$0, BMRU defines

$\pm \alpha$1

$\pm \alpha$2

$\pm \alpha$3

and the state update

$\pm \alpha$4

Here, $\pm \alpha$5 is the componentwise Heaviside step function, $\pm \alpha$6 is the sign function, and $\pm \alpha$7 denotes the Hadamard product (Brandoit et al., 12 May 2026).

This yields two operating modes per dimension. If $\pm \alpha$8, then $\pm \alpha$9 and the cell is in retain mode: $\mu$0 If $\mu$1, then $\mu$2 and the cell is in update mode: $\mu$3 The result is a bistable hidden state with hysteretic thresholds and exact persistence whenever threshold crossings do not occur (Brandoit et al., 12 May 2026).

CMRU changes only the update part of this mechanism. The candidate, threshold, and gate remain unchanged: $\mu$4 The new state equation is

$\mu$5

with scalar hyperparameter $\mu$6, restricted to $\mu$7 in most experiments (Brandoit et al., 12 May 2026).

Per dimension, retain mode is unchanged: $\mu$8 Update mode becomes

$\mu$9

BMRU therefore performs full replacement during updates, whereas CMRU performs a cumulative update, similar to a residual connection through time (Brandoit et al., 12 May 2026).

3. Gradient flow, memory regimes, and state-space interpretation

The principal theoretical advantage of CMRU is the restoration of gradient flow during updates. For BMRU, treating $h_t = A h_{t-1} + B x_t,$0 as constant in the local Jacobian gives

$h_t = A h_{t-1} + B x_t,$1

Hence, in retain mode,

$h_t = A h_{t-1} + B x_t,$2

but in update mode,

$h_t = A h_{t-1} + B x_t,$3

Any update event therefore cuts the gradient path from $h_t = A h_{t-1} + B x_t,$4 to $h_t = A h_{t-1} + B x_t,$5, which the paper links to poor convergence, high sensitivity to initialization, and brittle learning on long sequences (Brandoit et al., 12 May 2026).

For CMRU, the recurrence can be written as

$h_t = A h_{t-1} + B x_t,$6

so the Jacobian becomes

$h_t = A h_{t-1} + B x_t,$7

In update mode,

$h_t = A h_{t-1} + B x_t,$8

Thus, whenever $h_t = A h_{t-1} + B x_t,$9, updates no longer block gradients, and when $A$0 the gradient magnitude is exactly $A$1 in both retain and update modes, giving skip-connection-like behavior through time (Brandoit et al., 12 May 2026).

The paper interprets $A$2 as selecting among three regimes. When $A$3, CMRU reduces to the original BMRU, with bistable states $A$4 and gradient blocking preserved. When $A$5, the model implements event-driven fading memory: memory decays with the number of update events rather than with elapsed time, while retain intervals remain perfectly persistent. When $A$6, the cell becomes a quantized cumulative integrator: $A$7 in update mode, with identity Jacobian in both modes (Brandoit et al., 12 May 2026).

This modification also changes the reachable state structure. In BMRU, each dimension can only take $A$8, so the overall state space is limited to at most $A$9 points. In CMRU with $h_t$0,

$h_t$1

so reachable states form a lattice of integer multiples of $h_t$2, yielding countably infinite discrete attractors (Brandoit et al., 12 May 2026). The paper presents this as a substantial increase in representational capacity while preserving discrete spacing desirable for analog circuits.

CMRU also admits a selective state-space representation

$h_t$3

with

$h_t$4

In retain mode, $h_t$5 and all eigenvalues are $h_t$6, directly providing persistent memory. In update mode, $h_t$7, so all eigenvalues equal $h_t$8; for $h_t$9 they lie inside the unit circle, and for $z_t \in (0,1)$0 they remain $z_t \in (0,1)$1 (Brandoit et al., 12 May 2026). This places CMRU at an explicit interface between persistent-memory dynamics and the state-space vocabulary used for linear RNNs and SSMs.

4. $z_t \in (0,1)$2CMRU and representational trade-offs

The $z_t \in (0,1)$3CMRU variant relaxes the fixed quantization scale of CMRU by introducing an input-dependent scale

$z_t \in (0,1)$4

and update

$z_t \in (0,1)$5

The sign pattern remains discrete through $z_t \in (0,1)$6, but the magnitude can vary continuously with the input (Brandoit et al., 12 May 2026).

This changes the geometry of the memory state. In CMRU, hidden states remain tied to fixed multiples of a global $z_t \in (0,1)$7; in $z_t \in (0,1)$8CMRU, hidden states are no longer restricted to fixed multiples of a global $z_t \in (0,1)$9 and can densely cover $h_t \in \mathbb{R}^d$0 (Brandoit et al., 12 May 2026). The trade-off is explicit: strict quantization and explicit bistability at fixed levels are lost, but the model gains smooth, high-precision continuous memory while preserving event-driven accumulation and exact persistence in retain mode.

The paper positions the two variants differently. CMRU retains quantized states, persistent memory, and noise-resilient dynamics essential for analog implementation. $h_t \in \mathbb{R}^d$1CMRU is described as better suited for large-scale digital ML where quantization is not a hardware constraint (Brandoit et al., 12 May 2026). A common misconception is therefore to treat the two cells as interchangeable: the reported results indicate that they share the cumulative-update principle, but only CMRU preserves the strict discrete-state structure and explicit analog-hardware mapping of the BMRU lineage.

A second misconception is that the gains of CMRU arise simply from larger state expressivity. On sequential MNIST, the paper concludes instead that gradient flow from cumulative updates, not increased representational capacity, is the crucial improvement (Brandoit et al., 12 May 2026). This point matters especially when comparing CMRU to $h_t \in \mathbb{R}^d$2CMRU, because the fixed-scale CMRU still improves dramatically relative to BMRU once cumulative updates are enabled.

5. Training setup, architecture, and computational characteristics

The experimental architecture uses a common backbone into which all recurrent cells are plugged. The model dimension is $h_t \in \mathbb{R}^d$3, the recurrent cell has state dimension $h_t \in \mathbb{R}^d$4 varied across experiments from $h_t \in \mathbb{R}^d$5 to $h_t \in \mathbb{R}^d$6, and the number of recurrent blocks is $h_t \in \mathbb{R}^d$7 depending on the task (Brandoit et al., 12 May 2026). Each block contains a recurrent sublayer producing

$h_t \in \mathbb{R}^d$8

followed by a pointwise MLP with GLU,

$h_t \in \mathbb{R}^d$9

with hidden size $\pm \alpha_i$0, and skip connections with learned gain

$\pm \alpha_i$1

Pooling depends on the task. Last pooling is used for tasks explicitly testing long-range memory, namely sMNIST, copy-first, and parity, whereas mean pooling is used for LRA tasks and audio tasks (Brandoit et al., 12 May 2026). Standard sinusoidal positional encodings are concatenated to inputs and linearly projected.

Training uses AdamW with $\pm \alpha_i$2, $\pm \alpha_i$3, $\pm \alpha_i$4, and weight decay $\pm \alpha_i$5. The learning rate is $\pm \alpha_i$6 with $\pm \alpha_i$7 linear warmup and cosine decay to $\pm \alpha_i$8. Gradient norm is clipped to $\pm \alpha_i$9, batch size is $\alpha \in \mathbb{R}^d_{>0}$0, no task-specific hyperparameter tuning is performed, and each configuration is run with $\alpha \in \mathbb{R}^d_{>0}$1 seeds with mean $\alpha \in \mathbb{R}^d_{>0}$2 min–max reported (Brandoit et al., 12 May 2026).

Because the Heaviside and sign operators are non-differentiable, training uses surrogate gradients: $\alpha \in \mathbb{R}^d_{>0}$3 so the forward dynamics remain exactly discrete while the backward pass is smoothed (Brandoit et al., 12 May 2026). Computationally, CMRU, $\alpha \in \mathbb{R}^d_{>0}$4CMRU, BMRU, LRU, and minGRU are all affine in $\alpha \in \mathbb{R}^d_{>0}$5, which means they support associative scan over sequence length and can therefore be parallelized on GPUs and TPUs (Brandoit et al., 12 May 2026). This inherits the scan-compatibility emphasized in the earlier MRU formulation, where recurrences of the form $\alpha \in \mathbb{R}^d_{>0}$6 admit a Blelloch-style scan implementation (Geeter et al., 14 Jan 2026).

For models intended to be deployed with $\alpha \in \mathbb{R}^d_{>0}$7 on existing Schmitt-trigger BMRU hardware, the paper proposes $\alpha \in \mathbb{R}^d_{>0}$8-annealing: start at $\alpha \in \mathbb{R}^d_{>0}$9, keep it for $\pm \alpha$00 of steps, linearly decay to $\pm \alpha$01 over the next $\pm \alpha$02, and hold at $\pm \alpha$03 for the final $\pm \alpha$04; only checkpoints after $\pm \alpha$05 are saved and evaluated (Brandoit et al., 12 May 2026). This is presented as a way to preserve the deployment dynamics of BMRU while benefiting from the gradient flow of CMRU during training.

6. Empirical performance and benchmark profile

On sequential MNIST with last pooling, $\pm \alpha$06, and one layer, the effect of $\pm \alpha$07 is direct. For CMRU, $\pm \alpha$08, $\pm \alpha$09 yields $\pm \alpha$10 accuracy, high variance, whereas $\pm \alpha$11 yields $\pm \alpha$12 accuracy, low variance. For $\pm \alpha$13CMRU, $\pm \alpha$14, $\pm \alpha$15 yields $\pm \alpha$16 accuracy, and $\pm \alpha$17 yields $\pm \alpha$18 accuracy (Brandoit et al., 12 May 2026). The paper reports that increasing $\pm \alpha$19 monotonically improves both mean accuracy and cross-seed stability, with the largest gains on the harder sMNIST relative to pMNIST.

At small state dimension, the reported scaling behavior is notable. With $\pm \alpha$20 and a single layer on sMNIST, at $\pm \alpha$21, CMRU reaches $\pm \alpha$22 and $\pm \alpha$23CMRU $\pm \alpha$24, while LRU and minGRU are described as significantly worse and more variable. At $\pm \alpha$25, all methods are $\pm \alpha$26–$\pm \alpha$27, but the CMRU variants still show lower variance (Brandoit et al., 12 May 2026). This indicates that the cumulative memory mechanism is especially advantageous in the small-model regime emphasized by ultra-low-power applications.

The strongest specialization appears on copy-first tasks. In the discrete 15-class copy-first, noise-free setting with $\pm \alpha$28 and last pooling, CMRU ($\pm \alpha$29) achieves $\pm \alpha$30 accuracy up to sequences of length $\pm \alpha$31, while LRU/minGRU collapse to near random for large $\pm \alpha$32 (Brandoit et al., 12 May 2026). In the continuous, noise-free version, CMRU with $\pm \alpha$33 achieves MAE near the theoretical quantization bound $\pm \alpha$34, and $\pm \alpha$35CMRU achieves essentially zero MAE. In the continuous with heavy noise variant, with $\pm \alpha$36 and later inputs i.i.d. from the same distribution, $\pm \alpha$37CMRU with $\pm \alpha$38 remains robust across all $\pm \alpha$39 and noise with MAE $\pm \alpha$40, whereas LRU and minGRU degrade rapidly with $\pm \alpha$41 and noise and at $\pm \alpha$42 essentially fail, with MAE near the $\pm \alpha$43 baseline (Brandoit et al., 12 May 2026).

On larger-scale noisy copy-first with $\pm \alpha$44, $\pm \alpha$45, $\pm \alpha$46, the paper reports that CMRU and $\pm \alpha$47CMRU solve the task robustly for all seeds, while LRU and minGRU still fail entirely (Brandoit et al., 12 May 2026). The interpretation given is architectural rather than merely parametric: the advantage comes from persistent-memory structure rather than only from small-model efficiency.

On the Long Range Arena-style tasks, the performance profile is mixed. On IMDb sentiment, LRU has the best mean at $\pm \alpha$48, minGRU is slightly below, and CMRU/$\pm \alpha$49CMRU are $\pm \alpha$50–$\pm \alpha$51, within $\pm \alpha$52. On sCIFAR10, LRU and minGRU are best at $\pm \alpha$53–$\pm \alpha$54, especially at larger depth, while CMRU and $\pm \alpha$55CMRU are slightly behind but close, with the gap shrinking with depth (Brandoit et al., 12 May 2026). By contrast, on ListOps, at $\pm \alpha$56, CMRU and $\pm \alpha$57CMRU reach $\pm \alpha$58 accuracy, LRU is $\pm \alpha$59 and largely fails, and minGRU is intermediate at $\pm \alpha$60–$\pm \alpha$61. On Pathfinder, CMRU reaches $\pm \alpha$62 and $\pm \alpha$63CMRU $\pm \alpha$64, while LRU and minGRU are $\pm \alpha$65, which is chance at this small capacity of $\pm \alpha$66 (Brandoit et al., 12 May 2026). The paper interprets these results as evidence that persistent-memory cells have a clear advantage on discrete and long-range compositional tasks.

For character-level language modeling on Shakespeare, the reported cross-entropy losses are: $\pm \alpha$67CMRU: 1.443, CMRU: 1.455, LRU: 1.504, minGRU: 1.453, Hybrid minGRU+CMRU: 1.442, and Hybrid minGRU+$\pm \alpha$68CMRU: 1.441 (Brandoit et al., 12 May 2026). This supports the claim that CMRU and $\pm \alpha$69CMRU are competitive on smooth statistical tasks and that persistent + fading memory hybrids are complementary.

In the audio keyword spotting regime with very small state dimensions, the differences are small. On KWS Digits (11 classes), all models are $\pm \alpha$70–$\pm \alpha$71 accuracy even at $\pm \alpha$72; CMRU at $\pm \alpha$73 reaches $\pm \alpha$74, while LRU at $\pm \alpha$75 reaches $\pm \alpha$76. On KWS All (35 classes), at $\pm \alpha$77, CMRU is $\pm \alpha$78 and LRU/minGRU are $\pm \alpha$79–$\pm \alpha$80; at $\pm \alpha$81, CMRU is $\pm \alpha$82, LRU $\pm \alpha$83, and minGRU $\pm \alpha$84 (Brandoit et al., 12 May 2026). The important point emphasized is that a single-layer CMRU with $\pm \alpha$85 still fits existing hardware assumptions while maintaining competitive accuracy.

7. Analog implementation, model landscape, and related cumulative-memory frameworks

CMRU is explicitly designed for always-on sensors, implants, and related settings requiring sub-$\pm \alpha$86W operation for years (Brandoit et al., 12 May 2026). In BMRU, quantized bistability and hysteresis map directly to Schmitt trigger circuits, where states are stored as discrete logic levels and do not accumulate analog errors (Brandoit et al., 12 May 2026, Geeter et al., 14 Jan 2026). CMRU with $\pm \alpha$87 extends this hardware picture to a hysteretic charge accumulator: cumulative updates correspond to adding discrete charges or currents to an integrator, and the paper proposes a native analog circuit based on a cascade of BMRU cells interleaved with Differential Pair Integrators, with staircase states at multiples of $\pm \alpha$88 (Brandoit et al., 12 May 2026). The reported Cadence simulations show event-driven increments or decrements of output by $\pm \alpha$89 every $\pm \alpha$90 ms in update mode, perfect retain in retain mode, and instantaneous power in the deep nW regime, including $\pm \alpha$91 nW in the example.

Within the broader model landscape, the paper contrasts CMRU and $\pm \alpha$92CMRU with LRUs, SSMs, and minGRU. LRUs and SSMs are strong on tasks such as sCIFAR and IMDb, but are limited to fading memory and typically rely on digital implementations or capacitor-based analog realizations (Brandoit et al., 12 May 2026). minGRU is fully parallelizable and strong on sCIFAR, IMDb, and language modeling, but its gates remain in $\pm \alpha$93 and therefore do not provide true retain or overwrite modes (Brandoit et al., 12 May 2026). CMRU and $\pm \alpha$94CMRU, by contrast, supply explicit persistent memory via multistability and exact retain mode, and at $\pm \alpha$95 they provide perfect gradient flow through time while remaining affine in the previous state (Brandoit et al., 12 May 2026).

A useful clarification concerns the relation between CMRU and HiPPO. HiPPO-LegS defines a memory state as the coefficient vector of the optimal projection of the entire history onto a polynomial basis under the measure

$\pm \alpha$96

leading to the ODE

$\pm \alpha$97

and discrete recurrence

$\pm \alpha$98

for full-history cumulative memory (Gu et al., 2020). This is a different construction from the quantized, hysteretic CMRU introduced in 2026, but the juxtaposition is informative: HiPPO provides a cumulative-memory mechanism through optimal polynomial projection, whereas CMRU provides cumulative memory through event-driven discrete accumulation with hysteresis. This suggests two distinct technical meanings of cumulative recurrent memory: one based on analytically derived compression of all history, the other on quantized multistable state transitions.

The main contributions claimed for CMRU are therefore fourfold: diagnosis of gradient blocking in BMRU as the primary training bottleneck; introduction of cumulative updates that restore gradient flow while preserving persistent memory; a selective SSM representation in which $\pm \alpha$99 directly controls eigenvalues; and an explicit path toward ultra-low-power analog RNNs that combine quantized, noise-resilient persistent memory with gradient-friendly training (Brandoit et al., 12 May 2026). A plausible implication is that CMRU occupies a specific niche rather than replacing fading-memory models wholesale: it is strongest where discrete, long-range retention and hardware-constrained efficiency are both first-order design criteria.