Bistable Memory Recurrent Units (BMRUs)

• Bistable Memory Recurrent Units (BMRUs) are neural cells that quantize their hidden state to two stable levels using hysteresis for persistent memory.
• They use a hold-or-switch mechanism with adaptive thresholds to update states, enabling exact retention and scan-compatible affine recurrence.
• Cumulative variants (CMRU and αCMRU) overcome gradient-blocking issues, support stable training, and facilitate efficient analog hardware implementations.

Bistable Memory Recurrent Units (BMRUs) are recurrent neural network cells whose hidden state is quantized and hysteretic, so that each state coordinate occupies one of two stable levels and remains there until an input-dependent threshold is crossed. In the sequence-modeling literature, they occupy a specific niche: unlike monostable state-space models, they provide persistent memory through multistability; unlike conventional nonlinear RNNs, their recurrence is affine in the previous state and is therefore compatible with parallel scan algorithms. The architecture was introduced within the broader family of memory recurrent units (MRUs) as a scan-compatible realization of hysteretic dynamics, and it was subsequently reformulated for current-mode analog hardware and extended by cumulative variants that address training instabilities while preserving persistent-memory behavior (Geeter et al., 14 Jan 2026, Brandoit et al., 12 May 2026, Fyon et al., 12 May 2026).

1. Formal definition and state geometry

In its original vector form, the BMRU receives an input $x_t \in \mathbb{R}^m$ and updates a hidden state $h_t \in \mathbb{R}^d$ according to

$\begin{aligned} \hat{h}_t &= W_x x_t + b_x, \ \beta_t &= \left|W_\beta x_t + b_\beta\right|, \ z_t &= \HOp\left(|\hat{h}_t| - \beta_t\right), \ h_t &= z_t \odot \SOp(\hat{h}_t)\odot \alpha + (1 - z_t)\odot h_{t-1}. \end{aligned}$

Here $\HOp$ is the elementwise Heaviside step function, $\SOp$ is the elementwise sign function, and $\alpha \in \mathbb{R}^d$ is a learned scale vector. Each coordinate of the hidden state is constrained to the set $\{+\alpha_i,-\alpha_i\}$, so the cell implements a discrete two-well memory per dimension (Brandoit et al., 12 May 2026).

The update has two coordinatewise modes. In retain mode, $z_{t,i}=0$ and $h_{t,i}=h_{t-1,i}$. In update mode, $z_{t,i}=1$ and $h_t \in \mathbb{R}^d$0. The threshold $h_t \in \mathbb{R}^d$1 is adaptive and input dependent, so switching is not governed by a fixed global threshold but by a learned threshold field over the input sequence. In the original formulation, every coordinate has exactly two stable attractors $h_t \in \mathbb{R}^d$2, and the global state space has $h_t \in \mathbb{R}^d$3 fixed points; because updates always snap to $h_t \in \mathbb{R}^d$4, the reachable states lie on the discrete lattice $h_t \in \mathbb{R}^d$5 (Brandoit et al., 12 May 2026).

Within the MRU framework, BMRUs arise from a piecewise approximation of a bistable hysteresis relation. The resulting scalar rule is a hold-or-switch dynamic: if the input magnitude is below threshold, the previous state is held exactly; if the input magnitude exceeds threshold, the state is forced to one of two stable levels determined by the sign of the candidate drive. The multidimensional BMRU is the learnable realization of that principle (Geeter et al., 14 Jan 2026).

2. Hysteresis, persistent memory, and scan-compatible recurrence

The defining dynamical feature of a BMRU is hysteresis. A coordinate retains its state when the candidate magnitude remains within the threshold band and flips only when a sufficiently strong input crosses the relevant threshold. This yields persistent memory in the strict sense used by the cited work: once a state bit is set, it remains indefinitely until explicitly switched by a later input. There is no fading in the original formulation; memory persistence is controlled by thresholded switching rather than exponential decay (Brandoit et al., 12 May 2026).

This hysteretic mechanism also explains the model’s noise behavior. Because the state is quantized and bistable, small perturbations do not alter the stored state unless they push the system across a threshold window. In the analog-hardware interpretation, this is the same qualitative property exploited by Schmitt triggers: two stable output levels separated by a hysteresis band suppress spurious transitions caused by small current or voltage fluctuations (Fyon et al., 8 May 2026).

Despite its nonlinearity, the recurrence is affine in the previous state: $h_t \in \mathbb{R}^d$6 For a fixed input sequence, the coefficients $h_t \in \mathbb{R}^d$7 and the additive term are determined from the inputs alone, so the temporal evolution is a linear scan. In the generalized state-space view,

$h_t \in \mathbb{R}^d$8

for the cumulative variants, and $h_t \in \mathbb{R}^d$9 for the original BMRU. This is why BMRUs belong to the class of parallelizable RNNs: they can be evaluated with associative scan algorithms in the same general sense as LRUs and diagonal SSMs, while retaining multistable rather than purely fading-memory dynamics (Brandoit et al., 12 May 2026).

The contrast with monostable sequence models is central. The MRU formulation was motivated by the observation that strictly linear SSMs are monostable and therefore cannot realize persistent memory through stable internal states. BMRUs add multistability to scan-compatible recurrent computation, replacing transient-only storage by attractor-based storage (Geeter et al., 14 Jan 2026).

3. Optimization pathology and cumulative generalizations

The original BMRU’s principal training limitation is gradient blocking during state updates. If $\begin{aligned} \hat{h}_t &= W_x x_t + b_x, \ \beta_t &= \left|W_\beta x_t + b_\beta\right|, \ z_t &= \HOp\left(|\hat{h}_t| - \beta_t\right), \ h_t &= z_t \odot \SOp(\hat{h}_t)\odot \alpha + (1 - z_t)\odot h_{t-1}. \end{aligned}$0 is treated as a fixed gate during backpropagation, then

$\begin{aligned} \hat{h}_t &= W_x x_t + b_x, \ \beta_t &= \left|W_\beta x_t + b_\beta\right|, \ z_t &= \HOp\left(|\hat{h}_t| - \beta_t\right), \ h_t &= z_t \odot \SOp(\hat{h}_t)\odot \alpha + (1 - z_t)\odot h_{t-1}. \end{aligned}$1

Hence gradients propagate perfectly through retain steps, but are exactly zero through update steps. Over a long sequence,

$\begin{aligned} \hat{h}_t &= W_x x_t + b_x, \ \beta_t &= \left|W_\beta x_t + b_\beta\right|, \ z_t &= \HOp\left(|\hat{h}_t| - \beta_t\right), \ h_t &= z_t \odot \SOp(\hat{h}_t)\odot \alpha + (1 - z_t)\odot h_{t-1}. \end{aligned}$2

elementwise, so any open gate on a coordinate severs the backward path for that coordinate. The cited analysis identifies this overwrite-induced structural Jacobian, rather than only the non-differentiability of Heaviside and sign, as the key reason that the original formulation is hard to train under standard initialization (Brandoit et al., 12 May 2026).

To address this, the cumulative memory recurrent unit (CMRU) introduces a cumulative update term: $\begin{aligned} \hat{h}_t &= W_x x_t + b_x, \ \beta_t &= \left|W_\beta x_t + b_\beta\right|, \ z_t &= \HOp\left(|\hat{h}_t| - \beta_t\right), \ h_t &= z_t \odot \SOp(\hat{h}_t)\odot \alpha + (1 - z_t)\odot h_{t-1}. \end{aligned}$3 with $\begin{aligned} \hat{h}_t &= W_x x_t + b_x, \ \beta_t &= \left|W_\beta x_t + b_\beta\right|, \ z_t &= \HOp\left(|\hat{h}_t| - \beta_t\right), \ h_t &= z_t \odot \SOp(\hat{h}_t)\odot \alpha + (1 - z_t)\odot h_{t-1}. \end{aligned}$4. The relaxed $\begin{aligned} \hat{h}_t &= W_x x_t + b_x, \ \beta_t &= \left|W_\beta x_t + b_\beta\right|, \ z_t &= \HOp\left(|\hat{h}_t| - \beta_t\right), \ h_t &= z_t \odot \SOp(\hat{h}_t)\odot \alpha + (1 - z_t)\odot h_{t-1}. \end{aligned}$5CMRU replaces the fixed amplitude $\begin{aligned} \hat{h}_t &= W_x x_t + b_x, \ \beta_t &= \left|W_\beta x_t + b_\beta\right|, \ z_t &= \HOp\left(|\hat{h}_t| - \beta_t\right), \ h_t &= z_t \odot \SOp(\hat{h}_t)\odot \alpha + (1 - z_t)\odot h_{t-1}. \end{aligned}$6 by an input-dependent amplitude $\begin{aligned} \hat{h}_t &= W_x x_t + b_x, \ \beta_t &= \left|W_\beta x_t + b_\beta\right|, \ z_t &= \HOp\left(|\hat{h}_t| - \beta_t\right), \ h_t &= z_t \odot \SOp(\hat{h}_t)\odot \alpha + (1 - z_t)\odot h_{t-1}. \end{aligned}$7: $\begin{aligned} \hat{h}_t &= W_x x_t + b_x, \ \beta_t &= \left|W_\beta x_t + b_\beta\right|, \ z_t &= \HOp\left(|\hat{h}_t| - \beta_t\right), \ h_t &= z_t \odot \SOp(\hat{h}_t)\odot \alpha + (1 - z_t)\odot h_{t-1}. \end{aligned}$8 Under this formulation,

$\begin{aligned} \hat{h}_t &= W_x x_t + b_x, \ \beta_t &= \left|W_\beta x_t + b_\beta\right|, \ z_t &= \HOp\left(|\hat{h}_t| - \beta_t\right), \ h_t &= z_t \odot \SOp(\hat{h}_t)\odot \alpha + (1 - z_t)\odot h_{t-1}. \end{aligned}$9

so update steps no longer annihilate gradients unless $\HOp$0. For $\HOp$1, the Jacobian is exactly 1 in both retain and update modes, creating skip-connections through time (Brandoit et al., 12 May 2026).

The parameter $\HOp$2 determines the memory regime. At $\HOp$3, CMRU reduces exactly to the original BMRU. For $\HOp$4, old information decays only when update events occur, so forgetting depends on update count rather than wall-clock time. At $\HOp$5, the unit becomes a selective integrator,

$\HOp$6

which preserves exact gradient flow and accumulates signed increments (Brandoit et al., 12 May 2026).

The empirical effect is large. On sMNIST with state dimension $\HOp$7, CMRU with $\HOp$8 achieves about $\HOp$9 accuracy, CMRU with $\SOp$0 about $\SOp$1, and $\SOp$2CMRU with $\SOp$3 about $\SOp$4. The same study reports that cross-seed variability collapses as $\SOp$5, and that $\SOp$6 is consistently best for both performance and initialization robustness across multiple tasks. The same paper also describes $\SOp$7-annealing—training with $\SOp$8 and decaying toward $\SOp$9—as a software-training–hardware-deployment bridge, because it yields a gradient-friendly optimization phase while allowing final dynamics compatible with the original bistable hardware (Brandoit et al., 12 May 2026).

4. Analog realization and first-quadrant reformulation

BMRUs were designed explicitly for hardware–software co-design in ultra-low-power analog environments. Their quantized, bistable states map directly to Schmitt-trigger behavior, and this correspondence has been developed into current-mode CMOS implementations in which the recurrent unit’s transfer function is realized physically rather than only approximated numerically (Fyon et al., 8 May 2026, Fyon et al., 12 May 2026).

A key hardware development is a 9-transistor current-mode Schmitt-trigger cell built from two interdependent Heaviside-like elements. The first block implements thresholding at $\alpha \in \mathbb{R}^d$0; the second injects a feedback current $\alpha \in \mathbb{R}^d$1, lowering the effective falling-edge threshold and thereby creating hysteresis. The resulting switching points are

$\alpha \in \mathbb{R}^d$2

with a high output level approximately $\alpha \in \mathbb{R}^d$3. The cell stores a quantized state $\alpha \in \mathbb{R}^d$4, is static rather than capacitor-based, and was simulated in a 180 nm CMOS process with nanowatt-range power consumption and robust hysteresis under mismatch and temperature variation (Fyon et al., 8 May 2026).

To align the mathematical cell with unipolar current-mode hardware, a first-quadrant BMRU (FQ BMRU) was introduced. Its update is

$\alpha \in \mathbb{R}^d$5

so the state becomes unipolar and bistable, $\alpha \in \mathbb{R}^d$6, with a hold region between $\alpha \in \mathbb{R}^d$7 and $\alpha \in \mathbb{R}^d$8. The cited work proves that bipolar outputs $\alpha \in \mathbb{R}^d$9 and unipolar outputs $\{+\alpha_i,-\alpha_i\}$0 are equivalent up to reparameterization of subsequent linear layers, and that fixed-threshold FQ BMRUs preceded by a ReLU MLP can approximate networks based on the original BMRU arbitrarily closely on compact input sets (Fyon et al., 12 May 2026).

The hardware mapping is direct:

Mathematical quantity	Circuit quantity	Role
$\{+\alpha_i,-\alpha_i\}$1	$\{+\alpha_i,-\alpha_i\}$2	high switching threshold
$\{+\alpha_i,-\alpha_i\}$3	$\{+\alpha_i,-\alpha_i\}$4	low switching point
$\{+\alpha_i,-\alpha_i\}$5	$\{+\alpha_i,-\alpha_i\}$6	high output current

This one-to-one mapping supports a software model that serves as a high-fidelity simulator of the physical hardware. In transistor-level simulations of a two-layer FQ-BMRU keyword-spotting system in 180 nm CMOS, the candidate signal at the second recurrent layer exhibits mean absolute error of approximately $\{+\alpha_i,-\alpha_i\}$7 relative to software predictions, while the thresholded state error is approximately $\{+\alpha_i,-\alpha_i\}$8, corresponding to about $\{+\alpha_i,-\alpha_i\}$9 noise suppression at the cell boundary. The same study reports that the power cost of the BMRU cells scales linearly with state dimension whereas the feedforward backbone scales quadratically, so recurrence is added at linear marginal cost relative to the feedforward analog network (Fyon et al., 12 May 2026).

At the system level, the Schmitt-trigger-based BMRU line has been associated with keyword spotting at power budgets around $z_{t,i}=0$0, and the FQ-BMRU transistor-level implementation reports sub-microwatt inference at the RNN core (Brandoit et al., 12 May 2026, Fyon et al., 12 May 2026).

5. Empirical behavior and task profile

The empirical profile of BMRUs and their cumulative descendants is strongly task dependent. The clearest advantages appear on tasks that require discrete long-range retention rather than smooth fading-memory integration. On the discrete 15-class copy-first task with single-layer state dimension $z_{t,i}=0$1, CMRU with $z_{t,i}=0$2 and $z_{t,i}=0$3CMRU reach $z_{t,i}=0$4 accuracy up to sequence length $z_{t,i}=0$5, whereas LRU and minGRU degrade heavily with length and many seeds fail. On the continuous noisy copy-first regression task, CMRU with $z_{t,i}=0$6 reaches MAE near its theoretical quantization limit $z_{t,i}=0$7 up to $z_{t,i}=0$8, $z_{t,i}=0$9CMRU reaches near-zero MAE, and LRU/minGRU degrade toward random baselines as $h_{t,i}=h_{t-1,i}$0 grows. At larger scale $h_{t,i}=h_{t-1,i}$1, CMRU and $h_{t,i}=h_{t-1,i}$2CMRU still solve the noisy copy-first task for all seeds, whereas LRU and minGRU fail (Brandoit et al., 12 May 2026).

The original BMRU paper already emphasized this persistent-memory regime. On copy-first-input, BMRU retains information nearly independently of sequence length and generalizes to much longer sequences than those used for training, while LRU degrades as the horizon grows. Hybrid BMRU+LRU stacks were reported to combine the retention properties of BMRU with the transient-computation strengths of LRU on permuted sequential MNIST with long trailing blank intervals (Geeter et al., 14 Jan 2026).

On Long Range Arena-style benchmarks at small model sizes, the cumulative formulations exhibit a differentiated performance profile. On IMDb, LRU and minGRU slightly outperform CMRU and $h_{t,i}=h_{t-1,i}$3CMRU, with differences reported as $h_{t,i}=h_{t-1,i}$4. On sCIFAR-10, LRU and minGRU have an advantage at shallow depth, although CMRU and $h_{t,i}=h_{t-1,i}$5CMRU narrow the gap with deeper stacks. On ListOps, at depth $h_{t,i}=h_{t-1,i}$6, CMRU and $h_{t,i}=h_{t-1,i}$7CMRU outperform LRU by a wide margin, around $h_{t,i}=h_{t-1,i}$8 versus around $h_{t,i}=h_{t-1,i}$9, and match or slightly exceed minGRU. On Pathfinder at $z_{t,i}=1$0, CMRU and $z_{t,i}=1$1CMRU reach about $z_{t,i}=1$2 accuracy, whereas LRU and minGRU remain around $z_{t,i}=1$3, i.e. chance (Brandoit et al., 12 May 2026).

On Shakespeare character-level language modeling, the same study reports single-cell baselines of approximately $z_{t,i}=1$4 cross-entropy for LRU, $z_{t,i}=1$5 for minGRU, $z_{t,i}=1$6 for CMRU, and $z_{t,i}=1$7 for $z_{t,i}=1$8CMRU. Hybrid minGRU+CMRU and minGRU+$z_{t,i}=1$9CMRU models slightly outperform either component alone, supporting the conclusion that persistent and fading memories are complementary rather than mutually exclusive (Brandoit et al., 12 May 2026).

In keyword spotting, BMRU-style architectures were designed with ultra-low-capacity analog deployment in mind. The analog Schmitt-trigger work summarizes an external BMRU-based keyword-spotting experiment with accuracy greater than $h_t \in \mathbb{R}^d$00 at about $h_t \in \mathbb{R}^d$01, roughly three orders of magnitude below common TinyML implementations with similar accuracy. In the cumulative-study benchmarks on Google Speech Commands, single-layer CMRU with $h_t \in \mathbb{R}^d$02 reaches about $h_t \in \mathbb{R}^d$03 on digits, about $h_t \in \mathbb{R}^d$04 on the full 35-class task, and about $h_t \in \mathbb{R}^d$05 on Yes-vs-Others; LRU and minGRU are often $h_t \in \mathbb{R}^d$06 higher, but the differences are described as small and within cross-seed ranges (Fyon et al., 8 May 2026, Brandoit et al., 12 May 2026).

These results support a narrow but consistent interpretation. BMRU-type cells are strongest where the latent computation benefits from persistent discrete states, exact retention across long inactive spans, or analog noise resilience. They are weaker, or only competitive, on tasks dominated by smooth continuous temporal statistics.

6. Related bistable models, theoretical lineage, and conceptual boundaries

BMRUs belong to a broader lineage of bistable recurrent models, but they are not interchangeable with all such models. A closely related biological predecessor is the bistable recurrent cell (BRC), whose scalar dynamics

$h_t \in \mathbb{R}^d$07

are bistable when $h_t \in \mathbb{R}^d$08, with the transition derived as a supercritical pitchfork bifurcation. The BRC and its neuromodulated variant nBRC provide cell-level long-lasting memory and outperform GRU and LSTM on several long-memory benchmarks, especially when every training example requires long-range retention. However, their recurrence is not presented as scan-compatible and their internal memory mechanism is continuous self-feedback rather than discrete hysteretic switching (Vecoven et al., 2020).

An earlier attractor-network antecedent is the recurrent bistable gradient network (RBGN), in which each neuron follows continuous-time gradient descent in a double-well potential,

$h_t \in \mathbb{R}^d$09

That literature treats bistability as a local memory mechanism and studies learning rules for storing patterns as attractors. The cited analysis reports that Hebbian learning leads to unwanted behavior on overlapping patterns, whereas CD-1 yields a more promising recurrent network and supports MNIST pattern completion with a mean error rate of about $h_t \in \mathbb{R}^d$10 after reconstruction (Fischer et al., 2016).

Another theoretical precursor studies random neural networks with strong self-interaction,

$h_t \in \mathbb{R}^d$11

showing that units become bistable for $h_t \in \mathbb{R}^d$12 and that the network exhibits distinct regimes including transient but long-lived chaotic activity together with multiple stable fixed points. That work provides a general dynamical-systems template for understanding self-coupled bistable units, but not the scan-compatible algebraic update that characterizes BMRUs (Stern et al., 2014).

Within the contemporary sequence-model landscape, the distinction is sharper. Classical gated RNNs such as LSTM and GRU use continuous gates in $h_t \in \mathbb{R}^d$13, so retain and update are approximate rather than exact; parallelizable gated RNNs and diagonal SSMs provide fading memory through eigenvalues inside the unit circle; BMRU, CMRU, and $h_t \in \mathbb{R}^d$14CMRU instead supply a multistable persistent-memory primitive that remains parallelizable because the recurrence is affine in the previous state (Brandoit et al., 12 May 2026).

A common misconception is therefore to treat BMRUs as merely binary or quantized RNNs. The defining property is not quantization alone but hysteretic multistability with explicit retain regions. Conversely, not every bistable recurrent architecture is a BMRU: BRCs, RBGNs, and self-coupled random rate models all realize bistability, but BMRUs are distinguished by hold-or-switch algebraic updates, exact scan compatibility, and direct mapping to Schmitt-trigger-like analog primitives (Geeter et al., 14 Jan 2026, Fyon et al., 8 May 2026).

The broader theoretical implications remain active. The cumulative-study paper notes that BMRU-family cells can also realize regimes unavailable to standard fading-memory cells: with $h_t \in \mathbb{R}^d$15, CMRU and $h_t \in \mathbb{R}^d$16CMRU solve parity with perfect length generalization, while LRU, minGRU, and CMRU with $h_t \in \mathbb{R}^d$17 remain at about $h_t \in \mathbb{R}^d$18. In that discussion, Grazzi et al. (2025) are cited for the claim that state-tracking tasks like parity require negative eigenvalues in linear RNNs, and Merril et al. (2024) are cited for complexity-theoretic arguments about the limits of diagonal SSMs and Transformers. This suggests that the BMRU line is best understood not only as an analog-hardware architecture, but also as an explicit attempt to reintroduce multistable state-tracking dynamics into the family of parallelizable recurrent models (Brandoit et al., 12 May 2026).