Expressive Leaky Memory (ELM) Neurons

• Expressive Leaky Memory (ELM) neurons are phenomenological models that generate scalar outputs from rich internal states, combining fast reset dynamics with slow-decaying memory across multiple timescales.
• They employ nonlinear dendritic integration and learnable synaptic traces to efficiently capture both rapid transients and long-range dependencies for sequence processing.
• Achieving high parameter efficiency, ELM architectures demonstrate competitive performance on benchmarks and inspire hardware implementations using nanoscale leaky memcapacitive devices.

Searching arXiv for the cited papers to ground the article and verify bibliographic details. Expressive Leaky Memory (ELM) neurons are phenomenological neuron models in which scalar output activity is generated from rich internal state, multi-timescale leak dynamics, and nonlinear synaptic integration. Across the literature, the core motif appears in several closely related forms: as an age-and-memory-dependent point-process neuron with reset and leaky hidden state in mean-field Hawkes theory; as a recurrent cell with learnable synaptic traces, memory units, and a compact multilayer integrator for long-horizon sequence processing; as a network primitive whose per-neuron complexity, width, and connectivity can be traded under a fixed parameter budget; and as a nanoscale leaky memcapacitive device that physically co-localizes integration, leak, reset, and slow adaptation (Schmutz, 2020, Spieler et al., 2023, Spieler et al., 12 May 2026, Zhang et al., 2023). The unifying idea is that temporal computation is not delegated solely to network recurrence or external state-space structure, but is partly internalized within each neuron through explicitly leaky memory variables operating on multiple timescales.

1. Definition and formal lineage

A foundational formulation appears in the mean-field model of interacting point processes proposed by Schmutz, where each neuron carries two state variables: age $A_t^i\in\mathbb{R}_+$, the time elapsed since its last spike, and leaky memory $M_t^i\in\mathbb{R}^d$, a $d$-dimensional memory that accumulates past activity but decays between spikes. Spiking is governed by the stochastic intensity

$$\lambda^i(t)=f\bigl(A^i_{t-},\,M^i_{t-},\,X_t\bigr),$$

with bounded, Lipschitz-continuous $f$, and between spikes the state evolves according to

$$\frac{d}{dt}A^i_t = 1,\qquad \frac{d}{dt}M^i_t = b\bigl(M^i_t\bigr),$$

while at a spike the neuron resets by

$$A^i_t \mapsto 0,\qquad M^i_t\mapsto M^i_{t-}+\Gamma\bigl(M^i_{t-}\bigr).$$

Crucially, $\Gamma$ need not reset memory to a fixed value; it may depend on the current $M$ and thus encode cumulative adaptation or plasticity (Schmutz, 2020).

This formulation generalizes age-dependent Hawkes processes. Age-only renewal models retain only $a(t)$, so the intensity is $M_t^i\in\mathbb{R}^d$0 and the process forgets everything but the last interspike interval; memoryless Poisson models have no history dependence at all. By combining age and multidimensional leaky memory, the ELM motif can simultaneously represent instantaneous refractoriness and slower processes such as adaptation, facilitation, depression, and synaptic filtering. The examples given in the source material make this explicit: Erlang-kernel self-interaction yields a $M_t^i\in\mathbb{R}^d$1-dimensional adaptation cascade, while a $M_t^i\in\mathbb{R}^d$2 Tsodyks–Markram construction uses facilitation and depression variables with synaptic efficacy $M_t^i\in\mathbb{R}^d$3 (Schmutz, 2020).

A plausible implication is that “ELM” is best understood not as a single fixed equation set but as a design family. In that family, a neuron is expressive to the extent that it combines resettable fast state, slowly decaying internal state, and nonlinear coupling between them.

2. Single-neuron dynamics in the phenomenological ELM model

The 2023 ELM formulation for sequence modeling makes this family concrete as a recurrent cell. At each time $M_t^i\in\mathbb{R}^d$4, the neuron receives an input spike vector $M_t^i\in\mathbb{R}^d$5 and maintains a synaptic current trace $M_t^i\in\mathbb{R}^d$6, a memory-state vector $M_t^i\in\mathbb{R}^d$7, and a scalar scaling factor $M_t^i\in\mathbb{R}^d$8. The per-channel synaptic and per-unit memory timescales, $M_t^i\in\mathbb{R}^d$9 and $d$0, are learnable, with decay factors

$d$1

The update is organized into four stages (Spieler et al., 2023): $d$2

$d$3

$d$4

$d$5

The nonlinear dendritic integration step uses a small MLP. Concretely, with one hidden layer of size $d$6 and ReLU,

$d$7

This architecture couples filtered synaptic input with decayed internal state before the memory update is committed. The update can also be written as

$d$8

but the explicit per-unit gating is set by the learnable $d$9 rather than by a conventional learned gate vector (Spieler et al., 2023).

The source material emphasizes two consequences. First, because each memory unit $$\lambda^i(t)=f\bigl(A^i_{t-},\,M^i_{t-},\,X_t\bigr),$$0 has its own $$\lambda^i(t)=f\bigl(A^i_{t-},\,M^i_{t-},\,X_t\bigr),$$1, a single ELM neuron can track both fast and slow processes. Second, the synaptic side likewise uses per-channel $$\lambda^i(t)=f\bigl(A^i_{t-},\,M^i_{t-},\,X_t\bigr),$$2 to model input filtering. In this sense, temporal depth is distributed throughout the cell rather than concentrated in a single recurrent hidden state.

3. Architectural variants, parameter efficiency, and empirical performance

The main architectural hyperparameters are the input dimension $$\lambda^i(t)=f\bigl(A^i_{t-},\,M^i_{t-},\,X_t\bigr),$$3, the number of memory units $$\lambda^i(t)=f\bigl(A^i_{t-},\,M^i_{t-},\,X_t\bigr),$$4, and the MLP width. In the reported implementations, $$\lambda^i(t)=f\bigl(A^i_{t-},\,M^i_{t-},\,X_t\bigr),$$5 equals the number of input channels—1278 for NeuronIO, 700 cochlea channels for SHD, and 8–169 for LRA—while $$\lambda^i(t)=f\bigl(A^i_{t-},\,M^i_{t-},\,X_t\bigr),$$6 is task-dependent: 10–20 memory units are sufficient to fit NeuronIO, 150 are used on LRA, and up to 250 can be used. The standard ELM uses an MLP of width $$\lambda^i(t)=f\bigl(A^i_{t-},\,M^i_{t-},\,X_t\bigr),$$7 with one hidden layer; Branch-ELM inserts an intermediate branch grouping of the $$\lambda^i(t)=f\bigl(A^i_{t-},\,M^i_{t-},\,X_t\bigr),$$8 inputs into $$\lambda^i(t)=f\bigl(A^i_{t-},\,M^i_{t-},\,X_t\bigr),$$9 branches of size $f$0 before the MLP (Spieler et al., 2023).

This design yields unusually low parameter counts relative to the tasks considered. On NeuronIO, the total trainable parameters are approximately 53 K for ELM and approximately 8 K for Branch-ELM; on LRA with $f$1, the single-node ELM has approximately 100 K parameters. The NeuronIO fitting result is central: prior work found that a TCN with $f$2 M parameters was needed to reach a sufficient spike-prediction AUC of 0.991, whereas ELM with $f$3 achieves $f$4 and RMS-voltage error $f$5 mV with 53 K parameters, and Branch-ELM further reduces to 8 K parameters while crossing the AUC threshold (Spieler et al., 2023).

The reported benchmark results span both neuromorphic and long-range sequence tasks.

Setting	ELM result	Comparator(s)
NeuronIO	AUC≈0.992, RMS-voltage error ≈0.64 mV	TCN with >10 M parameters needed for AUC 0.991
SHD-Adding, 2 ms bins ($f$6)	ELM ∼ 0.82 accuracy	LSTM ∼ 0.50
LRA Image (1 K px)	49.6%	Chrono-LSTM 46.1%, Transformer 42.4%
LRA Pathfinder (1 K)	71.2%	70.8%, 71.4%
LRA Pathfinder-X (16 K)	77.3%	LSTM 70.8%, Transformer/Longformer fail
LRA Text	80.3%	75.4%, 64.3%
LRA Retrieval	84.9%	82.9%, 57.5%

On SHD-Adding, the comparison is ELM (186 K params), Branch-ELM (67 K), LSTM (956 K), and a learned-$f$7 LIF-SNN (51 K), with the summary that across all bin sizes, ELM $f$8 Branch-ELM $f$9 LSTM $$\frac{d}{dt}A^i_t = 1,\qquad \frac{d}{dt}M^i_t = b\bigl(M^i_t\bigr),$$0 SNN. On LRA, the comparison is between a single ELM neuron ($$\frac{d}{dt}A^i_t = 1,\qquad \frac{d}{dt}M^i_t = b\bigl(M^i_t\bigr),$$1, $$\frac{d}{dt}A^i_t = 1,\qquad \frac{d}{dt}M^i_t = b\bigl(M^i_t\bigr),$$2 K params), Chrono-LSTM ($$\frac{d}{dt}A^i_t = 1,\qquad \frac{d}{dt}M^i_t = b\bigl(M^i_t\bigr),$$3 K), Transformer/Longformer ($$\frac{d}{dt}A^i_t = 1,\qquad \frac{d}{dt}M^i_t = b\bigl(M^i_t\bigr),$$4 K), and SOTA S4/Mega ($$\frac{d}{dt}A^i_t = 1,\qquad \frac{d}{dt}M^i_t = b\bigl(M^i_t\bigr),$$5 K). Only deep, purpose-built state-space models outperform ELM, and the source states that a single ELM neuron is the only non–purpose-built model to reliably solve the 16 K horizon Pathfinder-X task (Spieler et al., 2023).

Ablations isolate the mechanisms responsible for these results. On NeuronIO, $$\frac{d}{dt}A^i_t = 1,\qquad \frac{d}{dt}M^i_t = b\bigl(M^i_t\bigr),$$6 memory units causes AUC to fall below 0.99, while $$\frac{d}{dt}A^i_t = 1,\qquad \frac{d}{dt}M^i_t = b\bigl(M^i_t\bigr),$$7 is sufficient; removing the MLP sharply degrades both spike- and voltage-prediction; the memory-timescale range $$\frac{d}{dt}A^i_t = 1,\qquad \frac{d}{dt}M^i_t = b\bigl(M^i_t\bigr),$$8 ms is ideal, and forbidding $$\frac{d}{dt}A^i_t = 1,\qquad \frac{d}{dt}M^i_t = b\bigl(M^i_t\bigr),$$9 ms hurts performance markedly. The scaling factor $$A^i_t \mapsto 0,\qquad M^i_t\mapsto M^i_{t-}+\Gamma\bigl(M^i_{t-}\bigr).$$0 controls the amplitude of memory updates: larger $$A^i_t \mapsto 0,\qquad M^i_t\mapsto M^i_{t-}+\Gamma\bigl(M^i_{t-}\bigr).$$1 helps fit rapid transients but can cause instabilities. Branch-ELM, using approximately 45 branches of approximately 65 synapses each, recovers nearly the full ELM performance at approximately $$A^i_t \mapsto 0,\qquad M^i_t\mapsto M^i_{t-}+\Gamma\bigl(M^i_{t-}\bigr).$$2 parameter reduction. On LRA Path-X, varying $$A^i_t \mapsto 0,\qquad M^i_t\mapsto M^i_{t-}+\Gamma\bigl(M^i_{t-}\bigr).$$3 gives a saturating curve: $$A^i_t \mapsto 0,\qquad M^i_t\mapsto M^i_{t-}+\Gamma\bigl(M^i_{t-}\bigr).$$4, $$A^i_t \mapsto 0,\qquad M^i_t\mapsto M^i_{t-}+\Gamma\bigl(M^i_{t-}\bigr).$$5, $$A^i_t \mapsto 0,\qquad M^i_t\mapsto M^i_{t-}+\Gamma\bigl(M^i_{t-}\bigr).$$6, $$A^i_t \mapsto 0,\qquad M^i_t\mapsto M^i_{t-}+\Gamma\bigl(M^i_{t-}\bigr).$$7, $$A^i_t \mapsto 0,\qquad M^i_t\mapsto M^i_{t-}+\Gamma\bigl(M^i_{t-}\bigr).$$8, $$A^i_t \mapsto 0,\qquad M^i_t\mapsto M^i_{t-}+\Gamma\bigl(M^i_{t-}\bigr).$$9, $\Gamma$0 (Spieler et al., 2023).

These results support a specific interpretation: long-horizon performance is not attributed merely to recurrence, but to the conjunction of long memory timescales and nonlinear synaptic integration within the neuron itself.

4. Network formulation and scaling tradeoffs

In the 2026 ELM Network, ELM neurons become explicit architectural knobs. Each neuron $\Gamma$1 at time $\Gamma$2 maintains a vector of $\Gamma$3 leaky memory states $\Gamma$4 and emits a scalar activity $\Gamma$5. Its input $\Gamma$6 is formed by selecting $\Gamma$7 synaptic channels from the concatenated feed-forward and recurrent activities. The per-neuron update is (Spieler et al., 12 May 2026)

$\Gamma$8

$\Gamma$9

$M$0

$M$1

$M$2

with

$M$3

Two architectural axes are then separated explicitly. Per-neuron complexity is

$M$4

while connectivity overhead is

$M$5

For a layer of width $M$6, the trainable-parameter budget satisfies

$M$7

A fixed Bernoulli mask with recurrent fraction $M$8 determines whether each synapse is feed-forward or recurrent, so effective connectivity per neuron is $M$9 (Spieler et al., 12 May 2026).

The paper positions ELM relative to canonical recurrent units. Compared with LSTM/GRU, ELM replaces explicit sigmoid gates and point-wise input combination with a vector-state of $a(t)$0 leak units, a learned nonlinear MLP, multi-timescale filtering, structured dendritic branching, and a high-pass output filter. Compared with LIF, ELM has multiple parallel leak channels and a smooth high-pass nonlinearity rather than a single or few state variables and a hard threshold. The subtraction of the EMA readout $a(t)$1 is identified as a high-pass filter that removes runaway DC components and stabilizes recurrent training (Spieler et al., 12 May 2026).

The central theoretical result is a closed-form information-theoretic tradeoff model. Modeling the layer as $a(t)$2 parallel noisy channels,

$a(t)$3

and assuming Gaussian signal plus noise, signal-covariance eigenvalues $a(t)$4, per-neuron noise variance $a(t)$5, and budget $a(t)$6, the task-relevant information is

$a(t)$7

In the sub-saturation regime,

$a(t)$8

Here $a(t)$9 quantifies single-neuron expressivity, $M_t^i\in\mathbb{R}^d$00 population-level redundancy, $M_t^i\in\mathbb{R}^d$01 the parameter-effectivity scale, and $M_t^i\in\mathbb{R}^d$02 the irreducible floor of per-neuron noise (Spieler et al., 12 May 2026).

Empirically, performance improves monotonically along each of the three axes individually—neuron count $M_t^i\in\mathbb{R}^d$03, complexity $M_t^i\in\mathbb{R}^d$04, and connectivity $M_t^i\in\mathbb{R}^d$05—on both SHD-Adding and Enwik8. Under fixed budget, however, a non-trivial optimum emerges in the tradeoff between many simple neurons and few complex neurons. The paper states that larger budgets favor both more and more complex neurons, that more synapses always help up to a point, and that beyond sufficient feed-forward inputs recurrent synapses dominate performance gains. The Pareto-frontier hyperparameter recipe is

$M_t^i\in\mathbb{R}^d$06

The practical guidance is correspondingly budget-sensitive: small-budget regimes favor more neurons of moderate $M_t^i\in\mathbb{R}^d$07, while large-budget regimes move toward deeper neurons while still scaling width (Spieler et al., 12 May 2026).

5. Population dynamics and mean-field description

The mean-field theory of age and leaky memory provides a population-level description of ELM-like neurons that is analytically explicit. In the $M_t^i\in\mathbb{R}^d$08 mean-field limit, with all-to-all coupling scaled by $M_t^i\in\mathbb{R}^d$09, the law of a typical neuron converges to the solution of a piecewise-deterministic SDE with self-consistency in $M_t^i\in\mathbb{R}^d$10. Equivalently, the population density $M_t^i\in\mathbb{R}^d$11 satisfies the nonlocal transport PDE (Schmutz, 2020)

$M_t^i\in\mathbb{R}^d$12

with boundary condition at $M_t^i\in\mathbb{R}^d$13

$M_t^i\in\mathbb{R}^d$14

and mean field

$M_t^i\in\mathbb{R}^d$15

The interpretation of the terms is direct. The left-hand side is pure transport in $M_t^i\in\mathbb{R}^d$16 driven by the drift $M_t^i\in\mathbb{R}^d$17; the sink $M_t^i\in\mathbb{R}^d$18 accounts for neurons that spike at $M_t^i\in\mathbb{R}^d$19 and leave that state; the boundary term re-inserts mass at $M_t^i\in\mathbb{R}^d$20 and updated memory $M_t^i\in\mathbb{R}^d$21; and the $M_t^i\in\mathbb{R}^d$22 equation closes the loop by allowing each spike to contribute through the kernel $M_t^i\in\mathbb{R}^d$23. The paper proves propagation of chaos and uses a path integral representation for the law of the limit process to derive this multidimensional nonlocal transport equation (Schmutz, 2020).

Several choices of $M_t^i\in\mathbb{R}^d$24, $M_t^i\in\mathbb{R}^d$25, and $M_t^i\in\mathbb{R}^d$26 illustrate how single-neuron mechanisms determine population dynamics. Exponential leak,

$M_t^i\in\mathbb{R}^d$27

gives exponentially decaying memory. The Erlang-kernel construction produces a $M_t^i\in\mathbb{R}^d$28th-order Poissonian cascade implementing the kernel

$M_t^i\in\mathbb{R}^d$29

One listed intensity is

$M_t^i\in\mathbb{R}^d$30

with $M_t^i\in\mathbb{R}^d$31 a refractoriness-modulated recovery curve, for example $M_t^i\in\mathbb{R}^d$32, and $M_t^i\in\mathbb{R}^d$33 an activation nonlinearity, for example sigmoid. The source further notes that power-law adaptation can arise if $M_t^i\in\mathbb{R}^d$34 is fractional. This suggests that the mean-field framework is not only a limit theorem for interacting point processes, but also a generative language for specifying distinct ELM temporal filters (Schmutz, 2020).

6. Physical realization, interpretation, and caveats

A hardware embodiment of the ELM motif is provided by the nanoscale leaky memcapacitor. The device is a parallel-plate capacitor whose top electrode is mounted on a compliant spring and moves vertically under electrostatic attraction. As charge $M_t^i\in\mathbb{R}^d$35 accumulates, the plate separation shrinks from $M_t^i\in\mathbb{R}^d$36 to $M_t^i\in\mathbb{R}^d$37, increasing capacitance; when $M_t^i\in\mathbb{R}^d$38 exceeds a contact threshold $M_t^i\in\mathbb{R}^d$39, ionic or tunneling conduction turns on, producing a leakage path with sharply falling resistance $M_t^i\in\mathbb{R}^d$40. Because both capacitance and leak depend on the same internal coordinate $M_t^i\in\mathbb{R}^d$41, the element is simultaneously a memcapacitor and a memristor connected in parallel (Zhang et al., 2023).

In the simplest analytical form,

$M_t^i\in\mathbb{R}^d$42

The more detailed equations used in the paper are

$M_t^i\in\mathbb{R}^d$43

$M_t^i\in\mathbb{R}^d$44

$M_t^i\in\mathbb{R}^d$45

$M_t^i\in\mathbb{R}^d$46

These equations define a closed three-dimensional dynamical system in $M_t^i\in\mathbb{R}^d$47, or equivalently $M_t^i\in\mathbb{R}^d$48 (Zhang et al., 2023).

The emergent behaviors parallel the ELM design logic. For constant drive in the range $M_t^i\in\mathbb{R}^d$49, the system converges to a limit cycle in $M_t^i\in\mathbb{R}^d$50-space rather than a static fixed point; a saddle-node-on-invariant-circle bifurcation at $M_t^i\in\mathbb{R}^d$51 nucleates the limit cycle, and a reverse saddle-node annihilation at $M_t^i\in\mathbb{R}^d$52 destroys it. By tuning $M_t^i\in\mathbb{R}^d$53, the device traverses three firing regimes—negative-going spikes, nearly sinusoidal oscillations, and positive-going spikes. Bursting can be added by replacing the series resistor with a slow memristor whose resistance evolves on a longer time scale, for example

$M_t^i\in\mathbb{R}^d$54

or by a thresholded rule with $M_t^i\in\mathbb{R}^d$55 when $M_t^i\in\mathbb{R}^d$56 and $M_t^i\in\mathbb{R}^d$57 otherwise (Zhang et al., 2023).

The hardware paper also makes a concrete energy claim: the net energy per spike can be very small, on the order of

$M_t^i\in\mathbb{R}^d$58

whereas standard CMOS leaky-integrate-and-fire cores often consume tens to hundreds of picojoules per spike. In the proposed network construction, outputs $M_t^i\in\mathbb{R}^d$59 are connected through programmable resistances, such as nanoscale memristor synapses, to the series-resistor inputs of other nodes; weighted presynaptic currents charge the postsynaptic capacitor, modifying both $M_t^i\in\mathbb{R}^d$60 and $M_t^i\in\mathbb{R}^d$61 until the device spikes (Zhang et al., 2023).

Two recurrent cautions qualify broader claims about ELM neurons. First, the 2023 ELM model is explicitly phenomenological: there is no direct biophysical interpretability of individual parameters. Second, training relies on BPTT and dendritic geometry is abstracted away, so biological plausibility is limited even when the model reproduces the input-output behavior of a detailed cortical pyramidal neuron. A common misconception is therefore to equate ELM with a biophysically faithful neuron model. The empirical record supports a narrower statement: multiple learnable, slowly decaying hidden states together with nonlinear synaptic integration are sufficient to reproduce detailed pyramidal-cell input-output relations efficiently and to support strong long-range sequence processing, but this does not by itself establish mechanistic identifiability of the underlying biology (Spieler et al., 2023).

Taken together, the theoretical, phenomenological, scaling-law, and hardware formulations converge on a single theme. ELM neurons treat per-neuron temporal computation as a primary modeling resource, alongside width and connectivity. The resulting models occupy an intermediate position between simple recurrent units and detailed compartmental neurons: they are substantially richer than standard leaky or gated cells, yet compact enough to admit mean-field analysis, large-scale hyperparameter scaling, and even device-level physical instantiation.