--- title: Expressive Leaky Memory (ELM) Neurons url: https://www.emergentmind.com/topics/expressive-leaky-memory-elm-neurons type: topic --- # Expressive Leaky Memory (ELM) Neurons Searching arXiv for the cited papers to ground the article and verify bibliographic details. Expressive Leaky Memory (ELM) neurons are phenomenological neuron models in which scalar output activity is generated from rich internal state, multi-timescale leak dynamics, and nonlinear synaptic integration. Across the literature, the core motif appears in several closely related forms: as an age-and-memory-dependent point-process neuron with reset and leaky hidden state in mean-field Hawkes theory; as a recurrent cell with learnable synaptic traces, memory units, and a compact multilayer integrator for long-horizon sequence processing; as a network primitive whose per-neuron complexity, width, and connectivity can be traded under a fixed parameter budget; and as a nanoscale leaky memcapacitive device that physically co-localizes integration, leak, reset, and slow adaptation [2003.13810] [2306.16922] [2605.12049] [2304.10899]. The unifying idea is that temporal computation is not delegated solely to network recurrence or external state-space structure, but is partly internalized within each neuron through explicitly leaky memory variables operating on multiple timescales. ## 1. Definition and formal lineage A foundational formulation appears in the mean-field model of interacting point processes proposed by Schmutz, where each neuron carries two state variables: age $A_t^i\in\mathbb{R}_+$, the time elapsed since its last spike, and leaky memory $M_t^i\in\mathbb{R}^d$, a $d$-dimensional memory that accumulates past activity but decays between spikes. Spiking is governed by the stochastic intensity \[ \lambda^i(t)=f\bigl(A^i_{t-},\,M^i_{t-},\,X_t\bigr), \] with bounded, Lipschitz-continuous $f$, and between spikes the state evolves according to \[ \frac{d}{dt}A^i_t = 1,\qquad \frac{d}{dt}M^i_t = b\bigl(M^i_t\bigr), \] while at a spike the neuron resets by \[ A^i_t \mapsto 0,\qquad M^i_t\mapsto M^i_{t-}+\Gamma\bigl(M^i_{t-}\bigr). \] Crucially, $\Gamma$ need not reset memory to a fixed value; it may depend on the current $M$ and thus encode cumulative adaptation or plasticity [2003.13810]. This formulation generalizes age-dependent Hawkes processes. Age-only renewal models retain only $a(t)$, so the intensity is $f(a)$ and the process forgets everything but the last interspike interval; memoryless Poisson models have no history dependence at all. By combining age and multidimensional leaky memory, the ELM motif can simultaneously represent instantaneous refractoriness and slower processes such as adaptation, facilitation, depression, and synaptic filtering. The examples given in the source material make this explicit: Erlang-kernel self-interaction yields a $d$-dimensional adaptation cascade, while a $d=2$ Tsodyks–Markram construction uses facilitation and depression variables with synaptic efficacy $h(t,a,m)=m(1)\,m(2)\,\bar h(t)$ [2003.13810]. A plausible implication is that “ELM” is best understood not as a single fixed equation set but as a design family. In that family, a neuron is expressive to the extent that it combines resettable fast state, slowly decaying internal state, and nonlinear coupling between them. ## 2. Single-neuron dynamics in the phenomenological ELM model The 2023 ELM formulation for sequence modeling makes this family concrete as a recurrent cell. At each time $t$, the neuron receives an input spike vector $x_t\in\mathbb{R}^{d_s}$ and maintains a synaptic current trace $s_t\in\mathbb{R}^{d_s}$, a memory-state vector $m_t\in\mathbb{R}^{d_m}$, and a scalar scaling factor $\lambda>0$. The per-channel synaptic and per-unit memory timescales, $\tau_s\in\mathbb{R}_+^{d_s}$ and $\tau_m\in\mathbb{R}_+^{d_m}$, are learnable, with decay factors \[ \kappa_s=\exp\!\bigl(-\tfrac{\Delta t}{\tau_s}\bigr),\qquad \kappa_m=\exp\!\bigl(-\tfrac{\Delta t}{\tau_m}\bigr). \] The update is organized into four stages [2306.16922]: \[ s_t=\kappa_s\odot s_{t-1}+w_s\odot x_t, \] \[ \Delta m_t=\tanh\bigl(\mathrm{MLP}_{w_p}[\,s_t;\,\kappa_m\odot m_{t-1}\,]\bigr), \] \[ m_t=\kappa_m\odot m_{t-1}+\lambda(1-\kappa_m)\odot\Delta m_t, \] \[ y_t=w_y\,m_t. \] The nonlinear dendritic integration step uses a small MLP. Concretely, with one hidden layer of size $2d_m$ and ReLU, \[ h=\mathrm{ReLU}\bigl(W^{(1)}[s_t;\,\kappa_m\odot m_{t-1}]+b^{(1)}\bigr),\qquad \mathrm{MLP}(\cdot)=W^{(2)}h+b^{(2)}. \] This architecture couples filtered synaptic input with decayed internal state before the memory update is committed. The update can also be written as \[ m_t=(1-\alpha_m)\,m_{t-1}+\alpha_m\,f(s_t,m_{t-1}),\qquad \alpha_m=1-\kappa_m, \] but the explicit per-unit gating is set by the learnable $\tau_m$ rather than by a conventional learned gate vector [2306.16922]. The source material emphasizes two consequences. First, because each memory unit $i$ has its own $\tau_{m,i}$, a single ELM neuron can track both fast and slow processes. Second, the synaptic side likewise uses per-channel $\tau_s$ to model input filtering. In this sense, temporal depth is distributed throughout the cell rather than concentrated in a single recurrent hidden state. ## 3. Architectural variants, parameter efficiency, and empirical performance The main architectural hyperparameters are the input dimension $d_s$, the number of memory units $d_m$, and the MLP width. In the reported implementations, $d_s$ equals the number of input channels—1278 for NeuronIO, 700 cochlea channels for SHD, and 8–169 for LRA—while $d_m$ is task-dependent: 10–20 memory units are sufficient to fit NeuronIO, 150 are used on LRA, and up to 250 can be used. The standard ELM uses an MLP of width $2d_m$ with one hidden layer; Branch-ELM inserts an intermediate branch grouping of the $d_s$ inputs into $d_{\rm tree}$ branches of size $d_{\rm branch}$ before the MLP [2306.16922]. This design yields unusually low parameter counts relative to the tasks considered. On NeuronIO, the total trainable parameters are approximately 53 K for ELM and approximately 8 K for Branch-ELM; on LRA with $d_m=150$, the single-node ELM has approximately 100 K parameters. The NeuronIO fitting result is central: prior work found that a TCN with $>10$ M parameters was needed to reach a sufficient spike-prediction AUC of 0.991, whereas ELM with $d_m=20$ achieves $\mathrm{AUC}\approx0.992$ and RMS-voltage error $\approx0.64$ mV with 53 K parameters, and Branch-ELM further reduces to 8 K parameters while crossing the AUC threshold [2306.16922]. The reported benchmark results span both neuromorphic and long-range sequence tasks. | Setting | ELM result | Comparator(s) | |---|---:|---| | NeuronIO | AUC≈0.992, RMS-voltage error ≈0.64 mV | TCN with >10 M parameters needed for AUC 0.991 | | SHD-Adding, 2 ms bins ($T=2000$) | ELM ∼ 0.82 accuracy | LSTM ∼ 0.50 | | LRA Image (1 K px) | 49.6% | Chrono-LSTM 46.1%, Transformer 42.4% | | LRA Pathfinder (1 K) | 71.2% | 70.8%, 71.4% | | LRA Pathfinder-X (16 K) | 77.3% | LSTM 70.8%, Transformer/Longformer fail | | LRA Text | 80.3% | 75.4%, 64.3% | | LRA Retrieval | 84.9% | 82.9%, 57.5% | On SHD-Adding, the comparison is ELM (186 K params), Branch-ELM (67 K), LSTM (956 K), and a learned-$\tau$ LIF-SNN (51 K), with the summary that across all bin sizes, ELM $\gtrsim$ Branch-ELM $\gg$ LSTM $\gg$ SNN. On LRA, the comparison is between a single ELM neuron ($d_m=150$, $\sim100$ K params), Chrono-LSTM ($\sim100$ K), Transformer/Longformer ($\sim600$ K), and SOTA S4/Mega ($\sim600$ K). Only deep, purpose-built state-space models outperform ELM, and the source states that a single ELM neuron is the only non–purpose-built model to reliably solve the 16 K horizon Pathfinder-X task [2306.16922]. Ablations isolate the mechanisms responsible for these results. On NeuronIO, $d_m<10$ memory units causes AUC to fall below 0.99, while $d_m\approx20$ is sufficient; removing the MLP sharply degrades both spike- and voltage-prediction; the memory-timescale range $[1,150]$ ms is ideal, and forbidding $\tau_m>25$ ms hurts performance markedly. The scaling factor $\lambda$ controls the amplitude of memory updates: larger $\lambda$ helps fit rapid transients but can cause instabilities. Branch-ELM, using approximately 45 branches of approximately 65 synapses each, recovers nearly the full ELM performance at approximately $7\times$ parameter reduction. On LRA Path-X, varying $d_m$ gives a saturating curve: $10\to\sim57.8\%$, $25\to63\%$, $50\to66.3\%$, $100\to69.3\%$, $150\to71.5\%$, $200\to73.0\%$, $300\to71.9\%$ [2306.16922]. These results support a specific interpretation: long-horizon performance is not attributed merely to recurrence, but to the conjunction of long memory timescales and nonlinear synaptic integration within the neuron itself. ## 4. Network formulation and scaling tradeoffs In the 2026 ELM Network, ELM neurons become explicit architectural knobs. Each neuron $i$ at time $t$ maintains a vector of $d_m$ leaky memory states $\mathbf{m}_t\in\mathbb{R}^{d_m}$ and emits a scalar activity $a_t\in\mathbb{R}$. Its input $\mathbf{z}_t\in\mathbb{R}^{d_s}$ is formed by selecting $d_s$ synaptic channels from the concatenated feed-forward and recurrent activities. The per-neuron update is [2605.12049] \[ \mathbf{b}_t=c\,\mathrm{branch\_sum}\bigl(\mathbf{z}_t\odot \mathbf{w}_s\bigr), \] \[ \Delta\mathbf{m}_t=\tanh\!\bigl(\mathrm{MLP}_{\mathbf{w}_p}[\,\mathbf{b}_t,\;\boldsymbol{\kappa}_m\odot\mathbf{m}_{t-1}\,]\bigr), \] \[ \mathbf{m}_t=\boldsymbol{\kappa}_m\odot\mathbf{m}_{t-1}+(1-\boldsymbol{\kappa}_\lambda)\odot\Delta\mathbf{m}_t, \] \[ r_t=\kappa_r\,r_{t-1}+(1-\kappa_r)\,\mathbf{w}_r^\top\mathbf{m}_t, \] \[ a_t=\mathrm{ReLU}\Bigl(b+\mathbf{w}_r^\top\mathbf{m}_t-r_t\Bigr), \] with \[ \kappa_{m,i}=\exp\!\bigl(-1/\tau_{m,i}\bigr),\qquad \kappa_{\lambda,i}=\exp\!\bigl(-\lambda/\tau_{m,i}\bigr),\qquad \kappa_r=\exp\!\bigl(-1/\tau_r\bigr). \] Two architectural axes are then separated explicitly. Per-neuron complexity is \[ k_e=\#\mathbf{w}_p+\#\mathbf{w}_r, \] while connectivity overhead is \[ k_c=\#\mathbf{w}_s. \] For a layer of width $N$, the trainable-parameter budget satisfies \[ P=\sum_{i=1}^N\bigl((\#\mathbf{w}_p+\#\mathbf{w}_r)+\#\mathbf{w}_s\bigr) \quad\Longrightarrow\quad P=N(k_e+k_c). \] A fixed Bernoulli mask with recurrent fraction $\rho_{\rm rec}$ determines whether each synapse is feed-forward or recurrent, so effective connectivity per neuron is $k_c=d_s$ [2605.12049]. The paper positions ELM relative to canonical recurrent units. Compared with LSTM/GRU, ELM replaces explicit sigmoid gates and point-wise input combination with a vector-state of $d_m$ leak units, a learned nonlinear MLP, multi-timescale filtering, structured dendritic branching, and a high-pass output filter. Compared with LIF, ELM has multiple parallel leak channels and a smooth high-pass nonlinearity rather than a single or few state variables and a hard threshold. The subtraction of the EMA readout $r_t$ is identified as a high-pass filter that removes runaway DC components and stabilizes recurrent training [2605.12049]. The central theoretical result is a closed-form information-theoretic tradeoff model. Modeling the layer as $N$ parallel noisy channels, \[ y_i=f_i(x)+n_i,\qquad n_i\sim\mathcal{N}(0,\sigma_n^2), \] and assuming Gaussian signal plus noise, signal-covariance eigenvalues $\lambda_i\propto i^{-\beta}$, per-neuron noise variance $\sigma_n^2(k_e)\propto\max\{(\gamma k_e)^{-\alpha},\,q_\infty\}$, and budget $P=N(k_e+k_c)$, the task-relevant information is \[ I_{\mathrm{rep}}(k_e)=\frac{1}{2}\sum_{i=1}^{N}\log_2\!\bigl(1+s(k_e)\,i^{-\beta}\bigr), \quad N=\frac{P}{k_e+k_c}, \quad s(k_e)=\min\!\bigl((\gamma k_e)^\alpha,\;q_\infty^{-1}\bigr). \] In the sub-saturation regime, \[ I_{\mathrm{rep}}\approx\frac{(\gamma k_e)^\alpha}{2\ln 2}\sum_{i=1}^N i^{-\beta}\propto k_e^\alpha. \] Here $\alpha$ quantifies single-neuron expressivity, $\beta$ population-level redundancy, $\gamma$ the parameter-effectivity scale, and $q_\infty$ the irreducible floor of per-neuron noise [2605.12049]. Empirically, performance improves monotonically along each of the three axes individually—neuron count $N$, complexity $k_e$, and connectivity $k_c$—on both SHD-Adding and Enwik8. Under fixed budget, however, a non-trivial optimum emerges in the tradeoff between many simple neurons and few complex neurons. The paper states that larger budgets favor both more and more complex neurons, that more synapses always help up to a point, and that beyond sufficient feed-forward inputs recurrent synapses dominate performance gains. The Pareto-frontier hyperparameter recipe is \[ d_m\propto\sqrt{N},\qquad d_{\rm mlp}=2d_m,\qquad d_{\rm tree}=2d_{\rm mlp},\qquad d_{\rm branch}=d_{\rm tree},\qquad k_c\propto N,\qquad \rho_{\rm rec}\approx \sqrt{\frac{N}{N+d_{\rm inp}}}. \] The practical guidance is correspondingly budget-sensitive: small-budget regimes favor more neurons of moderate $d_m$, while large-budget regimes move toward deeper neurons while still scaling width [2605.12049]. ## 5. Population dynamics and mean-field description The mean-field theory of age and leaky memory provides a population-level description of ELM-like neurons that is analytically explicit. In the $N\to\infty$ mean-field limit, with all-to-all coupling scaled by $1/N$, the law of a typical neuron converges to the solution of a piecewise-deterministic SDE with self-consistency in $x_t$. Equivalently, the population density $\rho_t(a,m)$ satisfies the nonlocal transport PDE [2003.13810] \[ \partial_t\rho_t(a,m) +\partial_a\,\rho_t(a,m) +\nabla_m\!\cdot\bigl[b(m)\,\rho_t(a,m)\bigr] = -\,f\bigl(a,m,x_t\bigr)\,\rho_t(a,m), \] with boundary condition at $a=0$ \[ \rho_t(0,m) = \bigl(\gamma_*\!\!\int_0^\infty f(a,\cdot,x_t)\,\rho_t(a,\cdot)\,da\bigr)(m), \] and mean field \[ x_t = \bar H_t +\int_0^t\int_{\mathbb{R}^d}\int_0^\infty h(t-s,a,m)\,f\bigl(a,m,x_s\bigr)\,\rho_s(a,m)\,da\,dm\,ds. \] The interpretation of the terms is direct. The left-hand side is pure transport in $(a,m)$ driven by the drift $(1,b(m))$; the sink $-f\rho$ accounts for neurons that spike at $(a,m)$ and leave that state; the boundary term re-inserts mass at $a=0$ and updated memory $\gamma(m)=m+\Gamma(m)$; and the $x_t$ equation closes the loop by allowing each spike to contribute through the kernel $h(\cdot,\cdot)$. The paper proves propagation of chaos and uses a path integral representation for the law of the limit process to derive this multidimensional nonlocal transport equation [2003.13810]. Several choices of $b$, $\Gamma$, and $f$ illustrate how single-neuron mechanisms determine population dynamics. Exponential leak, \[ b(m)=-\alpha m, \] gives exponentially decaying memory. The Erlang-kernel construction produces a $d$th-order Poissonian cascade implementing the kernel \[ t\mapsto c\,e^{-\alpha t}t^{d-1}/(d-1)!. \] One listed intensity is \[ f(a,m,x)=\bigl[m(1)m(2)\bigr]\phi(x)\psi(a), \] with $\psi(a)$ a refractoriness-modulated recovery curve, for example $\psi(a)=1-e^{-\kappa a}$, and $\phi(x)$ an activation nonlinearity, for example sigmoid. The source further notes that power-law adaptation can arise if $b$ is fractional. This suggests that the mean-field framework is not only a limit theorem for interacting point processes, but also a generative language for specifying distinct ELM temporal filters [2003.13810]. ## 6. Physical realization, interpretation, and caveats A hardware embodiment of the ELM motif is provided by the nanoscale leaky memcapacitor. The device is a parallel-plate capacitor whose top electrode is mounted on a compliant spring and moves vertically under electrostatic attraction. As charge $q$ accumulates, the plate separation shrinks from $d$ to $(d-x)$, increasing capacitance; when $x$ exceeds a contact threshold $x_c$, ionic or tunneling conduction turns on, producing a leakage path with sharply falling resistance $R(x)$. Because both capacitance and leak depend on the same internal coordinate $x$, the element is simultaneously a memcapacitor and a memristor connected in parallel [2304.10899]. In the simplest analytical form, \[ C(x)=C_0/[1-x/d],\qquad R(x)=R_0\exp(-\alpha x). \] The more detailed equations used in the paper are \[ q=\varepsilon A\,V_C/(d-x)\equiv C(x)V_C, \] \[ V_C=[R_m(1/2+\arctan \beta(x_c-x)/\pi)+\rho_0(d-x)/A]\,I_M\equiv R(x)I_M, \] \[ \gamma\,\dot x=q^2/(\varepsilon A)-k\,x-\frac{d}{dx}\Bigl[4\varepsilon_l\bigl((\sigma/(d-x))^{12}-(\sigma/(d-x))^6\bigr)\Bigr], \] \[ V(t)=r\bigl(\dot C(x)V_C+C'(x)V_C\dot x+I_M\bigr)+V_C. \] These equations define a closed three-dimensional dynamical system in $(x,q,I_M)$, or equivalently $(x,q,V_C)$ [2304.10899]. The emergent behaviors parallel the ELM design logic. For constant drive in the range $V_10$ when $I_r>I'$ and $\dot r<0$ otherwise [2304.10899]. The hardware paper also makes a concrete energy claim: the net energy per spike can be very small, on the order of \[ E_{\rm spike}\simeq \tfrac{1}{2}C_{\rm avg}V_{dc}^2-\text{(work recovered on recoil)}\lesssim 10~\mathrm{fJ}, \] whereas standard CMOS leaky-integrate-and-fire cores often consume tens to hundreds of picojoules per spike. In the proposed network construction, outputs $V_C$ are connected through programmable resistances, such as nanoscale memristor synapses, to the series-resistor inputs of other nodes; weighted presynaptic currents charge the postsynaptic capacitor, modifying both $q$ and $x$ until the device spikes [2304.10899]. Two recurrent cautions qualify broader claims about ELM neurons. First, the 2023 ELM model is explicitly phenomenological: there is no direct biophysical interpretability of individual parameters. Second, training relies on BPTT and dendritic geometry is abstracted away, so biological plausibility is limited even when the model reproduces the input-output behavior of a detailed cortical pyramidal neuron. A common misconception is therefore to equate ELM with a biophysically faithful neuron model. The empirical record supports a narrower statement: multiple learnable, slowly decaying hidden states together with nonlinear synaptic integration are sufficient to reproduce detailed pyramidal-cell input-output relations efficiently and to support strong long-range sequence processing, but this does not by itself establish mechanistic identifiability of the underlying biology [2306.16922]. Taken together, the theoretical, phenomenological, scaling-law, and hardware formulations converge on a single theme. ELM neurons treat per-neuron temporal computation as a primary modeling resource, alongside width and connectivity. The resulting models occupy an intermediate position between simple recurrent units and detailed compartmental neurons: they are substantially richer than standard leaky or gated cells, yet compact enough to admit mean-field analysis, large-scale hyperparameter scaling, and even device-level physical instantiation. Source: https://www.emergentmind.com/topics/expressive-leaky-memory-elm-neurons