Memory & Nonlinear Capacity in Dynamical Systems

• Memory capacity and nonlinear capacity are fundamental metrics that quantify how dynamical systems store, process, and reconstruct temporal data using linear and nonlinear methods.
• They are measured via techniques such as linear regression, polynomial expansions, and spectral analyses to delineate forecasting and prediction capabilities.
• Trade-offs between linear and nonlinear regimes inform optimal designs in recurrent networks, photonic systems, and reservoir computing for robust temporal processing.

Memory capacity (MC) and nonlinear capacity are fundamental concepts for quantifying how dynamical systems—including recurrent neural networks, reservoir computers, and communication channels—store, process, and utilize temporal information. MC generally captures the ability of a system to reconstruct past inputs from its present state, while nonlinear capacity extends this notion to the reconstruction (or computation) of nonlinear functions of past inputs. Theoretical and empirical studies have established a diverse taxonomy of memory-related capacities, with sharp distinctions between linear and nonlinear regimes, channel vs. network contexts, and storage vs. forecasting vs. prediction objectives.

1. Formal Definitions: Memory Capacity and Nonlinear Capacity

The classical definition of memory capacity, introduced for linear echo state networks and extended to general stationary inputs and nonlinear recurrent networks, relies on the proportion of variance in an input that can be linearly reconstructed given the network's internal state at each lag. For a real-valued, variance-stationary input process $Z_t$ (with $\mathrm{Var}(Z_t) = \gamma(0)$), and state $X_t$ generated recursively,

$$x_t = F(x_{t-1}, Z_t), \qquad y_t = W^\top x_t + a,$$

the $\tau$-lag memory capacity is

$$MC_\tau = 1 - \frac{1}{\mathrm{Var}(Z_t)} \min_{W, a} \mathbb{E}\left[ \left(Z_{t+\tau} - W^\top X_t - a \right)^2 \right],$$

with total memory capacity as $MC = \sum_{\tau \leq 0} MC_\tau$. For nonlinear or dependent inputs, the same structure holds, with extensions to forecasting capacity (FC), where the network's state is used to linearly predict future input values (Gonon et al., 2020).

In networks where the induced mapping can be expanded in Volterra or Hermite polynomial bases—such as in the framework of reservoir computing (Sheldon et al., 2020, Jüngling et al., 2021, Castro et al., 2024, Takasu et al., 28 Apr 2025)—the total computational capacity decomposes into linear (memory) capacity and capacities for higher-order (nonlinear) target functions: $$C[z] = 1 - \text{nMSE}[z] = 1 - \frac{\langle (z - \hat{z})^2 \rangle}{\langle z^2 \rangle},$$ where $z$ may be a nonlinear function of past inputs, and $\hat{z}$ the optimal linear readout from the system's state. Linear memory capacity is the sum over first-order delays, while nonlinear capacity includes contributions from higher-order orthogonal polynomials (e.g., quadratic or cubic functions of delayed inputs) (Sheldon et al., 2020, Jüngling et al., 2021, Castro et al., 2024, Takasu et al., 28 Apr 2025). In this formalism, total capacity is additive across orthogonal target subspaces.

2. Theoretical Bounds and Scaling Laws

Memory capacity admits tight theoretical bounds, rooted in system dimensionality and the statistical properties of the input process. For a nonlinear recurrent network of $\mathrm{Var}(Z_t) = \gamma(0)$0 neurons, with invertible state covariance matrix $\mathrm{Var}(Z_t) = \gamma(0)$1, generic bounds for memory and forecasting capacities are expressed via the input autocovariance function $\mathrm{Var}(Z_t) = \gamma(0)$2 or its spectral density $\mathrm{Var}(Z_t) = \gamma(0)$3. Specifically,

$\mathrm{Var}(Z_t) = \gamma(0)$4

where $\mathrm{Var}(Z_t) = \gamma(0)$5 is the spectral radius of the infinite Toeplitz matrix associated with input autocovariance (Gonon et al., 2020). For white inputs, $\mathrm{Var}(Z_t) = \gamma(0)$6 if $\mathrm{Var}(Z_t) = \gamma(0)$7, yielding the sharp bound $\mathrm{Var}(Z_t) = \gamma(0)$8. These bounds generalize to forecasting capacity and relate to the network's ability to exploit input dependence (i.e., autocorrelation).

For classical linear networks, MC is exactly the rank of the Kalman controllability matrix—almost surely maximal, $\mathrm{Var}(Z_t) = \gamma(0)$9, under random matrix draws (Ballarin et al., 7 Feb 2025). In nonlinear networks, MC can be tuned arbitrarily in $X_t$0 by scaling the input variance, and is thus not intrinsic to the network architecture per se.

Nonlinear capacity scaling exhibits distinct regimes. For certain dense associative memory networks and dynamic energy networks, the capacity (number of stored patterns or sequences) can scale exponentially with the number of neurons, $X_t$1, if the nonlinearities (e.g., softmax or threshold functions) are appropriately chosen (Karuvally et al., 28 Oct 2025, Kafraj et al., 2 Jan 2026). For polynomial-threshold maps, the critical number of linearly separable associations also exhibits exponential dependence on input dimension when sparsity is high (Baldi et al., 2021). Conversely, in standard recurrent reservoirs or hardware implementations, sublinear scaling of MC with system size arises when neuronal correlations are non-negligible: $X_t$2, where $X_t$3 quantifies the average pairwise correlation (Takasu et al., 28 Apr 2025).

3. Measurement Methodologies: Linear vs. Nonlinear Capacity

Quantifying linear and nonlinear capacities involves a suite of algorithmic procedures:

• Linear memory capacity is measured by regressing the system state onto delayed copies of the input, maximizing squared correlation at each lag, then summing.
• Nonlinear capacity is measured by reconstructing orthogonal nonlinear functions (e.g., Legendre or Hermite polynomials of delayed inputs) from the system state.
• Consistency spectrum analysis (replica tests) measures the fraction of system variance attributable to a subset of input lags or channels, yielding a hierarchy of differential capacities for higher-order interactions (Jüngling et al., 2021).
• Volterra expansion in continuous-time dynamical systems enables a decomposition of capacity into orders, with the total computational capacity approaching the number of effective state degrees of freedom in the uncorrelated limit (Sheldon et al., 2020).
• Replica experiments and whitening/covariance decompositions robustly separate linear from nonlinear memory contributions in practice (Jüngling et al., 2021, Takasu et al., 28 Apr 2025).

In physical or photonic systems (e.g., microring resonator-based time-delay reservoirs), capacity is assessed by reconstructing a set of delayed inputs and their polynomial functions, with measurement protocols based on ridge regression and orthogonal polynomial expansion (Castro et al., 2024).

4. Special Cases: Nonlinear Channels and High-capacity Architectures

In nonlinear optical channels, capacity analysis requires detailed modeling of channel memory. When the finite memory of a fiber-optic link is properly accounted for, channel capacity does not exhibit the "peaky" behavior predicted by infinite-memory (i.i.d. input) Gaussian-noise models; coding strategies that temporally concentrate power can exploit memory to achieve strictly non-decreasing capacity with respect to average input power, saturating to a positive asymptote at high power (Agrell et al., 2014). This highlights the centrality of memory length as a limiting resource for nonlinear communication systems.

Associative memory and energy-based neural architectures with suitable nonlinearities (e.g., thresholding or high-order softmax) enable exponential storage capacity, with each network attractor corresponding to a combinatorial selection of hidden or visible neuron states (Karuvally et al., 28 Oct 2025, Kafraj et al., 2 Jan 2026). In such regimes, the network is robust to noise and supports efficient decoding.

In polynomial threshold systems, sparsity in output or hidden representations increases capacity via geometric effects on linear separability (random polytope phase transitions), with critical numbers of storable patterns scaling as $X_t$4 in the extreme sparse regime (Baldi et al., 2021).

5. Trade-offs and Impact of Nonlinearity

Nonlinearity is fundamental for boosting capacity beyond the linear regime, but entails critical trade-offs:

• In purely linear networks, capacity is bounded by the effective state dimension, and is often saturated for generic architectures.
• Strong nonlinearity increases the set of linearly separable patterns/functions and supports higher-order computational capacity, but may also increase correlations among units, causing sublinear MC scaling and earlier onset of capacity saturation (Takasu et al., 28 Apr 2025).
• In physical or hardware systems, the balance between nonlinearity and memory is determined by design parameters: for photonic reservoirs, increasing the nonlinear response (e.g., via free-carrier lifetime) trades linear memory for nonlinear computational breadth, with optimal capacity at an intermediate point before system instability (Castro et al., 2024).
• In continuous-time reservoirs (e.g., models of time cells), maximizing MC and maximizing predictive (forecasting) capacity are generally distinct, leading to different optimal timescales for the underlying filters (Hsu et al., 2020). Biological architectures may resolve these objectives by hybridizing memory and prediction pathways.

6. Limitations, Critiques, and Future Perspectives

Recent work has questioned the intrinsic value of conventional MC for practical network comparison. In both the linear and nonlinear settings, MC can be tuned to any value within $X_t$5 by manipulating input statistics (scaling, autocorrelation), limiting its discriminative power as a network property (Ballarin et al., 7 Feb 2025). Consequently, modern perspectives advocate for alternative, geometry- and information-aware metrics that (i) normalize out input variance, (ii) are invariant to scaling, and (iii) reflect the actual computational or functional geometry of the network (Ballarin et al., 7 Feb 2025).

Novel directions include:

• Information-theoretic and task-specific capacity metrics,
• Geometric/topological methods that probe the structure of state trajectories,
• Hierarchical and task-aligned decompositions (e.g., balanced memory profiles, cross-order consistency tests).

Empirical and theoretical advances in hardware reservoir computing, neural interfaces, and nonlinear channel design continue to motivate the refinement and extension of capacity concepts to more realistic, non-i.i.d., and task-based contexts. These efforts aim to robustly quantify and design memory and computational architectures matched to practical and biological information-processing demands.