Threshold Power-Law RNN Dynamics

• The paper demonstrates that threshold power-law RNNs with p≠1 exhibit inherent scale invariance, enabling identical dynamical behavior regardless of coupling strength g.
• The methodology employs variable rescaling and dynamic mean field theory to systematically compare dynamics across threshold power-law, ReLU (p=1), and sigmoidal RNN models.
• The findings offer practical design guidelines for reservoir computing, ensuring consistent trainability by appropriately rescaling network inputs and outputs.

Threshold power-law recurrent neural networks (RNNs) constitute a class of dynamical neural network models characterized by a transfer function in which unit activations are zero below a specified threshold and rise as a power law above this threshold. These networks, relevant for both reservoir computing and broader recurrent neural modeling, display distinctive dynamical invariances and bifurcations depending on the power-law exponent and the scaling of their recurrent coupling. The qualitative independence of their dynamics and learning performance from coupling strength (with critical exceptions) distinguishes them from classical sigmoidal and rectified linear unit (ReLU) RNNs, thereby providing new perspectives and practical guarantees for the design of untrained reservoirs in machine learning and theoretical neuroscience (Nicola, 30 Nov 2025).

1. Model Formulation and Threshold Power-Law Transfer Functions

The canonical threshold power-law RNN is described for $N$ units with internal states $\mathbf z(t)\in\mathbb{R}^N$, where the dynamics evolve as

$$\frac{d z_i}{d t} = -z_i + g \sum_{j=1}^N \omega_{ij}\,f(z_j), \quad i=1,\ldots,N$$

with $g > 0$ the global coupling constant and $W = (\omega_{ij})$ a random recurrent connectivity matrix. The transfer function is

$f(x) = \max(0,\,x-\phi)^{p}$

with threshold $\phi$ and power $p>0$. Discrete-time analogues incorporate a leak-rate $\alpha$:

$$\mathbf z(t+1) = (1-\alpha) \mathbf z(t) + \alpha\, g\, W\, f(\mathbf z(t)), \quad \alpha\in(0,1].$$

The focus is primarily on the continuous-time model. This class encompasses both classical sublinear, linear (ReLU), and supralinear threshold nonlinearities, allowing systematic exploration of their dynamical effects (Nicola, 30 Nov 2025).

2. The Coupling Constant $g$ as a Scale Parameter

For all exponents $p\neq 1$, the coupling constant $g$ is strictly a scale parameter for the system, not influencing qualitative network dynamics. This follows from the homogeneous scaling property of the threshold power-law function: under the change of variables $\mathbf y = g^{1/(p-1)} \mathbf z$, the dynamics become

$$\frac{d \mathbf y}{dt} = -\mathbf y + W f(\mathbf y)$$

independent of $g$. As a result, all dynamical solutions—fixed points, periodic orbits, and chaotic attractors—are mapped onto each other by amplitude rescaling as $g$ varies. The qualitative geometry in state space and all stability properties remain invariant across $g > 0$ (Nicola, 30 Nov 2025).

3. Dynamics, Chaos, and Absence of "Edge of Chaos" Tuning

Dynamic mean field theory (Kadmon & Sompolinsky, 2015; Omri et al., 2018) establishes that for $p \le 1/2$, chaotic trajectories exist for arbitrarily small nonzero $g$ in the limit $N \to \infty$. In threshold power-law RNNs ($p\neq 1$), due to scale invariance, any chaotic trajectory at some $g^*$ can be mapped to a corresponding chaotic trajectory at any $\bar g$ by the relation

$$\mathbf z^{(\bar g)}(t) = (\bar g/g^*)^{-1/(p-1)}\,\mathbf z^*(t).$$

The Lyapunov spectrum, including the maximal Lyapunov exponent $\lambda_{\max}(g)$, remains constant for all $g > 0$. There is consequently no "edge of chaos" bifurcation with respect to $g$, and chaos can be neither tamed nor forced by tuning $g$ in this regime. All dynamical stability types are preserved up to amplitude scaling (Nicola, 30 Nov 2025).

4. The Singular ReLU Case: $p=1$

When the power-law exponent $p=1$, producing the ReLU transfer function $f(x) = \max(0, x - \phi)$, the scale transformation that removes $g$ is singular. In this scenario,

$$\frac{d\mathbf z}{dt} = -\mathbf z + g\, W\, f(\mathbf z)$$

explicitly retains the coupling strength, and tuning $g$ induces genuine bifurcations between dynamical regimes. As $g$ passes through critical values, the system transitions between quiescent states, periodic orbits, and chaos. Here, both the untrained network's dynamics and the convergence/performance of training algorithms are directly dependent on the value of $g$. The ReLU case is thus exceptional within the threshold power-law family (Nicola, 30 Nov 2025).

5. Comparison with Classical Sigmoidal RNNs

Traditional RNNs with sigmoidal activations ($f(z) = \tanh(z)$) do not allow elimination of $g$ via variable rescaling, as the activation is not homogeneous of any degree. The system,

$$\frac{d\mathbf z}{dt} = -\mathbf z + g\, W\, \tanh(\mathbf z),$$

exhibits qualitatively distinct dynamical regimes as $g$ varies: for $g < 1$ the network is quiescent, for $g > 1$ it becomes chaotic, and the "edge of chaos" near $g \approx 1$ is empirically favored for reservoir computing. Consequently, training metrics, memory capacity, and algorithmic stability hinge nontrivially on precise $g$ tuning in sigmoidal networks, in marked contrast with the scale-invariant regime of threshold power-law RNNs for $p\neq 1$ (Nicola, 30 Nov 2025).

6. Theoretical and Practical Implications for Training

A central result states that if a threshold power-law reservoir ($p\neq 1$) can be successfully trained at a single $g^*$ to approximate any supervised signal (using arbitrary encoder and decoder), there exist rescaled encoders and decoders for any $\bar g > 0$ guaranteeing identical accuracy. Explicitly, the mapping

$$\hat\phi = \phi\,(\bar g/g^*)^{p/(p-1)}, \quad \hat\eta = \eta\,(g^*/\bar g)^{1/(p-1)}$$

preserves the test loss $L^*$. Thus, if chaos is tamed (training converges) for one $g^*$, it is universally possible for all $g > 0$, modulo trivial output/input scaling. This result guarantees "no-tuning" of $g$ in sub-ReLU ($p<1$) threshold power-law reservoir implementations. Empirical evidence from training FORCED-based networks on oscillator and chaotic targets corroborates this theoretical invariance, with test errors remaining constant across a wide $g$ range after rescaling (Nicola, 30 Nov 2025).

7. Summary and Broader Significance

Threshold power-law RNNs with $p\neq 1$ exhibit pure scale invariance in their coupling parameter $g$, leading to invariant dynamical structure and trainability under amplitude rescaling. This property sharply contrasts with both ReLU and sigmoidal RNNs, in which $g$ controls stability, chaos, and computational capacity. These findings refine the theoretical understanding of non-sigmoidal RNNs and provide practical design guarantees, especially for reservoir computing, simplifying hyperparameter selection and removing the need to tune the global coupling strength except in the ReLU ($p=1$) regime (Nicola, 30 Nov 2025). A plausible implication is substantial robustness and efficiency for sub-ReLU threshold power-law reservoir architectures in large-scale, untrained recurrent network applications.