---
title: Threshold Power-Law RNN Dynamics
url: https://www.emergentmind.com/topics/threshold-power-law-recurrent-neural-networks
type: topic
---

# Threshold Power-Law RNN Dynamics

Threshold power-law recurrent neural networks (RNNs) constitute a class of dynamical neural network models characterized by a transfer function in which unit activations are zero below a specified threshold and rise as a power law above this threshold. These networks, relevant for both reservoir computing and broader recurrent neural modeling, display distinctive dynamical invariances and bifurcations depending on the power-law exponent and the scaling of their recurrent coupling. The qualitative independence of their dynamics and learning performance from coupling strength (with critical exceptions) distinguishes them from classical sigmoidal and rectified linear unit (ReLU) RNNs, thereby providing new perspectives and practical guarantees for the design of untrained reservoirs in machine learning and theoretical neuroscience [2512.01006].

## 1. Model Formulation and Threshold Power-Law Transfer Functions

The canonical threshold power-law RNN is described for $N$ units with internal states $\mathbf z(t)\in\mathbb{R}^N$, where the dynamics evolve as
$$
\frac{d z_i}{d t} = -z_i + g \sum_{j=1}^N \omega_{ij}\,f(z_j), \quad i=1,\ldots,N
$$
with $g > 0$ the global coupling constant and $W = (\omega_{ij})$ a random recurrent connectivity matrix. The transfer function is
$$
f(x) = \max(0,\,x-\phi)^{p}
$$
with threshold $\phi$ and power $p>0$. Discrete-time analogues incorporate a leak-rate $\alpha$:
$$
\mathbf z(t+1) = (1-\alpha) \mathbf z(t) + \alpha\, g\, W\, f(\mathbf z(t)), \quad \alpha\in(0,1].
$$
The focus is primarily on the continuous-time model. This class encompasses both classical sublinear, linear (ReLU), and supralinear threshold nonlinearities, allowing systematic exploration of their dynamical effects [2512.01006].

## 2. The Coupling Constant $g$ as a Scale Parameter

For all exponents $p\neq 1$, the coupling constant $g$ is strictly a scale parameter for the system, not influencing qualitative network dynamics. This follows from the homogeneous scaling property of the threshold power-law function: under the change of variables $\mathbf y = g^{1/(p-1)} \mathbf z$, the dynamics become
$$
\frac{d \mathbf y}{dt} = -\mathbf y + W f(\mathbf y)
$$
independent of $g$. As a result, all dynamical solutions—fixed points, periodic orbits, and chaotic attractors—are mapped onto each other by amplitude rescaling as $g$ varies. The qualitative geometry in state space and all stability properties remain invariant across $g > 0$ [2512.01006].

## 3. Dynamics, Chaos, and Absence of "Edge of Chaos" Tuning

Dynamic mean field theory (Kadmon & Sompolinsky, 2015; Omri et al., 2018) establishes that for $p \le 1/2$, chaotic trajectories exist for arbitrarily small nonzero $g$ in the limit $N \to \infty$. In threshold power-law RNNs ($p\neq 1$), due to scale invariance, any chaotic trajectory at some $g^*$ can be mapped to a corresponding chaotic trajectory at any $\bar g$ by the relation
$$
\mathbf z^{(\bar g)}(t) = (\bar g/g^*)^{-1/(p-1)}\,\mathbf z^*(t).
$$
The Lyapunov spectrum, including the maximal Lyapunov exponent $\lambda_{\max}(g)$, remains constant for all $g > 0$. There is consequently no "edge of chaos" bifurcation with respect to $g$, and chaos can be neither tamed nor forced by tuning $g$ in this regime. All dynamical stability types are preserved up to amplitude scaling [2512.01006].

## 4. The Singular ReLU Case: $p=1$

When the power-law exponent $p=1$, producing the ReLU transfer function $f(x) = \max(0, x - \phi)$, the scale transformation that removes $g$ is singular. In this scenario,
$$
\frac{d\mathbf z}{dt} = -\mathbf z + g\, W\, f(\mathbf z)
$$
explicitly retains the coupling strength, and tuning $g$ induces genuine bifurcations between dynamical regimes. As $g$ passes through critical values, the system transitions between quiescent states, periodic orbits, and chaos. Here, both the untrained network's dynamics and the convergence/performance of training algorithms are directly dependent on the value of $g$. The ReLU case is thus exceptional within the threshold power-law family [2512.01006].

## 5. Comparison with Classical Sigmoidal RNNs

Traditional RNNs with sigmoidal activations ($f(z) = \tanh(z)$) do not allow elimination of $g$ via variable rescaling, as the activation is not homogeneous of any degree. The system,
$$
\frac{d\mathbf z}{dt} = -\mathbf z + g\, W\, \tanh(\mathbf z),
$$
exhibits qualitatively distinct dynamical regimes as $g$ varies: for $g < 1$ the network is quiescent, for $g > 1$ it becomes chaotic, and the "edge of chaos" near $g \approx 1$ is empirically favored for reservoir computing. Consequently, training metrics, memory capacity, and algorithmic stability hinge nontrivially on precise $g$ tuning in sigmoidal networks, in marked contrast with the scale-invariant regime of threshold power-law RNNs for $p\neq 1$ [2512.01006].

## 6. Theoretical and Practical Implications for Training

A central result states that if a threshold power-law reservoir ($p\neq 1$) can be successfully trained at a single $g^*$ to approximate any supervised signal (using arbitrary encoder and decoder), there exist rescaled encoders and decoders for any $\bar g > 0$ guaranteeing identical accuracy. Explicitly, the mapping
$$
\hat\phi = \phi\,(\bar g/g^*)^{p/(p-1)}, \quad
\hat\eta = \eta\,(g^*/\bar g)^{1/(p-1)}
$$
preserves the test loss $L^*$. Thus, if chaos is tamed (training converges) for one $g^*$, it is universally possible for all $g > 0$, modulo trivial output/input scaling. This result guarantees "no-tuning" of $g$ in sub-ReLU ($p<1$) threshold power-law reservoir implementations. Empirical evidence from training FORCED-based networks on oscillator and chaotic targets corroborates this theoretical invariance, with test errors remaining constant across a wide $g$ range after rescaling [2512.01006].

## 7. Summary and Broader Significance

Threshold power-law RNNs with $p\neq 1$ exhibit pure scale invariance in their coupling parameter $g$, leading to invariant dynamical structure and trainability under amplitude rescaling. This property sharply contrasts with both ReLU and sigmoidal RNNs, in which $g$ controls stability, chaos, and computational capacity. These findings refine the theoretical understanding of non-sigmoidal RNNs and provide practical design guarantees, especially for reservoir computing, simplifying hyperparameter selection and removing the need to tune the global coupling strength except in the ReLU ($p=1$) regime [2512.01006]. A plausible implication is substantial robustness and efficiency for sub-ReLU threshold power-law reservoir architectures in large-scale, untrained recurrent network applications.

Source: https://www.emergentmind.com/topics/threshold-power-law-recurrent-neural-networks