Almost-Linear RNNs: Design and Tradeoffs

• Almost-Linear RNNs are recurrent models that maintain a linear state transition with controlled, sparse nonlinearity to enable efficient computation and parallelization.
• They include architectures like RWKV and LMN, which balance linear memory cores with minimal nonlinear components for fast inference and dynamical systems analysis.
• Key trade-offs involve managing information bottlenecks and over-fragmentation while exploiting linear recurrence to improve scalability and interpretability.

Searching arXiv for recent and foundational papers on Almost-Linear RNNs and related architectures. Search query: "Almost-Linear RNNs RWKV linear RNN parallelizable fixed-point recurrent neural networks" Almost-Linear RNNs (AL-RNNs) are a class of recurrent sequence models in which the dominant recurrent computation remains linear, affine, or piecewise linear, while nonlinearity is either restricted to carefully delimited components or made sparse enough to preserve tractable computation and analysis. Across recent literature, the term has at least three closely related uses: architectures with transformer-like parallel training and RNN-like linear-time, constant-memory decoding; linear-core recurrent models whose updates admit associative prefix-scan evaluation; and piecewise linear recurrent models with only a small number of nonlinear units, yielding parsimonious symbolic decompositions of dynamical systems (Peng et al., 2023, Merrill et al., 4 Mar 2026, Brenner et al., 2024). This shared emphasis on restricted recurrence places AL-RNNs at the intersection of efficient language modeling, circuit complexity, interpretable dynamical-systems reconstruction, and mechanistic studies of memory and nonlinearity.

1. Terminological scope and defining principles

The term AL-RNN does not denote a single canonical architecture. In the language-modeling and linear-RNN literature, AL-RNNs are recurrent architectures whose core state evolution remains linear in the previous state, thereby admitting parallel prefix-scan evaluation, while allowing carefully placed, limited nonlinearity that does not destroy the associativity required for scan (Merrill et al., 4 Mar 2026). In the RWKV formulation, AL-RNNs are sequence models whose memory and compute scale linearly, or almost linearly, with sequence length $T$ during inference, while retaining the expressive capacity and parallelizable training typical of transformer architectures (Peng et al., 2023). In the dynamical-systems literature, AL-RNNs are recurrent neural networks whose state update is piecewise linear but uses only a small number $P$ of nonlinear (ReLU) units while the remaining $M-P$ units are linear, so that the dynamics decompose into $2^P$ linear subregions (Brenner et al., 2024).

A concise comparison of these usages is given below.

Research strand	Defining recurrent structure	Primary emphasis
RWKV	Exact linear attention with learned per-channel decay and constant-size state	$O(Td)$ time and $O(d)$ space at inference
Linear-core AL-RNN theory	$h_t=A_t h_{t-1}+b_t$ with associative composition	Parallel prefix-scan and circuit complexity
PWL AL-RNNs for DSR	Linear units plus sparse ReLU switching units	Parsimony, symbolic coding, topological analysis
LMN-style models	Strictly linear memory with nonlinear functional transform	Separation of memory and nonlinear computation

This diversity creates a common misconception that AL-RNNs are simply “linear RNNs.” The literature is more specific. The linearity requirement is usually imposed on the recurrent state transition itself, not necessarily on input parameterization, normalization, readout, feedforward sublayers, or piecewise switching structure (Merrill et al., 4 Mar 2026, Bacciu et al., 2018). Conversely, another misconception is that any use of sparse nonlinearity preserves scanability; the theoretical literature explicitly distinguishes safe local nonlinearity from branched dependence on $h_{t-1}$, which breaks associativity and defeats parallel prefix (Merrill et al., 4 Mar 2026).

2. RWKV as a modern AL-RNN for language modeling

RWKV, “Receptance Weighted Key Value,” is the most explicit large-scale language-model instantiation of the AL-RNN idea. It combines the efficient parallelizable training of transformers with the efficient inference of RNNs by reformulating attention as a linear attention mechanism with learned per-channel time decay (Peng et al., 2023). The architecture is built from stacked residual blocks, each containing a time-mixing sub-block that performs attention-like temporal aggregation with learned time decay and a channel-mixing sub-block that acts as an MLP-like component with gating.

RWKV uses token-shift time mixing before projection. For time mixing,

$$\begin{aligned} r_t &= W_r \cdot \big(\mu_r \odot x_t + (1 - \mu_r)\odot x_{t-1}\big), \ k_t &= W_k \cdot \big(\mu_k \odot x_t + (1 - \mu_k)\odot x_{t-1}\big), \ v_t &= W_v \cdot \big(\mu_v \odot x_t + (1 - \mu_v)\odot x_{t-1}\big), \end{aligned}$$

and for channel mixing,

$$\begin{aligned} r'_t &= W'_r \cdot \big(\mu'_r \odot x_t + (1 - \mu'_r)\odot x_{t-1}\big), \ k'_t &= W'_k \cdot \big(\mu'_k \odot x_t + (1 - \mu'_k)\odot x_{t-1}\big). \end{aligned}$$

Its attention-free WKV operator uses per-channel decay weights satisfying

$P$0

with aggregation

$P$1

The output applies a receptance gate,

$P$2

while channel mixing uses squared ReLU and gating,

$P$3

The central AL-RNN property is that the time-mixing block admits an exact recurrent formulation with constant-size state. Defining per-channel states $P$4,

$P$5

A numerically stable version introduces normalized states $P$6 and a shared exponent $P$7 to prevent overflow while preserving exactness. Per layer, the recurrent state consists of the current time-mix input $P$8, the current channel-mix input $P$9, numerator $M-P$0, denominator $M-P$1, and auxiliary exponent $M-P$2, for a total state of $M-P$3 per layer, or $M-P$4 if $M-P$5 is ignored in infinite precision (Peng et al., 2023).

The complexity profile is the clearest operational marker of RWKV’s AL-RNN status. At inference, RWKV achieves $M-P$6 time and $M-P$7 space per layer, compared with $M-P$8 time and $M-P$9 space for a transformer, $2^P$0 time and $2^P$1 space for Reformer, and $2^P$2 time with $2^P$3 space for linear transformers (Peng et al., 2023). During training, the $2^P$4 projections over all tokens are parallelizable with cost $2^P$5, while the WKV scan is $2^P$6 and can be implemented with a custom CUDA kernel; future work noted in the paper suggests parallel scan could reduce the WKV step to $2^P$7.

Empirically, RWKV models from $2^P$8M to $2^P$9B parameters were trained on The Pile ($O(Td)$0B tokens), with Adam, bfloat16, context length $O(Td)$1, exponential learning-rate decay, and an auxiliary PaLM loss encouraging the softmax normalizer to be near zero (Peng et al., 2023). Across twelve tasks—ARC Easy/Challenge, BoolQ, COPA, HeadQA, HellaSwag, LAMBADA, OpenBookQA, PIQA, ReCoRD, SciQ, and Winogrande—RWKV is reported as competitive with similarly sized transformers in matched-FLOP regimes. On Long Range Arena, it performs second only to S4 on five datasets, with strong text-task results including ListOps $O(Td)$2, Text $O(Td)$3, and Retrieval $O(Td)$4, while underperforming S4 on Image, Pathfinder, and Path-X. The paper also reports linear scaling of cumulative generation time and lower CPU/GPU memory usage during generation, as well as favorable Enwik8 bits-per-character behavior in small configurations.

The main limitation stated for RWKV is information funneling: compressing the past into a single per-channel vector can limit exact recall of fine-grained details over very long contexts relative to quadratic self-attention. The same source also notes prompt sensitivity and weaker performance on some non-text long-range tasks (Peng et al., 2023).

3. Associative linear cores, circuit complexity, and the parallelization frontier

A second line of work formalizes why certain almost-linear recurrent models can be parallelized nearly as well as transformers. In this account, a linear RNN updates its recurrent state by

$O(Td)$5

where $O(Td)$6 and $O(Td)$7 may depend on the current token through closed-form arithmetic functions, but the update remains affine in $O(Td)$8 (Merrill et al., 4 Mar 2026). An AL-RNN preserves this associative linear core while permitting limited local nonlinearity—such as layer normalization implemented via closed-form arithmetic at bounded precision, feedforward sublayers between LRNN blocks, or output thresholding—provided that such operations do not make the transition depend on the current state in a branched way.

The crucial algebraic object is the associative combine operator on transition pairs: $O(Td)$9 Associativity follows directly from matrix associativity, and it yields a closed-form prefix solution

$O(d)$0

where $O(d)$1. Because prefixes can be computed by a balanced binary tree or Blelloch scan, all states can be evaluated in depth $O(d)$2 with total work $O(d)$3 (Merrill et al., 4 Mar 2026). This is the sense in which linear-core AL-RNNs are “easy to parallelize in practice as transformers”: the recurrent computation is reducible to a scan over an associative monoid, and scan is a standard primitive on GPUs and TPUs.

The complexity-theoretic results place this observation on a sharper footing. The paper proves that LRNNs over $O(d)$4 lie in $O(d)$5, namely bounded fan-in, poly-size arithmetic circuits of depth $O(d)$6 with positivity test at the output. It further states that transformers are in $O(d)$7, so LRNNs incur at most an additional $O(d)$8 factor in depth relative to transformers (Merrill et al., 4 Mar 2026). Under log precision, the upper bound sharpens to $O(d)$9.

This paper’s negative results are equally central to the AL-RNN concept. A general nonlinear RNN with state-dependent branching can recognize much harder languages. Under polynomial precision, a one-layer MLP-gated RNN can simulate multi-stack machines; the paper states a Turing-completeness theorem and a corollary that there exists a one-layer MLP RNN whose language is $h_t=A_t h_{t-1}+b_t$0-complete under FO reductions. Under log precision, a one-layer log-precision MLP RNN can solve sorted deterministic graph connectivity, an $h_t=A_t h_{t-1}+b_t$1-complete problem (Merrill et al., 4 Mar 2026). This yields a precise barrier: nonlinear recurrence may be more expressive, but that same state-dependent branching is what resists efficient parallelization.

The same framework distinguishes common linear variants. Permutation-diagonal LRNNs, with $h_t=A_t h_{t-1}+b_t$2, are shown to be $h_t=A_t h_{t-1}+b_t$3-complete and, in the single-layer case, exactly equivalent to deterministic weighted finite automata. Diagonal-plus-low-rank LRNNs, exemplified by DeltaNet and RWKV-7 forms, are shown to be $h_t=A_t h_{t-1}+b_t$4-complete; the paper gives 4-layer RWKV-7 and 4-layer DeltaNet constructions for iterated $h_t=A_t h_{t-1}+b_t$5 matrix multiplication, and states that DPLR LRNNs can simulate weighted finite automata in four layers (Merrill et al., 4 Mar 2026). This suggests a principled AL-RNN design rule: preserve the affine associative core, but enrich $h_t=A_t h_{t-1}+b_t$6 structurally—PD for exact state tracking with minimal overhead, DPLR for stronger arithmetic aggregation.

4. Piecewise-linear AL-RNNs for dynamical-systems reconstruction

In a distinct but related usage, AL-RNNs are introduced as parsimonious piecewise-linear recurrent models for reconstructing nonlinear dynamical systems from time-series data. Here the latent state $h_t=A_t h_{t-1}+b_t$7 evolves according to

$h_t=A_t h_{t-1}+b_t$8

where $h_t=A_t h_{t-1}+b_t$9 is diagonal, $h_{t-1}$0 is dense, $h_{t-1}$1 is a bias, and only the last $h_{t-1}$2 coordinates of $h_{t-1}$3 are nonlinear: $h_{t-1}$4 Thus the state update is piecewise linear, but the nonlinearity budget is explicitly controlled by $h_{t-1}$5 (Brenner et al., 2024).

The induced partition of state space is defined by the sign pattern of the $h_{t-1}$6 ReLU coordinates. If $h_{t-1}$7 is the diagonal indicator matrix of active coordinates, then

$h_{t-1}$8

There are at most $h_{t-1}$9 distinct diagonal patterns, each corresponding to a linear subregion. The switching manifolds are the hyperplanes

$$\begin{aligned} r_t &= W_r \cdot \big(\mu_r \odot x_t + (1 - \mu_r)\odot x_{t-1}\big), \ k_t &= W_k \cdot \big(\mu_k \odot x_t + (1 - \mu_k)\odot x_{t-1}\big), \ v_t &= W_v \cdot \big(\mu_v \odot x_t + (1 - \mu_v)\odot x_{t-1}\big), \end{aligned}$$0

for the ReLU coordinates, and the regions $$\begin{aligned} r_t &= W_r \cdot \big(\mu_r \odot x_t + (1 - \mu_r)\odot x_{t-1}\big), \ k_t &= W_k \cdot \big(\mu_k \odot x_t + (1 - \mu_k)\odot x_{t-1}\big), \ v_t &= W_v \cdot \big(\mu_v \odot x_t + (1 - \mu_v)\odot x_{t-1}\big), \end{aligned}$$1 are indexed by binary sign codes $$\begin{aligned} r_t &= W_r \cdot \big(\mu_r \odot x_t + (1 - \mu_r)\odot x_{t-1}\big), \ k_t &= W_k \cdot \big(\mu_k \odot x_t + (1 - \mu_k)\odot x_{t-1}\big), \ v_t &= W_v \cdot \big(\mu_v \odot x_t + (1 - \mu_v)\odot x_{t-1}\big), \end{aligned}$$2 (Brenner et al., 2024).

A major contribution of this formulation is a symbolic encoding of dynamics. Assigning a symbol $$\begin{aligned} r_t &= W_r \cdot \big(\mu_r \odot x_t + (1 - \mu_r)\odot x_{t-1}\big), \ k_t &= W_k \cdot \big(\mu_k \odot x_t + (1 - \mu_k)\odot x_{t-1}\big), \ v_t &= W_v \cdot \big(\mu_v \odot x_t + (1 - \mu_v)\odot x_{t-1}\big), \end{aligned}$$3 to each region $$\begin{aligned} r_t &= W_r \cdot \big(\mu_r \odot x_t + (1 - \mu_r)\odot x_{t-1}\big), \ k_t &= W_k \cdot \big(\mu_k \odot x_t + (1 - \mu_k)\odot x_{t-1}\big), \ v_t &= W_v \cdot \big(\mu_v \odot x_t + (1 - \mu_v)\odot x_{t-1}\big), \end{aligned}$$4, a trajectory $$\begin{aligned} r_t &= W_r \cdot \big(\mu_r \odot x_t + (1 - \mu_r)\odot x_{t-1}\big), \ k_t &= W_k \cdot \big(\mu_k \odot x_t + (1 - \mu_k)\odot x_{t-1}\big), \ v_t &= W_v \cdot \big(\mu_v \odot x_t + (1 - \mu_v)\odot x_{t-1}\big), \end{aligned}$$5 induces a symbolic sequence $$\begin{aligned} r_t &= W_r \cdot \big(\mu_r \odot x_t + (1 - \mu_r)\odot x_{t-1}\big), \ k_t &= W_k \cdot \big(\mu_k \odot x_t + (1 - \mu_k)\odot x_{t-1}\big), \ v_t &= W_v \cdot \big(\mu_v \odot x_t + (1 - \mu_v)\odot x_{t-1}\big), \end{aligned}$$6 with $$\begin{aligned} r_t &= W_r \cdot \big(\mu_r \odot x_t + (1 - \mu_r)\odot x_{t-1}\big), \ k_t &= W_k \cdot \big(\mu_k \odot x_t + (1 - \mu_k)\odot x_{t-1}\big), \ v_t &= W_v \cdot \big(\mu_v \odot x_t + (1 - \mu_v)\odot x_{t-1}\big), \end{aligned}$$7 iff $$\begin{aligned} r_t &= W_r \cdot \big(\mu_r \odot x_t + (1 - \mu_r)\odot x_{t-1}\big), \ k_t &= W_k \cdot \big(\mu_k \odot x_t + (1 - \mu_k)\odot x_{t-1}\big), \ v_t &= W_v \cdot \big(\mu_v \odot x_t + (1 - \mu_v)\odot x_{t-1}\big), \end{aligned}$$8, and the left-shift map $$\begin{aligned} r_t &= W_r \cdot \big(\mu_r \odot x_t + (1 - \mu_r)\odot x_{t-1}\big), \ k_t &= W_k \cdot \big(\mu_k \odot x_t + (1 - \mu_k)\odot x_{t-1}\big), \ v_t &= W_v \cdot \big(\mu_v \odot x_t + (1 - \mu_v)\odot x_{t-1}\big), \end{aligned}$$9 acts on these sequences. Under assumptions of uniform hyperbolicity in each subregion and non-global divergence, the paper states that asymptotically fixed orbits correspond exactly to eventually fixed symbolic sequences, asymptotically $$\begin{aligned} r'_t &= W'_r \cdot \big(\mu'_r \odot x_t + (1 - \mu'_r)\odot x_{t-1}\big), \ k'_t &= W'_k \cdot \big(\mu'_k \odot x_t + (1 - \mu'_k)\odot x_{t-1}\big). \end{aligned}$$0-periodic orbits correspond to eventually $$\begin{aligned} r'_t &= W'_r \cdot \big(\mu'_r \odot x_t + (1 - \mu'_r)\odot x_{t-1}\big), \ k'_t &= W'_k \cdot \big(\mu'_k \odot x_t + (1 - \mu'_k)\odot x_{t-1}\big). \end{aligned}$$1-periodic sequences, and asymptotically aperiodic orbits correspond to aperiodic symbolic sequences (Brenner et al., 2024). This makes the sparse switching structure not only interpretable but topologically meaningful.

Training uses identity teacher forcing with a simple observation model, $$\begin{aligned} r'_t &= W'_r \cdot \big(\mu'_r \odot x_t + (1 - \mu'_r)\odot x_{t-1}\big), \ k'_t &= W'_k \cdot \big(\mu'_k \odot x_t + (1 - \mu'_k)\odot x_{t-1}\big). \end{aligned}$$2, and an MSE loss

$$\begin{aligned} r'_t &= W'_r \cdot \big(\mu'_r \odot x_t + (1 - \mu'_r)\odot x_{t-1}\big), \ k'_t &= W'_k \cdot \big(\mu'_k \odot x_t + (1 - \mu'_k)\odot x_{t-1}\big). \end{aligned}$$3

An optional parsimony regularizer replaces ReLU with leaky-ReLU slopes $$\begin{aligned} r'_t &= W'_r \cdot \big(\mu'_r \odot x_t + (1 - \mu'_r)\odot x_{t-1}\big), \ k'_t &= W'_k \cdot \big(\mu'_k \odot x_t + (1 - \mu'_k)\odot x_{t-1}\big). \end{aligned}$$4 and penalizes deviations from linearity,

$$\begin{aligned} r'_t &= W'_r \cdot \big(\mu'_r \odot x_t + (1 - \mu'_r)\odot x_{t-1}\big), \ k'_t &= W'_k \cdot \big(\mu'_k \odot x_t + (1 - \mu'_k)\odot x_{t-1}\big). \end{aligned}$$5

Optimization uses RADAM; initialization sets $$\begin{aligned} r'_t &= W'_r \cdot \big(\mu'_r \odot x_t + (1 - \mu'_r)\odot x_{t-1}\big), \ k'_t &= W'_k \cdot \big(\mu'_k \odot x_t + (1 - \mu'_k)\odot x_{t-1}\big). \end{aligned}$$6 Gaussian with $$\begin{aligned} r'_t &= W'_r \cdot \big(\mu'_r \odot x_t + (1 - \mu'_r)\odot x_{t-1}\big), \ k'_t &= W'_k \cdot \big(\mu'_k \odot x_t + (1 - \mu'_k)\odot x_{t-1}\big). \end{aligned}$$7, $$\begin{aligned} r'_t &= W'_r \cdot \big(\mu'_r \odot x_t + (1 - \mu'_r)\odot x_{t-1}\big), \ k'_t &= W'_k \cdot \big(\mu'_k \odot x_t + (1 - \mu'_k)\odot x_{t-1}\big). \end{aligned}$$8, and $$\begin{aligned} r'_t &= W'_r \cdot \big(\mu'_r \odot x_t + (1 - \mu'_r)\odot x_{t-1}\big), \ k'_t &= W'_k \cdot \big(\mu'_k \odot x_t + (1 - \mu'_k)\odot x_{t-1}\big). \end{aligned}$$9 from a normalized positive-definite random matrix, with the initial state $P$00 and learned $P$01 (Brenner et al., 2024).

Empirically, this AL-RNN formulation is reported to discover topologically minimal piecewise-linear reconstructions for chaotic systems. For Lorenz-63, $P$02 effectively yields three key subregions: two unstable spirals in the lobes separated by a saddle region near the origin. For Rössler, two subregions suffice: one unstable spiral in the $P$03 plane and a “half-spiral” in $P$04 (Brenner et al., 2024). On long Rössler simulations with $P$05, the paper reports $P$06 and $P$07, with visited regions saturating well below $P$08. On ECG, with $P$09, the model captures nearly periodic yet chaotic patterns and yields a maximum Lyapunov estimate $P$10 versus ground truth $P$11. On fMRI, with $P$12 and approximately $P$13 regions, the model captures local dynamics, while a task-alignment setting with $P$14 and readouts reset every $P$15 steps achieves average classification accuracy $P$16 across subjects (Brenner et al., 2024).

The paper contrasts this formulation with SLDS, classical piecewise-affine identification, standard PLRNNs, reservoir computing, Neural ODE/SDE models, Koopman approaches, and SINDy. Its main claim is not merely that sparse ReLU recurrence is sufficient for reconstruction, but that few nonlinearities make the switching structure explicit, reduce the number of visited regions, and facilitate fixed-point, cycle, and symbolic-graph analysis (Brenner et al., 2024).

5. Linear memory networks and the separation of memory from nonlinear computation

An earlier precursor to the AL-RNN design philosophy is the Linear Memory Network (LMN), which explicitly separates nonlinear input-output transformation from strictly linear recurrent memory. The functional component computes

$P$17

while the memory component follows the strictly linear recurrence

$P$18

Two output variants are defined: $P$19 The paper states that “the memory is entirely linear while the feedforward component allows to model nonlinear dependencies” (Bacciu et al., 2018).

The memory mechanism is derived from a linear autoencoder for sequences. Given inputs $P$20, the autoencoder state satisfies

$P$21

with reconstruction

$P$22

Training proceeds by forming a matrix of reversed subsequences, performing a truncated SVD, and constructing $P$23, $P$24, and $P$25 in closed form. The paper states that “the training algorithm guarantees an optimal encoding when $P$26” (Bacciu et al., 2018). In LMNs, this autoencoder is trained on hidden-state sequences rather than raw inputs, so the linear memory stores a compressed history of nonlinear functional activations.

A dedicated pretraining scheme exploits equivalence between unfolded feedforward networks and recurrent networks. First, an unfolded finite-horizon model is trained to produce hidden activations $P$27. Second, a linear autoencoder is fitted to the sequence of hidden states. Third, the LMN parameters are initialized by transferring unfolded and autoencoder parameters, with

$P$28

and corresponding formulas for $P$29 and $P$30 (Bacciu et al., 2018). This is an important historical design lesson for AL-RNNs: the recurrence can remain linear and analyzable, while the nonlinearity is concentrated in tokenwise or feedforward maps that consume the memory rather than redefining it.

The empirical results reported for polyphonic music datasets show that LMN-B is competitive with gated recurrent networks and other state-of-the-art models. Test frame-level accuracies include JSB Chorales $P$31 for LMN-B and $P$32 for pret-LMN-B, MuseData $P$33 and $P$34, Nottingham $P$35 and $P$36, and Piano MIDI $P$37 and $P$38 (Bacciu et al., 2018). The paper further states that, for a fixed total hidden size, LMN requires fewer parameters than LSTM or GRU, and highlights the absence of “unwanted exponential decay effects due to the presence of gates.” A stated limitation is that if the memory component is trained only with backpropagation, there are no theoretical guarantees on memorization properties; burn-in behavior at the beginning of sequences is also observed (Bacciu et al., 2018).

6. Minimal nonlinearity as a computational resource

Later work uses AL-RNNs not only as models but as probes for the functional role of nonlinearity in memory. In this formulation, the state update is

$P$39

where the first $P$40 units are linear, the last $P$41 units are ReLU, and

$P$42

The principal measure of nonlinearity is the number $P$43 of PWL units. Effective use of nonlinearity is quantified by a bitcode

$P$44

so that small support of $P$45 indicates that the model uses few subregions (Brenner et al., 9 Jun 2025).

This work introduces Manifold Attractor Regularization (MAR),

$P$46

to encourage near-perfect integrators and long time scales. Within a given bitcode-defined subregion, the masked interaction matrix is $P$47, the local Jacobian is

$P$48

fixed points satisfy

$P$49

and local stability is determined by $P$50. The same paper analyzes Lyapunov exponents and gradient flow through $P$51 powers, making the piecewise-linear structure a direct vehicle for mechanistic interpretation (Brenner et al., 9 Jun 2025).

Across tasks, the reported findings support a “minimal nonlinearity” thesis. On IMDb, performance is minimally affected by $P$52 and dynamics are dominated by a single slow mode. On sequential MNIST, linear models are already strong, with maximum eigenvalues slightly above $P$53, specifically $P$54; accuracy improves with intermediate $P$55 and reaches up to $P$56, while fully nonlinear models degrade. The paper states that for $P$57, final states within a class share nearly identical bitcodes, with average variation $P$58 (Brenner et al., 9 Jun 2025).

Tasks that require explicit routing show a sharper need for nonlinearity. On the copy task with $P$59, $P$60, and $P$61, purely linear models perform above chance but are fragile; $P$62 often yields perfect recall, and the mechanism is described as a two-subregion separation between encoding/decoding and autonomous delay-phase transients, with a $P$63-step global $P$64-cycle and neutral Lyapunov exponent $P$65 (Brenner et al., 9 Jun 2025). On the addition problem, linear models fail, while $P$66 drastically reduces error and performance plateaus near $P$67. On contextual multistability, linear models remain at $P$68, whereas $P$69 solves the task nearly perfectly, up to $P$70 accuracy. In a joint task-plus-spiking model for CRCNS PFC-1, nonlinearity improves both task accuracy and spike reconstruction, with mean spike correlation $P$71 close to a Poisson upper bound of $P$72 (Brenner et al., 9 Jun 2025).

A related development, Fixed-Point RNNs, addresses a different expressive bottleneck: dense state mixing. The target dense linear RNN

$P$73

is realized as the fixed point of a parallelizable diagonal recurrence

$P$74

If $P$75, then $P$76, recovering a dense transition from diagonal scans plus channel mixing (Movahedi et al., 13 Mar 2025). Bedouin accelerates the iteration by letting each fixed-point step sweep through time,

$P$77

The paper states that a single-layer Bedouin solves $P$78 and $P$79, succeeds on copying with $P$80 length generalization where Mamba and Mamba-2 struggle, and that a 2-layer Bedouin with $P$81 reaches $P$82 scaled accuracy on modular arithmetic with brackets (Movahedi et al., 13 Mar 2025). This line extends the AL-RNN program from sparse nonlinearity to dense expressivity emerging from fixed-point solutions of linear scan operators.

7. Limitations, design tradeoffs, and emerging synthesis

Across these strands, AL-RNNs are defined less by a single architecture than by a recurring tradeoff: preserve a recurrent core that is linear, affine, piecewise linear, or otherwise scan-compatible, and use nonlinearity sparingly enough that parallelism, stability, or interpretability are not lost. The major benefit is computational. RWKV yields $P$83 inference with constant per-layer state (Peng et al., 2023). Linear-core AL-RNN theory yields $P$84 scan depth and a near-transformer parallel depth overhead (Merrill et al., 4 Mar 2026). Piecewise-linear AL-RNNs yield explicit symbolic partitions and tractable local linear analysis (Brenner et al., 2024). LMNs yield closed-form memory training and an explicit separation of memory from nonlinear transformation (Bacciu et al., 2018). Minimal-nonlinearity AL-RNNs expose when gating, attractor switching, neutral cycles, or context-dependent routing are computationally necessary (Brenner et al., 9 Jun 2025).

The limitations are similarly consistent. Linear or compressed state summaries can create information bottlenecks, as stated explicitly for RWKV (Peng et al., 2023). Piecewise-linear approaches can overfragment when $P$85 is too large and underfit when $P$86 is too small (Brenner et al., 2024). Minimal-nonlinearity studies report degraded generalization in fully nonlinear regimes because fragmented subregion usage destabilizes learning (Brenner et al., 9 Jun 2025). Circuit-theoretic work emphasizes that the very source of extra power in nonlinear RNNs—state-dependent branching—is also what blocks efficient parallelization under standard complexity conjectures (Merrill et al., 4 Mar 2026). Fixed-point linear densification introduces its own tradeoff between contraction and capacity, and its convergence speed remains an open issue in the worst case (Movahedi et al., 13 Mar 2025).

A reasonable synthesis suggested by these papers is that AL-RNNs define an expressivity/parallelism/interpretablity frontier rather than a single model family. One end of that frontier is occupied by exact linear-recurrence scan models such as RWKV and other LRNNs; another by sparse-switching PWL systems for dynamical reconstruction; another by architectures such as LMNs and Fixed-Point RNNs that preserve linear memory or linear scans while relocating or restructuring nonlinearity (Peng et al., 2023, Merrill et al., 4 Mar 2026, Brenner et al., 2024, Bacciu et al., 2018, Movahedi et al., 13 Mar 2025). This suggests that “almost-linear” is best understood as a design discipline: keep the recurrent backbone as close as possible to associative linear evolution, and spend nonlinearity only where gating, switching, compositional routing, or task-dependent context truly require it.