---
title: 'Almost-Linear RNNs: Design and Tradeoffs'
url: https://www.emergentmind.com/topics/almost-linear-rnns-al-rnns
type: topic
---

# Almost-Linear RNNs: Design and Tradeoffs

Searching arXiv for recent and foundational papers on Almost-Linear RNNs and related architectures.
Search query: "Almost-Linear RNNs RWKV linear RNN parallelizable fixed-point recurrent neural networks"
Almost-Linear RNNs (AL-RNNs) are a class of recurrent sequence models in which the dominant recurrent computation remains linear, affine, or piecewise linear, while nonlinearity is either restricted to carefully delimited components or made sparse enough to preserve tractable computation and analysis. Across recent literature, the term has at least three closely related uses: architectures with transformer-like parallel training and RNN-like linear-time, constant-memory decoding; linear-core recurrent models whose updates admit associative prefix-scan evaluation; and piecewise linear recurrent models with only a small number of nonlinear units, yielding parsimonious symbolic decompositions of dynamical systems [2305.13048] [2603.03612] [2410.14240]. This shared emphasis on restricted recurrence places AL-RNNs at the intersection of efficient language modeling, circuit complexity, interpretable dynamical-systems reconstruction, and mechanistic studies of memory and nonlinearity.

## 1. Terminological scope and defining principles

The term AL-RNN does not denote a single canonical architecture. In the language-modeling and linear-RNN literature, AL-RNNs are recurrent architectures whose core state evolution remains linear in the previous state, thereby admitting parallel prefix-scan evaluation, while allowing carefully placed, limited nonlinearity that does not destroy the associativity required for scan [2603.03612]. In the RWKV formulation, AL-RNNs are sequence models whose memory and compute scale linearly, or almost linearly, with sequence length $T$ during inference, while retaining the expressive capacity and parallelizable training typical of transformer architectures [2305.13048]. In the dynamical-systems literature, AL-RNNs are recurrent neural networks whose state update is piecewise linear but uses only a small number $P$ of nonlinear (ReLU) units while the remaining $M-P$ units are linear, so that the dynamics decompose into $2^P$ linear subregions [2410.14240].

A concise comparison of these usages is given below.

| Research strand | Defining recurrent structure | Primary emphasis |
|---|---|---|
| RWKV | Exact linear attention with learned per-channel decay and constant-size state | $O(Td)$ time and $O(d)$ space at inference |
| Linear-core AL-RNN theory | $h_t=A_t h_{t-1}+b_t$ with associative composition | Parallel prefix-scan and circuit complexity |
| PWL AL-RNNs for DSR | Linear units plus sparse ReLU switching units | Parsimony, symbolic coding, topological analysis |
| LMN-style models | Strictly linear memory with nonlinear functional transform | Separation of memory and nonlinear computation |

This diversity creates a common misconception that AL-RNNs are simply “linear RNNs.” The literature is more specific. The linearity requirement is usually imposed on the recurrent state transition itself, not necessarily on input parameterization, normalization, readout, feedforward sublayers, or piecewise switching structure [2603.03612] [1811.03356]. Conversely, another misconception is that any use of sparse nonlinearity preserves scanability; the theoretical literature explicitly distinguishes safe local nonlinearity from branched dependence on $h_{t-1}$, which breaks associativity and defeats parallel prefix [2603.03612].

## 2. RWKV as a modern AL-RNN for language modeling

RWKV, “Receptance Weighted Key Value,” is the most explicit large-scale language-model instantiation of the AL-RNN idea. It combines the efficient parallelizable training of transformers with the efficient inference of RNNs by reformulating attention as a linear attention mechanism with learned per-channel time decay [2305.13048]. The architecture is built from stacked residual blocks, each containing a time-mixing sub-block that performs attention-like temporal aggregation with learned time decay and a channel-mixing sub-block that acts as an MLP-like component with gating.

RWKV uses token-shift time mixing before projection. For time mixing,
\[
\begin{aligned}
r_t &= W_r \cdot \big(\mu_r \odot x_t + (1 - \mu_r)\odot x_{t-1}\big), \\
k_t &= W_k \cdot \big(\mu_k \odot x_t + (1 - \mu_k)\odot x_{t-1}\big), \\
v_t &= W_v \cdot \big(\mu_v \odot x_t + (1 - \mu_v)\odot x_{t-1}\big),
\end{aligned}
\]
and for channel mixing,
\[
\begin{aligned}
r'_t &= W'_r \cdot \big(\mu'_r \odot x_t + (1 - \mu'_r)\odot x_{t-1}\big), \\
k'_t &= W'_k \cdot \big(\mu'_k \odot x_t + (1 - \mu'_k)\odot x_{t-1}\big).
\end{aligned}
\]
Its attention-free WKV operator uses per-channel decay weights satisfying
\[
w_{t,i}=-(t-i)w,\quad w\in\mathbb{R}_{\ge 0}^d,
\]
with aggregation
\[
wkv_t \;=\; \frac{ \sum_{i=1}^{t-1} e^{-(t-1-i)w+k_i} \odot v_i \;+\; e^{u+k_t} \odot v_t }{ \sum_{i=1}^{t-1} e^{-(t-1-i)w+k_i} \;+\; e^{u+k_t} }.
\]
The output applies a receptance gate,
\[
o_t \;=\; W_o \cdot \big(\sigma(r_t) \odot wkv_t\big),
\]
while channel mixing uses squared ReLU and gating,
\[
o'_t \;=\; \sigma(r'_t) \odot \big(W'_v \cdot \max(k'_t, 0)^2\big).
\]

The central AL-RNN property is that the time-mixing block admits an exact recurrent formulation with constant-size state. Defining per-channel states $a_t,b_t\in\mathbb{R}^d$,
\[
\begin{aligned}
a_0, b_0 &= 0, \\
wkv_t &= \frac{a_{t-1} + e^{u + k_t} \odot v_t}{b_{t-1} + e^{u + k_t}}, \\
a_t &= e^{-w} \odot a_{t-1} + e^{k_t} \odot v_t, \\
b_t &= e^{-w} \odot b_{t-1} + e^{k_t}.
\end{aligned}
\]
A numerically stable version introduces normalized states $a'_t,b'_t$ and a shared exponent $p_t$ to prevent overflow while preserving exactness. Per layer, the recurrent state consists of the current time-mix input $x_t$, the current channel-mix input $y_t$, numerator $a'_t$, denominator $b'_t$, and auxiliary exponent $p_t$, for a total state of $5D$ per layer, or $4D$ if $p_t$ is ignored in infinite precision [2305.13048].

The complexity profile is the clearest operational marker of RWKV’s AL-RNN status. At inference, RWKV achieves $O(Td)$ time and $O(d)$ space per layer, compared with $O(T^2d)$ time and $O(T^2+Td)$ space for a transformer, $O(T\log T\cdot d)$ time and $O(T\log T+Td)$ space for Reformer, and $O(Td^2)$ time with $O(Td+d^2)$ space for linear transformers [2305.13048]. During training, the $r/k/v/o$ projections over all tokens are parallelizable with cost $O(BTd^2)$, while the WKV scan is $O(BTd)$ and can be implemented with a custom CUDA kernel; future work noted in the paper suggests parallel scan could reduce the WKV step to $O(B\log T\cdot d)$.

Empirically, RWKV models from $169$M to $14$B parameters were trained on The Pile ($330$B tokens), with Adam, bfloat16, context length $1024$, exponential learning-rate decay, and an auxiliary PaLM loss encouraging the softmax normalizer to be near zero [2305.13048]. Across twelve tasks—ARC Easy/Challenge, BoolQ, COPA, HeadQA, HellaSwag, LAMBADA, OpenBookQA, PIQA, ReCoRD, SciQ, and Winogrande—RWKV is reported as competitive with similarly sized transformers in matched-FLOP regimes. On Long Range Arena, it performs second only to S4 on five datasets, with strong text-task results including ListOps $55.88$, Text $86.04$, and Retrieval $88.34$, while underperforming S4 on Image, Pathfinder, and Path-X. The paper also reports linear scaling of cumulative generation time and lower CPU/GPU memory usage during generation, as well as favorable Enwik8 bits-per-character behavior in small configurations.

The main limitation stated for RWKV is information funneling: compressing the past into a single per-channel vector can limit exact recall of fine-grained details over very long contexts relative to quadratic self-attention. The same source also notes prompt sensitivity and weaker performance on some non-text long-range tasks [2305.13048].

## 3. Associative linear cores, circuit complexity, and the parallelization frontier

A second line of work formalizes why certain almost-linear recurrent models can be parallelized nearly as well as transformers. In this account, a linear RNN updates its recurrent state by
\[
h_t = A_t h_{t-1} + b_t,
\]
where $A_t$ and $b_t$ may depend on the current token through closed-form arithmetic functions, but the update remains affine in $h_{t-1}$ [2603.03612]. An AL-RNN preserves this associative linear core while permitting limited local nonlinearity—such as layer normalization implemented via closed-form arithmetic at bounded precision, feedforward sublayers between LRNN blocks, or output thresholding—provided that such operations do not make the transition depend on the current state in a branched way.

The crucial algebraic object is the associative combine operator on transition pairs:
\[
(A_2,b_2)\circ(A_1,b_1):=(A_2A_1,\;A_2b_1+b_2).
\]
Associativity follows directly from matrix associativity, and it yields a closed-form prefix solution
\[
h_t = \Pi_t h_0 + \sum_{i=1}^t (A_tA_{t-1}\cdots A_{i+1})b_i,
\]
where $\Pi_t=A_tA_{t-1}\cdots A_1$. Because prefixes can be computed by a balanced binary tree or Blelloch scan, all states can be evaluated in depth $O(\log T)$ with total work $O(T)$ [2603.03612]. This is the sense in which linear-core AL-RNNs are “easy to parallelize in practice as transformers”: the recurrent computation is reducible to a scan over an associative monoid, and scan is a standard primitive on GPUs and TPUs.

The complexity-theoretic results place this observation on a sharper footing. The paper proves that LRNNs over $\mathbb{Q}$ lie in $\mathsf{PNC}^1$, namely bounded fan-in, poly-size arithmetic circuits of depth $O(\log n)$ with positivity test at the output. It further states that transformers are in $\mathsf{TC}^0\subseteq\mathsf{NC}^1$, so LRNNs incur at most an additional $O(\log^* n)$ factor in depth relative to transformers [2603.03612]. Under log precision, the upper bound sharpens to $\mathsf{AC}^0[\mathsf{ENC}^1]$.

This paper’s negative results are equally central to the AL-RNN concept. A general nonlinear RNN with state-dependent branching can recognize much harder languages. Under polynomial precision, a one-layer MLP-gated RNN can simulate multi-stack machines; the paper states a Turing-completeness theorem and a corollary that there exists a one-layer MLP RNN whose language is $\mathsf{P}$-complete under FO reductions. Under log precision, a one-layer log-precision MLP RNN can solve sorted deterministic graph connectivity, an $\mathsf{L}$-complete problem [2603.03612]. This yields a precise barrier: nonlinear recurrence may be more expressive, but that same state-dependent branching is what resists efficient parallelization.

The same framework distinguishes common linear variants. Permutation-diagonal LRNNs, with $A_t=P_tD_t$, are shown to be $\mathsf{NC}^1$-complete and, in the single-layer case, exactly equivalent to deterministic weighted finite automata. Diagonal-plus-low-rank LRNNs, exemplified by DeltaNet and RWKV-7 forms, are shown to be $\mathsf{PNC}^1$-complete; the paper gives 4-layer RWKV-7 and 4-layer DeltaNet constructions for iterated $3\times 3$ matrix multiplication, and states that DPLR LRNNs can simulate weighted finite automata in four layers [2603.03612]. This suggests a principled AL-RNN design rule: preserve the affine associative core, but enrich $A_t$ structurally—PD for exact state tracking with minimal overhead, DPLR for stronger arithmetic aggregation.

## 4. Piecewise-linear AL-RNNs for dynamical-systems reconstruction

In a distinct but related usage, AL-RNNs are introduced as parsimonious piecewise-linear recurrent models for reconstructing nonlinear dynamical systems from time-series data. Here the latent state $\mathbf{z}_t\in\mathbb{R}^M$ evolves according to
\[
\mathbf{z}_{t} \;=\; \mathbf{A}\,\mathbf{z}_{t-1}\;+\;\mathbf{W}\,\Phi^*(\mathbf{z}_{t-1})\;+\;\mathbf{h},
\]
where $\mathbf{A}\in\mathbb{R}^{M\times M}$ is diagonal, $\mathbf{W}\in\mathbb{R}^{M\times M}$ is dense, $\mathbf{h}\in\mathbb{R}^M$ is a bias, and only the last $P$ coordinates of $\Phi^*$ are nonlinear:
\[
\Phi^*(\mathbf{z}_{t}) \;=\; \big[z_{1,t},\ldots,z_{M-P,t},\;\max(0,z_{M-P+1,t}),\ldots,\max(0,z_{M,t})\big]^\top.
\]
Thus the state update is piecewise linear, but the nonlinearity budget is explicitly controlled by $P$ [2410.14240].

The induced partition of state space is defined by the sign pattern of the $P$ ReLU coordinates. If $D_{\mathcal{O}(t)}$ is the diagonal indicator matrix of active coordinates, then
\[
\mathbf{z}_{t} \;=\; \big(\mathbf{A}+\mathbf{W}\,\mathbf{D}_{\mathcal{O}(t-1)}\big)\,\mathbf{z}_{t-1}\;+\;\mathbf{h}
\;:=\; \mathbf{F}_{\mathcal{O}(t-1)}\,\mathbf{z}_{t-1}\;+\;\mathbf{h}.
\]
There are at most $2^P$ distinct diagonal patterns, each corresponding to a linear subregion. The switching manifolds are the hyperplanes
\[
\Sigma_m = \{\mathbf{z}\in\mathbb{R}^M \mid z_m=0\},
\]
for the ReLU coordinates, and the regions $U_e$ are indexed by binary sign codes $e\in\{0,1\}^P$ [2410.14240].

A major contribution of this formulation is a symbolic encoding of dynamics. Assigning a symbol $a_e$ to each region $U_e$, a trajectory $\{\mathbf{z}_t\}$ induces a symbolic sequence $(a_t)_{t\in\mathbb{N}}$ with $a_t=a_e$ iff $\mathbf{z}_t\in U_e$, and the left-shift map $\sigma$ acts on these sequences. Under assumptions of uniform hyperbolicity in each subregion and non-global divergence, the paper states that asymptotically fixed orbits correspond exactly to eventually fixed symbolic sequences, asymptotically $p$-periodic orbits correspond to eventually $p$-periodic sequences, and asymptotically aperiodic orbits correspond to aperiodic symbolic sequences [2410.14240]. This makes the sparse switching structure not only interpretable but topologically meaningful.

Training uses identity teacher forcing with a simple observation model, $\hat{\mathbf{x}}_t=\mathcal{I}\mathbf{z}_t$, and an MSE loss
\[
\ell_{\mathrm{MSE}}(\hat{\mathbf{X}},\mathbf{X})=\frac{1}{NT}\sum_{t=1}^T \|\hat{\mathbf{x}}_t-\mathbf{x}_t\|_2^2.
\]
An optional parsimony regularizer replaces ReLU with leaky-ReLU slopes $\alpha_i=\sigma(\gamma_i)$ and penalizes deviations from linearity,
\[
\mathcal{L}_{\mathrm{lin}}=\lambda_{\mathrm{lin}}\sum_{i=1}^M |\alpha_i-1|.
\]
Optimization uses RADAM; initialization sets $\mathbf{W}$ Gaussian with $\sigma=0.01$, $\mathbf{h}=0$, and $\mathbf{A}$ from a normalized positive-definite random matrix, with the initial state $\mathbf{z}_1=[\mathbf{x}_1,\mathbf{L}\mathbf{x}_1]^\top$ and learned $\mathbf{L}$ [2410.14240].

Empirically, this AL-RNN formulation is reported to discover topologically minimal piecewise-linear reconstructions for chaotic systems. For Lorenz-63, $P=3$ effectively yields three key subregions: two unstable spirals in the lobes separated by a saddle region near the origin. For Rössler, two subregions suffice: one unstable spiral in the $(x,y)$ plane and a “half-spiral” in $z$ [2410.14240]. On long Rössler simulations with $P=10$, the paper reports $D_{\mathrm{stsp}}=0.08$ and $D_H=0.06$, with visited regions saturating well below $2^P$. On ECG, with $P=2$, the model captures nearly periodic yet chaotic patterns and yields a maximum Lyapunov estimate $\lambda_{\max}\approx 1.96\,\mathrm{s}^{-1}$ versus ground truth $\approx 2.19\,\mathrm{s}^{-1}$. On fMRI, with $P=3$ and approximately $8$ regions, the model captures local dynamics, while a task-alignment setting with $P=2$ and readouts reset every $7$ steps achieves average classification accuracy $\approx 0.78\pm 0.05$ across subjects [2410.14240].

The paper contrasts this formulation with SLDS, classical piecewise-affine identification, standard PLRNNs, reservoir computing, Neural ODE/SDE models, Koopman approaches, and SINDy. Its main claim is not merely that sparse ReLU recurrence is sufficient for reconstruction, but that few nonlinearities make the switching structure explicit, reduce the number of visited regions, and facilitate fixed-point, cycle, and symbolic-graph analysis [2410.14240].

## 5. Linear memory networks and the separation of memory from nonlinear computation

An earlier precursor to the AL-RNN design philosophy is the Linear Memory Network (LMN), which explicitly separates nonlinear input-output transformation from strictly linear recurrent memory. The functional component computes
\[
h_t = \sigma(W^{xh}x_t + W^{mh}h^m_{t-1}),
\]
while the memory component follows the strictly linear recurrence
\[
h^m_t = W^{hm}h_t + W^{mm}h^m_{t-1}.
\]
Two output variants are defined:
\[
y^m_t = \sigma(W^{mo}h^m_t),\qquad
y^h_t = \sigma(W^{ho}h_t).
\]
The paper states that “the memory is entirely linear while the feedforward component allows to model nonlinear dependencies” [1811.03356].

The memory mechanism is derived from a linear autoencoder for sequences. Given inputs $\{x_t\}$, the autoencoder state satisfies
\[
y_t = Ax_t + By_{t-1},
\]
with reconstruction
\[
\begin{bmatrix}x_t\\ y_{t-1}\end{bmatrix}=Cy_t.
\]
Training proceeds by forming a matrix of reversed subsequences, performing a truncated SVD, and constructing $A$, $B$, and $C$ in closed form. The paper states that “the training algorithm guarantees an optimal encoding when $p=\operatorname{rank}(\Xi)$” [1811.03356]. In LMNs, this autoencoder is trained on hidden-state sequences rather than raw inputs, so the linear memory stores a compressed history of nonlinear functional activations.

A dedicated pretraining scheme exploits equivalence between unfolded feedforward networks and recurrent networks. First, an unfolded finite-horizon model is trained to produce hidden activations $h^t$. Second, a linear autoencoder is fitted to the sequence of hidden states. Third, the LMN parameters are initialized by transferring unfolded and autoencoder parameters, with
\[
W^{hm}=A,\qquad W^{mm}=B,
\]
and corresponding formulas for $W^{mh}$ and $W^{mo}$ [1811.03356]. This is an important historical design lesson for AL-RNNs: the recurrence can remain linear and analyzable, while the nonlinearity is concentrated in tokenwise or feedforward maps that consume the memory rather than redefining it.

The empirical results reported for polyphonic music datasets show that LMN-B is competitive with gated recurrent networks and other state-of-the-art models. Test frame-level accuracies include JSB Chorales $33.98$ for LMN-B and $34.49$ for pret-LMN-B, MuseData $35.56$ and $35.66$, Nottingham $72.71$ and $74.16$, and Piano MIDI $28.00$ and $28.79$ [1811.03356]. The paper further states that, for a fixed total hidden size, LMN requires fewer parameters than LSTM or GRU, and highlights the absence of “unwanted exponential decay effects due to the presence of gates.” A stated limitation is that if the memory component is trained only with backpropagation, there are no theoretical guarantees on memorization properties; burn-in behavior at the beginning of sequences is also observed [1811.03356].

## 6. Minimal nonlinearity as a computational resource

Later work uses AL-RNNs not only as models but as probes for the functional role of nonlinearity in memory. In this formulation, the state update is
\[
\mathbf{z}_t = \mathbf{A}\mathbf{z}_{t-1} + \mathbf{W}\,\Phi^*(\mathbf{z}_{t-1}) + \mathbf{C}\,\mathbf{s}_t + \mathbf{h},
\]
where the first $M-P$ units are linear, the last $P$ units are ReLU, and
\[
\mathbf{A}=\mathrm{diag}(0,\ldots,0,\;a_{M-P+1},\ldots,a_M).
\]
The principal measure of nonlinearity is the number $P$ of PWL units. Effective use of nonlinearity is quantified by a bitcode
\[
b_t=\sum_{i=1}^P \mathbb{I}[z_{t,i}>0]\,2^{P-i},
\qquad
p(b)=\frac{n(b)}{\sum_{b'\in B} n(b')},
\]
so that small support of $p(b)$ indicates that the model uses few subregions [2506.07919].

This work introduces Manifold Attractor Regularization (MAR),
\[
\mathcal{L}_{\mathrm{reg}}=\tau\left[
\sum_{i=1}^{M_\mathrm{reg}}(\tilde{A}_{ii}-1)^2
+\sum_{i=1}^{M_\mathrm{reg}}\sum_{j\neq i}^M W_{ij}^2
+\sum_{i=1}^{M_\mathrm{reg}} h_i^2
\right],
\]
to encourage near-perfect integrators and long time scales. Within a given bitcode-defined subregion, the masked interaction matrix is $\mathbf{W}_{\mathrm{masked}}=\mathbf{W}\odot\mathbf{M}(\text{bitcode})$, the local Jacobian is
\[
\mathbf{J}=\mathbf{A}+\mathbf{W}_{\mathrm{masked}},
\]
fixed points satisfy
\[
(\mathbf{A}+\mathbf{W}_{\mathrm{masked}}-\mathbf{I})\mathbf{z}^*=-\mathbf{h},
\]
and local stability is determined by $|\lambda_i(\mathbf{J})|<1$. The same paper analyzes Lyapunov exponents and gradient flow through $\mathbf{J}^\top$ powers, making the piecewise-linear structure a direct vehicle for mechanistic interpretation [2506.07919].

Across tasks, the reported findings support a “minimal nonlinearity” thesis. On IMDb, performance is minimally affected by $P$ and dynamics are dominated by a single slow mode. On sequential MNIST, linear models are already strong, with maximum eigenvalues slightly above $1$, specifically $\lambda_{\max}\approx 1.003\pm 0.003$; accuracy improves with intermediate $P$ and reaches up to $98.6\%$, while fully nonlinear models degrade. The paper states that for $P=50$, final states within a class share nearly identical bitcodes, with average variation $0.06\pm 0.02$ [2506.07919].

Tasks that require explicit routing show a sharper need for nonlinearity. On the copy task with $N_{\mathrm{sym}}=4$, $N_{\mathrm{seq}}=8$, and $D=200$, purely linear models perform above chance but are fragile; $P=1$ often yields perfect recall, and the mechanism is described as a two-subregion separation between encoding/decoding and autonomous delay-phase transients, with a $100$-step global $k$-cycle and neutral Lyapunov exponent $\lambda_{\max}\approx 0$ [2506.07919]. On the addition problem, linear models fail, while $P=1$ drastically reduces error and performance plateaus near $P\approx 3$. On contextual multistability, linear models remain at $50\%$, whereas $P=1$ solves the task nearly perfectly, up to $96\%$ accuracy. In a joint task-plus-spiking model for CRCNS PFC-1, nonlinearity improves both task accuracy and spike reconstruction, with mean spike correlation $r=0.359\pm 0.002$ close to a Poisson upper bound of $r=0.371\pm 0.057$ [2506.07919].

A related development, Fixed-Point RNNs, addresses a different expressive bottleneck: dense state mixing. The target dense linear RNN
\[
h_t^* = A_t h_{t-1}^* + B_t x_t
\]
is realized as the fixed point of a parallelizable diagonal recurrence
\[
f_\theta(x,h)_t = \Lambda_t f_\theta(x,h)_{t-1} + Q_tB_tx_t + (I-Q_t)h_t.
\]
If $h^*=f_\theta(x,h^*)$, then $h_t^*=Q_t^{-1}\Lambda_t h_{t-1}^*+B_tx_t$, recovering a dense transition from diagonal scans plus channel mixing [2503.10799]. Bedouin accelerates the iteration by letting each fixed-point step sweep through time,
\[
h_t^{(\ell)} = \lambda_t \odot h_{t-1}^{(\ell)} + Q_tB_tx_t + (I-Q_t)h_t^{(\ell-1)}.
\]
The paper states that a single-layer Bedouin solves $A_5$ and $S_5$, succeeds on copying with $2\times$ length generalization where Mamba and Mamba-2 struggle, and that a 2-layer Bedouin with $r=4$ reaches $0.28$ scaled accuracy on modular arithmetic with brackets [2503.10799]. This line extends the AL-RNN program from sparse nonlinearity to dense expressivity emerging from fixed-point solutions of linear scan operators.

## 7. Limitations, design tradeoffs, and emerging synthesis

Across these strands, AL-RNNs are defined less by a single architecture than by a recurring tradeoff: preserve a recurrent core that is linear, affine, piecewise linear, or otherwise scan-compatible, and use nonlinearity sparingly enough that parallelism, stability, or interpretability are not lost. The major benefit is computational. RWKV yields $O(Td)$ inference with constant per-layer state [2305.13048]. Linear-core AL-RNN theory yields $O(\log T)$ scan depth and a near-transformer parallel depth overhead [2603.03612]. Piecewise-linear AL-RNNs yield explicit symbolic partitions and tractable local linear analysis [2410.14240]. LMNs yield closed-form memory training and an explicit separation of memory from nonlinear transformation [1811.03356]. Minimal-nonlinearity AL-RNNs expose when gating, attractor switching, neutral cycles, or context-dependent routing are computationally necessary [2506.07919].

The limitations are similarly consistent. Linear or compressed state summaries can create information bottlenecks, as stated explicitly for RWKV [2305.13048]. Piecewise-linear approaches can overfragment when $P$ is too large and underfit when $P$ is too small [2410.14240]. Minimal-nonlinearity studies report degraded generalization in fully nonlinear regimes because fragmented subregion usage destabilizes learning [2506.07919]. Circuit-theoretic work emphasizes that the very source of extra power in nonlinear RNNs—state-dependent branching—is also what blocks efficient parallelization under standard complexity conjectures [2603.03612]. Fixed-point linear densification introduces its own tradeoff between contraction and capacity, and its convergence speed remains an open issue in the worst case [2503.10799].

A reasonable synthesis suggested by these papers is that AL-RNNs define an expressivity/parallelism/interpretablity frontier rather than a single model family. One end of that frontier is occupied by exact linear-recurrence scan models such as RWKV and other LRNNs; another by sparse-switching PWL systems for dynamical reconstruction; another by architectures such as LMNs and Fixed-Point RNNs that preserve linear memory or linear scans while relocating or restructuring nonlinearity [2305.13048] [2603.03612] [2410.14240] [1811.03356] [2503.10799]. This suggests that “almost-linear” is best understood as a design discipline: keep the recurrent backbone as close as possible to associative linear evolution, and spend nonlinearity only where gating, switching, compositional routing, or task-dependent context truly require it.

Source: https://www.emergentmind.com/topics/almost-linear-rnns-al-rnns