Linear Recurrent Autoencoder Networks

• Linear Recurrent Autoencoder Networks (LRANs) are neural architectures that encode high-dimensional states into a compact latent space with linear recurrence to approximate nonlinear dynamics.
• They leverage a data-adaptive encoder and nonlinear decoder to capture Koopman-invariant subspaces, mitigating overfitting compared to fixed dictionary methods like EDMD.
• Multi-step training with linear latent dynamics enables LRANs to produce fast inference and short-term accurate predictions in complex dynamical systems.

Linearly Recurrent Autoencoder Networks (LRANs) are neural architectures for learning low-dimensional approximations of nonlinear dynamical systems by combining a learned encoder, linear recurrent dynamics in latent space, and a nonlinear decoder. In the formulation introduced by Otto and Rowley, LRANs are explicitly motivated by neural-network approximations of the Koopman operator and by limitations of Extended Dynamic Mode Decomposition (EDMD), particularly the trade-off between dictionary expressivity and overfitting under finite data (Otto et al., 2017). The defining premise is that a compact latent state can evolve linearly while the full state need not be linearly reconstructible from that latent representation, because reconstruction is handled by a nonlinear decoder.

1. Koopman-theoretic setting and motivation

LRANs are formulated for discrete-time systems of the form $x_{t+1} = F(x_t)$ on a state space $M \subset \mathbb{R}^n$, and also admit a continuous-time interpretation through the Koopman semigroup. For an observable $f : M \to \mathbb{C}$, the discrete-time Koopman operator acts by composition,

$(\mathcal{K} f)(x) = f(F(x)),$

and remains linear even when the underlying state dynamics are nonlinear. In continuous time, with flow $\Phi_t(x)$, the Koopman semigroup satisfies

$\mathcal{K}_t \psi(x) = \psi(\Phi_t(x)),$

with infinitesimal generator

$\mathcal{K} \psi(x) = \nabla_x \psi(x) \cdot f(x).$

Koopman eigenfunctions $\varphi_i$ evolve linearly under the operator, with eigenvalues $\mu_i$ in discrete time or $\lambda_i$ in continuous time, and $M \subset \mathbb{R}^n$0 under sampling.

The central motivation for LRANs is the observation that EDMD approximates the Koopman operator on a finite dictionary $M \subset \mathbb{R}^n$1, but larger dictionaries do not automatically improve generalization. The cited work identifies a fundamental trade-off: richer dictionaries increase representational capacity, yet finite data impose an effective rank bound and may induce overfitting. A counterexample described in the source shows that adding a single superfluous feature to a dictionary that already contains the true eigenfunctions can cause severe overfitting, with incorrect learned eigenvalues and eigenfunctions despite near-zero training residuals (Otto et al., 2017).

LRANs address this difficulty by replacing a fixed hand-designed dictionary with a learned encoder that maps observations into a low-dimensional latent space intended to approximate a Koopman-invariant subspace. This suggests a reinterpretation of Koopman learning: rather than selecting a large observable library and then regularizing the resulting operator estimate, one may learn a compact observable coordinate system jointly with the linear latent dynamics and a reconstruction map.

2. Core architecture and mathematical formulation

The canonical LRAN consists of three components: an encoder, linear recurrent latent dynamics, and a decoder. The encoder maps the state to a latent code,

$M \subset \mathbb{R}^n$2

where $M \subset \mathbb{R}^n$3 is a low latent dimension. The latent recurrence is linear and time-invariant,

$M \subset \mathbb{R}^n$4

so that multi-step predictions take the form

$M \subset \mathbb{R}^n$5

For continuous-time data with irregular sampling, the latent state evolves as

$M \subset \mathbb{R}^n$6

The decoder reconstructs the physical state from the latent coordinate,

$M \subset \mathbb{R}^n$7

This decoder is nonlinear. That design choice is a defining distinction from standard DMD- or EDMD-style reconstructions, which typically assume the full state is linearly represented in an invariant subspace. In LRANs, the latent space is required to support linear evolution, but the state need not itself belong to that linear span.

When the learned matrix $M \subset \mathbb{R}^n$8 is diagonalizable, the encoded coordinates define an approximate Koopman-invariant subspace. If $M \subset \mathbb{R}^n$9, $f : M \to \mathbb{C}$0, and $f : M \to \mathbb{C}$1 denote right eigenvectors, left eigenvectors, and eigenvalues of $f : M \to \mathbb{C}$2, then approximate Koopman eigenfunctions are linear functionals of the encoded state,

$f : M \to \mathbb{C}$3

and satisfy

$f : M \to \mathbb{C}$4

Thus the network learns observables whose temporal evolution is linear, even though the underlying dynamics and reconstruction map remain nonlinear.

A related reconstruction framework in the same work is partially linear multi-kernel regression,

$f : M \to \mathbb{C}$5

used primarily with KDMD reduced-order models. This combines a dominant linear dependence on $f : M \to \mathbb{C}$6 with a nonlinear correction in an RKHS, and is intended for cases in which the data lie near a curved low-dimensional manifold (Otto et al., 2017).

3. Training objectives, optimization, and spectral structure

LRAN training is explicitly multi-step. Given a sequence $f : M \to \mathbb{C}$7, the model computes encoded states

$f : M \to \mathbb{C}$8

predicted latent states

$f : M \to \mathbb{C}$9

and decoded predictions

$(\mathcal{K} f)(x) = f(F(x)),$0

The loss combines normalized reconstruction error in state space with normalized error in latent space, weighted across the rollout horizon by a discount factor $(\mathcal{K} f)(x) = f(F(x)),$1 and relative weighting parameter $(\mathcal{K} f)(x) = f(F(x)),$2:

$(\mathcal{K} f)(x) = f(F(x)),$3

Here $(\mathcal{K} f)(x) = f(F(x)),$4 and $(\mathcal{K} f)(x) = f(F(x)),$5 normalize the temporal weights, $(\mathcal{K} f)(x) = f(F(x)),$6 and $(\mathcal{K} f)(x) = f(F(x)),$7 prevent divide-by-zero, and $(\mathcal{K} f)(x) = f(F(x)),$8 is an optional regularizer. The experiments reported in the source did not require an explicit regularizer; instead, small latent dimension $(\mathcal{K} f)(x) = f(F(x)),$9 acted as regularization (Otto et al., 2017).

Optimization uses ADAM with initial learning rate $\Phi_t(x)$0 and geometric decay, with mini-batches of 50 sequences. Sequence lengths in the reported experiments are dataset-specific: Duffing uses $\Phi_t(x)$1, cylinder wake uses $\Phi_t(x)$2, and Kuramoto–Sivashinsky uses $\Phi_t(x)$3. Training durations are reported as $\Phi_t(x)$4–$\Phi_t(x)$5 optimization steps depending on the dataset.

The matrix $\Phi_t(x)$6 is initialized using $\Phi_t(x)$7 blocks of the form $$\begin{bmatrix} \sigma & \omega\ -\omega & \sigma \end{bmatrix},$$\Phi_t(x)$8\lambda = \sigma \pm i\omega$\Phi_t(x)$9r = 0.8$\mathcal{K}_t \psi(x) = \psi(\Phi_t(x)),$0K$ to impose known Koopman eigenvalues. In the Duffing experiments, this constraint is used to embed known eigenvalues and thereby improve interpretability. No explicit stability regularization beyond the multi-step loss and low latent dimension is reported in the experiments.

A common misconception is that Koopman-oriented models must reconstruct the full state linearly from a finite set of observables. LRANs explicitly reject that requirement: the latent space is linearly recurrent, but reconstruction may be nonlinear. The source presents this as a mechanism for capturing richer state structure from very few encoded observables.

4. Relation to EDMD, KDMD, and balanced model reduction

EDMD approximates the Koopman operator on a chosen feature dictionary

$\mathcal{K}_t \psi(x) = \psi(\Phi_t(x)),$1

using snapshot pairs $\mathcal{K}_t \psi(x) = \psi(\Phi_t(x)),$2 with $\mathcal{K}_t \psi(x) = \psi(\Phi_t(x)),$3. The estimator is

$\mathcal{K}_t \psi(x) = \psi(\Phi_t(x)),$4

with $$G = \frac{1}{M}\sum_{i=1}^M \Psi(x_i)\Psi(x_i)^*, \qquad A = \frac{1}{M}\sum_{i=1}^M \Psi(x_i)\Psi(y_i)^*.</p> <p>$$\mathcal{K}_t \psi(x) = \psi(\Phi_t(x)),$5\Psi_X = U\Sigma V^{*$\mathcal{K}_t \psi(x) = \psi(\Phi_t(x)),$$6}</p> <p>\hat{K} = \Sigma⁺ V^* \Psi_Y^* U.$$

The problem, as framed in the LRAN paper, is not that EDMD or KDMD are invalid, but that their success depends heavily on dictionary design and on having sufficient data relative to the effective feature dimension. LRANs offer an alternative in which the dictionary is data-adapted and low-dimensional from the outset.

The same paper also develops balanced model reduction for over-specified EDMD or KDMD systems in feature space. For a feature-state linear system with reconstruction operator $\mathcal{K}_t \psi(x) = \psi(\Phi_t(x)),$7, the infinite-horizon observability and controllability Gramians are $$W_o = \sum_{t=0}^{\infty} (\hat{K})^t C^* C (\hat{K}^{*)^t,} \qquad W_c = \frac{1}{M}\sum_{t=0}^{\infty} (\hat{K}^{*)^t} \Sigma² (\hat{K})^t.</p> <p>$$\mathcal{K}_t \psi(x) = \psi(\Phi_t(x)),$8H = A^*B$\mathcal{K}_t \psi(x) = \psi(\Phi_t(x)),9</sup></p><p>H=UHΣHVH^∗,</p><p>9</sup></p> <p>H = U_H \Sigma_H V_H^*,</p> <p>\mathcal{K} \psi(x) = \nabla_x \psi(x) \cdot f(x).0</p><p>T=BVHΣH^−1/2,S=AUHΣH^−1/2.</p><p>0</p> <p>T = B V_H \Sigma_H^{-1/2}, \qquad S = A U_H \Sigma_H^{-1/2}.</p> <p>\mathcal{K} \psi(x) = \nabla_x \psi(x) \cdot f(x).$1d$\mathcal{K} \psi(x) = \nabla_x \psi(x) \cdot f(x).2</p><p>zt+1=Sd^∗K^^∗Tdzt+1MSd^∗Σut,xt≈CTdzt,z=Sd^∗ΨU(x).2</p> <p>z_{t+1} = S_d^* \hat{K}^* T_d z_t + \frac{1}{\sqrt{M}} S_d^* \Sigma u_t, \qquad x_t \approx C T_d z_t, \qquad z = S_d^* \Psi_U(x).

Finite-horizon Balanced Proper Orthogonal Decomposition (BPOD) is used when the system is unstable or when a finite prediction window is the object of interest. The LRAN paper presents BPOD-reduced KDMD models as a complementary pipeline rather than a direct replacement. The reported comparison is nuanced: KDMD with BPOD can recover coherent spectral structure, but inference requires kernel evaluations scaling with the number of training samples unless support vectors are pruned, whereas LRAN inference is a single encoder pass, linear latent rollout, and decoder evaluation (Otto et al., 2017).

5. Experimental demonstrations

The reported experiments span three dynamical systems of increasing complexity: the unforced Duffing oscillator, an unstable cylinder wake at $\mathcal{K} \psi(x) = \nabla_x \psi(x) \cdot f(x).$3, and the Kuramoto–Sivashinsky equation at the onset of chaos (Otto et al., 2017).

For the unforced Duffing oscillator, $$\ddot{x} = -\delta \dot{x} - x(\beta + \alpha x^2), \qquad \delta = 0.5,\; \beta = -1,\; \alpha = 1,</p> <p>$$\mathcal{K} \psi(x) = \nabla_x \psi(x) \cdot f(x).$4x=\pm 1$\mathcal{K} \psi(x) = \nabla_x \psi(x) \cdot f(x).$$5</p> <p>\lambda_{1,2} = \frac{1}{4}\left(-1 \pm \sqrt{31}\, i\right).$$\mathcal{K} \psi(x) = \nabla_x \psi(x) \cdot f(x).$6\lambda_0 = 0$\mathcal{K} \psi(x) = \nabla_x \psi(x) \cdot f(x).$7M = 10^4$\mathcal{K} \psi(x) = \nabla_x \psi(x) \cdot f(x).8Δ</sup>t=0.258\Delta</sup> t = 0.25\mathcal{K} \psi(x) = \nabla_x \psi(x) \cdot f(x).$9d=3$\varphi_i$0K$\varphi_i192.74%192.74\%\varphi_i294.88%294.88\%\varphi_i396.50%396.50\% for the KDMD ROM. For prediction error versus horizon, the free LRAN achieves the lowest mean square relative error, followed by the constrained LRAN; in this case the KDMD ROM’s nonlinear reconstruction does not help and is described as sensitive to overfitting.

For the unstable cylinder wake, the data consist of 2000 velocity snapshots sampled every $\varphi_i$4, split into 1000/500/500 train/evaluation/test snapshots. The data are projected onto the 200 most energetic POD modes and then time-delay embedded with two delays to produce 400-dimensional inputs. The LRAN uses latent dimension $\varphi_i$5 and sequence length $\varphi_i$6; the KDMD ROM is reduced to the same dimension by BPOD over the same horizon. The leading continuous-time eigenvalues of the two models agree closely; the first mode captures the vortex-shedding pattern, KDMD yields a second eigenvalue near $\varphi_i$7 consistent with a harmonic, and LRAN produces a second eigenvalue close to $\varphi_i$8, which the source interprets as suggesting combinations that capture the slow transition. Linear reconstructions via Koopman modes are comparable across the models, but the nonlinear LRAN decoder yields roughly an order-of-magnitude lower mean relative error than the KDMD ROM’s kernel reconstructor over many steps within the training horizon.

For the Kuramoto–Sivashinsky equation, $$u_t + u_{xx} + u_{xxxx} + u u_x = 0, \qquad x \in [0,L],\; L = 8\pi,$$\varphi_i$9\Delta t = 1.0$\mu_i$0d=16$\mu_i$1T=5$. The learned Koopman spectra include slightly unstable modes, consistent with the unstable Fourier modes. LRAN modestly outperforms the KDMD ROM across horizons, both models remain accurate for short times up to five steps, and the nonlinear decoder is reported as essential because linear modes alone do not reconstruct the state well.

These experiments collectively support three recurring claims from the source: LRANs can identify meaningful Koopman eigenfunctions, can produce accurate low-dimensional models for systems with intrinsically low-dimensional organization, and are primarily accurate within the training horizon, with degradation beyond that window.

6. Relation to Linear Memory Networks and broader interpretation

The 2018 paper on Linear Memory Networks (LMNs) provides a closely related but distinct architecture in which nonlinear computation and linear memory are explicitly separated (Bacciu et al., 2018). The LMN functional state and memory state are defined by

$\mu_i$2

$\mu_i$3

Its memory block is a linear, input-driven recurrence equivalent in form to a linear recurrent autoencoder for sequences,

$\mu_i$4

with decoder

$\mu_i$5

The LMN paper shows that this memory mechanism can be trained in closed form via truncated SVD of a reversed-subsequence data matrix, yielding explicit formulas

$\mu_i$6

and an optimal encoding when the latent dimension equals the rank of the constructed sequence matrix.

The relation to LRANs is structural. In LRAN notation, one may identify the latent state with the LMN memory, $\mu_i$7, and write the LMN linear memory as

$\mu_i$8

with $\mu_i$9, $\lambda_i$0, and input $\lambda_i$1. Both frameworks therefore employ linear time-invariant latent dynamics together with learned nonlinear maps. The difference lies in what is encoded and reconstructed. LRANs typically encode the observed state $\lambda_i$2 and decode the full state through a nonlinear decoder trained end-to-end. LMNs instead feed a nonlinear functional representation $\lambda_i$3 into a linear sequence autoencoder so that the memory reconstructs past functional states rather than necessarily the raw observation. The LMN paper emphasizes closed-form pretraining of the memory followed by end-to-end fine-tuning, whereas LRANs are presented primarily as end-to-end trained encoder–linear dynamics–decoder systems.

This comparison clarifies a broader interpretation of linear recurrent autoencoder architectures. They can be viewed as instances of a general design pattern: nonlinear feature extraction or encoding, linear latent evolution, and a reconstruction or readout mechanism. In Koopman-oriented LRANs, the latent recurrence is intended to approximate a Koopman-invariant subspace of observables. In LMNs, the same pattern is applied to recurrent sequence modeling through a linear memory trained to reconstruct histories of internal functional activations. A plausible implication is that LRANs and LMNs occupy neighboring points in the design space of models that decouple nonlinear representation learning from linear latent propagation.

7. Limitations, scope, and common points of confusion

The source material presents LRANs as a response to specific limitations of EDMD and KDMD, not as a universal replacement. Their principal advantage is the ability to learn a compact, informative latent space with linear evolution and nonlinear state reconstruction, thereby reducing dependence on fixed dictionaries and enabling fast inference. At the same time, several limitations are stated explicitly.

First, prediction accuracy is strongest within the training horizon $\lambda_i$4. The reported error curves typically remain low up to that horizon and then grow beyond it. For more complex systems such as Kuramoto–Sivashinsky, extending the useful horizon requires either larger latent dimension or an extended training curriculum. Second, LRAN training is computationally expensive, involving hundreds of thousands of ADAM steps, even though inference is fast. Third, nonlinear reconstruction is powerful but not immune to overfitting in general; while the LRAN decoder did not exhibit the reconstruction failures observed for the kernel regressor in the Duffing example, the broader issue of reconstruction regularization remains relevant.

Another common confusion concerns interpretability. LRANs do not guarantee that every latent coordinate is itself a Koopman eigenfunction. Rather, if $\lambda_i$5 is diagonalizable, eigenfunctions are obtained as linear functionals of the encoded state through the eigenvectors of $\lambda_i$6. This distinction matters when interpreting latent trajectories, spectra, or decoder behavior.

Finally, LRANs should not be conflated with autonomous latent-linear models only. The canonical LRAN in the cited work uses autonomous latent evolution, $\lambda_i$7, but the comparison with LMNs shows that closely related architectures can incorporate driven linear latent recurrences. This suggests a broader family of linearly recurrent autoencoder models whose differences lie in whether the latent dynamics are autonomous or input-driven, whether the encoded quantity is the raw state or an internal representation, and whether the linear recurrent component is trained by gradient descent, closed-form linear algebra, or a combination of both.