DeepKoopFormer: Koopman-Enhanced Transformer

• DeepKoopFormer is a Koopman-enhanced Transformer architecture that encodes high-dimensional time-series data into a latent space governed by a linear, spectrally constrained operator.
• It decouples expressive, non-linear representation learning from the stable, linear temporal propagation, ensuring robustness to noise and long-range prediction stability.
• Extensive experiments across synthetic and real-world datasets demonstrate its superior performance against standard LSTM and baseline Transformer models.

DeepKoopFormer is a Koopman-enhanced Transformer architecture for multivariate time-series forecasting that combines a Transformer encoder with a latent linear propagator and a linear decoder. Introduced in “DeepKoopFormer: A Koopman Enhanced Transformer Based Architecture for Time Series Forecasting” (Forootani et al., 4 Aug 2025), it is designed for settings in which long-range prediction, robustness to noise, and dynamical stability are as important as raw forecast accuracy. Its central premise is that high-dimensional and nonlinear observations can be encoded into a latent space in which temporal evolution is governed by a spectrally constrained linear Koopman operator, thereby coupling expressive sequence representation with an explicitly analyzable dynamical law.

1. Problem formulation and conceptual basis

DeepKoopFormer addresses supervised forecasting from a context window of length $P$ to a prediction horizon of length $H$. For a multivariate sequence

$$\mathbf{x} = [\mathbf{x}_1,\mathbf{x}_2,\dots,\mathbf{x}_T] \in \mathbb{R}^{T \times d},$$

with $\mathbf{x}_t \in \mathbb{R}^d$, the input-output pair at index $t$ is

$$\mathbf{X}_t = [\mathbf{x}_t,\mathbf{x}_{t+1},\dots,\mathbf{x}_{t+P-1}] \in \mathbb{R}^{P \times d},$$

$$\mathbf{Y}_t = [\mathbf{x}_{t+P},\mathbf{x}_{t+P+1},\dots,\mathbf{x}_{t+P+H-1}] \in \mathbb{R}^{H \times d},$$

and the learned map is

$$f_\theta:\mathbb{R}^{P\times d}\to \mathbb{R}^{H\times d}, \qquad \hat{\mathbf{Y}}_t = f_\theta(\mathbf{X}_t).$$

This formulation is standard for multistep forecasting, but DeepKoopFormer departs from purely autoregressive or purely attention-based forecasters by imposing a structured latent transition (Forootani et al., 4 Aug 2025).

The model is motivated by two paired claims. First, Transformer forecasters are strong at long-range representation learning but may suffer from interpretability issues and instability under noise or dynamical uncertainty. Second, Koopman operator theory offers a principled way to represent nonlinear dynamics through linear evolution in a lifted space of observables. DeepKoopFormer therefore uses the Transformer to construct latent observables and the Koopman operator to advance them linearly. A plausible implication is that the architecture is intended to decouple representational capacity from rollout stability: the encoder remains highly expressive, while temporal propagation is constrained by a simple operator law rather than a fully unconstrained deep recurrence.

2. Encoder–propagator–decoder architecture

The architecture is modular. A Transformer-based encoder maps the input history to a latent state,

$\mathbf{z}_t = \mathcal{E}_\theta(\mathbf{X}_t),$

although the preliminaries also write the encoder on a single observation as $\mathbf{z}_t = \mathcal{E}_\theta(\mathbf{x}_t)$. In forecasting usage, the operative form is the context-based encoding. Temporal dynamics are then advanced by a linear latent operator,

$H$0

and each propagated latent state is decoded by a linear readout,

$H$1

so that

$H$2

Collecting these outputs over $H$3 yields the final forecast sequence (Forootani et al., 4 Aug 2025).

The Transformer is therefore not the rollout mechanism. It serves as a nonlinear history encoder that extracts a latent state from the past, after which future evolution is generated exclusively by powers of $H$4. This is one of the model’s defining design decisions. Standard Transformer blocks appear in the encoder, with input embedding

$H$5

positional augmentation

$H$6

and pooled hidden summaries such as

$H$7

The paper states that, throughout experiments, $H$8 for simplicity (Forootani et al., 4 Aug 2025).

DeepKoopFormer is instantiated with three Transformer backbones: PatchTST, Informer, and Autoformer. The PatchTST version uses non-overlapping patches $H$9, where $$\mathbf{x} = [\mathbf{x}_1,\mathbf{x}_2,\dots,\mathbf{x}_T] \in \mathbb{R}^{T \times d},$$0. The Informer version uses ProbSparse attention. The Autoformer version decomposes the input into trend and seasonal components,

$$\mathbf{x} = [\mathbf{x}_1,\mathbf{x}_2,\dots,\mathbf{x}_T] \in \mathbb{R}^{T \times d},$$1

The paper also emphasizes a channel-independent encoding strategy, especially in the PatchTST-style setting. This places DeepKoopFormer closer to a plug-and-play latent dynamical wrapper over Transformer encoders than to a monolithic new attention mechanism.

3. Koopman operator parameterization and stability mechanisms

The architectural core is the latent propagator

$$\mathbf{x} = [\mathbf{x}_1,\mathbf{x}_2,\dots,\mathbf{x}_T] \in \mathbb{R}^{T \times d},$$2

with orthogonality constraints

$$\mathbf{x} = [\mathbf{x}_1,\mathbf{x}_2,\dots,\mathbf{x}_T] \in \mathbb{R}^{T \times d},$$3

and singular values parameterized as

$$\mathbf{x} = [\mathbf{x}_1,\mathbf{x}_2,\dots,\mathbf{x}_T] \in \mathbb{R}^{T \times d},$$4

This orthogonal–diagonal–orthogonal parameterization isolates contraction in the diagonal factor while keeping the basis changes well conditioned (Forootani et al., 4 Aug 2025).

The paper describes this construction as enforcing bounded spectral radius and stability; the explicit bound given is

$$\mathbf{x} = [\mathbf{x}_1,\mathbf{x}_2,\dots,\mathbf{x}_T] \in \mathbb{R}^{T \times d},$$5

A direct implication is that $$\mathbf{x} = [\mathbf{x}_1,\mathbf{x}_2,\dots,\mathbf{x}_T] \in \mathbb{R}^{T \times d},$$6, so repeated propagation decays geometrically: $$\mathbf{x} = [\mathbf{x}_1,\mathbf{x}_2,\dots,\mathbf{x}_T] \in \mathbb{R}^{T \times d},$$7 The paper further states a global exponential stability corollary,

$$\mathbf{x} = [\mathbf{x}_1,\mathbf{x}_2,\dots,\mathbf{x}_T] \in \mathbb{R}^{T \times d},$$8

DeepKoopFormer supplements the hard operator constraint with Lyapunov-based energy regularization,

$$\mathbf{x} = [\mathbf{x}_1,\mathbf{x}_2,\dots,\mathbf{x}_T] \in \mathbb{R}^{T \times d},$$9

using $\mathbf{x}_t \in \mathbb{R}^d$0 as the Lyapunov candidate. If this term is driven to zero during training, the realized latent trajectory satisfies

$\mathbf{x}_t \in \mathbb{R}^d$1

which is a per-step contractivity condition rather than merely an asymptotic bound. The paper also gives a perturbation result: $\mathbf{x}_t \in \mathbb{R}^d$2 indicating geometrically decaying sensitivity to latent perturbations (Forootani et al., 4 Aug 2025).

Orthogonality is maintained exactly by reprojecting $\mathbf{x}_t \in \mathbb{R}^d$3 and $\mathbf{x}_t \in \mathbb{R}^d$4 onto the Stiefel manifold using Householder QR on every forward pass. The paper also notes that if a strictly normal operator is desired, one may tie the factors by enforcing $\mathbf{x}_t \in \mathbb{R}^d$5, but this is presented as optional rather than required.

4. Learning objective and optimization regime

DeepKoopFormer is trained with a predictive-only objective rather than the reconstruction-heavy objectives common in many Koopman autoencoders: $\mathbf{x}_t \in \mathbb{R}^d$6 The first term is a multistep forecast loss, assembled from recursively propagated latent states and decoded predictions. The second term is the Lyapunov regularizer. The paper explicitly contrasts this choice with prior Koopman autoencoder formulations that balance prediction against reconstruction (Forootani et al., 4 Aug 2025).

Optimization is performed end to end with Adam. For the Van der Pol and Lorenz experiments, the learning rate is $\mathbf{x}_t \in \mathbb{R}^d$7; for climate, cryptocurrency, and electricity experiments it is $\mathbf{x}_t \in \mathbb{R}^d$8. Epoch counts extend up to $\mathbf{x}_t \in \mathbb{R}^d$9, $t$0, or $t$1 depending on the dataset. Typical stability settings are $t$2 and $t$3. In many experiments sinusoidal positional encodings are used.

Representative synthetic settings are fully specified. The Van der Pol system is discretized by Euler with $t$4, total time $t$5, additive Gaussian noise $t$6, patch length $t$7, forecast horizon $t$8, $t$9, 2 attention heads, and 2 encoder layers. The Lorenz system uses Euler discretization with $$\mathbf{X}_t = [\mathbf{x}_t,\mathbf{x}_{t+1},\dots,\mathbf{x}_{t+P-1}] \in \mathbb{R}^{P \times d},$$0, total time $$\mathbf{X}_t = [\mathbf{x}_t,\mathbf{x}_{t+1},\dots,\mathbf{x}_{t+P-1}] \in \mathbb{R}^{P \times d},$$1, additive Gaussian noise $$\mathbf{X}_t = [\mathbf{x}_t,\mathbf{x}_{t+1},\dots,\mathbf{x}_{t+P-1}] \in \mathbb{R}^{P \times d},$$2, patch length $$\mathbf{X}_t = [\mathbf{x}_t,\mathbf{x}_{t+1},\dots,\mathbf{x}_{t+P-1}] \in \mathbb{R}^{P \times d},$$3, forecast horizon $$\mathbf{X}_t = [\mathbf{x}_t,\mathbf{x}_{t+1},\dots,\mathbf{x}_{t+P-1}] \in \mathbb{R}^{P \times d},$$4, $$\mathbf{X}_t = [\mathbf{x}_t,\mathbf{x}_{t+1},\dots,\mathbf{x}_{t+P-1}] \in \mathbb{R}^{P \times d},$$5, 2 attention heads, FFN width 64, and training for 3000 epochs (Forootani et al., 4 Aug 2025).

5. Experimental domains and reported behavior

The evaluation spans synthetic dynamical systems, climate reanalysis and projection data, financial time series, and electricity generation. Synthetic benchmarks comprise the Van der Pol oscillator

$$\mathbf{X}_t = [\mathbf{x}_t,\mathbf{x}_{t+1},\dots,\mathbf{x}_{t+P-1}] \in \mathbb{R}^{P \times d},$$6

with $$\mathbf{X}_t = [\mathbf{x}_t,\mathbf{x}_{t+1},\dots,\mathbf{x}_{t+P-1}] \in \mathbb{R}^{P \times d},$$7, and the Lorenz system

$$\mathbf{X}_t = [\mathbf{x}_t,\mathbf{x}_{t+1},\dots,\mathbf{x}_{t+P-1}] \in \mathbb{R}^{P \times d},$$8

with $$\mathbf{X}_t = [\mathbf{x}_t,\mathbf{x}_{t+1},\dots,\mathbf{x}_{t+P-1}] \in \mathbb{R}^{P \times d},$$9, $$\mathbf{Y}_t = [\mathbf{x}_{t+P},\mathbf{x}_{t+P+1},\dots,\mathbf{x}_{t+P+H-1}] \in \mathbb{R}^{H \times d},$$0, and $$\mathbf{Y}_t = [\mathbf{x}_{t+P},\mathbf{x}_{t+P+1},\dots,\mathbf{x}_{t+P+H-1}] \in \mathbb{R}^{H \times d},$$1. Real-world evaluations use CMIP6 and ERA5 climate data for wind speed and surface pressure forecasting, a cryptocurrency dataset with 27,585 daily records from the top 10 cryptocurrencies on Binance, and a six-channel Spanish electricity generation dataset (Forootani et al., 4 Aug 2025).

On wind-speed forecasting, the paper reports that DeepKoopFormer-PatchTST and DeepKoopFormer-Informer consistently outperform LSTM in both MSE and MAE, with the advantage increasing at longer horizons. On surface-pressure forecasting, results are more nuanced: all models perform well, MSE values are often on the order of $$\mathbf{Y}_t = [\mathbf{x}_{t+P},\mathbf{x}_{t+P+1},\dots,\mathbf{x}_{t+P+H-1}] \in \mathbb{R}^{H \times d},$$2 to $$\mathbf{Y}_t = [\mathbf{x}_{t+P},\mathbf{x}_{t+P+1},\dots,\mathbf{x}_{t+P+H-1}] \in \mathbb{R}^{H \times d},$$3, MAE remains below $$\mathbf{Y}_t = [\mathbf{x}_{t+P},\mathbf{x}_{t+P+1},\dots,\mathbf{x}_{t+P+H-1}] \in \mathbb{R}^{H \times d},$$4, and LSTM can be competitive at shorter horizons and smaller patch sizes because the signal is smoother. On ERA5 wind speed at 10 locations over Germany, all DeepKoopFormer variants are described as closely tracking ground truth, with PatchTST and Informer again strongest, while Autoformer sometimes oversmooths.

The financial and electricity experiments are reported as particularly favorable to the Koopman-enhanced variants. On cryptocurrency forecasting, LSTM attains very low training error but degrades more strongly on test settings as patch length and horizon increase, while DeepKoopFormer variants—especially PatchTST—maintain lower MSE and MAE across a wider range of configurations. On electricity generation, PatchTST-based DeepKoopFormer is reported to generalize best, Informer remains competitive, Autoformer degrades more at longer horizons, and LSTM is worst overall except in some very short-horizon cases such as $$\mathbf{Y}_t = [\mathbf{x}_{t+P},\mathbf{x}_{t+P+1},\dots,\mathbf{x}_{t+P+H-1}] \in \mathbb{R}^{H \times d},$$5. Across the synthetic and real datasets, the paper’s abstract summarizes the overall claim as consistent outperformance over standard LSTM and baseline Transformer models in accuracy, robustness to noise, and long-term forecasting stability (Forootani et al., 4 Aug 2025).

At the same time, the detailed exposition makes the strongest direct empirical comparisons against LSTM. This suggests that the paper’s principal quantitative evidence is centered on the benefits of adding a constrained Koopman propagator to strong encoder backbones, rather than on exhaustive head-to-head ablations against unconstrained versions of every Transformer backbone.

6. Position within deep Koopman research

DeepKoopFormer belongs to a broader line of work that learns latent coordinates with approximately linear evolution, but it occupies a specific position within that literature. Earlier deep Koopman identification methods learned latent lifted state-space models with multi-step simulation loss and structured latent propagation, yet used feedforward encoders over state or fixed I/O histories rather than Transformer-based context encoding (Iacob et al., 2021). Koopman Neural Forecaster introduced global and local time-varying operators for temporal distribution shift and used a Transformer only to generate a local operator, not as the backbone sequence encoder (Wang et al., 2022). Koopa addressed non-stationary forecasting through decomposition into time-invariant and time-variant components with separate Koopman predictors and direct forecast loss, but without attention (Liu et al., 2023). Delay-aware deep Koopman work later used an LSTM front-end to summarize history before lifting into a latent linear system, establishing the general principle of history-conditioned Koopman embeddings for delayed dynamics (Valábek et al., 6 Nov 2025). A different branch of the literature has focused on interpretability of learned observables through post-hoc symbolic dictionary readout rather than on Transformer integration (Slote et al., 7 May 2026).

Against that background, DeepKoopFormer is distinguished by four features. It uses Transformer encoders as the representation-learning front end; it retains a strictly linear latent propagator rather than a state-dependent or locally recomputed one; it enforces stability structurally through orthogonal parameterization and a hard contraction bound; and it trains with a predictive-only objective augmented by Lyapunov regularization (Forootani et al., 4 Aug 2025). The resulting notion of interpretability is structural rather than semantic. The latent dynamics are explicitly inspectable through $$\mathbf{Y}_t = [\mathbf{x}_{t+P},\mathbf{x}_{t+P+1},\dots,\mathbf{x}_{t+P+H-1}] \in \mathbb{R}^{H \times d},$$6, its singular values, and its contraction behavior, but the paper does not establish that individual latent coordinates correspond to physically meaningful modes in the stronger scientific sense pursued by symbolic Koopman discovery.

Several limitations are also explicit or immediate. The method assumes that a useful latent linearization exists for the target process. Practical performance depends on hyperparameters such as backbone choice, patch length, latent dimension, $$\mathbf{Y}_t = [\mathbf{x}_{t+P},\mathbf{x}_{t+P+1},\dots,\mathbf{x}_{t+P+H-1}] \in \mathbb{R}^{H \times d},$$7, and $$\mathbf{Y}_t = [\mathbf{x}_{t+P},\mathbf{x}_{t+P+1},\dots,\mathbf{x}_{t+P+H-1}] \in \mathbb{R}^{H \times d},$$8. The paper identifies future directions in control tasks, irregular time series, and spatiotemporal graph structures. A plausible implication is that DeepKoopFormer should be understood less as a universal replacement for sequence models than as a structured forecasting framework for regimes where latent dynamical regularity is valuable and where stability under rollout is a first-class requirement.