Transformers for dynamical systems learn transfer operators in-context

Abstract: Large-scale foundation models for scientific machine learning adapt to physical settings unseen during training, such as zero-shot transfer between turbulent scales. This phenomenon, in-context learning, challenges conventional understanding of learning and adaptation in physical systems. Here, we study in-context learning of dynamical systems in a minimal setting: we train a small two-layer, single-head transformer to forecast one dynamical system, and then evaluate its ability to forecast a different dynamical system without retraining. We discover an early tradeoff in training between in-distribution and out-of-distribution performance, which manifests as a secondary double descent phenomenon. We discover that attention-based models apply a transfer-operator forecasting strategy in-context. They (1) lift low-dimensional time series using delay embedding, to detect the system's higher-dimensional dynamical manifold, and (2) identify and forecast long-lived invariant sets that characterize the global flow on this manifold. Our results clarify the mechanism enabling large pretrained models to forecast unseen physical systems at test without retraining, and they illustrate the unique ability of attention-based models to leverage global attractor information in service of short-term forecasts.

• The paper demonstrates that transformers implicitly reconstruct transfer operators through time-delay embeddings to accurately forecast unseen dynamical systems.
• The paper rigorously analyzes double descent behavior in both in-distribution and out-of-distribution forecasting, revealing emergent generalization properties.
• The approach leverages attention rollout and state propagation analyses, establishing a mechanistic basis for operator-based inference in scientific machine learning.

In-Context Learning in Transformers: Emergence of Transfer Operator Dynamics for Unseen Systems

Introduction

This work systematically investigates the mechanistic basis for in-context learning (ICL) in transformers applied to dynamical system forecasting, emphasizing the model's ability to generalize to systems and distributions not encountered during pretraining. By precisely characterizing the inductive biases that emerge within small attention-based models, the study demonstrates that transformers do not merely memorize time series, but instead adaptively infer latent state-space structure and implicitly reconstruct transfer operators associated with the underlying dynamical system. This analysis provides theoretical and empirical insights into the surprising out-of-distribution (OOD) generalization observed in state-of-the-art foundation models for scientific machine learning.

Training Protocol and Problem Setting

A small, two-layer, single-head GPT-style transformer is trained via next-token prediction on quantized univariate time series originating from a single dynamical system (Train-ID). The architecture involves relative positional encodings and causal masking, providing a highly interpretable testbed for mechanistic analysis. The model is evaluated on two scenarios:

1. Forecasting unseen trajectories generated by the identical dynamical system as in training (Test-ID, in-distribution);
2. Forecasting trajectories from completely different dynamical systems (Test-OOD, out-of-distribution), both selected randomly from a large ODE database.

The transformer is trained and evaluated across 100 distinct pairings of train/test systems to ensure statistical robustness.

Empirical Characterization of Double Descent and Generalization

An initial analysis of model learning dynamics reveals a two-phase double descent phenomenon: the canonical regime where the model initially generalizes well before overfitting to ID data, and an earlier, secondary double descent in OOD forecasting loss. Notably, the OOD double descent curve indicates that an internal mechanism capable of extracting system-independent information emerges prior to full memorization of the training set.

Figure 1: Double descent in training and validation loss demonstrates distinct ID and OOD generalization regimes, with OOD performance peaking prior to ID overfitting.

This observation suggests that transformers encode transferable features useful for forecasting novel systems, which is not explainable by classical supervised learning paradigms alone.

Emergent Time Delay Embedding and Attractor Manifold Inference

Detailed probing of the transformer's conditional next-token distributions across various input lags reveals that the model performs empirical time-delay embedding during inference. Specifically, when applied to a new univariate OOD time series, the distribution $p_\text{model}(x_{t+1} | x_{t-k})$ closely recapitulates the attractor structure of the unseen system. The order of Markov dependence inferred by the transformer scales with the intrinsic attractor dimension, despite the univariate nature of the observation. This behavior is theoretically consistent with Takens' theorem for topological embedding of dynamical attractors, even though the transformer is never explicitly exposed to the multidimensional system states.

Figure 2: Transformers recover attractor geometry and Markov order matching the true dynamical manifold via in-context time-delay embedding.

Attention rollout analysis reveals that the effective latent dimension of the model's internal representations increases concomitantly with the attractor's intrinsic dimension (as verified with Lorenz-96 systems of varying dimensionality), even when the true system state is unobserved.

Implicit Learning of Transfer Operators

The study further analyzes how the transformer propagates state information in context. By examining transition matrices formed from model output across lagged embeddings, the authors demonstrate that the transformer implements an approximation of the Perron-Frobenius transfer (propagator) operator for the reconstructed attractor. The spectrum of eigenvalues and leading eigenfunctions, both for invariant (stationary) and metastable (almost-invariant) sets, closely match between the model-inferred operator and the true operator estimated from the fully observed system. The Kullback-Leibler divergence between true and model-inferred invariant distributions tightly tracks OOD forecasting loss during training.

Figure 3: The transformer constructs empirical transfer operators, closely matching the true system's leading eigenfunctions and eigenvalue spectrum, thereby capturing invariant and metastable state distributions.

This mechanistic analysis establishes that the out-of-distribution forecasting capability of transformers arises from in-context estimation and propagation of reconstructed state-space measures and not from naive sequence copying or lower-complexity baselines.

Manifold Dimension Estimation and Scaling

As an auxiliary experiment, manifold dimension estimation performed via Grassberger-Procaccia analysis on Lorenz-96 systems confirms that the intrinsic attractor dimension scales linearly with system dimension. The transformer adaptively adjusts its internal latent dimension, as deduced from the stable rank of attention rollout matrices, to reflect manifold complexity.

Figure 4: Estimated intrinsic manifold dimension of Lorenz-96 trajectories increases linearly with the number of dynamical variables.

Implications and Future Research Directions

This work provides a precise mechanistic account for in-context generalization in transformers applied to dynamical systems. The ability of these models to perform automated time delay embedding and to reconstruct transfer operators from univariate observations enables robust OOD forecasting on previously unseen systems. These findings directly clarify recent empirical results on zero-shot transfer and cross-PDE generalization in large scientific foundation models.

From a theoretical perspective, this demonstrates that even compact transformers, provided with sufficient context length and training, develop operator-theoretic inference capabilities in context, leveraging their quadratic attention structure. This positions attention-based models as uniquely equipped for scientific settings characterized by partial observability and the need for adaptive, data-driven state reconstruction.

In terms of practical and future developments, the approach outlined here suggests that scaling context windows and embedding dimensionality, in concert with development of architectures that can efficiently capture and propagate long-range dependencies, will be central to further advances in foundation models for physical and dynamical domains. Exploration of these phenomena in multivariate systems, higher-dimensional PDEs, and in real-world experimental data remains an open direction.

Conclusion

In summary, transformers trained on univariate time series from dynamical systems emerge as in-context learners of dynamical state reconstruction and transfer operators, achieving out-of-distribution forecasting by implicitly embedding and propagating the underlying attractor manifold—a capability not accounted for by traditional supervised learning or parametric system identification. This mechanistic insight refines the understanding of generalization in modern scientific machine learning and sets the stage for principled development and scaling of operator-based foundation models for complex dynamical tasks.