Transformer Learning of Chaotic Collective Dynamics in Many-Body Systems

Abstract: Learning reduced descriptions of chaotic many-body dynamics is fundamentally challenging: although microscopic equations are Markovian, collective observables exhibit strong memory and exponential sensitivity to initial conditions and prediction errors. We show that a self-attention-based transformer framework provides an effective approach for modeling such chaotic collective dynamics directly from time-series data. By selectively reweighting long-range temporal correlations, the transformer learns a non-Markovian reduced description that overcomes intrinsic limitations of conventional recurrent architectures. As a concrete demonstration, we study the one-dimensional semiclassical Holstein model, where interaction quenches induce strongly nonlinear and chaotic dynamics of the charge-density-wave order parameter. While pointwise predictions inevitably diverge at long times, the transformer faithfully reproduces the statistical "climate" of the chaos, including temporal correlations and characteristic decay scales. Our results establish self-attention as a powerful mechanism for learning effective reduced dynamics in chaotic many-body systems.

• The paper introduces a transformer-based method combining embedding dynamics into a latent space with self-attention for learning chaotic collective dynamics, achieving prediction accuracy of time-discretized order parameter $\Delta.).$
• The authors demonstrate the transformer’s efficacy by successfully predicting correlation functions, fluctuation amplitudes, and characteristic decay scales, which hold beyond trajectory-level accuracy
• This study emphasizes revealing memory structures in chaotic reduced descriptions, showcasing cases using a quenched Holstein model.

Motivation and problem setting

Learning reduced descriptions of many-body dynamics from time-series data is well established for dissipative and open quantum systems, where neural networks effectively learn dynamical maps or implicit memory kernels induced by an environment. The authors address a qualitatively different regime: closed, energy-conserving hybrid quantum–classical systems whose microscopic equations of motion are strictly Markovian, but whose collective observables are generically non-Markovian once microscopic degrees of freedom are projected out. In such reduced descriptions, memory arises from information redistribution within the system itself rather than from dissipation. Chaos compounds the difficulty: exponential sensitivity to initial conditions and to prediction errors makes long-time pointwise forecasting fundamentally impossible, so the appropriate target is not trajectory-level accuracy but the statistical "climate" of the dynamics—correlation functions, fluctuation amplitudes, and characteristic decay scales.

The paper argues that recurrent architectures, which have been successful at predicting chaotic dynamics when trained on full Markovian system states, are intrinsically limited for reduced observables because they cannot selectively reweight long-range temporal correlations. Self-attention provides this capability: attention weights act as a discrete, adaptive approximation to the Mori–Zwanzig memory kernel $K(t)$, learned directly from data without imposing a fixed functional form or timescale.

Framework

The formal target is a next-step predictor of the form $$\Delta_{T+1} = \mathcal{F}(\Delta_1,\dots,\Delta_T)$$ for a scalar order parameter $\Delta(t)$, motivated by the Mori–Zwanzig projection formalism in which the reduced dynamics contains a linear term, a convolutional memory kernel, and a fluctuating force. The architecture follows prior transformer implementations for dissipative quantum dynamics (Cuesta et al., 2023) but with modifications specific to chaotic regimes: scalar order-parameter values are embedded into a $d_p$-dimensional latent space via a linear layer, augmented with trigonometric positional encoding built from relative time offsets (appropriate since post-transient chaotic dynamics has no meaningful absolute time origin), and processed by two transformer blocks with multi-head scaled dot-product self-attention, residual connections, layer normalization, and position-wise feed-forward networks.

Two architectural findings are emphasized. First, enlarging the input window $T$ from 41 to 201–401 was essential for stable training; small windows fail to converge. Second, the embedding dimension used in earlier work ($d_p=64$) causes collapse onto trivial constant predictions; $d_p=500$ yields training losses on the order of $10^{-5}$–$10^{-6}$ after 3200 epochs (Adam optimizer, MSE loss, batch size 128, single attention head, NVIDIA RTX A6000). These sensitivity results are practically important: they indicate that learning chaotic reduced dynamics requires substantially more temporal context and representational capacity than dissipative problems.

Testbed: quenched Holstein model

The benchmark is the one-dimensional spinless-fermion Holstein model treated semiclassically via single-trajectory Ehrenfest dynamics: classical Newton equations for Einstein phonons coupled self-consistently to a von Neumann equation for the electronic density matrix. This is a closed, energy-conserving, strictly Markovian microscopic evolution (a single-trajectory limit of the truncated Wigner approximation that neglects phonon quantum fluctuations—a limitation the authors acknowledge explicitly).

A deep quench from $g=0$ to finite coupling drives the homogeneous initial state far from equilibrium. CDW order nucleates stochastically, fragmenting the lattice into domains separated by mobile kink defects whose collisions and reorganization produce erratic, aperiodic fluctuations of the staggered order parameter $$\Delta_{T+1} = \mathcal{F}(\Delta_1,\dots,\Delta_T)$$0. Two quenches with identical parameters yield substantially different trajectories, confirming strong sensitivity to defect configurations.

Results

One-step accuracy: predicted values cluster tightly along the diagonal against exact Ehrenfest results across the full range of $$\Delta_{T+1} = \mathcal{F}(\Delta_1,\dots,\Delta_T)$$1, with a sharply peaked error distribution centered at zero. This establishes the transformer as a reliable local predictor despite strongly nonlinear underlying dynamics.

Autoregressive trajectories: when iterated by feeding its own predictions back as input, the model tracks exact trajectories closely at short times but diverges at long times—inevitable given chaos. Crucially, however, the generated trajectories reproduce the correct amplitude, fluctuation scale, and temporal variability of $$\Delta_{T+1} = \mathcal{F}(\Delta_1,\dots,\Delta_T)$$2.

Statistical climate: the normalized autocorrelation function computed from transformer-generated trajectories matches the exact result over all delays, including the rapid initial decay, the negative minimum, and the damped oscillatory recovery. Since the autocorrelation is phase-insensitive, this agreement constitutes a stringent test that the model has captured the effective reduced dynamics—including its memory structure—rather than merely mimicking short-time behavior. The implication is that self-attention can learn a faithful non-Markovian reduced description whose validity extends beyond the predictability horizon of individual trajectories.

Relation to variational approaches

The framework is complementary to time-dependent neural quantum state methods, which parametrize the wave function itself and evolve it variationally within a restricted manifold. Here no microscopic state is represented; the model learns an effective dynamical map directly from observable time series. NQS targets faithful microscopic evolution, whereas this approach targets emergent statistical structure of collective variables without any wave-function description—an advantage when only experimentally accessible observables are available.

Limitations and open questions

Several caveats bear directly on the strength of the claims. The training data come from single-trajectory Ehrenfest dynamics, which neglects phonon entanglement and quantum fluctuations; whether the learned reduced description remains accurate for TWA-sampled or fully quantum data is asserted as feasible but not demonstrated. The demonstration uses a single scalar observable, a single attention head, one parameter point $$\Delta_{T+1} = \mathcal{F}(\Delta_1,\dots,\Delta_T)$$3, and one quench protocol, leaving open how the approach scales to multi-component or spatially resolved order parameters and to stronger chaos. The claim that attention weights approximate the memory kernel $$\Delta_{T+1} = \mathcal{F}(\Delta_1,\dots,\Delta_T)$$4 is argued schematically but not validated quantitatively against an independently computed kernel. Finally, the sensitivity of training to window length and embedding dimension suggests the hyperparameter requirements for more complex systems remain uncharacterized.

Conclusion

This paper establishes self-attention as an effective mechanism for learning non-Markovian reduced dynamics emerging from Markovian microscopic laws in a chaotic, energy-conserving many-body system. While individual trajectories diverge beyond the predictability horizon, the transformer reproduces the full autocorrelation structure of the CDW order parameter following an interaction quench. The work leaves open quantitative validation of the learned memory kernel, extension beyond semiclassical single-trajectory data, and scaling to higher-dimensional order-parameter spaces.