LSTM Variants for Chaotic Dynamical Systems: An Empirical Study on the Lorenz Attractor

Abstract: Forecasting chaotic dynamical systems such as the Lorenz attractor is notoriously difficult: small numerical errors are amplified exponentially over long autoregressive rollouts. We study seven recurrent and convolutional architectures for the AI-DEEDS 2026 Chaotic Systems Challenge: a vanilla LSTM, an LSTM with additive attention, a Bidirectional LSTM (BiLSTM), a BiLSTM trained with the Huber loss, a Temporal Convolutional Network (TCN), a CNN front-end followed by an LSTM, and a CNN front-end followed by a BiLSTM. All models share the same pre-processing, sequence length, and rollout procedure, isolating the contribution of each design choice. The challenge scores predictions on a 0-100 scale where higher is better. We obtain leaderboard scores between 45.72 and 58.81, with the BiLSTM trained with Huber loss being the strongest configuration. Two findings stand out: (i) adding additive attention to the unidirectional baseline degraded performance by over ten points, and (ii) prepending a CNN front-end to either an LSTM or a BiLSTM did not help and slightly hurt the score. Per-pair RMSE measurements confirm that the BiLSTM family generalizes better in the harder pairs (6-7), while the LSTM + Attention model collapses there (RMSE up to 8.94 on pair 6). We discuss why bidirectional context and a robust loss help in chaotic regimes while attention and CNN front-ends fail in this setting.

• The paper demonstrates that a BiLSTM with Huber loss achieves the best results, scoring 58.81 and a low RMSE of 1.176 in long-horizon predictions.
• It rigorously evaluates seven model variants (including vanilla LSTM, attention-augmented, and CNN hybrids) using standardized preprocessing and a fixed input window.
• Results highlight that attention and convolutional modifications may hinder performance, emphasizing robust model design for chaotic time-series forecasting.

Empirical Analysis of LSTM Architectures for Chaotic Time-Series Forecasting

Overview and Motivation

This study rigorously evaluates the efficacy of seven recurrent and convolutional neural architectures for forecasting the trajectories of chaotic dynamical systems, specifically the Lorenz attractor, within the competitive AI-DEEDS 2026 Chaotic Systems Challenge (2606.22662). The primary aim is to isolate the contributions of attention mechanisms, bidirectionality, robust losses, and convolutional feature extractors in RNN-based models under a standardized pipeline for autoregressive time-series prediction. The Lorenz attractor, featuring high sensitivity to initial conditions and exponential error growth due to its positive Lyapunov exponent, presents an archetypal testbed for scrutinizing long-horizon forecasting robustness in sequential deep learning. All models were assessed with controlled preprocessing, window length, and rollout procedures, ensuring that outcomes reflect strict architectural and loss-function effects.

Methods and Model Variants

The benchmark comprises seven methods:

• Vanilla LSTM: A two-layer unidirectional LSTM baseline trained with MSE loss.
• LSTM + Attention: Augmentation with Bahdanau-style additive attention, concatenated with the final hidden state and an MLP head.
• Bidirectional LSTM (BiLSTM): Processes sequences in both temporal directions within each window, enriching local representations.
• BiLSTM + Huber loss: Substitution of MSE with Huber loss ($\delta=1.0$), mitigating gradient domination by outlier errors.
• Temporal Convolutional Network (TCN): Four-layer dilated causal CNN with weight normalization and dropout, trained on delta targets with Huber loss.
• CNN + LSTM: 1D CNN front-end to augment each timestep with learned local features before an LSTM and Huber loss head.
• CNN + BiLSTM: Analogous to the above, but replaces the LSTM backbone with a bidirectional variant.

All models share standardized preprocessing and an input window of length 50. Forecasts are scored on a 0-100 scale; 100 denotes perfect long-horizon prediction. Metrics include leaderboard scores, mean epoch training losses, and mean test-batch RMSEs (in original coordinates).

Results and Numerical Findings

Model performance is dominated by the BiLSTM + Huber loss configuration, achieving the highest score (58.81), lowest mean training loss (0.0149), and lowest mean RMSE (1.176). Bidirectionality alone (BiLSTM) outperforms the unidirectional LSTM, which itself surpasses all convolutional and attention-augmented variants. Notably:

• LSTM + Attention degrades performance significantly, scoring only 45.72 and suffering catastrophic RMSE spikes (up to 8.94) in harder evaluation pairs, counter to its usual application in sequence modeling.
• CNN front-ends do not yield improvements; both CNN + LSTM and CNN + BiLSTM lag behind their non-CNN counterparts in leaderboard scores despite competitive training losses, indicative of overfitting to features that do not generalize under long autoregressive rollouts.
• TCN fails to outperform recurrent models, reflecting the limitations of causal convolutional architectures in short-window chaotic forecasting.

Per-pair RMSE analysis confirms these trends: BiLSTM + Huber maintains robust error control across challenging regimes (pair 6 RMSE: 1.34, 3.12, 2.10), while attention-augmented and CNN-hybrid models collapse under distributional shifts.

Mechanistic Insights and Architectural Implications

• Bidirectional context enriches local dynamical encoding: Even in autoregressive settings where future context is unavailable during rollout, the backward pass captures within-window trends (e.g., lobe-switching), heightening regime identification and improving stability.
• Huber loss regularizes error propagation: Its linear tail attenuates the dominance of sporadic large residuals, fostering gradient signals representative of typical-case forecast accuracy, a crucial property in chaotic rollouts where outlier errors are frequent.
• Attention and CNN augmentations fail in near-Markovian, short-window settings: Attention introduces undue parametric complexity, vulnerable to overfitting and neglectful of the system's inherent temporal locality. CNN front-ends tend to oversmooth sharp transitions vital for accurate lobe-switching forecasts, resulting in amplified error drift.
• TCN's parallelism and receptive field are underutilized in low-dimensional, short-window Lorenz dynamics: The recurrent backbone’s inherent recency bias and adaptive forgetting mechanisms suit the system’s Markovian structure better than stacked convolutions.

Practical and Theoretical Implications

This study substantiates that for short input windows and lengthy autoregressive rollouts of chaotic dynamical systems, the simplest sequence model—a BiLSTM with robust loss—exhibits superior generalization and predictive stability relative to purported improvements popular in broader time-series domains. Results contradict the default expectation that additive attention or convolutional feature extractors universally augment recurrent models.

Implications extend to general scientific ML benchmarking: architectural innovations effective in conventional tasks may fail in settings with exponential error amplification or near-Markovian structure. The work echoes findings in other domains (e.g., nuclear reactor modeling, seismic wavefield prediction) where hidden-test-set CTFs reveal gaps between one-step training and long-horizon generalization performance.

Potential Future Directions

Several avenues are proposed:

• State-delta prediction with BiLSTM, possibly stabilizing rollout trajectories further.
• Application of multi-scale sequence models (e.g., TimeMixer) to explicitly model fast oscillations vs. slow lobe-switching.
• Hybrid approaches (e.g., SINDy + RNN) that fuse sparse polynomial regression with sequence modeling, leveraging underlying system physics.
• Ensembling BiLSTM + Huber models for variance reduction and score improvement.
• Exploration of alternate attention mechanisms (multi-head, scaled-dot-product) and per-horizon hyperparameter tuning.

Conclusion

The empirical findings robustly demonstrate that, in the context of forecasting chaotic systems under rigorous, standardized conditions, bidirectional recurrent architectures augmented with robust loss functions decisively outperform attention-based enhancements, convolutional hybridizations, and fully convolutional sequence models. The results serve as a cautionary note for transposing sequence modeling advances without consideration of system-specific error dynamics, window length, and Markovian properties, and highlight the importance of physics-informed model selection and robust benchmarking protocols for scientific AI applications.