Learning Chaotic Systems and Long-Term Predictions with Neural Jump ODEs (2407.18808v1)

Abstract: The Path-dependent Neural Jump ODE (PD-NJ-ODE) is a model for online prediction of generic (possibly non-Markovian) stochastic processes with irregular (in time) and potentially incomplete (with respect to coordinates) observations. It is a model for which convergence to the $L^2$-optimal predictor, which is given by the conditional expectation, is established theoretically. Thereby, the training of the model is solely based on a dataset of realizations of the underlying stochastic process, without the need of knowledge of the law of the process. In the case where the underlying process is deterministic, the conditional expectation coincides with the process itself. Therefore, this framework can equivalently be used to learn the dynamics of ODE or PDE systems solely from realizations of the dynamical system with different initial conditions. We showcase the potential of our method by applying it to the chaotic system of a double pendulum. When training the standard PD-NJ-ODE method, we see that the prediction starts to diverge from the true path after about half of the evaluation time. In this work we enhance the model with two novel ideas, which independently of each other improve the performance of our modelling setup. The resulting dynamics match the true dynamics of the chaotic system very closely. The same enhancements can be used to provably enable the PD-NJ-ODE to learn long-term predictions for general stochastic datasets, where the standard model fails. This is verified in several experiments.

• The paper introduces the PD-NJ-ODE model that converges to the L2-optimal predictor for incomplete, path-dependent observations.
• It enhances training by integrating output feedback and input skipping, which significantly improve long-term prediction accuracy in chaotic and stochastic datasets.
• Empirical results and theoretical guarantees highlight the model's potential for practical applications in finance and complex dynamic systems.

Learning Chaotic Systems and Long-Term Predictions with Neural Jump ODEs

Authors: Florian Krach, Josef Teichmann

Overview

The paper "Learning Chaotic Systems and Long-Term Predictions with Neural Jump ODEs" introduces the Path-dependent Neural Jump ODE (PD-NJ-ODE) model. This model is designed for online prediction of generic, potentially non-Markovian, stochastic processes with irregular and incomplete observations. The theoretical underpinnings of the model guarantee convergence to the $L^2$-optimal predictor, characterized by the conditional expectation given the current information. This model neither requires knowledge of the underlying process's law nor the observation framework's specifics. The authors focus on enhancing the PD-NJ-ODE for more accurate long-term predictions, particularly for deterministic chaotic systems.

Main Contributions

1. PD-NJ-ODE Model: The paper outlines a model for predicting stochastic processes with irregular observations. The PD-NJ-ODE leverages neural jump ODEs to process path-dependent data, accommodating incomplete observations.
2. Theoretical Guarantees: Convergence to the $L^2$-optimal predictor is established, ensuring that the model's output aligns with the conditional expectation of the target process.
3. Training Enhancements: Two novel methods are proposed to improve the model's performance:
   • Output Feedback: Incorporating the model's previous outputs as inputs for future predictions to stabilize training.
   • Input Skipping: Randomly omitting intermediate observations during training to encourage long-term prediction accuracy.
4. Empirical Validation: The effectiveness of these methods is demonstrated through extensive experiments, particularly on the chaotic system of a double pendulum and various stochastic datasets.

Technical Details

Path-dependent Neural Jump ODE (PD-NJ-ODE)

The PD-NJ-ODE processes a discrete-time sequence of observations $(X_{t_i})$ with possibly incomplete data, ensuring that the neural model approximates the conditional expectation of the target process. The architecture involves an encoder to map observations to a latent space, a neural ODE to model the dynamics in this space, and a readout network to produce predictions.

Theoretical Convergence

The authors extend existing results, proving under mild assumptions that the output of the PD-NJ-ODE,

$Y^{\theta^{\min}_{m,N_m}},$

converges to the $L^2$-optimal predictor,

$\hat{X} = (E[X_t | \mathcal{A}_t ])_{t \in [0,T]}.$

Learning Deterministic Systems

In deterministic settings, such as chaotic ODE or PDE systems, the conditional expectation coincides with the actual process: $E[X_t | \mathcal{A}_t] = X_t.$ The authors show that using only the initial condition $X_0$ as input and employing all observations in the loss function ensures convergence to the optimal predictor. This method improves the inductive bias for long-term predictions in chaotic systems.

Training Enhancements

Output Feedback: The model iteratively uses its own predictions as part of the input, which is known to stabilize the training of dynamical systems.

Input Skipping: By randomly choosing a subset of observations to omit during training, the model learns to make accurate long-term predictions even when intermediate data is sparse. The effectiveness of this method is shown empirically across multiple datasets.

Experimental Results

The authors conduct experiments on both deterministic and stochastic datasets:

1. Chaotic Systems: The training enhancements are tested on the double pendulum system. The results indicate significant improvements in long-term prediction accuracy when using output feedback and input skipping.
2. Stochastic Datasets: Similar improvements are observed on geometric Brownian motion datasets with both constant and time-dependent drifts. These experiments validate the applicability of the proposed methods to generic stochastic processes.

Implications and Future Directions

The paper's contributions have both practical and theoretical implications:

• Practical Implications: The enhanced PD-NJ-ODE model allows for more reliable and accurate long-term predictions in settings where intermediate data is irregular or incomplete. This is valuable in fields such as finance, where long-term forecasts are crucial.
• Theoretical Implications: The convergence guarantees extend the applicability of neural ODE models to a broader class of stochastic and deterministic processes. This provides a robust framework for learning from irregularly observed data.
• Future Developments: Potential future work includes exploring alternative methods for selecting input observations and extending the model to handle multiple time horizons and higher-dimensional dynamic systems.

Conclusion

The enhancements proposed in this paper address key limitations in long-term prediction accuracy, particularly for chaotic systems. By utilizing output feedback and input skipping, the PD-NJ-ODE model achieves significant improvements while maintaining theoretical guarantees. This work represents a valuable step forward in the application of neural ODEs to predictive modeling in stochastic and deterministic contexts.