Efficient Transformed Gaussian Process State-Space Models for Non-Stationary High-Dimensional Dynamical Systems (2503.18309v3)

Abstract: Gaussian process state-space models (GPSSMs) offer a principled framework for learning and inference in nonlinear dynamical systems with uncertainty quantification. However, existing GPSSMs are limited by the use of multiple independent stationary Gaussian processes (GPs), leading to prohibitive computational and parametric complexity in high-dimensional settings and restricted modeling capacity for non-stationary dynamics. To address these challenges, we propose an efficient transformed Gaussian process state-space model (ETGPSSM) for scalable and flexible modeling of high-dimensional, non-stationary dynamical systems. Specifically, our ETGPSSM integrates a single shared GP with input-dependent normalizing flows, yielding an expressive implicit process prior that captures complex, non-stationary transition dynamics while significantly reducing model complexity. For the inference of the implicit process, we develop a variational inference algorithm that jointly approximates the posterior over the underlying GP and the neural network parameters defining the normalizing flows. To avoid explicit variational parameterization of the latent states, we further incorporate the ensemble Kalman filter (EnKF) into the variational framework, enabling accurate and efficient state estimation. Extensive empirical evaluations on synthetic and real-world datasets demonstrate the superior performance of our ETGPSSM in system dynamics learning, high-dimensional state estimation, and time-series forecasting, outperforming existing GPSSMs and neural network-based SSMs in terms of computational efficiency and accuracy.

• The paper presents efficient Transformed Gaussian Process State-Space Models (GPSSMs) for tackling high-dimensional, non-stationary systems with novel methods.
• Key methods include using the Linear Model of Coregionalization (LMC) and integrating Input-Dependent Normalizing Flows to model interdependencies and non-stationarity.
• The paper explores Variational Inference techniques like Non-Mean-Field VI with EnKF and Mean-Field VI with Mamba SSMs to improve inference efficiency and capture non-Markovian dynamics.

Gaussian Process State-Space Models for High-Dimensional Dynamical Systems

Gaussian Process State-Space Models (GPSSMs) represent an intersection of Gaussian Processes (GPs) and State-Space Models (SSMs), leveraging the flexibility of GPs to model the underlying dynamics of systems with latent states. The paper discussed here elaborates on GPSSMs, particularly focusing on the challenges and advancements associated with high-dimensional latent spaces.

Core Formulation and Methodology

The traditional GPSSM model integrates a GP prior over the transition function while employing parametric models for the emission function. This method effectively captures the dynamical processes by modeling the latent state transitions with GPs, providing a robust probabilistic framework suitable for systems where modeling flexibility and uncertainty quantification are significant.

The formulation challenges include modeling dependencies within high-dimensional outputs, often leading to a decoupling of output dimensions which may result in loss of inductive biases and degraded performance, particularly under partial observability. Additionally, scalability issues arise from parameter growth and computational complexity, both accelerating with dimensionality increases.

Advances and Solution Approaches

The paper outlines innovative modeling techniques that mitigate these limitations:

• Linear Model of Coregionalization (LMC): The use of LMC allows the characterization of dependencies between outputs without resorting to treating output dimensions independently. This enhancement maintains the interdependencies among different dimensions, improving the model's performance and generalization capabilities.
• Normalizing Flows in GPSSMs: By incorporating normalizing flows, the paper presents a novel approach for transforming Gaussian processes resulting in correlated outputs, addressing the dependency issues in multi-dimensional spaces. This approach holds computational complexity and parameter count linearly with dimensionality, gaining efficiency in application to large-scale systems.
• Input-Dependent Normalizing Flows: The adoption of input-dependent normalizing flows marks a further development. These flows adjust based on the preceding state, introducing non-stationarity and interdependencies within the GP transitions, addressing flexibility without exponential parameter growth.

Computational Aspects

The paper critically examines the inference methodologies using Variational Inference (VI) in GPSSMs. Two approaches are explored:

• Non-Mean-Field VI: This involves modeling dependencies in latent variables using Ensemble Kalman Filtering (EnKF), enabling efficient inference while mitigating computational bottlenecks typical in traditional VI approaches.
• Mean-Field VI with Mamba SSM: Emphasizing historical latent states rather than a strict Markov assumption, this method enhances inference capacity by utilizing Mamba state-space models, aligning with the non-Markovian dynamics inherent in many real-world systems.

Implications and Future Directions

The advancements presented signify substantial improvements in GPSSMs, allowing their application to complex, high-dimensional dynamical systems such as climate modeling, mechanical system diagnostics, and beyond. The paper highlights a trajectory for future research aimed at further optimizing these models, exploring deeper integration with neural networks for dynamic parameter setting, and refining inference strategies.

Furthermore, these innovations bear practical significance in real-time processing and scaling applications where computational constraints and system complexity have presented historic barriers. The integration of efficient computation techniques within GPSSMs could pave the way for their broader adoption across various scientific and engineering domains.

In conclusion, this paper contributes significantly to the understanding and application potential of GPSSMs in high-dimensional settings, providing a fertile ground for future exploration in model flexibility, computational efficiency, and practical applicability in AI-driven dynamical systems analysis.