Infinite-Horizon Gaussian Processes (1811.06588v1)

Abstract: Gaussian processes provide a flexible framework for forecasting, removing noise, and interpreting long temporal datasets. State space modelling (Kalman filtering) enables these non-parametric models to be deployed on long datasets by reducing the complexity to linear in the number of data points. The complexity is still cubic in the state dimension $m$ which is an impediment to practical application. In certain special cases (Gaussian likelihood, regular spacing) the GP posterior will reach a steady posterior state when the data are very long. We leverage this and formulate an inference scheme for GPs with general likelihoods, where inference is based on single-sweep EP (assumed density filtering). The infinite-horizon model tackles the cubic cost in the state dimensionality and reduces the cost in the state dimension $m$ to $\mathcal{O}(m^2)$ per data point. The model is extended to online-learning of hyperparameters. We show examples for large finite-length modelling problems, and present how the method runs in real-time on a smartphone on a continuous data stream updated at 100~Hz.

• The paper reduces computational complexity from cubic to quadratic per data point by exploiting steady-state conditions in an infinite-horizon GP framework.
• It applies state-space modeling with Kalman filtering and single-sweep EP to efficiently handle non-Gaussian likelihoods in large temporal datasets.
• Empirical results demonstrate real-time prediction capabilities and robust uncertainty estimation, broadening GP applications in dynamic environments.

An Overview of "Infinite-Horizon Gaussian Processes"

The paper discusses the formulation of a novel approach to Gaussian Processes (GPs) titled "Infinite-Horizon Gaussian Processes" (IHGP). The primary focus of this work is to mitigate the computational constraints associated with traditional GP regression and prediction on large temporal datasets. By leveraging state-space models and infinite-horizon approximations, the authors propose a method that significantly reduces the computational complexity with respect to the state dimension.

Background and Motivation

Gaussian Processes are a powerful statistical tool for modeling and forecasting time-series data due to their flexibility and ability to provide uncertainty estimates. However, the cubic computational burden concerning the number of data points and state dimensions severely limits their practical application. The paper leverages state space models, particularly Kalman filtering techniques, for GP applications. Typically, state space models reduce complexity to linear in terms of the data size ($\mathcal{O}(n)$ where $n$ is the number of data points). However, complexity remains cubic ($\mathcal{O}(m^3)$) concerning state dimensions, $m$, which presents an obstacle in handling high-dimensional state spaces efficiently.

For special cases where the GP posterior reaches a steady state for long datasets, the paper posits the use of single-sweep Expectation Propagation (EP) as a solution to formulate inference schemes suitable for GPs with general likelihoods. The approach transforms the standard complexity from cubic to quadratic ($\mathcal{O}(m^2)$) per data point, thus making it more feasible for practical applications requiring real-time processing.

Methodology

The proposed model extends the concept of Kalman filtering into the infinite horizon. By introducing the IHGP, the authors aim to achieve computational efficiency through:

1. Steady-State Kalman Filtering: The model taps into the notion of steady-state conditions for Kalman filters, which turns the computation into a time-invariant process after certain assumptions about noise and data spacing hold true over long timeframes. This exploits the invariant properties of the state covariance and gain as time tends toward infinity in equidistant settings.
2. Handling General Likelihoods: The IHGP allows for the utilization of non-Gaussian likelihoods via moment matching approaches integrated within the ADF-EP framework. This is accomplished by interpolating the Riccati equation solutions for predictive state covariance.
3. Real-Time Online Learning: A substantial portion of the paper is dedicated to discussing how IHGP can be extended for online learning scenarios where hyperparameters must adapt over streaming data without iterative computations over existing data points. This feature is tailored for applications characterized by non-stationary data streams.

Experimental Results

The authors present comprehensive experiments to verify the IHGP against standard GP models in both simulated and real-world scenarios. A key takeaway from the results is the demonstrable efficiency of IHGP in real-time processing settings, significantly reducing the runtime and maintaining a competitive prediction and uncertainty estimation accuracy.

The empirical results (as demonstrated in the timing experiments) show that IHGP offers a practical advantage when dealing with high-dimensional state spaces, with the running times scaling more favorably than standard methods.

Implications and Future Work

The implications of an IHGP framework are profound in fields where real-time decision-making and forecasting are critical, such as finance, autonomous systems, and smart grid technologies. By relaxing the computational constraints, IHGP widens the scope of GP applications potential.

Future research can concentrate on fine-tuning the model for various classes of covariance functions and exploring extensions into more diverse and complex data domains. Additionally, improvements around algorithm stability and convergence in varying environmental contexts would further enhance its capabilities.

In conclusion, this paper introduces an innovative approach to Gaussian Processes that addresses significant computational concerns while retaining robust predictive capabilities. The IHGP framework represents a practical advancement, translating the theoretical benefits of GPs into tangible, scalable solutions.