Stationarity without mean reversion in improper Gaussian processes (2310.02877v2)

Abstract: The behavior of a GP regression depends on the choice of covariance function. Stationary covariance functions are preferred in machine learning applications. However, (non-periodic) stationary covariance functions are always mean reverting and can therefore exhibit pathological behavior when applied to data that does not relax to a fixed global mean value. In this paper we show that it is possible to use improper GP priors with infinite variance to define processes that are stationary but not mean reverting. To this aim, we use of non-positive kernels that can only be defined in this limit regime. The resulting posterior distributions can be computed analytically and it involves a simple correction of the usual formulas. The main contribution of the paper is the introduction of a large family of smooth non-reverting covariance functions that closely resemble the kernels commonly used in the GP literature (e.g. squared exponential and Mat\'ern class). By analyzing both synthetic and real data, we demonstrate that these non-positive kernels solve some known pathologies of mean reverting GP regression while retaining most of the favorable properties of ordinary smooth stationary kernels.

• The paper presents a novel improper GP regression framework that forgoes finite variance assumptions to enable modeling of non-mean-reverting stationary processes.
• The authors derive analytical posterior formulas and introduce innovative kernels, including the Smooth Walk and generalized improper Matérn kernels.
• Empirical evaluations reveal enhanced performance in synthetic scenarios and real-world tasks, notably improving stock price forecasting and reducing prediction errors.

An Analysis of Improper Gaussian Process Regression and Kernels

This paper addresses a limitation in the existing application of Gaussian Process (GP) regression, particularly within machine learning and statistical signal processing, by introducing a framework for improper GP regression using improper kernels. Traditionally, stationary covariance functions, while popular, exhibit mean-reverting behavior that can misrepresent data processes which do not revert to a global mean. The authors propose an innovative approach using improper GP priors with infinite variance to construct stationary processes devoid of mean reversion, presenting improper kernels such as the Smooth Walk kernel and variations of the Matérn kernels in this context.

Conceptual Foundation

Gaussian processes offer a versatile, non-parametric approach to regression and classification, relying heavily on the properties of their covariance function. Stationarity and isotropy are common in these functions but inherently impose a mean-reversion characteristic not suitable for all data types, such as financial time series. The authors circumvent this by employing improper Gaussian Process priors whose stationary covariance functions have infinite variance everywhere, allowing for the modeling of data without mean-reversion tendencies.

Improper Kernels

The introduction of the Smooth Walk kernel exemplifies the potential of improper kernels. It generates infinitely smooth outputs and is complemented by the proposed improper Matérn kernels, generalized to be differentiable any number of times specified by the user. These kernels operate within an improper regime, only definable through a conscious removal of finite variance assumptions, thereby expanding the scope of regression models to produce non-mean-reverting outputs.

Theoretical Contributions

Theoretical advances include deriving analytical posterior distributions for regressions using these improper kernels. The paper provides clear closed-form matrix formulas for posterior computation, factoring in the infinite constant shift in covariance to yield a well-defined posterior even from an improper prior. These contributions extend the flexibility of GP models, presenting a novel approach to non-mean-reverting processes while retaining the favorable properties seen with smooth stationary kernels.

Performance Evaluation

Empirical analyses underscore the utility of improper kernels, demonstrating enhanced performance in both synthetic scenarios and real-world applications like stock price forecasting. Improper kernels performed significantly better than traditional stationary kernels, shown by robust calibration of probabilistic intervals for stock predictions and superior mean squared error performance across numerous multivariate regression tasks from the UCI dataset repository.

Practical Implications and Future Work

The proposed framework enhances GP regression's adaptability to real-world phenomena that deviate from mean-reversion assumptions, showing promising applicability in fields such as econometrics and geostatistics. By using Spectral Density analysis tied with distribution theory, the paper illuminates the mathematical nuances of improper kernels, suggesting a vast potential for exploration in high-dimensional spaces through isotropic adaptations.

While the success of improper kernels in regression tasks is demonstrated, further work is recommended to extend theoretical understanding, particularly regarding their high-dimensional efficacy, where current proofs rely on numerical analysis. Moreover, a systematic exploration of other improper kernels could unveil additional families with distinct operational advantages.

In summary, improper GP regression stands as a compelling evolution of Gaussian processes, providing a nuanced toolset for modeling complex, non-mean-reverting data, promising substantial impact on both theoretical advancements and practical applications in machine learning and related fields.