High-Dimensional Gaussian Process Regression with Soft Kernel Interpolation

Abstract: We introduce Soft Kernel Interpolation (SoftKI), a method that combines aspects of Structured Kernel Interpolation (SKI) and variational inducing point methods, to achieve scalable Gaussian Process (GP) regression on high-dimensional datasets. SoftKI approximates a kernel via softmax interpolation from a smaller number of interpolation points learned by optimizing a combination of the SoftKI marginal log-likelihood (MLL), and when needed, an approximate MLL for improved numerical stability. Consequently, it can overcome the dimensionality scaling challenges that SKI faces when interpolating from a dense and static lattice while retaining the flexibility of variational methods to adapt inducing points to the dataset. We demonstrate the effectiveness of SoftKI across various examples and show that it is competitive with other approximated GP methods when the data dimensionality is modest (around 10).

• The paper introduces SoftKI to mitigate Gaussian Process regression's O(n^3) scaling, decoupling computational cost from data dimensionality.
• It employs softmax-based interpolation and learns inducing point locations, enabling flexible kernel approximations without fixed grid constraints.
• Empirical evaluations on UCI and molecular benchmarks demonstrate SoftKI's superior performance and scalability over traditional GP methods.

High-Dimensional Gaussian Process Regression with Soft Kernel Interpolation

The paper by Camaño and Huang introduces Soft Kernel Interpolation (SoftKI) as a method to address the scalability limitations of Gaussian Process (GP) regression, particularly for high-dimensional datasets. Traditional Gaussian Processes, while powerful, suffer from computational inefficiencies characterized by $\mathcal{O}(n^3)$ time complexity for exact inference, where $n$ is the number of data points. This complexity arises from the necessity to solve linear systems and form covariance matrices that grow quadratically with the dataset size. SoftKI offers a novel route to mitigate these challenges by merging strategies from existing inducing point methods and Structured Kernel Interpolation (SKI).

Technical Contributions and Methodology

SoftKI is inspired by SKI, which leverages rectilinear grids to approximate GP kernels through interpolation. Unlike SKI, which ties the number of inducing points directly to the dimensionality of the data, SoftKI employs a softmax interpolation process to derive kernel approximations. This innovative approach departs from the fixed lattice structure used in SKI, allowing flexibility by learning the optimal positions of inducing points. As a consequence, the computational cost in SoftKI becomes independent of the data's dimensionality, which is a significant advancement for high-dimensional datasets.

SoftKI achieves its aims through several key steps:

1. Soft Kernel Interpolation: The paper uses softmax weights to interpolate between inducing points, ensuring that each data point can be represented more flexibly without being confined to a grid structure.
2. Inducing Point Learning: SoftKI treats the inducing points as hyperparameters, optimizing their locations using a stochastic gradient-based method. This strategy is enhanced through the use of a pseudo-marginal likelihood known as the Hutchinson pseudoloss, which stabilizes the optimization process.
3. Posterior Inference: Given the low-rank nature of the approximated kernel matrix, SoftKI employs matrix decomposition techniques to perform efficient posterior inference, maintaining its computational advantages.

A notable takeaway is that SoftKI supports GPU acceleration and stochastic optimization, broadening its applicability to massive datasets found in practical settings.

Empirical Evaluation

The empirical evaluation conducted by the authors highlights SoftKI's superior performance on various datasets, particularly when dealing with moderate to high-dimensional data. On UCI benchmark datasets, SoftKI not only surpassed other scalable GP methods like SGPR and SVGP in test RMSE for datasets with modest dimensionality but also demonstrated scalability to datasets of larger dimensions. Moreover, its application to high-dimensional molecular dataset benchmarks further validates its versatility and effectiveness where standard methods struggle due to computational constraints.

Implications and Future Directions

The practical implications of SoftKI are substantial. By reducing the computational dependency on data dimensionality, SoftKI extends the applicability of Gaussian Processes to domains that were previously unfeasible, such as high-resolution spatial modeling and complex systems biology where data dimensionality is inherently high. The ability to automatically learn inducing point locations is particularly beneficial in these complex problem spaces, potentially improving model interpretability and prediction robustness.

Theoretically, this work paves the way for future research into more sophisticated kernel approximation methods that dynamically adapt to data characteristics. Further exploration could investigate the integration of SoftKI with derivative observations, which are crucial in fields like chemistry for energy landscape predictions. Another promising direction would be the exploration of SoftKI's integration with ensemble methods to enhance predictive performance further.

In conclusion, the paper presents a substantial contribution to scalable Gaussian Process regression. By strategically leveraging kernel interpolation and inducing point learning without traditional dimensionality constraints, SoftKI embodies a forward-looking approach to handling complex, high-volume datasets, fostering advancements across numerous scientific and engineering domains.