Residual Deep Gaussian Processes on Manifolds (2411.00161v2)

Abstract: We propose practical deep Gaussian process models on Riemannian manifolds, similar in spirit to residual neural networks. With manifold-to-manifold hidden layers and an arbitrary last layer, they can model manifold- and scalar-valued functions, as well as vector fields. We target data inherently supported on manifolds, which is too complex for shallow Gaussian processes thereon. For example, while the latter perform well on high-altitude wind data, they struggle with the more intricate, nonstationary patterns at low altitudes. Our models significantly improve performance in these settings, enhancing prediction quality and uncertainty calibration, and remain robust to overfitting, reverting to shallow models when additional complexity is unneeded. We further showcase our models on Bayesian optimisation problems on manifolds, using stylised examples motivated by robotics, and obtain substantial improvements in later stages of the optimisation process. Finally, we show our models to have potential for speeding up inference for non-manifold data, when, and if, it can be mapped to a proxy manifold well enough.

• The paper introduces a novel framework that extends deep Gaussian processes to Riemannian manifolds through residual connections.
• It leverages manifold-to-manifold layers via Gaussian vector fields and exponential maps to overcome Euclidean limitations.
• Empirical tests on tasks like wind interpolation and robotic navigation demonstrate enhanced performance over traditional GP methods.

Residual Deep Gaussian Processes on Manifolds

In this paper, the authors propose a sophisticated model class termed residual deep Gaussian processes on manifolds, specifically designed to enhance performance in tasks where data naturally resides on non-Euclidean domains. These tasks include complex patterns in climatic wind data, robotics pathfinding, and potentially non-manifold data approximations using suitable manifold embeddings. The research primarily focuses on exploiting Riemannian geometry to extend existing deep Gaussian processes (DGPs) from Euclidean spaces to manifolds, improving both predictive accuracy and uncertainty estimation.

Key Contributions

The paper contributes a novel paradigm where DGPs retain manifold structure across layers, each implementing manifold-to-manifold transformations via Gaussian vector fields (GVFs) and exponential maps. This approach bypasses the inherent Euclidean constraints of standard GPs, ensuring compatibility with the manifold inputs and outputs. The paper introduces the concept of residual connections, akin to those in neural networks, where each layer output is a perturbation of its input, governed by a GVF.

Important elements introduced include:

• Manifold-to-Manifold Layers: Unlike the conventional construction of DGPs, the proposed architecture models each hidden layer as a composition of manifold-valued Gaussian processes, rendering them adept at dealing with Riemannian manifold data.
• Gaussian Vector Fields (GVF): The authors explore several constructions of GVFs to represent layers, emphasizing Hodge-compositional Matérn GVFs on spaces like spheres, which naturally cater to diverse applications such as climate modeling.

Methodology

To make these models tractable and efficient, the authors employ a variational inference scheme tailored for GPUs on manifolds, inspired by the doubly stochastic inference framework in Euclidean deep GPs. The model employs both inducing variables and interdomain inducing variables with an emphasis on spectral methods to facilitate computation on compact manifolds efficiently.

Empirical Evaluation

Through rigorous synthetic and real-world benchmarks, the efficacy of residual deep GPs is validated. In Bayesian optimization scenarios, these models demonstrate meaningful improvements over shallow manifold-aware GPs when optimizing irregular and singular functions. Further experiments on wind velocity interpolation on the globe demonstrate the superiority of the proposed model, especially in capturing complex patterns at lower altitudes where traditional GPs falter.

Implications and Future Directions

Residual deep GPs on manifolds open various directions for future research. Their ability to capture complex manifold structures paves the way for advances in domains requiring robust geometric understanding, such as robotics navigation in intricate terrains or three-dimensional shape analysis. There is also potential to apply this framework to speed up inference for inherently Euclidean datasets by embedding them into proxy manifolds, provided the mappings preserve relevant geometric properties.

The research presented can significantly influence how manifold data is processed, offering a robust alternative that leans on deep structures while maintaining the integrity of manifold representations. This pioneering step suggests manifold-directed enhancements in scalable Bayesian architectures that focus on achieving better approximation while preserving computational efficiency.

In aggregate, this work sets the stage for a new class of models that effectively marry deep learning's hierarchical advantages with the geometric insights of GPs, potentially influencing future methodologies in manifold-based data analytics and the science of machine learning on non-Euclidean domains.