Gaussian Process Frameworks

• Gaussian Process frameworks are probabilistic models defined by a mean function and a covariance kernel for uncertainty-aware function learning.
• They employ scalable methods such as sparse approximations and deep kernel learning to efficiently handle high-dimensional and nonstationary data.
• They integrate physical constraints and full Bayesian marginalization to deliver robust predictions and calibrated uncertainty in diverse applications.

A Gaussian Process (GP) framework is a principled probabilistic model for distributions over functions, defined by a mean function and a positive-definite covariance (kernel) function. GP frameworks underpin a wide spectrum of machine learning, scientific computing, and uncertainty quantification methodologies by combining analytical tractability, expressiveness in function learning, and inherent representation of predictive uncertainty. Recent years have witnessed the development of numerous variants and hybridizations of GP frameworks to address scalability, physical constraints, non-stationarity, high-dimensional modeling, multi-fidelity learning, dynamical systems, and compositional inference.

1. Gaussian Processes: Core Structure and Regression

A Gaussian process is a stochastic process $f(x)$ such that, for any collection of input points $X = \{x_1,\dots,x_n\}$, the vector $f(X)$ follows a multivariate normal distribution:

$f(X) \sim \mathcal{N}(m(X), K(X,X))$

where $m(x)$ is the mean function and $K(x, x')$ is the kernel function encoding covariance structure. For observed data $(X, y)$, and assuming $y = f(X) + \epsilon$ with $\epsilon \sim \mathcal{N}(0, \sigma_n^2 I)$, the GP posterior mean and covariance for predictions at new inputs $X_*$ are:

$\mu_* = K(X_*, X)[K(X,X) + \sigma_n^2 I]^{-1} y$

$$\Sigma_* = K(X_*,X_*) - K(X_*,X)[K(X,X) + \sigma_n^2 I]^{-1} K(X,X_*)$$

This enables uncertainty quantification for predictions and hyperparameter learning via the log-marginal likelihood, typically optimized or marginalized in a Bayesian framework (Tazi et al., 2023).

Covariance kernels include stationary (e.g., squared-exponential, Matérn) and more complex, input-dependent forms to capture various types of correlation, smoothness, and prior knowledge.

2. Approximation, Scaling, and Modularization

Naive GP regression scales cubically ($O(n^3)$) with data size, motivating frameworks for efficiency:

• Sparse/Inducing-Point Methods: Place $M \ll N$ inducing variables (pseudo-points) to approximate the full GP posterior, reducing complexity to $O(M^2 N)$. Methods such as Variational Free Energy (VFE), FITC, and Power Expectation Propagation (Power EP) offer a spectrum of approximations by minimizing different divergences or matching moments in a variational or EP setting (Bui et al., 2016).
• Power EP Unification: Provides a general inference-time framework encompassing VFE ($\alpha\to 0$), EP/FITC ($\alpha=1$), and intermediates, balancing predictive accuracy and calibrated uncertainty.
• Interdomain and Multioutput GPs: Use arbitrary linear transformations of the latent function—e.g., derivative, integral, or convolution—to define inducing variables, enabling scalable inference for vector-valued functions, convolutional GPs, and multi-task learning. Modular software (e.g., GPflow) implements these abstractions for extensibility and efficiency (Wilk et al., 2020).
• Scalable Structures: Inducing-sparse GPs, Kronecker-structured methods, composite experts (Bayesian Committee Machines), and stochastic variational inference are combined for performance on large and/or high-dimensional datasets (Tazi et al., 2023, Vakayil et al., 2023).

3. Advanced Covariance Models: Nonstationarity and Deep Kernels

To model data with spatially or contextually varying behavior, several frameworks introduce nonstationary kernels:

• Neural Network Parameterization: Kernel parameters—variance, lengthscale, noise—are made functions of the input via a feedforward neural network. This admits spatially or contextually variant covariance structure, with network weights and GP parameters trained jointly via backpropagation through the marginal likelihood (James et al., 16 Jul 2025).
• Deep Kernel Learning: Maps inputs through a learned neural network before applying a base kernel (e.g., squared-exponential), increasing representational flexibility and enabling adaptation to complex data features (Chang et al., 2022).
• Hybrid Data-Physics Kernels: GP covariance structure is simultaneously shaped by data and encoded physical constraints—such as through Boltzmann–Gibbs factors that regularize predictions to be consistent with PDEs (Chang et al., 2022).

Models are trained via maximum-likelihood or evidence maximization, with uncertainty quantified via the posterior predictive distribution.

4. Physics, Operators, and Boundary-Constrained GP Frameworks

GP frameworks are extended to incorporate physical knowledge, partial differential equations (PDEs), and operator learning:

• Physics-Constrained GPs: Incorporate physical constraints (e.g., PDEs, boundary conditions) directly in the prior or as additional terms in the loss, such as via co-kriging of both function and differential operator outputs, or by spectral expansion using eigenfunctions satisfying boundary constraints. This tightens uncertainty estimates and yields physically consistent surrogates (Gulian et al., 2020, Chang et al., 2022).
• Hybrid Neural Operator–GPs: GP priors are constructed around neural operators (e.g., Wavelet Neural Operator) as mean functions, combining expressive mapping with Bayesian uncertainty quantification (NOGaP framework) (Kumar et al., 2024).
• Quasi-Gaussianity and Dynamics-Informed GPs: For stochastic fluid dynamics (e.g., 2D stochastic Navier–Stokes), the GP prior is derived from the stationary covariance of the linearized (Ornstein–Uhlenbeck) process, justified by measure equivalence between the true and linearized invariant measures. This grounds the prior in dynamical theory, not just empirical data (Hamzi et al., 26 Nov 2025).

5. Extensions: Non-Standard Inputs, Compositionality, and Prediction Frameworks

GP frameworks address a variety of input/output and system modeling scenarios:

• GPs on Probability Distributions: Inputs are probability measures rather than vectors; kernels are defined via distances (e.g., Wasserstein, Hellinger) between distributions, lifting the GP model to distribution space (Dolgov et al., 2018).
• Stacked/Composite GP Architectures: Organized networks of GPs propagate uncertainty through intermediate variables, enabling model composition, cascading predictions, and uncertainty quantification in dynamical systems or emulations (Abdelfatah et al., 2016).
• Global-Local and Patchwork Approximations: Approaches such as TwinGP combine global and local kernel components, and subset selection strategies for scalable, accurate emulation in massive data settings (Vakayil et al., 2023).
• Resource and Feedback Control: GPs model latent parameters (e.g., snap feedforward for motion control, time-varying interference for communication systems) as smooth functions with uncertainty, enabling adaptive prediction and resource allocation (Haren et al., 2022, Shah et al., 23 Jan 2025).

6. Full Bayesian Treatment and Kernel Uncertainty

• Generalized GP Frameworks for Inference Stability: The Generalized GP (Gen GP) approach treats all kernel hyperparameters—including the Matérn smoothness $\nu$—as free variables, imposing full Bayesian marginalization rather than optimization. This avoids artificial overconfidence, producing more robust and consistent function estimates and uncertainty bands (e.g., in cosmological inference for $H_0$), especially when the mean function encodes parametric physical models (Ruchika et al., 4 Oct 2025).
• Marginalization vs. MAP Optimization: Empirical evidence demonstrates that uncertainty estimates and function reconstructions diverge significantly between maximum a posteriori and marginalized hyperparameter treatments when standard GP frameworks are used. The Gen GP paradigm enforces methodological consistency and honest error quantification.

Framework	Key Idea	Reference
Sparse/Inducing-Point	$O(M^2 N)$ power-EP/VFE/EP approximations for scalability	(Bui et al., 2016)
Interdomain/Multioutput	Arbitrary linear transforms (e.g., convolution, derivative); deep/multioutput GPs; modular software	(Wilk et al., 2020)
Nonstationary GP	Input-dependent kernel parameters via neural nets	(James et al., 16 Jul 2025)
Physics-constrained GP	PDEs/Boundary conditions via spectral kernels and co-kriging	(Gulian et al., 2020, Chang et al., 2022)
Hybrid Neural Operator	Operator-learned mean with GP correction, Kronecker kernels	(Kumar et al., 2024)
Gen GP	Bayesian full marginalization over Matérn $\nu$ and kernel params	(Ruchika et al., 4 Oct 2025)
Dynamics-Informed GP	OU-based spectral prior matched to SPDE invariant measure	(Hamzi et al., 26 Nov 2025)

7. Empirical Validation, Practical Guidance, and Limitations

• Case studies demonstrate that GP frameworks, with appropriate kernel selection and scalable inference, can match or exceed the predictive accuracy and uncertainty calibration of non-Gaussian or black-box models in diverse applications, including UQ in high-dimension, control, environmental modeling, emulation, and cosmological inference (Tazi et al., 2023, Vakayil et al., 2023, Chang et al., 2022, Ruchika et al., 4 Oct 2025).
• Modern frameworks (e.g., GPflow, GPyTorch) provide extensible and computationally optimized modules implementing these approaches.
• Limitations include: computational cost for exact inference, extrapolation instability in deep/neural-kernel variants, narrow support for prior knowledge in nonphysics-based models, and challenges of hyperparameter selection in very high dimensions or with small data.

8. Outlook and Theoretical Advances

Recent developments in Gaussian process frameworks emphasize:

• Integration of physics and data for data-efficient and physically consistent surrogates,
• Modular, software-driven composition for rapid experimentation and expansion to new problem domains,
• Full Bayesian treatment (marginalization instead of fixed or optimized kernels) to maintain rigor in uncertainty quantification and model selection,
• Direct correspondence between dynamical system invariants and probabilistic priors for SPDEs and turbulent flows.

Collectively, modern GP frameworks provide unified, extensible methodologies for principled Bayesian learning, prediction, and uncertainty characterization in increasingly complex data and modeling environments.