Deep Kernel Physics-Constrained GPR

• DK-PC-GPR is a hybrid approach combining deep neural feature extraction with Gaussian process regression to embed physics constraints derived from PDEs.
• It constructs a composite kernel by mapping high-dimensional inputs to a lower-dimensional latent space while incorporating differential operator effects for accuracy and uncertainty quantification.
• The framework demonstrates robust performance on various PDE benchmarks through staged training and posterior sampling, offering efficient parameter estimation and scalability.

Deep Kernel Physics-Constrained Gaussian Process Regression (DK-PC-GPR) designates a class of machine learning surrogates that integrate deep neural network-based kernel learning with explicit partial differential equation (PDE) or physics-based constraints within the Gaussian process regression (GPR) paradigm. These frameworks are developed to address inference and uncertainty quantification in high-dimensional, PDE-governed systems under data scarcity and substantial model complexity. By combining the nonlinear representational power of deep neural networks with the probabilistic structure and uncertainty quantification capabilities of GPs, while embedding physical constraints, DK-PC-GPR methods offer robust, efficient surrogates for parameter estimation, forward uncertainty propagation, and scientific machine learning in challenging settings (Yan et al., 17 Sep 2025, Chang et al., 2022, Yan et al., 30 Jan 2025).

1. Model Architecture and Deep Kernel Formulation

DK-PC-GPR models synthesize two principal components: a deep neural network feature extractor and a GPR defined by a composite "deep kernel." The neural network $\phi(x;w)$, parameterized by weights $w$, maps high-dimensional inputs $x\in\mathbb{R}^d$ into a lower-dimensional feature space $\mathbb{R}^p$. The GP is defined over this latent space using a base, positive-definite kernel $k_0$ (typically squared exponential/RBF) with hyperparameters $\theta$. The deep kernel takes the form

$$k(x, x'; \theta, w) = k_0(\phi(x;w), \phi(x';w); \theta)$$

For an RBF base kernel with length-scales $\ell\in\mathbb{R}^p$ and variance $\sigma^2$,

$$k_0(z, z'; \theta) = \sigma^2 \exp\left(-\frac{1}{2} \| (z-z') / \ell \|^2\right)$$

which yields

$w$0

(Yan et al., 17 Sep 2025, Yan et al., 30 Jan 2025, Chang et al., 2022).

Physics constraints associated with the solution $w$1 of a PDE, $w$2 (with $w$3 unknown parameters), are incorporated by leveraging linear operator properties of GPs: if $w$4, then $w$5 is also a GP with covariance $w$6. The joint GP prior is thus defined on state and residual, with cross-covariances $w$7. Differentiation of $w$8 through $w$9 is implemented by automatic differentiation to compute operator effects (Yan et al., 17 Sep 2025, Yan et al., 30 Jan 2025).

2. Incorporation of Physics Constraints

DK-PC-GPR frameworks systematically encode physical knowledge at the core of the learning objective. Approaches include:

• Soft physic-constraint penalties: Physics constraints are imposed via penalization terms in the training objective, e.g., residuals of the governing PDE evaluated at collocation points, acting as soft regularizers.
• Joint GP likelihoods: The GP prior is augmented to include both the field of interest and its PDE residual, with a block-structured covariance incorporating operator effects, yielding an exact likelihood over observed data and physical constraints (Yan et al., 17 Sep 2025, Yan et al., 30 Jan 2025).
• Boltzmann–Gibbs prior: Physical constraints may also be formulated through an energy functional $x\in\mathbb{R}^d$0 associated with the governing physics, incorporated as a Boltzmann–Gibbs prior $x\in\mathbb{R}^d$1 in the objective (Chang et al., 2022).
• Hybrid generative models: Some formulations introduce latent forcing/source processes for non-homogeneous operators, modeled by additional GPs and marginalized via variational inference, yielding regularized model evidence lower bounds (Wang et al., 2020).

The resulting objectives jointly balance data fit, physics consistency, and GP marginal likelihood, typically formulated as

$x\in\mathbb{R}^d$2

with weights tuning the influence of each contribution (Yan et al., 17 Sep 2025, Chang et al., 2022).

3. Inference and Training Methodologies

A defining feature is the separation of inference into staged procedures that leverage the different roles of neural and kernel parameters:

• Stage 1: Physics-constrained DKL training optimizes neural network weights $x\in\mathbb{R}^d$3, kernel hyperparameters $x\in\mathbb{R}^d$4, and PDE parameters $x\in\mathbb{R}^d$5 by minimizing a composite loss via stochastic gradient descent (with all derivatives, including operator actions, computed by automatic differentiation). Empirically, the neural network discovers a low-dimensional manifold, improving scalability and data efficiency (Yan et al., 17 Sep 2025, Yan et al., 30 Jan 2025, Chang et al., 2022).
• Stage 2: Posterior sampling is often performed for the hyperparameters $x\in\mathbb{R}^d$6 with neural features fixed. In (Yan et al., 17 Sep 2025), Hamiltonian Monte Carlo (HMC) is used to sample the joint posterior $x\in\mathbb{R}^d$7, using the block-GP Gaussian likelihood structure. This division cuts the computational cost, as high-dimensional neural weight space is avoided during MCMC (Yan et al., 17 Sep 2025).
• Marginal likelihood maximization: For methods with no Bayesian parameter learning, all parameters are learned via direct maximization of the NLML or ELBO, with automatic differentiation propagated through the kernel and operator-induced dependencies (Yan et al., 30 Jan 2025, Chang et al., 2022, Wang et al., 2020).

Empirical practice uses mini-batch strategies, automatic differentiation for all gradients, and stochastic evaluation of physics penalties at random collocation points to further scale training (Chang et al., 2022, Wang et al., 2020).

4. Uncertainty Quantification and Predictive Inference

For any setting of (kernel, physics) hyperparameters, the predictive distribution at a new location $x\in\mathbb{R}^d$8 is Gaussian with mean and variance determined by standard GP regression formulas, but with the kernel incorporating the learned feature map and operator effects:

$x\in\mathbb{R}^d$9

$\mathbb{R}^p$0

(Yan et al., 17 Sep 2025, Chang et al., 2022, Yan et al., 30 Jan 2025). When hyperparameter uncertainty is retained (as in HMC-based approaches), predictive uncertainty is propagated via Monte Carlo, averaging predictions from samples $\mathbb{R}^p$1:

$\mathbb{R}^p$2

(Yan et al., 17 Sep 2025). This procedure enables full Bayesian uncertainty quantification over both function values and inferred PDE parameters.

5. Empirical Performance and Scaling Characteristics

The DK-PC-GPR methodology has shown strong empirical performance on canonical and high-dimensional inverse PDE benchmarks:

Problem (Dimensionality)	Accuracy Metric	Main Finding
1D heat equation (unknown $\mathbb{R}^p$3)	Max-abs error $\mathbb{R}^p$4	Posterior over $\mathbb{R}^p$5 sharply peaks at true value (Yan et al., 17 Sep 2025)
50D heat equation (3 unknowns)	Posterior variance	Posterior concentrates around the true vector; low-dimensional learned manifold mitigates curse of dimensionality (Yan et al., 17 Sep 2025)
50D advection–diffusion–reaction	Marginal posteriors	Credible intervals tightly contain true $\mathbb{R}^p$6 (Yan et al., 17 Sep 2025)
32x32, 64x64 PDE benchmarks	Validation MSE, Marginals	Physics prior essential: pure deep nets fail; GPR with physics matches true marginals with $\mathbb{R}^p$7 labeled samples (Chang et al., 2022)
Poisson, ADR, heat eq. ($\mathbb{R}^p$8)	Relative $\mathbb{R}^p$9 errors ($k_0$0)	Deep kernel approach outperforms physics-only GPs, especially in high dimensions (Yan et al., 30 Jan 2025)

Training is scalable to high-D with costs dominated by $k_0$1 solves (for $k_0$2 total observations), but with reduced sample complexity via manifold learning. HMC is restricted to hyperparameter/PDE parameter space, keeping computational demands controlled (Yan et al., 17 Sep 2025, Yan et al., 30 Jan 2025). Fast stochastic approaches (e.g. inducing points, mini-batches) enable large-scale regimes (Chang et al., 2022).

6. Relationship to Alternative Physics-Informed Methods

DK-PC-GPR is distinguished by its explicit, joint incorporation of deep feature learning, probabilistic regression, and operator constraints, as contrasted with:

• PINNs (Physics-Informed Neural Networks): Pures neural surrogates enforcing physics via residual losses, lacking built-in uncertainty quantification and often suffering from data-inefficiency in high-D (Yan et al., 30 Jan 2025, Chang et al., 2022).
• Shallow GP surrogates: GPs with standard RBF kernels (no deep feature extraction) scale poorly with D, rapidly lose accuracy, and cannot flexibly represent nonlinear structure (Yan et al., 30 Jan 2025, Chang et al., 2022).
• Latent force models / hybrid Bayesian frameworks: Latent-force and source models address operator constraints via convolutional or generative modeling, but typically operate in relatively low-D and are less effective with sparse data (Wang et al., 2020).
• PDE-constrained kernels: GPs incorporating linear operators into kernel design but without deep feature extractors struggle with complex, high-D nonlinearities (Yan et al., 30 Jan 2025).

DK-PC-GPR unifies neural dimensionality reduction with exact Bayesian UQ and joint PDE constraint modeling.

7. Limitations, Extensions, and Open Challenges

Key limitations of current DK-PC-GPR frameworks include:

• Cubic scaling in dataset size: $k_0$3 inverses/gp-factorizations limit full applicability to very large data; inducing-point, SKI/KISS-GP, or random feature approximations are indicated for future scaling (Yan et al., 30 Jan 2025, Chang et al., 2022).
• Restriction to (quasi-)linear PDEs: Many theoretical guarantees depend on operator linearity to preserve GP structure under physics constraints. Extensions to nonlinear PDEs require linearization or variational approximations (Yan et al., 30 Jan 2025, Wang et al., 2020).
• Hyperparameter/architecture tuning: Proper balancing of data and physics losses, choice of latent/feature map dimension, regularization, and network flexibility are crucial for robust performance and model identifiability (Yan et al., 30 Jan 2025).
• Physics-agnostic networks: Incorporating physical symmetries or conservation laws into network architectures may enhance physical fidelity and generalization (Yan et al., 30 Jan 2025).

Proposed extensions include multi-output surrogates, scalable GP variants, joint physics-data hybrid losses, and frameworks for nonlinear and hierarchical physics. A plausible implication is that DK-PC-GPR defines a general blueprint for uncertainty-aware scientific models in challenging, data-constrained, high-dimensional domains (Yan et al., 17 Sep 2025, Yan et al., 30 Jan 2025, Chang et al., 2022, Wang et al., 2020).