Physics-Based Deep Kernel Learning

• Physics-Based Deep Kernel Learning is a method that combines neural feature extraction with Gaussian processes and physical constraints to robustly model complex, scarce-data regimes.
• It incorporates physical structures such as differential equations and PDE-constrained covariance, enabling the embedding of governing laws directly into the kernel design.
• The approach supports active learning and scalable uncertainty quantification, making it valuable for high-dimensional inverse problems and simulation-driven scientific applications.

Physics-Based Deep Kernel Learning (DKL) denotes a class of models in which the standard DKL construction—a neural feature extractor coupled to a Gaussian process (GP)—is combined with physical structure, physically interpreted observables, or governing equations. In the literature surveyed here, that combination appears in several distinct forms: posterior regularization by differential equations with latent sources, PDE-constrained joint GP models over solutions and forcing terms, patch-based regression on physically meaningful microscopy channels, and active-learning surrogates for simulator-defined physical objectives. Across these variants, the recurring motivation is the same: to retain nonlinear representation learning while preserving uncertainty quantification in regimes characterized by scarce data, extrapolation, high-dimensional inputs, or expensive physical evaluation (Wang et al., 2020, Yan et al., 30 Jan 2025, Liu et al., 2021).

1. Canonical DKL backbone

At the methodological core, physics-based variants inherit the standard DKL decomposition in which a neural network maps the original input into a latent representation and a GP operates on that learned space. One explicit formulation writes

$k_{\text{DKL}(\mathbf{q},\mathbf{q}';\boldsymbol{\theta},\boldsymbol{\omega}) = \sigma^2 \exp\!\left( -\frac12 (\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})-\mathbf{h}^L(\mathbf{q}';\boldsymbol{\omega}))^\top \operatorname{diag}(\boldsymbol{\ell}^2)^{-1} (\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})-\mathbf{h}^L(\mathbf{q}';\boldsymbol{\omega})) \right),$

where $\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})$ is the final neural latent layer and $\boldsymbol{\theta}=\{\sigma^2,\boldsymbol{\ell}\}$ are ARD kernel hyperparameters. A closely related regression form is

$$y_n = f(x_n) + \epsilon_n, \qquad f \sim \mathrm{GP}(0,K_{\mathrm{DKL}}),$$

with

$K_{\mathrm{DKL}(x,x' \mid w,\theta) = K_{\mathrm{RBF}(g(x\mid w), g(x'\mid w)\mid \theta),$

so the GP acts on a learned embedding $g(x\mid w)$ rather than on raw coordinates (Yan et al., 30 Jan 2025, Liu et al., 2021).

This GP layer preserves the standard predictive structure of kernel regression. For test points $X_*$, one representative posterior is

$f_* \sim \mathrm{MVN}(\mu_{\text{post}, \Sigma_{\text{post}),$

with

$\mu_{\text{post} = K(X_*,X)\big(K(X,X)+\sigma^2 I\big)^{-1} y,$

and

$\Sigma_{\text{post} = K(X_*,X_*) - K(X_*,X)\big(K(X,X)+\sigma^2 I\big)^{-1}K(X,X_*).$

Physics-based DKL does not replace this GP machinery; rather, it changes what the inputs mean, how the kernel is constrained, and how the posterior is trained or interpreted (Liu et al., 2021).

2. Modes of physical integration

The literature uses “physics-based” in more than one sense. In some works, physics is encoded directly in the probabilistic model; in others, it enters through the semantics of observables, causal assumptions, or simulator-generated objectives.

Mode of integration	Mechanism	Representative papers
Differential-equation posterior regularization	Latent-source generative component and collapsed ELBO	PI-DKL (Wang et al., 2020)
PDE-constrained covariance structure	Joint GP over $\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})$0 and $\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})$1	PDE-DKL (Yan et al., 30 Jan 2025), inverse PDE framework (Yan et al., 17 Sep 2025)
Physically interpreted multimodal observables	Causal scaffold and patch-to-property prediction	Ferroelectric microscopy (Liu et al., 2021)
Physics through simulator and objectives	Active learning on physical trajectories and targets	FerroSIM study (Valleti et al., 2023)

The strongest form of integration appears when the governing operator acts directly on the kernel or on the posterior sample used in the training objective. In PI-DKL, the physical law is written as

$\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})$2

with $\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})$3 modeled as a latent source GP, and posterior samples of the target GP are pushed through $\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})$4 inside a Bayesian hybrid model (Wang et al., 2020). In PDE-DKL and in the later inverse-PDE framework, the operator is embedded in the covariance itself, so that the GP prior over the solution induces a GP over the PDE-transformed quantity (Yan et al., 30 Jan 2025, Yan et al., 17 Sep 2025).

A weaker but still explicitly physical sense appears when the model is built around observables that already have a mechanistic interpretation. In multimodal BE-PFM, DKL is used after a physically motivated causal framing in which morphology, piezoresponse, resonance frequency, $\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})$5-factor, and a nonlinearity proxy are treated as nanoscale physical observables linked by known microscopy physics and materials assumptions (Liu et al., 2021). By contrast, the FerroSIM optimization study is “physics-based” primarily because the inputs are simulator-generated electric-field trajectories and the targets are physical outputs such as curl, normalized curl, and total polarization, not because physics is embedded in the kernel or loss itself (Valleti et al., 2023).

3. Differential equations, PDE constraints, and inverse problems

A major branch of physics-based DKL uses differential equations as the primary source of inductive bias. PI-DKL starts from a deep-kernel GP over the target function and augments it with a latent-source equation

$\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})$6

where $\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})$7 itself carries a GP prior. The model uses a posterior surrogate sample

$\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})$8

and applies the differential functional to obtain

$\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})$9

After marginalizing the latent variables, learning maximizes the collapsed ELBO

$\boldsymbol{\theta}=\{\sigma^2,\boldsymbol{\ell}\}$0

This yields a soft posterior regularization scheme rather than a hard PDE constraint, and the paper positions it as a route to improved prediction accuracy and uncertainty quantification under scarce data and extrapolation (Wang et al., 2020).

PDE-DKL makes the PDE constraint more explicit. It considers a linear operator

$\boldsymbol{\theta}=\{\sigma^2,\boldsymbol{\ell}\}$1

over joint spatial–temporal–parametric coordinates $\boldsymbol{\theta}=\{\sigma^2,\boldsymbol{\ell}\}$2. The deep kernel is constructed on a latent representation $\boldsymbol{\theta}=\{\sigma^2,\boldsymbol{\ell}\}$3, but the PDE acts with respect to the original coordinates. The resulting joint covariance is

$\boldsymbol{\theta}=\{\sigma^2,\boldsymbol{\ell}\}$4

and training minimizes the negative log marginal likelihood of the combined solution and forcing observations. Boundary and initial conditions enter as observations of $\boldsymbol{\theta}=\{\sigma^2,\boldsymbol{\ell}\}$5, while forcing information enters as observations of $\boldsymbol{\theta}=\{\sigma^2,\boldsymbol{\ell}\}$6. The reported numerical results show large gains over PDE-GP in $\boldsymbol{\theta}=\{\sigma^2,\boldsymbol{\ell}\}$7-dimensional settings and feasibility in $\boldsymbol{\theta}=\{\sigma^2,\boldsymbol{\ell}\}$8-dimensional cases where PDE-GP is omitted as computationally infeasible; for example, on the $\boldsymbol{\theta}=\{\sigma^2,\boldsymbol{\ell}\}$9-dimensional Poisson equation the reported errors are $$y_n = f(x_n) + \epsilon_n, \qquad f \sim \mathrm{GP}(0,K_{\mathrm{DKL}}),$$0, $$y_n = f(x_n) + \epsilon_n, \qquad f \sim \mathrm{GP}(0,K_{\mathrm{DKL}}),$$1 for PDE-GP and $$y_n = f(x_n) + \epsilon_n, \qquad f \sim \mathrm{GP}(0,K_{\mathrm{DKL}}),$$2, $$y_n = f(x_n) + \epsilon_n, \qquad f \sim \mathrm{GP}(0,K_{\mathrm{DKL}}),$$3 for PDE-DKL, while at $$y_n = f(x_n) + \epsilon_n, \qquad f \sim \mathrm{GP}(0,K_{\mathrm{DKL}}),$$4 PDE-DKL reports $$y_n = f(x_n) + \epsilon_n, \qquad f \sim \mathrm{GP}(0,K_{\mathrm{DKL}}),$$5, $$y_n = f(x_n) + \epsilon_n, \qquad f \sim \mathrm{GP}(0,K_{\mathrm{DKL}}),$$6 (Yan et al., 30 Jan 2025).

The inverse-problem formulation in high-dimensional PDEs extends this logic from solution reconstruction to parameter inference. It assumes a linear time-dependent PDE with unknown parameters $$y_n = f(x_n) + \epsilon_n, \qquad f \sim \mathrm{GP}(0,K_{\mathrm{DKL}}),$$7, sparse exact observations of both $$y_n = f(x_n) + \epsilon_n, \qquad f \sim \mathrm{GP}(0,K_{\mathrm{DKL}}),$$8 and $$y_n = f(x_n) + \epsilon_n, \qquad f \sim \mathrm{GP}(0,K_{\mathrm{DKL}}),$$9, and a two-stage Bayesian pipeline. Stage 1 jointly optimizes $K_{\mathrm{DKL}(x,x' \mid w,\theta) = K_{\mathrm{RBF}(g(x\mid w), g(x'\mid w)\mid \theta),$0 via a composite loss

$K_{\mathrm{DKL}(x,x' \mid w,\theta) = K_{\mathrm{RBF}(g(x\mid w), g(x'\mid w)\mid \theta),$1

where the three terms respectively fit observed state values, enforce PDE consistency at collocation points, and penalize the joint GP negative log marginal likelihood. Stage 2 fixes the neural feature extractor and uses Hamiltonian Monte Carlo to sample the posterior over PDE parameters and kernel hyperparameters. This separates nonlinear representation learning from posterior sampling and yields posterior means, variances, and $K_{\mathrm{DKL}(x,x' \mid w,\theta) = K_{\mathrm{RBF}(g(x\mid w), g(x'\mid w)\mid \theta),$2 credible intervals for the physical parameters (Yan et al., 17 Sep 2025).

4. Physically interpreted observables and scientific applications

Not all physics-based DKL formulations are PDE solvers. In ferroelectric microscopy, DKL is embedded in a pipeline that combines a linear non-Gaussian acyclic model (LiNGAM) with patch-based GP regression on BE-PFM images from PbTiO$K_{\mathrm{DKL}(x,x' \mid w,\theta) = K_{\mathrm{RBF}(g(x\mid w), g(x'\mid w)\mid \theta),$3 thin films. The central premise is that known materials structure can serve as prior information for predicting unmeasured or partially observed physical properties. The variables include morphology, morphology curvature, amplitude, phase, frequency, and $K_{\mathrm{DKL}(x,x' \mid w,\theta) = K_{\mathrm{RBF}(g(x\mid w), g(x'\mid w)\mid \theta),$4-factor, and the paper explicitly distinguishes intrinsic descriptors, ideal observables, and measured channels. The DKL model then uses local structural patches from channels such as amplitude, phase, topography, and frequency to predict targets including morphology, amplitude, piezoresponse, polarization, resonance frequency, and a nonlinearity proxy (Liu et al., 2021).

The physical interpretation is central rather than incidental. The study reports that DKL can predict morphology from amplitude, phase, or frequency with good agreement to ground truth; predict amplitude from the polarization map even when trained on only $K_{\mathrm{DKL}(x,x' \mid w,\theta) = K_{\mathrm{RBF}(g(x\mid w), g(x'\mid w)\mid \theta),$5 of the data; predict amplitude, phase, and polarization from topography; and predict resonance frequency and SHO-fit $K_{\mathrm{DKL}(x,x' \mid w,\theta) = K_{\mathrm{RBF}(g(x\mid w), g(x'\mid w)\mid \theta),$6 from polarization. It also reports that generally less than $K_{\mathrm{DKL}(x,x' \mid w,\theta) = K_{\mathrm{RBF}(g(x\mid w), g(x'\mid w)\mid \theta),$7 of data used for DKL training is sufficient for good reconstruction of the full image. The uncertainty maps are physically interpreted: larger uncertainty in large in-plane $K_{\mathrm{DKL}(x,x' \mid w,\theta) = K_{\mathrm{RBF}(g(x\mid w), g(x'\mid w)\mid \theta),$8-domains is attributed to the fact that vertical PFM incompletely captures in-plane polarization. This is a case where the DKL core is mathematically standard, but the workflow is physically anchored by microscopy structure, materials priors, and causal assumptions (Liu et al., 2021).

A different scientific application appears in process optimization with a ferroelectric simulator. There the inputs are $K_{\mathrm{DKL}(x,x' \mid w,\theta) = K_{\mathrm{RBF}(g(x\mid w), g(x'\mid w)\mid \theta),$9 electric-field curves of the form

$g(x\mid w)$0

and the outputs are physical objectives derived from FerroSIM: curl, normalized curl, and total polarization. The study argues that DKL with active learning constructs latent spaces that are more compact and smooth for optimization than VAE-derived latent spaces, because the latent manifold is shaped jointly by the input data and the target functionality. In the active setting, the model starts from $g(x\mid w)$1 measured trajectories and adds $g(x\mid w)$2 Bayesian-optimization evaluations, for a total of $g(x\mid w)$3 of $g(x\mid w)$4 candidate curves. Here physics enters through simulator-defined trajectories and objectives rather than through explicit mechanistic constraints in the kernel (Valleti et al., 2023).

5. Uncertainty quantification, active learning, and design utility

The GP layer gives physics-based DKL its characteristic uncertainty semantics. In regression settings this appears as posterior variance on reconstructed fields or properties; in inverse problems it extends to posterior distributions over physical parameters; in design problems it supports exploration–exploitation policies. This is one reason the literature repeatedly positions DKL between conventional neural surrogates and more explicitly Bayesian scientific models (Yan et al., 17 Sep 2025, Valleti et al., 2023).

The FerroSIM study makes the active-learning role explicit. Its predictive posterior

$g(x\mid w)$5

drives a UCB-style acquisition

$g(x\mid w)$6

and the authors argue that the latent manifold forms as a result of active learning balancing the data distribution and target functionalities. In the reported examples, high-performing trajectories collapse into a small number of broad, connected clusters in latent space, which the paper interprets as more conducive to optimization in feature-rich, target-poor regimes common in domain sciences such as materials synthesis, energy storage, and molecular discovery (Valleti et al., 2023).

In PI-DKL, uncertainty is also treated as a first-class object, but with an extrapolative emphasis. The paper reports that physics regularization reduces overfitting near the training region, yields more uniform and smaller uncertainty in extrapolation regions, and improves test log-likelihood. In the synthetic ODE and diffusion examples, ordinary SKL/DKL exhibit rapidly increasing uncertainty away from observed data, whereas PI-DKL produces smaller and more spatially uniform predictive standard deviations (Wang et al., 2020).

For high-dimensional inverse PDEs, posterior uncertainty is split into two layers: uncertainty conditional on kernel and PDE parameters, and uncertainty induced by the posterior over those parameters. The two-stage HMC framework writes predictive uncertainty through Bayesian model averaging and the law of total expectation and total covariance. This is methodologically important because it distinguishes uncertainty in the field solution from uncertainty in the physical parameters themselves (Yan et al., 17 Sep 2025).

6. Failure modes and methodological debates

A central debate in the DKL literature concerns whether the GP layer really preserves the Bayesian robustness often attributed to kernel methods. One influential result is that DKL trained by maximum marginal likelihood or ELBO optimization can overfit badly in the overparameterized regime, sometimes more severely than a deterministic neural network trained by ordinary maximum likelihood. The argument is that DKL is only “somewhat Bayesian”: the latent function is integrated out, but the neural feature map and kernel hyperparameters are point-estimated. In that regime, the model can improve marginal likelihood by “hacking” the Gram matrix through over-correlation in feature space, so marginal likelihood or ELBO need not function as a trustworthy model-selection criterion. Fully Bayesian treatments using HMC or SGLD were reported to rectify these failures and restore the desired performance improvements over standard neural networks and Gaussian processes (Ober et al., 2021).

A closely related pathology is feature collapse. In sparse inducing-point DKL, unconstrained feature learning can map far-away data points to the same features as those of training-set points, producing unreliable uncertainty estimates even when predictive accuracy remains strong. DUE addresses this by constraining the feature extractor to be approximately bi-Lipschitz through spectral normalization, residual connections, and normalization-aware Lipschitz control. The paper reports that, on CIFAR-10 versus SVHN OOD detection, unconstrained SV-DKL attains AUROC $g(x\mid w)$7 while the bi-Lipschitz variant attains AUROC $g(x\mid w)$8; for GPDNN, the corresponding figures are $g(x\mid w)$9 without the constraint and $X_*$0 with it (Amersfoort et al., 2021).

Guided Deep Kernel Learning reframes the same issue as a guidance problem. It uses an infinite-width NNGP as a Bayesian reference and trains DKL with a hybrid objective consisting of a predictive term plus a KL regularizer toward the guide posterior. In one form, the core objective is

$X_*$1

The paper’s interpretation is that the NNGP adapts the DKL target confidence when it encounters novel data points, thereby reducing overconfidence while preserving predictive performance. This suggests that physics-based DKL cannot rely on “NN features + GP” alone when extrapolation and uncertainty calibration matter; additional geometric or Bayesian control may be necessary (Achituve et al., 2023).

7. Scalability, architectural variants, and open directions

The main scalability bottleneck in DKL is often not the raw input dimension alone, but the dimension and structure of the GP layer after feature extraction. One architectural response is the Deep Additive Kernel (DAK) model, which replaces a high-dimensional last-layer GP by an additive collection of one-dimensional GP units over latent channels:

$X_*$2

With induced prior approximation, each scalar GP unit becomes a Bayesian last-layer component,

$X_*$3

so the full model is reinterpreted as a last-layer Bayesian neural network. The paper reports complexity claims of $X_*$4 inference and $X_*$5 training for DAK-MC, and $X_*$6 inference and $X_*$7 training for DAK-CF. The work is not physics-informed, but it explicitly argues that additivity, low-dimensional latent partitioning, uncertainty-aware last layers, and scalable variational training map naturally onto many physics-informed settings (Zhao et al., 14 Feb 2025).

Another transferable pattern is bilevel adaptation of deep-kernel GPs across related tasks. ADKF-IFT meta-learns a shared feature extractor while adapting GP hyperparameters per task via an implicit-function-theorem hypergradient. The paper is not physics-informed, but it is presented as highly relevant to scientific modeling because it separates what is shared across tasks from what must adapt locally. A plausible implication is that related experiments, parameter regimes, or simulator settings in physics could benefit from the same partition: transferable latent geometry, but task-specific noise, amplitude, or correlation structure (Chen et al., 2022).

The open problems in explicitly physics-based DKL are more concrete. PDE-DKL and the inverse-PDE framework are formulated for linear operators, and the latter uses exact observations with nugget stabilization. Both retain GP cubic scaling in the number of observations even while addressing high input dimensionality. PI-DKL avoids analytically transformed kernels and can handle nonlinear operators, but it retains full-GP costs of $X_*$8. Taken together, these results indicate that the current literature is strongest in low-data, physics-informed regimes where uncertainty quantification is essential, while nonlinear operators, noisy and biased observations, broader calibration analysis, and more scalable posterior approximations remain active directions rather than settled components of the field (Yan et al., 30 Jan 2025, Yan et al., 17 Sep 2025, Wang et al., 2020).