---
title: Physics-Based Deep Kernel Learning
url: https://www.emergentmind.com/topics/physics-based-deep-kernel-learning-dkl
type: topic
---

# Physics-Based Deep Kernel Learning

Physics-Based Deep Kernel Learning (DKL) denotes a class of models in which the standard DKL construction—a neural feature extractor coupled to a Gaussian process (GP)—is combined with physical structure, physically interpreted observables, or governing equations. In the literature surveyed here, that combination appears in several distinct forms: posterior regularization by differential equations with latent sources, PDE-constrained joint GP models over solutions and forcing terms, patch-based regression on physically meaningful microscopy channels, and active-learning surrogates for simulator-defined physical objectives. Across these variants, the recurring motivation is the same: to retain nonlinear representation learning while preserving uncertainty quantification in regimes characterized by scarce data, extrapolation, high-dimensional inputs, or expensive physical evaluation [2006.04976][2501.18258][2110.06888].

## 1. Canonical DKL backbone

At the methodological core, physics-based variants inherit the standard DKL decomposition in which a neural network maps the original input into a latent representation and a GP operates on that learned space. One explicit formulation writes
$$
k_{\text{DKL}(\mathbf{q},\mathbf{q}';\boldsymbol{\theta},\boldsymbol{\omega}) = \sigma^2 \exp\!\left( -\frac12 (\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})-\mathbf{h}^L(\mathbf{q}';\boldsymbol{\omega}))^\top \operatorname{diag}(\boldsymbol{\ell}^2)^{-1} (\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})-\mathbf{h}^L(\mathbf{q}';\boldsymbol{\omega})) \right),
$$
where \(\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})\) is the final neural latent layer and \(\boldsymbol{\theta}=\{\sigma^2,\boldsymbol{\ell}\}\) are ARD kernel hyperparameters. A closely related regression form is
$$
y_n = f(x_n) + \epsilon_n, \qquad f \sim \mathrm{GP}(0,K_{\mathrm{DKL}}),
$$
with
$$
K_{\mathrm{DKL}(x,x' \mid w,\theta) = K_{\mathrm{RBF}(g(x\mid w), g(x'\mid w)\mid \theta),
$$
so the GP acts on a learned embedding \(g(x\mid w)\) rather than on raw coordinates [2501.18258][2110.06888].

This GP layer preserves the standard predictive structure of kernel regression. For test points \(X_*\), one representative posterior is
$$
f_* \sim \mathrm{MVN}(\mu_{\text{post}, \Sigma_{\text{post}),
$$
with
$$
\mu_{\text{post} = K(X_*,X)\big(K(X,X)+\sigma^2 I\big)^{-1} y,
$$
and
$$
\Sigma_{\text{post} = K(X_*,X_*) - K(X_*,X)\big(K(X,X)+\sigma^2 I\big)^{-1}K(X,X_*).
$$
Physics-based DKL does not replace this GP machinery; rather, it changes what the inputs mean, how the kernel is constrained, and how the posterior is trained or interpreted [2110.06888].

## 2. Modes of physical integration

The literature uses “physics-based” in more than one sense. In some works, physics is encoded directly in the probabilistic model; in others, it enters through the semantics of observables, causal assumptions, or simulator-generated objectives.

| Mode of integration | Mechanism | Representative papers |
|---|---|---|
| Differential-equation posterior regularization | Latent-source generative component and collapsed ELBO | PI-DKL [2006.04976] |
| PDE-constrained covariance structure | Joint GP over \(u\) and \(f=\mathcal{A}u\) | PDE-DKL [2501.18258], inverse PDE framework [2509.14054] |
| Physically interpreted multimodal observables | Causal scaffold and patch-to-property prediction | Ferroelectric microscopy [2110.06888] |
| Physics through simulator and objectives | Active learning on physical trajectories and targets | FerroSIM study [2303.14554] |

The strongest form of integration appears when the governing operator acts directly on the kernel or on the posterior sample used in the training objective. In PI-DKL, the physical law is written as
$$
\psi[f(\mathbf x)] = g(\mathbf x),
$$
with \(g\) modeled as a latent source GP, and posterior samples of the target GP are pushed through \(\psi\) inside a Bayesian hybrid model [2006.04976]. In PDE-DKL and in the later inverse-PDE framework, the operator is embedded in the covariance itself, so that the GP prior over the solution induces a GP over the PDE-transformed quantity [2501.18258][2509.14054].

A weaker but still explicitly physical sense appears when the model is built around observables that already have a mechanistic interpretation. In multimodal BE-PFM, DKL is used after a physically motivated causal framing in which morphology, piezoresponse, resonance frequency, \(Q\)-factor, and a nonlinearity proxy are treated as nanoscale physical observables linked by known microscopy physics and materials assumptions [2110.06888]. By contrast, the FerroSIM optimization study is “physics-based” primarily because the inputs are simulator-generated electric-field trajectories and the targets are physical outputs such as curl, normalized curl, and total polarization, not because physics is embedded in the kernel or loss itself [2303.14554].

## 3. Differential equations, PDE constraints, and inverse problems

A major branch of physics-based DKL uses differential equations as the primary source of inductive bias. PI-DKL starts from a deep-kernel GP over the target function and augments it with a latent-source equation
$$
\psi[f(\mathbf x)] = g(\mathbf x),
$$
where \(g\) itself carries a GP prior. The model uses a posterior surrogate sample
$$
f(\mathbf x)=\mu(\mathbf x)+\epsilon \sqrt{v(\mathbf x)}, \qquad \epsilon \sim \mathcal N(0,1),
$$
and applies the differential functional to obtain
$$
g(\cdot)=h(\cdot,\epsilon)=\psi\big[\mu(\cdot)+\epsilon\sqrt{v(\cdot)}\big].
$$
After marginalizing the latent variables, learning maximizes the collapsed ELBO
$$
L = \log \mathcal N(\mathbf y\mid \mathbf 0,K+\tau^{-1}I)
+ \gamma\, \mathbb E_{p(Z)}\mathbb E_{\epsilon\sim\mathcal N(0,1)}
\left[ \log \mathcal N(h(Z,\epsilon)\mid \mathbf 0,G) \right].
$$
This yields a soft posterior regularization scheme rather than a hard PDE constraint, and the paper positions it as a route to improved prediction accuracy and uncertainty quantification under scarce data and extrapolation [2006.04976].

PDE-DKL makes the PDE constraint more explicit. It considers a linear operator
$$
\mathcal{A}[u] := \partial_t u - \mathcal{L}_{\mathbf{x}[u;\boldsymbol{\mu}],
\qquad \mathcal{A}[u(\mathbf{q})] = f(\mathbf{q}),
$$
over joint spatial–temporal–parametric coordinates \(\mathbf{q}=(\mathbf{x},t,\boldsymbol{\mu})\). The deep kernel is constructed on a latent representation \(\mathbf{h}^L(\mathbf{q};\boldsymbol{\omega})\), but the PDE acts with respect to the original coordinates. The resulting joint covariance is
$$
\mathbf{K}_{\text{DKL} = \begin{bmatrix}
k_{\text{DKL}(\mathbf{Q}_u,\mathbf{Q}_u) + \sigma_u^2 \mathbf{I} &
k_{\text{DKL}(\mathbf{Q}_u,\mathbf{Q}_f)\mathcal{A}^\top \\
\mathcal{A}k_{\text{DKL}(\mathbf{Q}_f,\mathbf{Q}_u) &
\mathcal{A}k_{\text{DKL}(\mathbf{Q}_f,\mathbf{Q}_f)\mathcal{A}^\top + \sigma_f^2 \mathbf{I}
\end{bmatrix},
$$
and training minimizes the negative log marginal likelihood of the combined solution and forcing observations. Boundary and initial conditions enter as observations of \(u\), while forcing information enters as observations of \(f\). The reported numerical results show large gains over PDE-GP in \(10\)-dimensional settings and feasibility in \(50\)-dimensional cases where PDE-GP is omitted as computationally infeasible; for example, on the \(10\)-dimensional Poisson equation the reported errors are \(e_u=6.21\%\), \(e_f=6.38\%\) for PDE-GP and \(e_u=0.90\%\), \(e_f=1.06\%\) for PDE-DKL, while at \(d=50\) PDE-DKL reports \(e_u=3.38\%\), \(e_f=4.27\%\) [2501.18258].

The inverse-problem formulation in high-dimensional PDEs extends this logic from solution reconstruction to parameter inference. It assumes a linear time-dependent PDE with unknown parameters \(\boldsymbol{\phi}\), sparse exact observations of both \(u\) and \(f\), and a two-stage Bayesian pipeline. Stage 1 jointly optimizes \((\boldsymbol{\theta},\boldsymbol{\psi},\boldsymbol{\phi})\) via a composite loss
$$
\mathcal{L}(\boldsymbol{\theta}, \boldsymbol{\psi}, \boldsymbol{\phi})
= w_{\text{data} \mathcal{L}_{\text{data}
+ w_{\text{PDE} \mathcal{L}_{\text{PDE}
+ w_{\text{GP} \mathcal{L}_{\text{GP},
$$
where the three terms respectively fit observed state values, enforce PDE consistency at collocation points, and penalize the joint GP negative log marginal likelihood. Stage 2 fixes the neural feature extractor and uses Hamiltonian Monte Carlo to sample the posterior over PDE parameters and kernel hyperparameters. This separates nonlinear representation learning from posterior sampling and yields posterior means, variances, and \(95\%\) credible intervals for the physical parameters [2509.14054].

## 4. Physically interpreted observables and scientific applications

Not all physics-based DKL formulations are PDE solvers. In ferroelectric microscopy, DKL is embedded in a pipeline that combines a linear non-Gaussian acyclic model (LiNGAM) with patch-based GP regression on BE-PFM images from PbTiO\(_3\) thin films. The central premise is that known materials structure can serve as prior information for predicting unmeasured or partially observed physical properties. The variables include morphology, morphology curvature, amplitude, phase, frequency, and \(Q\)-factor, and the paper explicitly distinguishes intrinsic descriptors, ideal observables, and measured channels. The DKL model then uses local structural patches from channels such as amplitude, phase, topography, and frequency to predict targets including morphology, amplitude, piezoresponse, polarization, resonance frequency, and a nonlinearity proxy [2110.06888].

The physical interpretation is central rather than incidental. The study reports that DKL can predict morphology from amplitude, phase, or frequency with good agreement to ground truth; predict amplitude from the polarization map even when trained on only \(1\%\) of the data; predict amplitude, phase, and polarization from topography; and predict resonance frequency and SHO-fit \(R^2\) from polarization. It also reports that generally less than \(10\%\) of data used for DKL training is sufficient for good reconstruction of the full image. The uncertainty maps are physically interpreted: larger uncertainty in large in-plane \(a\)-domains is attributed to the fact that vertical PFM incompletely captures in-plane polarization. This is a case where the DKL core is mathematically standard, but the workflow is physically anchored by microscopy structure, materials priors, and causal assumptions [2110.06888].

A different scientific application appears in process optimization with a ferroelectric simulator. There the inputs are \(7{,}500\) electric-field curves of the form
$$
A e^{a t}\sin(\omega t)+B,
$$
and the outputs are physical objectives derived from FerroSIM: curl, normalized curl, and total polarization. The study argues that DKL with active learning constructs latent spaces that are more compact and smooth for optimization than VAE-derived latent spaces, because the latent manifold is shaped jointly by the input data and the target functionality. In the active setting, the model starts from \(100\) measured trajectories and adds \(500\) Bayesian-optimization evaluations, for a total of \(600\) of \(7{,}500\) candidate curves. Here physics enters through simulator-defined trajectories and objectives rather than through explicit mechanistic constraints in the kernel [2303.14554].

## 5. Uncertainty quantification, active learning, and design utility

The GP layer gives physics-based DKL its characteristic uncertainty semantics. In regression settings this appears as posterior variance on reconstructed fields or properties; in inverse problems it extends to posterior distributions over physical parameters; in design problems it supports exploration–exploitation policies. This is one reason the literature repeatedly positions DKL between conventional neural surrogates and more explicitly Bayesian scientific models [2509.14054][2303.14554].

The FerroSIM study makes the active-learning role explicit. Its predictive posterior
$$
p(\tilde y \mid \tilde X,\gamma, X,y)=\mathcal N(\mu,\Sigma)
$$
drives a UCB-style acquisition
$$
\mathrm{acq}(x)=\mu(x)+10\,\sigma(x),
$$
and the authors argue that the latent manifold forms as a result of active learning balancing the data distribution and target functionalities. In the reported examples, high-performing trajectories collapse into a small number of broad, connected clusters in latent space, which the paper interprets as more conducive to optimization in feature-rich, target-poor regimes common in domain sciences such as materials synthesis, energy storage, and molecular discovery [2303.14554].

In PI-DKL, uncertainty is also treated as a first-class object, but with an extrapolative emphasis. The paper reports that physics regularization reduces overfitting near the training region, yields more uniform and smaller uncertainty in extrapolation regions, and improves test log-likelihood. In the synthetic ODE and diffusion examples, ordinary SKL/DKL exhibit rapidly increasing uncertainty away from observed data, whereas PI-DKL produces smaller and more spatially uniform predictive standard deviations [2006.04976].

For high-dimensional inverse PDEs, posterior uncertainty is split into two layers: uncertainty conditional on kernel and PDE parameters, and uncertainty induced by the posterior over those parameters. The two-stage HMC framework writes predictive uncertainty through Bayesian model averaging and the law of total expectation and total covariance. This is methodologically important because it distinguishes uncertainty in the field solution from uncertainty in the physical parameters themselves [2509.14054].

## 6. Failure modes and methodological debates

A central debate in the DKL literature concerns whether the GP layer really preserves the Bayesian robustness often attributed to kernel methods. One influential result is that DKL trained by maximum marginal likelihood or ELBO optimization can overfit badly in the overparameterized regime, sometimes more severely than a deterministic neural network trained by ordinary maximum likelihood. The argument is that DKL is only “somewhat Bayesian”: the latent function is integrated out, but the neural feature map and kernel hyperparameters are point-estimated. In that regime, the model can improve marginal likelihood by “hacking” the Gram matrix through over-correlation in feature space, so marginal likelihood or ELBO need not function as a trustworthy model-selection criterion. Fully Bayesian treatments using HMC or SGLD were reported to rectify these failures and restore the desired performance improvements over standard neural networks and Gaussian processes [2102.12108].

A closely related pathology is feature collapse. In sparse inducing-point DKL, unconstrained feature learning can map far-away data points to the same features as those of training-set points, producing unreliable uncertainty estimates even when predictive accuracy remains strong. DUE addresses this by constraining the feature extractor to be approximately bi-Lipschitz through spectral normalization, residual connections, and normalization-aware Lipschitz control. The paper reports that, on CIFAR-10 versus SVHN OOD detection, unconstrained SV-DKL attains AUROC \(0.498 \pm 0.001\) while the bi-Lipschitz variant attains AUROC \(0.959 \pm 0.001\); for GPDNN, the corresponding figures are \(0.876 \pm 0.004\) without the constraint and \(0.958 \pm 0.005\) with it [2102.11409].

Guided Deep Kernel Learning reframes the same issue as a guidance problem. It uses an infinite-width NNGP as a Bayesian reference and trains DKL with a hybrid objective consisting of a predictive term plus a KL regularizer toward the guide posterior. In one form, the core objective is
$$
\mathbb E_{(x_*, y_*)\sim \mathcal D_2} D_{KL}\big[q_\theta(f_* \mid x_*, \mathcal D_1)\,\|\, p(f_* \mid x_*, y_*, \mathcal D_1)\big].
$$
The paper’s interpretation is that the NNGP adapts the DKL target confidence when it encounters novel data points, thereby reducing overconfidence while preserving predictive performance. This suggests that physics-based DKL cannot rely on “NN features + GP” alone when extrapolation and uncertainty calibration matter; additional geometric or Bayesian control may be necessary [2302.09574].

## 7. Scalability, architectural variants, and open directions

The main scalability bottleneck in DKL is often not the raw input dimension alone, but the dimension and structure of the GP layer after feature extraction. One architectural response is the Deep Additive Kernel (DAK) model, which replaces a high-dimensional last-layer GP by an additive collection of one-dimensional GP units over latent channels:
$$
f (X) =\sum_{p=1}^{P} \sigma_{p}g_{p} \Big( h^{[p]}_{\psi}\left( X \right) \Big) + \mu.
$$
With induced prior approximation, each scalar GP unit becomes a Bayesian last-layer component,
$$
\tilde{g}_{p}(\cdot)=\phi(\cdot) z_p,
\qquad z_p \sim \mathcal N(\mathbf 0,\mathbf I_M),
$$
so the full model is reinterpreted as a last-layer Bayesian neural network. The paper reports complexity claims of \(O(SM)\) inference and \(O(SPMB + PM)\) training for DAK-MC, and \(O(M)\) inference and \(O(PMB + PM)\) training for DAK-CF. The work is not physics-informed, but it explicitly argues that additivity, low-dimensional latent partitioning, uncertainty-aware last layers, and scalable variational training map naturally onto many physics-informed settings [2502.10540].

Another transferable pattern is bilevel adaptation of deep-kernel GPs across related tasks. ADKF-IFT meta-learns a shared feature extractor while adapting GP hyperparameters per task via an implicit-function-theorem hypergradient. The paper is not physics-informed, but it is presented as highly relevant to scientific modeling because it separates what is shared across tasks from what must adapt locally. A plausible implication is that related experiments, parameter regimes, or simulator settings in physics could benefit from the same partition: transferable latent geometry, but task-specific noise, amplitude, or correlation structure [2205.02708].

The open problems in explicitly physics-based DKL are more concrete. PDE-DKL and the inverse-PDE framework are formulated for linear operators, and the latter uses exact observations with nugget stabilization. Both retain GP cubic scaling in the number of observations even while addressing high input dimensionality. PI-DKL avoids analytically transformed kernels and can handle nonlinear operators, but it retains full-GP costs of \(O(N^3+m^3)\). Taken together, these results indicate that the current literature is strongest in low-data, physics-informed regimes where uncertainty quantification is essential, while nonlinear operators, noisy and biased observations, broader calibration analysis, and more scalable posterior approximations remain active directions rather than settled components of the field [2501.18258][2509.14054][2006.04976].

Source: https://www.emergentmind.com/topics/physics-based-deep-kernel-learning-dkl