Theory-Informed Kernel Overview

• Theory-informed kernels are Gaussian process kernels that embed explicit theoretical priors into function-space geometry.
• They are constructed using methods like Bayesian meta-learning and physics-informed approaches to integrate prior covariance and operator dynamics.
• Applications span from retinal data analysis to plasma physics, improving interpretability and aligning learned representations with theoretical constraints.

Searching arXiv for recent and foundational papers on theory-informed kernels and adjacent kernel-theoretic formulations. A theory-informed kernel is a kernel whose inductive bias is specified by an explicit theory source rather than by a generic similarity choice. In the most explicit formulation, a Theory-Informed Kernel (TIK) is a Gaussian process kernel that encodes a theory-derived inductive bias, learned by Bayesian meta-learning from synthetic tasks generated by a normative theory; it is a prior over functions, not a direct predictor (Zakirov et al., 29 Sep 2025). Across adjacent literatures, the same design logic appears in kernels derived from prior covariance or posterior covariance in kernel regression, from linear differential operators in physics-informed learning, from nonlocal transport closures in plasma physics, from system-theoretic priors in LTI identification, and from information-theoretic or algebraic characterizations of feature spaces (Simon, 2022).

1. Definition and conceptual scope

In the retina case study that introduced the name “Theory-Informed Kernel,” the kernel is built from three components: a shared feature extractor $\phi$ learned by meta-learning, a task-specific linear head $h_i$, and a task-specific RBF GP layer. The resulting kernel for task $i$ is

$$K_{\text{TIK},i}(x,x') = \sigma_{f,i}\exp\left(-\frac{\|h_i(\phi(x)) - h_i(\phi(x'))\|^2}{2\ell_i^2}\right).$$

The theory is not encoded as a closed-form probabilistic model in parameter space; it is converted into a function-space prior that can be used both for prediction and for exact Bayesian model selection (Zakirov et al., 29 Sep 2025).

A closely related theoretical statement appears in kernel regression with data-dependent kernels. When the kernel is fixed before observing the dataset, the optimal kernel is the prior covariance

$\Kprior(x,x') \equiv \E_{f \sim \mu_f}[f(x)f(x')],$

whereas when the kernel may be modified after seeing the dataset, the optimal kernel is the posterior covariance

$\Kpost(x,x') \equiv \E_{f \sim \mu_f | \D}[f(x)f(x')].$

This makes kernel choice itself a theory-bearing object: the kernel should reflect either the prior over target functions or its posterior conditioning on the observed data (Simon, 2022).

In regularized LTI system identification, the same principle is stated in domain-specific terms. Kernel design should be guided by the kind of prior knowledge one has. The paper distinguishes a machine-learning perspective, which encodes decay and local smoothness or varying rate, from a system-theory perspective, which encodes stability, time constants, damping, and resonant modes. This supports a broad reading of theory-informed kernels as kernels designed to embed substantive prior knowledge into the RKHS geometry (Chen, 2016).

2. Mathematical forms and kernel mechanisms

The unifying structural feature is that the kernel is not an incidental positive semidefinite function; it is the formal carrier of theoretical constraints. In the TIK construction, the theory-informed metric appears inside a deep-kernel RBF GP and is learned from theory-consistent synthetic tasks. For theory validation, the paper defines a mixture kernel

$$K_\beta(x,x') = \beta K_{\text{TIK}}(x,x') + (1-\beta)K_{\text{RBF}}(x,x'),$$

with

$\beta^* = \arg\max_{\beta\in[0,1]} P(Y|X,\beta),$

so that theory match is estimated continuously rather than treated as a binary judgment (Zakirov et al., 29 Sep 2025).

In plasma heat transport, the kernel appears inside a physically fixed operator. The nonlocal closure is

$$Q_e(x) = \int Q_{\rm SH}(x')\, \mathcal{W}(x, x')\, dx',$$

and the learned replacement is a time-dependent kernel

$\mathcal{W} = \mathcal{W}(X,\lambda,t).$

The method preserves the kernel-convolution structure of nonlocal transport theory, uses the physically motivated variables $h_i$0 and $h_i$1, incorporates time as an explicit input, and enforces a positive kernel weight output. The theory-informed aspect is therefore not a generic neural regressor but a learned kernel inside a pre-existing transport operator (Luo et al., 19 Jun 2025).

In operator-informed NTK theory, the same idea is expressed through differential operators. For losses involving a linear differential operator $h_i$2, the relevant kernel is

$h_i$3

The operator does not create an unrelated kernel family; it differentiates the standard NTK in both arguments. This gives an explicit mathematical description of how a physical operator deforms the learning kernel (Gan et al., 14 Mar 2025).

3. Construction paradigms

The literature contains several distinct construction routes, each of which turns theory into kernel geometry rather than leaving it as an external interpretation.

Paradigm	Kernel form or object	Theory source
Bayesian meta-learning	$h_i$4	Normative theory via synthetic tasks
Physics-constrained kernel learning	$h_i$5	Nonlocal heat transport theory
Bayesian-optimal kernel regression	$h_i$6 or $h_i$7	Prior or posterior over target functions
System-identification kernels	AMLS and SI kernels	Decay, smoothness, stability, damping, resonant modes
Information-theoretic subspace kernels	projection kernel $h_i$8 and maximal correlation kernel	Feature subspace and H-score optimality

In Bayesian meta-learning, the process has three stages: generate synthetic task data from the theory, meta-learn a kernel on these theory-consistent tasks, and apply the learned prior to real tasks. What is meta-learned is the geometry of the input space such that theory-consistent functions become easy to model with a GP. In the efficient-coding example, the authors train an efficient-coding autoencoder on 60,000 natural image patches from Van Hateren, construct 490 synthetic meta-train tasks, and then transfer the learned feature extractor to ex vivo retinal data (Zakirov et al., 29 Sep 2025).

In nonlocal plasma transport, the kernel is learned from 1D OSIRIS PIC simulations with binary collisions. The study samples 829 data points across parameter space, uses mirrored samples due to symmetry, and trains with a 70:15:15 train/validation/test split. Here the kernel is not a GP prior but a learned constitutive object inside a convolution law, which suggests that theory-informed kernels need not share a single probabilistic formalism (Luo et al., 19 Jun 2025).

Information-theoretic and algebraic constructions provide a third route. In the feature-subspace view, every feature subspace induces a projection kernel

$h_i$9

and the associated operator acts as the orthogonal projection onto that subspace. The maximal correlation kernel is the projection kernel associated with the maximal correlation functions, and its H-score is optimal. In algebraic duality, polynomial rings correspond to feature spaces, ideals correspond to manifolds, and cross-kernel matrices expose both intrinsic and extrinsic structure; IPCA and AVICA are then used for simultaneous manifold and feature learning (Xu et al., 2023, Király et al., 2014).

A fourth route concerns interpretability in the original feature space. In wide-feature settings with $i$0, the exact same predicted values of a kernel method can be re-expressed as

$i$1

with a special ridge penalty

$i$2

The penalty is derived from the kernel geometry itself, so the kernel’s nonlinear structure is transferred into a quadratic form on original coefficients rather than discarded (Groenen et al., 21 Aug 2025).

4. Physics-informed and scientific-computing formulations

Physics-informed kernel formulations are especially diverse because the theory source may be a PDE, a Green function, an NTK limit, or a residual evolution law.

For PINNs in the large-width, large-data limit, the governing object is not the bare PDE but the Neurally-Informed Equation (NIE),

$i$3

The kernel term reflects architecture choices and leads to spectral bias and kernel-task alignment effects. The relevant operator for alignment is $i$4, and the source term is filtered through its eigenspectrum (Seroussi et al., 2023).

A related operator-based NTK analysis shows that when the loss contains a linear differential operator, the training dynamics of $i$5 are governed by the differentiated NTK $i$6. The paper’s main theoretical conclusion is that, for a broad class of symmetric operators, the eigenvalue decay rate is not faster than in the standard $i$7 case, so adding differential operators does not generally strengthen spectral bias (Gan et al., 14 Mar 2025).

In cPIKANs, the kernel becomes block-structured because the physics-informed loss combines data and residual terms:

$i$8

The analysis covers the steady-state Helmholtz equation, transient diffusion and Allen-Cahn equations, and forced vibrations governed by the Euler-Bernoulli beam equation. The reported interpretation is that cPIKANs exhibit more structured and tractable kernel behavior than standard PINNs, and that spectral trends reveal when domain decomposition improves training (Faroughi et al., 9 Jun 2025).

Green-kernel learning provides a more direct kernel object. GsPINN parameterizes the fundamental solution $i$9 and augments the PINN loss with a linearized symmetric condition

$$K_{\text{TIK},i}(x,x') = \sigma_{f,i}\exp\left(-\frac{\|h_i(\phi(x)) - h_i(\phi(x'))\|^2}{2\ell_i^2}\right).$$0

This symmetry residual is first order in $$K_{\text{TIK},i}(x,x') = \sigma_{f,i}\exp\left(-\frac{\|h_i(\phi(x)) - h_i(\phi(x'))\|^2}{2\ell_i^2}\right).$$1, whereas the original PDE residual may require higher-order derivatives. Numerical simulations are reported to achieve up to a 50\% reduction in training time while maintaining good accuracy (Jiao et al., 2024).

SPIKE uses a moving reproducing-kernel representation for one-dimensional inviscid hyperbolic conservation laws,

$$K_{\text{TIK},i}(x,x') = \sigma_{f,i}\exp\left(-\frac{\|h_i(\phi(x)) - h_i(\phi(x'))\|^2}{2\ell_i^2}\right).$$2

with Tikhonov-regularized parameter evolution. The claimed consequence is that strong-form residual minimization can recover weak shock solutions while automatically maintaining conservation, tracking characteristics, and capturing shocks satisfying Rankine-Hugoniot conditions, without explicit shock detection or artificial viscosity (Su et al., 21 Oct 2025).

At large scale, theory-informed regression can also be implemented through Fourier representations. In Sobolev, physics-informed, and additive models, the theory enters through the choice of function space and quadratic penalty on Fourier coefficients, while NUFFT and FFT structure reduce the computational cost to $$K_{\text{TIK},i}(x,x') = \sigma_{f,i}\exp\left(-\frac{\|h_i(\phi(x)) - h_i(\phi(x'))\|^2}{2\ell_i^2}\right).$$3. In the physics-informed case, a known linear differential operator becomes a diagonal Fourier multiplier inside the regularized least-squares problem (Doumèche et al., 2 Sep 2025).

5. Statistical, representational, and interpretability viewpoints

One line of work treats kernels as explicit representations of feature subspaces. If $$K_{\text{TIK},i}(x,x') = \sigma_{f,i}\exp\left(-\frac{\|h_i(\phi(x)) - h_i(\phi(x'))\|^2}{2\ell_i^2}\right).$$4 is a $$K_{\text{TIK},i}(x,x') = \sigma_{f,i}\exp\left(-\frac{\|h_i(\phi(x)) - h_i(\phi(x'))\|^2}{2\ell_i^2}\right).$$5-dimensional subspace of the feature space, the projection kernel $$K_{\text{TIK},i}(x,x') = \sigma_{f,i}\exp\left(-\frac{\|h_i(\phi(x)) - h_i(\phi(x'))\|^2}{2\ell_i^2}\right).$$6 induces an integral operator that is exactly the orthogonal projection onto $$K_{\text{TIK},i}(x,x') = \sigma_{f,i}\exp\left(-\frac{\|h_i(\phi(x)) - h_i(\phi(x'))\|^2}{2\ell_i^2}\right).$$7. This yields a one-to-one correspondence between feature subspaces and kernels. The H-score then becomes a kernel-quality measure, and the maximal correlation kernel is optimal because it induces the most informative subspace for predicting $$K_{\text{TIK},i}(x,x') = \sigma_{f,i}\exp\left(-\frac{\|h_i(\phi(x)) - h_i(\phi(x'))\|^2}{2\ell_i^2}\right).$$8 (Xu et al., 2023).

Another line treats kernels as dual objects to symbolic algebra. For polynomial kernels, the polynomial ring is spanned by kernel sections from generic points, vanishing ideals are orthogonal complements of manifold-generated feature spans, and cross-kernel matrices recover interpolation spaces and vanishing features. This makes kernel learning simultaneously discriminative and generative: the kernel is used not only to separate observations but to reconstruct the manifold itself through ideal structure (Király et al., 2014).

Interpretability can also be recovered after a kernel model is fit. In the exact wide case $$K_{\text{TIK},i}(x,x') = \sigma_{f,i}\exp\left(-\frac{\|h_i(\phi(x)) - h_i(\phi(x'))\|^2}{2\ell_i^2}\right).$$9, kernel solutions can be rewritten in the original feature space with coefficients $\Kprior(x,x') \equiv \E_{f \sim \mu_f}[f(x)f(x')],$0 and a non-Euclidean quadratic penalty derived from the kernel approximation. In the approximate case $\Kprior(x,x') \equiv \E_{f \sim \mu_f}[f(x)f(x')],$1, the approximation quality is measured by

$\Kprior(x,x') \equiv \E_{f \sim \mu_f}[f(x)f(x')],$2

This shows that kernel geometry need not be incompatible with regression-style interpretability (Groenen et al., 21 Aug 2025).

The neural tangent kernel literature adds a dynamic perspective. During finite-width training, the NTK can evolve to become more aligned with the target function. The centered alignment metric

$\Kprior(x,x') \equiv \E_{f \sim \mu_f}[f(x)f(x')],$3

is reported to increase during training, and toy as well as analytic models show that the evolved kernel acquires target-aligned anisotropy. This suggests that some theory-informed kernels are not fixed priors but adaptive objects whose structure is shaped by optimization itself (Shan et al., 2021).

A more formal generalization appears in the Unified Neural Kernel, which interpolates between NTK-like and NNGP-like regimes. UNK is defined through gradient-descent dynamics with parameter anchoring and is stated to behave like NTK with a finite learning step while converging to NNGP as the learning step approaches infinity. This extends the meaning of theory-informed kernels from externally specified priors to kernels derived from the learning dynamics of the model itself (Zhang et al., 2024).

6. Limitations, misconceptions, and open directions

A common misconception is that a theory-informed kernel is simply a hard-coded symbolic theory. The TIK formulation explicitly rejects that identification: the kernel is a probabilistic model representing theory predictions, and the theory is embedded as a function-space prior learned from theory-consistent tasks rather than as a hand-written forward model (Zakirov et al., 29 Sep 2025).

Another misconception is that any data-dependent kernel improvement is automatically practical. The posterior kernel is optimal in the data-dependent setting, but the same paper states that it does not provide a practical algorithm for computing $\Kprior(x,x') \equiv \E_{f \sim \mu_f}[f(x)f(x')],$4 in real tasks, since that would require knowing the posterior over target functions. The result is therefore a theoretical optimum and a design principle, not an immediate recipe (Simon, 2022).

In physics-informed learning, it is also incorrect to assume that placing higher-order differential operators in the loss must intensify spectral bias. The operator-informed NTK analysis states the opposite for a broad class of symmetric operators: the eigenvalue decay rate is not faster than in the standard setting, and experiments report that higher-order operators tend to produce slower eigenvalue decay (Gan et al., 14 Mar 2025).

The most mature validation framework in the literature couples theory-informed kernels to Bayesian evidence. In the retinal example, exact Bayesian model selection is used with the mixture kernel $\Kprior(x,x') \equiv \E_{f \sim \mu_f}[f(x)f(x')],$5, and $\Kprior(x,x') \equiv \E_{f \sim \mu_f}[f(x)f(x')],$6 is fitted by grid search over 100 values in $\Kprior(x,x') \equiv \E_{f \sim \mu_f}[f(x)f(x')],$7. On synthetic data, $\Kprior(x,x') \equiv \E_{f \sim \mu_f}[f(x)f(x')],$8 correlated strongly with known ground-truth optimality; on biological neurons, it provided a graded estimate of how efficient-coding-like each neuron was. This makes theory match an inferential target rather than a post hoc narrative (Zakirov et al., 29 Sep 2025).

A more speculative research direction treats the kernel itself as a dynamical variable in kernel space. Under path entropy maximization, a kernel trajectory $\Kprior(x,x') \equiv \E_{f \sim \mu_f}[f(x)f(x')],$9 is optimized under constraints involving thermodynamic cost, mutual information, and model mismatch. The work states a lower bound

$\Kpost(x,x') \equiv \E_{f \sim \mu_f | \D}[f(x)f(x')].$0

for kernel change that unlocks additional mutual information, and it proposes NTK evolution during deep network training as a candidate empirical instantiation. The paper explicitly separates formal results from structured correspondences and conjectural bridges, so these claims should be read as a research program rather than as an established equivalence (Das, 29 Mar 2026).

Taken together, these strands indicate that “theory-informed kernel” is not the name of a single method. It denotes a family of constructions in which the kernel is the locus where prior theory, operator structure, synthetic task generation, system dynamics, or representational principles are made mathematically operative. This suggests that the central problem is not merely choosing a similarity function, but determining which theory source should define the geometry of the function class.