Deep Kernel Processes

• Deep Kernel Processes (DKPs) are a probabilistic framework that builds hierarchies of positive-definite Gram matrices using alternating deterministic transforms and stochastic updates.
• They generalize infinite-width limits of Bayesian neural networks and deep Gaussian processes, enabling robust, data-dependent feature learning and uncertainty quantification.
• Recent variants leverage infinite-variance priors and Wishart-based sampling to overcome deterministic kernel recursions, yielding improved performance on regression and classification benchmarks.

Deep Kernel Processes (DKPs) constitute a probabilistic framework for expressing hierarchies of positive-definite Gram matrices, providing a unifying view of deep learning models in function space. DKPs generalize the infinite-width limit of Bayesian neural networks (BNNs), deep Gaussian processes (DGPs), and feature-space models by constructing deep, hierarchical priors over kernels via alternating deterministic transformations and stochastic updates. The DKP hierarchy enables richer, data-dependent feature learning and stochasticity in representation, overcoming the limitations of deterministic kernel recursion associated with infinite-width BNNs with bounded-variance priors. Recent advances leverage infinite-variance, heavy-tailed priors to yield α–stable processes, as well as Wishart and inverse-Wishart distributions over kernel matrices to maintain layerwise stochasticity and tractable posterior inference.

1. Mathematical Definition and Formal Structure

A Deep Kernel Process is defined as a stochastic hierarchy of positive-definite Gram matrices

$\{\Sigma^{(1)},\Sigma^{(2)},\dots,\Sigma^{(L)}\}$

associated with $L$ layers and recursively linked. Each $\Sigma^{(\ell)}$ encodes the pairwise kernel evaluations among $n$ inputs at layer $\ell$. The generative structure for a canonical DKP consists of two interleaved operations:

• Kernel Transformation: Given a Gram matrix $G^{(\ell)} \succ 0$ at layer $\ell$, apply a parameterized kernel function $k_\phi$ to obtain $K^{(\ell)} = k_\phi(G^{(\ell)})$, where $$[k_\phi(G^{(\ell)})]_{ij} = k_\phi(G^{(\ell)}_{ii}, G^{(\ell)}_{ij}, G^{(\ell)}_{jj})$$ for kernels depending only on diagonal and off-diagonal entries (e.g., RBF, arc-cosine).
• (Inverse-)Wishart Sampling: Sample a new Gram matrix $L$0 via a (possibly inverse) Wishart distribution with scale set by $L$1:
  • Wishart: $L$2
  • Inverse Wishart: $L$3

At each layer, the mean $L$4, so purely deterministic recursion is recovered as the degrees of freedom go to infinity or the noise vanishes. Stochasticity is introduced by finite width or by explicit noise-injection (Aitchison et al., 2020, Ober et al., 2021).

2. Relationship to Deep Gaussian Processes and BNNs

DKPs generalize the infinite-width limits of BNNs and DGPs:

• Finite-variance BNNs and DGPs: For BNNs with Gaussian weight priors and widths tending to infinity, each layer’s output converges to a GP with deterministic, recursively defined kernel $L$5 given by the Cho–Saul recursion (Loría et al., 2024). In DGPs, Gram matrices at each layer follow Wishart distributions (if widths are finite) and transition to deterministic transforms in the infinite limit (Aitchison et al., 2020).
• Degeneracy of Deterministic Recursions: In standard infinite-width BNNs and DGPs, the kernel at each layer is a deterministic function of the data and priors, precluding data-dependent representation learning in the posterior (Loría et al., 2024). All posterior stochasticity is lost beyond the output GP layer.
• Deep Wishart and Inverse-Wishart Processes: DKPs recover stochasticity by introducing layerwise noise via Wishart or inverse-Wishart sampling. The Deep Wishart Process (DWP) and Deep Inverse Wishart Process (DIWP) retain stochastic kernels at every layer, with the level of noise controlled by degrees of freedom hyperparameters (Aitchison et al., 2020, Ober et al., 2021).

3. Overcoming Deterministic Kernel Limitations: Infinite-Variance Priors and α–Stable DKPs

Recent advancements address the representational degeneracy by considering infinite-variance, heavy-tailed priors:

• α–Stable Distributions: If BNN weights at each layer are drawn from elliptical α–stable distributions with index $L$6, the infinite-width limit yields marginal pre-activation processes with α–stable laws (infinite variance for $L$7). Each pre-activation vector $L$8 admits a Gaussian mixture representation conditional on a random positive α/2–stable scaling variable $L$9 (Loría et al., 2024).
• Stochastic Recursive Kernels: Conditioned on these scales, the covariance matrices $\Sigma^{(\ell)}$0 follow the modified Cho–Saul recursion:

$\Sigma^{(\ell)}$1

with $\Sigma^{(\ell)}$2 (for ReLU activation).

• Feature Learning and Non-Gaussianity: For $\Sigma^{(\ell)}$3, the posterior over features is data-dependent, restoring the ability to learn non-degenerate, data-adaptive representations (absent in Gaussian cases). This approach yields α–stable processes marginally and enables modeling discontinuities and heavy-tails that GPs cannot capture (Loría et al., 2024).

4. Inference and Variational Posterior Schemes

Practical inference in DKPs often centers on variational methods and doubly-stochastic inducing-point schemes that operate in the kernel/Gram-matrix domain:

• Doubly-Stochastic Inducing Point Methods: By introducing inducing Gram matrices at a reduced set of points, large-scale inference becomes tractable. The joint prior is factored hierarchically over inducing and training/test Gram matrices, with conditionals given by standard GP and Wishart/IW conditioning (Aitchison et al., 2020, Ober et al., 2021).
• Flexible Variational Posteriors: The generalized Bartlett decomposition is used to parameterize an expressive variational family over positive semi-definite Gram matrices, allowing independent control over mean and variance by decoupling Gamma and Gaussian components in the lower-triangular Cholesky factors (Ober et al., 2021).
• Layerwise Stochasticity: In DIWP, the inverse-Wishart degrees of freedom $\Sigma^{(\ell)}$4 are learned (by maximizing the ELBO), and determine the amount of kernel noise per layer; $\Sigma^{(\ell)}$5 recovers a deterministic transform, while small $\Sigma^{(\ell)}$6 increases variability (Aitchison et al., 2020).

5. Empirical Performance and Statistical Properties

DKPs have demonstrated superior empirical behavior and uncertainty quantification relative to deterministic DGPs and NNGPs:

• Benchmarks: On standard regression and classification benchmarks (Boston, Energy, Yacht, MNIST, CIFAR-10), DKPs based on the DIWP and α–stable constructions outperform both DGP and NNGP baselines, attaining tighter evidence lower bounds, lower test set RMSE/MAE, and better-calibrated predictive intervals (Loría et al., 2024, Aitchison et al., 2020, Ober et al., 2021).
• Feature Learning: For $\Sigma^{(\ell)}$7 in α–DKPs, the posterior over features remains data-dependent, enabling true feature learning, which is lost in the degenerate Gaussian process case ($\Sigma^{(\ell)}$8) (Loría et al., 2024).
• Robust Uncertainty and Non-smooth Targets: DKPs can capture non-smooth and jump functions, with predictive uncertainty intervals achieving nominal coverage in settings where GP-based methods systematically undercover (Loría et al., 2024).
• Computational Efficiency: Operating in Gram-matrix space avoids the curse of dimensionality present in feature-space MCMC approaches ($\Sigma^{(\ell)}$9 per-iteration cost with DIWP, compared to $n$0 for earlier shallow stable models) (Loría et al., 2024).

6. Hyperparameters, Depth, and Model Design

Tuning DKPs involves control over kernel functions, degree-of-freedom parameters, and architectural design:

• Kernel Choice: Squared-exponential, ReLU, Matérn, and arc-cosine kernels can be used, provided they depend solely on Gram-matrix entries (Aitchison et al., 2020).
• Depth: Depths of two hidden layers plus output suffice to exceed shallow NNGP/DGP performance. Additional depth can be used, but depth L=2+1 is empirically competitive (Aitchison et al., 2020).
• Degrees of Freedom and Scale: The degrees of freedom in Wishart/IW steps set stochasticity; learning these by ELBO maximization enables adaptive layerwise behavior, interpolating between deterministic and stochastic regimes. Practical initializations set $n$1 (with $n$2 data points) and $n$3 (Aitchison et al., 2020).
• Learning Stochasticity: Gradient-based optimization allows the model to adaptively control per-layer stochasticity and transitions between infinite-width (deterministic) and finite-width (stochastic) behaviors.

7. Connections, Limitations, and Future Directions

DKPs provide a function-space generalization of deep learning models with the following properties:

• Unified Kernel-centric View: DKPs encompass DGPs, finite and infinite-width BNNs, and hybrid models (with bottlenecks) within a flexible, kernel-matrix-based hierarchical framework (Aitchison et al., 2020).
• Stochasticity and Expressiveness: Layerwise introduction of stochasticity facilitates both representation learning and better modeling of heavy-tailed, discontinuous structure that is inaccessible to deterministic GPs (Loría et al., 2024).
• Computational Tractability: All operations are performed in kernel space, avoiding high-dimensional, rotation-symmetric feature representations and associated inference difficulties (Aitchison et al., 2020, Ober et al., 2021).
• Open Questions: For α–stable-based DKPs, the interpretation of marginal α–stable processes without covariance functions remains nuanced; practical implementation relies on conditional Gaussian mixture representations. Injection of layerwise kernel noise in DIWP remains somewhat ad hoc, although infinite-variance BNN limits now offer a natural stochastic kernel process (Loría et al., 2024).

A plausible implication is that DKPs open avenues for principled, scalable, and expressive nonparametric inference with deep architectures, with robust uncertainty quantification and adaptation to non-Gaussian, non-smooth target functions.

---

Key Literature:

• "Deep Kernel Posterior Learning under Infinite Variance Prior Weights" (Loría et al., 2024)
• "Deep kernel processes" (Aitchison et al., 2020)
• "A variational approximate posterior for the deep Wishart process" (Ober et al., 2021)