---
title: Deep Kernel Processes
url: https://www.emergentmind.com/topics/deep-kernel-processes-dkps
type: topic
---

# Deep Kernel Processes

Deep Kernel Processes (DKPs) constitute a probabilistic framework for expressing hierarchies of positive-definite Gram matrices, providing a unifying view of deep learning models in function space. DKPs generalize the infinite-width limit of Bayesian neural networks (BNNs), deep Gaussian processes (DGPs), and feature-space models by constructing deep, hierarchical priors over kernels via alternating deterministic transformations and stochastic updates. The DKP hierarchy enables richer, data-dependent feature learning and stochasticity in representation, overcoming the limitations of deterministic kernel recursion associated with infinite-width BNNs with bounded-variance priors. Recent advances leverage infinite-variance, heavy-tailed priors to yield α–stable processes, as well as Wishart and inverse-Wishart distributions over kernel matrices to maintain layerwise stochasticity and tractable posterior inference.

## 1. Mathematical Definition and Formal Structure

A Deep Kernel Process is defined as a stochastic hierarchy of positive-definite Gram matrices
$$
\{\Sigma^{(1)},\Sigma^{(2)},\dots,\Sigma^{(L)}\}
$$
associated with $L$ layers and recursively linked. Each $\Sigma^{(\ell)}$ encodes the pairwise kernel evaluations among $n$ inputs at layer $\ell$. The generative structure for a canonical DKP consists of two interleaved operations:
- **Kernel Transformation:** Given a Gram matrix $G^{(\ell)} \succ 0$ at layer $\ell$, apply a parameterized kernel function $k_\phi$ to obtain $K^{(\ell)} = k_\phi(G^{(\ell)})$, where $[k_\phi(G^{(\ell)})]_{ij} = k_\phi(G^{(\ell)}_{ii}, G^{(\ell)}_{ij}, G^{(\ell)}_{jj})$ for kernels depending only on diagonal and off-diagonal entries (e.g., RBF, arc-cosine).
- **(Inverse-)Wishart Sampling:** Sample a new Gram matrix $G^{(\ell+1)}$ via a (possibly inverse) Wishart distribution with scale set by $K^{(\ell)}$:
  - Wishart: $G^{(\ell+1)} \sim \text{Wishart}(\Sigma = K^{(\ell)}/N_{\ell+1}, \nu = N_{\ell+1})$
  - Inverse Wishart: $G^{(\ell+1)} \sim \text{IW}(\Psi = \beta_{\ell+1} K^{(\ell)}, \nu = \alpha_{\ell+1})$

At each layer, the mean $\mathbb{E}[G^{(\ell+1)}|K^{(\ell)}] = K^{(\ell)}$, so purely deterministic recursion is recovered as the degrees of freedom go to infinity or the noise vanishes. Stochasticity is introduced by finite width or by explicit noise-injection [2010.01590, 2107.10125].

## 2. Relationship to Deep Gaussian Processes and BNNs

DKPs generalize the infinite-width limits of BNNs and DGPs:
- **Finite-variance BNNs and DGPs:** For BNNs with Gaussian weight priors and widths tending to infinity, each layer’s output converges to a GP with deterministic, recursively defined kernel $K^{(\ell)}(x, x')$ given by the Cho–Saul recursion [2410.01284]. In DGPs, Gram matrices at each layer follow Wishart distributions (if widths are finite) and transition to deterministic transforms in the infinite limit [2010.01590].
- **Degeneracy of Deterministic Recursions:** In standard infinite-width BNNs and DGPs, the kernel at each layer is a deterministic function of the data and priors, precluding data-dependent representation learning in the posterior [2410.01284]. All posterior stochasticity is lost beyond the output GP layer.
- **Deep Wishart and Inverse-Wishart Processes:** DKPs recover stochasticity by introducing layerwise noise via Wishart or inverse-Wishart sampling. The Deep Wishart Process (DWP) and Deep Inverse Wishart Process (DIWP) retain stochastic kernels at every layer, with the level of noise controlled by degrees of freedom hyperparameters [2010.01590, 2107.10125].

## 3. Overcoming Deterministic Kernel Limitations: Infinite-Variance Priors and α–Stable DKPs

Recent advancements address the representational degeneracy by considering infinite-variance, heavy-tailed priors:
- **α–Stable Distributions:** If BNN weights at each layer are drawn from elliptical α–stable distributions with index $\alpha \in (0,2]$, the infinite-width limit yields marginal pre-activation processes with α–stable laws (infinite variance for $\alpha<2$). Each pre-activation vector $\mathbf{z}^{(\ell)}$ admits a Gaussian mixture representation conditional on a random positive α/2–stable scaling variable $s^{(\ell)}_+$ [2410.01284].
- **Stochastic Recursive Kernels:** Conditioned on these scales, the covariance matrices $\Sigma^{(\ell)}$ follow the modified Cho–Saul recursion:
  $$
  \Sigma_{k,h}^{(\ell)} = \frac{1}{\pi} \sqrt{(1 + s^{(\ell-1)} \Sigma^{(\ell-1)}_{k,k})(1 + s^{(\ell-1)} \Sigma^{(\ell-1)}_{h,h})} \left[ \sin\theta + (\pi-\theta)\cos\theta \right]
  $$
  with $\cos\theta = \dfrac{1 + s^{(\ell-1)} \Sigma^{(\ell-1)}_{k,h}}{\sqrt{(1 + s^{(\ell-1)} \Sigma^{(\ell-1)}_{k,k})(1 + s^{(\ell-1)} \Sigma^{(\ell-1)}_{h,h})}}$ (for ReLU activation).
- **Feature Learning and Non-Gaussianity:** For $\alpha<2$, the posterior over features is data-dependent, restoring the ability to learn non-degenerate, data-adaptive representations (absent in Gaussian cases). This approach yields α–stable processes marginally and enables modeling discontinuities and heavy-tails that GPs cannot capture [2410.01284].

## 4. Inference and Variational Posterior Schemes

Practical inference in DKPs often centers on variational methods and doubly-stochastic inducing-point schemes that operate in the kernel/Gram-matrix domain:
- **Doubly-Stochastic Inducing Point Methods:** By introducing inducing Gram matrices at a reduced set of points, large-scale inference becomes tractable. The joint prior is factored hierarchically over inducing and training/test Gram matrices, with conditionals given by standard GP and Wishart/IW conditioning [2010.01590, 2107.10125].
- **Flexible Variational Posteriors:** The generalized Bartlett decomposition is used to parameterize an expressive variational family over positive semi-definite Gram matrices, allowing independent control over mean and variance by decoupling Gamma and Gaussian components in the lower-triangular Cholesky factors [2107.10125].
- **Layerwise Stochasticity:** In DIWP, the inverse-Wishart degrees of freedom $\nu_\ell$ are learned (by maximizing the ELBO), and determine the amount of kernel noise per layer; $\nu_\ell \rightarrow \infty$ recovers a deterministic transform, while small $\nu_\ell$ increases variability [2010.01590].

## 5. Empirical Performance and Statistical Properties

DKPs have demonstrated superior empirical behavior and uncertainty quantification relative to deterministic DGPs and NNGPs:
- **Benchmarks:** On standard regression and classification benchmarks (Boston, Energy, Yacht, MNIST, CIFAR-10), DKPs based on the DIWP and α–stable constructions outperform both DGP and NNGP baselines, attaining tighter evidence lower bounds, lower test set RMSE/MAE, and better-calibrated predictive intervals [2410.01284, 2010.01590, 2107.10125].
- **Feature Learning:** For $\alpha<2$ in α–DKPs, the posterior over features remains data-dependent, enabling true feature learning, which is lost in the degenerate Gaussian process case ($\alpha=2$) [2410.01284].
- **Robust Uncertainty and Non-smooth Targets:** DKPs can capture non-smooth and jump functions, with predictive uncertainty intervals achieving nominal coverage in settings where GP-based methods systematically undercover [2410.01284].
- **Computational Efficiency:** Operating in Gram-matrix space avoids the curse of dimensionality present in feature-space MCMC approaches ($O(n^3)$ per-iteration cost with DIWP, compared to $O(n^{I+2})$ for earlier shallow stable models) [2410.01284].

## 6. Hyperparameters, Depth, and Model Design

Tuning DKPs involves control over kernel functions, degree-of-freedom parameters, and architectural design:
- **Kernel Choice:** Squared-exponential, ReLU, Matérn, and arc-cosine kernels can be used, provided they depend solely on Gram-matrix entries [2010.01590].
- **Depth:** Depths of two hidden layers plus output suffice to exceed shallow NNGP/DGP performance. Additional depth can be used, but depth L=2+1 is empirically competitive [2010.01590].
- **Degrees of Freedom and Scale:** The degrees of freedom in Wishart/IW steps set stochasticity; learning these by ELBO maximization enables adaptive layerwise behavior, interpolating between deterministic and stochastic regimes. Practical initializations set $\nu_\ell \approx P+2$ (with $P$ data points) and $\beta_\ell=1$ [2010.01590].
- **Learning Stochasticity:** Gradient-based optimization allows the model to adaptively control per-layer stochasticity and transitions between infinite-width (deterministic) and finite-width (stochastic) behaviors.

## 7. Connections, Limitations, and Future Directions

DKPs provide a function-space generalization of deep learning models with the following properties:
- **Unified Kernel-centric View:** DKPs encompass DGPs, finite and infinite-width BNNs, and hybrid models (with bottlenecks) within a flexible, kernel-matrix-based hierarchical framework [2010.01590].
- **Stochasticity and Expressiveness:** Layerwise introduction of stochasticity facilitates both representation learning and better modeling of heavy-tailed, discontinuous structure that is inaccessible to deterministic GPs [2410.01284].
- **Computational Tractability:** All operations are performed in kernel space, avoiding high-dimensional, rotation-symmetric feature representations and associated inference difficulties [2010.01590, 2107.10125].
- **Open Questions:** For α–stable-based DKPs, the interpretation of marginal α–stable processes without covariance functions remains nuanced; practical implementation relies on conditional Gaussian mixture representations. Injection of layerwise kernel noise in DIWP remains somewhat ad hoc, although infinite-variance BNN limits now offer a natural stochastic kernel process [2410.01284].

A plausible implication is that DKPs open avenues for principled, scalable, and expressive nonparametric inference with deep architectures, with robust uncertainty quantification and adaptation to non-Gaussian, non-smooth target functions.

---

**Key Literature:**  
- "Deep Kernel Posterior Learning under Infinite Variance Prior Weights" [2410.01284]  
- "Deep kernel processes" [2010.01590]  
- "A variational approximate posterior for the deep Wishart process" [2107.10125]

Source: https://www.emergentmind.com/topics/deep-kernel-processes-dkps