---
title: Inducing Variables in Gaussian Processes
url: https://www.emergentmind.com/topics/inducing-variables
type: topic
---

# Inducing Variables in Gaussian Processes

Inducing variables are latent constructs in Gaussian process (GP) modeling introduced to enable scalable inference by summarizing the information present in the full GP with a lower-dimensional set of “pseudo-points.” In standard and deep Gaussian processes, inducing variables can take the form of point-evaluations, inter-domain projections, or more general functionals, and are central to state-of-the-art scalable variational and fully Bayesian inference frameworks. Their selection, parametrization, and inference directly impact both predictive performance and computational complexity.

## 1. Formulation and Role of Inducing Variables in Sparse GPs

In the standard sparse variational GP (SVGP) approach, a collection of $M$ inducing inputs $Z = \{ z_m\}_{m=1}^M$ is chosen in the input space, and their corresponding function values $u = \{ f(z_m)\}_{m=1}^M$ define the inducing variables. The joint prior over training function values $f$ and $u$ in a GP with kernel $k$ is given by

$$
p(f, u) = p(f \mid u) \, p(u),
$$
where
$$
p(u) = \mathcal{N}(u \mid 0, K_{ZZ}), \\
p(f \mid u) = \mathcal{N}(f \mid K_{XZ} K_{ZZ}^{-1} u, K_{XX} - K_{XZ} K_{ZZ}^{-1} K_{ZX}),
$$
with $K_{AB}$ denoting the matrix $[k(a, b)]$ over sets $A, B$ [2407.17033][2010.09370][2304.14034][2207.07654][1807.02537][2003.03080].

The inducing variables act as a low-rank summary of the GP posterior, reducing the required computational effort from $O(N^3)$ to $O(N M^2)$, with $N$ the data size and $M \ll N$.

## 2. Variational and Bayesian Inference with Inducing Variables

### 2.1 Stochastic Variational Inference

The variational framework introduces an approximate posterior, typically Gaussian,
$$
q(u) = \mathcal{N}(u \mid m, S),
$$
and seeks to maximize the evidence lower bound (ELBO):
$$
\mathcal{L} = \mathbb{E}_{q(u)} \left[ \sum_{n=1}^N \mathbb{E}_{p(f_n|u)} \log p(y_n|f_n) \right] - \mathrm{KL}[q(u) \, \|\, p(u)].
$$
Both the inducing locations $Z$ and variational parameters $(m, S)$ are optimized jointly [2207.07654][2304.14034][1807.02537].

### 2.2 Bayesian Treatments

A fully Bayesian approach places priors on the inducing inputs $Z$ and kernel hyperparameters, treating them as random variables. Stochastic gradient Hamiltonian Monte Carlo (SGHMC) is used to jointly sample from the posterior over $\{u, Z, \theta\}$, improving uncertainty quantification and predictive accuracy over point-estimate or purely variational approaches [2003.03080]. This Bayesian treatment also addresses the sensitivity to inducing point placement [2010.09370].

### 2.3 Extensions: Point-Process Priors and Variable-Size Sets

To make the number and selection of inducing points part of the model, point-process priors (e.g., $p_\alpha(Z) \propto \exp(-\alpha |Z|^2)$) are used, with the inclusion of each candidate location parameterized by $\lambda_k \in [0, 1]$. The variational posterior $q(Z)$ is then fit under the ELBO, allowing the model to determine both number and placement of inducing points [2010.09370].

## 3. Generalizations: Inter-Domain and Orthogonal Inducing Variables

Inducing variables are not restricted to point evaluations. Inter-domain inducing variables are defined as linear functionals,
$$
u_m = L_m[f] = \int f(x) \phi_m(x) dx,
$$
where $\phi_m(x)$ are chosen basis functions in the RKHS of $k$. These generalize classical inducing points and can be constructed from, for example, spherical harmonics (zonal functions) or neural-style features [2304.14034].

Orthogonally-decoupled GP frameworks decompose the kernel as $k(x,x') = Q(x,x') + R(x,x')$, with $Q$ spanned by the inducing feature map and $R$ covering the orthogonal complement. Two families of inducing variables $u$ (principal) and $v$ (orthogonal) are introduced, enabling the simultaneous, independent enrichment of the mean and variance approximations at reduced computational cost [2304.14034].

## 4. Scalable Approximations and Subspace Inducing Inputs

For high-dimensional data, the cost of kernel computations involving full-dimensional $Z$ becomes prohibitive. A scalable alternative is to constrain inducing inputs to a low-rank subspace, $Z = A \widetilde{X}$, with $A \in \mathbb{R}^{M \times R}$ and $\widetilde{X} \in \mathbb{R}^{R \times D}$ a data-driven orthonormal basis (e.g., top right singular vectors). This reduces the computational cost of kernel evaluations from $O(D M^2)$ to $O(R M^2)$ per iteration while retaining flexibility [1807.02537]. Numerically stable “kernel-preconditioned” parametrizations of $q(u)$, such as $m=K_{Z,Z}\alpha$ and $S=(K_{Z,Z}^{-1}+\Sigma^{-1})^{-1}$ with $\Sigma$ diagonal, avoid instabilities and reduce the requirement for explicit regularization.

## 5. Inducing Variables in Deep Gaussian Processes

Deep Gaussian processes (DGPs) stack multiple GPs, with each hidden layer $\ell$ having its own set of $M$ inducing inputs $Z^{(\ell)}$ and latent outputs $U^{(\ell)}$. The full joint prior becomes
$$
p(y, F^{(1:L)}, U^{(1:L)}) = p(y \mid F^{(L)}) \prod_{\ell=1}^L p(F^{(\ell)} \mid F^{(\ell-1)}, U^{(\ell)}) p(U^{(\ell)}),
$$
with $p(U^{(\ell)})$ Gaussian [2407.17033]. Inducing variables at each layer render otherwise intractable integrals and marginalizations computationally feasible at $O(N M^2 L)$ cost.

Recent advances address the challenge of accurate posterior inference over these multi-layer inducing variables, including

- Probabilistic selection of per-layer $Z^{(\ell)}$ via point-process priors [2010.09370];
- Fully Bayesian DGPs using SGHMC for all $U^{(\ell)}, Z^{(\ell)}, \theta^{(\ell)}$ [2003.03080];
- Denoising Diffusion Variational Inference (DDVI), in which the posterior $q(U)$ is represented as the terminal distribution of a reverse-time diffusion SDE parameterized by a neural network score function. This enables expressive, multimodal posteriors and provides a tractable, explicit ELBO [2407.17033].

## 6. Algorithmic Details and Empirical Comparisons

### 6.1 Classical and Advanced Inference Schemes

- **DSVI (Doubly Stochastic VI):** Takes $q(u) = \mathcal{N}(m, S)$, efficient but potentially biased for complex posteriors.
- **IPVI (Implicit VI):** Uses neural-network samplers, adversarial losses, does not yield explicit ELBO.
- **DDVI:** Employs reverse-time diffusion processes, optimizing an explicit path-space KL and corresponding ELBO, judged to outperform DSVI and IPVI in both expressivity and stability [2407.17033].

### 6.2 Empirical Findings

- DDVI-based DGPs achieve superior test RMSE, NLL, and calibration on UCI regression benchmarks (datasets up to $N=2 \times 10^6$) and achieve state-of-the-art accuracy on MNIST, Fashion-MNIST, CIFAR-10, SUSY, and HIGGS datasets [2407.17033].
- Probabilistic selection methods empirically show that as inducing points become less informative, the model prunes unnecessary points, trading off sparsity and predictive fit [2010.09370].
- Subspace inducing inputs offer computational speed-ups of $2\times$–$4\times$ without significant accuracy loss in extreme multi-label and high-dimensional settings [1807.02537].
- Mean-field variational GP approaches, when contrasted with Bayesian and DDVI methods, can exhibit underestimation of uncertainty and higher test errors, especially when inducing variable posteriors are complex [2207.07654][2003.03080][2407.17033].

## 7. Selection, Initialization, and Design of Inducing Variables

Proper initialization and adaptive learning of $Z$ are critical for model performance. Random selection from training points or guided methods (e.g., k-means, farthest-point) can be employed; variationally optimized $Z$ tend to outperform fixed sets, especially in representing out-of-distribution or unseen test points [2207.07654]. In inter-domain and orthogonally-decoupled setups, expressive basis design (e.g., spherical harmonics, neural network features) is important to cover both mean and covariance structures efficiently [2304.14034].

Point-process-based variational inference provides an additional layer of adaptivity, allowing the model to jointly infer both which locations to include and the number of inducing variables, automatically matching model complexity to data structure [2010.09370].

---

**References:**  
- [2407.17033]: "Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inference"  
- [2010.09370]: "Probabilistic selection of inducing points in sparse Gaussian processes"  
- [2304.14034]: "Spherical Inducing Features for Orthogonally-Decoupled Gaussian Processes"  
- [2207.07654]: "Learning inducing points and uncertainty on molecular data by scalable variational Gaussian processes"  
- [1807.02537]: "Fully Scalable Gaussian Processes using Subspace Inducing Inputs"  
- [2003.03080]: "Sparse Gaussian Processes Revisited: Bayesian Approaches to Inducing-Variable Approximations"

Source: https://www.emergentmind.com/topics/inducing-variables