Variance-Based Inducing-Input Placement in Sparse GPs

• Variance-based inducing-input placement is a method for selecting inducing points in sparse Gaussian processes to preserve the uncertainty structure and enhance prediction quality.
• It employs criteria such as maximum variance, mutual information, and weighted integrated MSE, along with both continuous and discrete optimization strategies.
• The approach balances global coverage with task-specific focus and adapts to model complexity by optimizing both the locations and the number of inducing points.

Variance-based inducing-input placement denotes the selection of inducing inputs $Z=\{z_j\}_{j=1}^M$ in sparse Gaussian processes so that a reduced set of inducing variables preserves the uncertainty structure of the latent function while controlling the dominant computational terms of training and prediction. In the literature, the same design problem appears under several closely related formulations: posterior-variance reduction in sparse GP regression, mutual-information or DPP-based subset selection, integrated variance minimization for sensor placement, weighted local IMSE for transductive local surrogates, and adaptive infilling rules that reduce the conditional-prior component of predictive variance in non-Gaussian sparse GP models. The unifying theme is that the inducing set is not only a compression device but also the principal determinant of approximation fidelity, especially through its effect on predictive variance, Nyström approximation quality, and the trade-off between global coverage and task-specific focus (Moss et al., 2022, Uhrenholt et al., 2020, Jakkala et al., 2023, Cole et al., 2020, Nicolas et al., 30 Jun 2026).

1. Sparse-GP foundations and the role of inducing inputs

Let $f:X\to\mathbb{R}$ be modeled as a Gaussian process with prior $f(x)\sim \mathcal{GP}(m(x),k(x,x'))$, and let observations satisfy $y=f(x)+\epsilon$ with Gaussian noise $\epsilon\sim \mathcal{N}(0,\sigma_n^2)$. Sparse GP constructions introduce inducing inputs $Z$ and inducing variables $u=f(Z)$, with kernel blocks such as $K_{ZZ}$ and $K_{XZ}$. In FITC, the covariance is approximated by a low-rank plus diagonal correction, yielding $O(NM^2)$ training complexity and prediction cost $f:X\to\mathbb{R}$0 for the mean and $f:X\to\mathbb{R}$1 for the variance per test point. In VFE/SVGP, one places a variational posterior $f:X\to\mathbb{R}$2 and optimizes an ELBO; with Gaussian likelihoods, the implied predictive posterior remains Gaussian in closed form and supports minibatched optimization (Moss et al., 2022, Uhrenholt et al., 2020).

Under the Gaussian-likelihood SVGP parameterization, the predictive equations are

$f:X\to\mathbb{R}$3

and

$f:X\to\mathbb{R}$4

Equivalent forms appear in the sparse-regression and sensor-placement literature, sometimes written as

$f:X\to\mathbb{R}$5

These expressions make the dependence on $f:X\to\mathbb{R}$6 explicit: the inducing set controls the projection $f:X\to\mathbb{R}$7, the low-rank correction, and therefore both the mean approximation and the residual uncertainty. Training scales as $f:X\to\mathbb{R}$8, or $f:X\to\mathbb{R}$9 per minibatch in SVGP, plus $f(x)\sim \mathcal{GP}(m(x),k(x,x'))$0 for factorization of $f(x)\sim \mathcal{GP}(m(x),k(x,x'))$1; prediction remains $f(x)\sim \mathcal{GP}(m(x),k(x,x'))$2 for the mean and $f(x)\sim \mathcal{GP}(m(x),k(x,x'))$3 for the variance unless additional structure or caching is used (Moss et al., 2022, Jakkala et al., 2023).

2. Classical variance criteria and their information-theoretic interpretation

Classical inducing-point placement seeks to reduce posterior uncertainty globally. One standard greedy rule is maximum variance, also described as conditional variance reduction (CVR),

$f(x)\sim \mathcal{GP}(m(x),k(x,x'))$4

while integrated variance reduction selects $f(x)\sim \mathcal{GP}(m(x),k(x,x'))$5 by minimizing

$f(x)\sim \mathcal{GP}(m(x),k(x,x'))$6

A more explicitly information-theoretic criterion selects $f(x)\sim \mathcal{GP}(m(x),k(x,x'))$7 to maximize mutual information with the latent function,

$f(x)\sim \mathcal{GP}(m(x),k(x,x'))$8

For a GP with Gaussian likelihood, this reduces to a log-determinant objective, and the MAP solution corresponds to a determinantal point process with kernel $f(x)\sim \mathcal{GP}(m(x),k(x,x'))$9 or a closely related form. Exact MAP is NP-hard, but greedy MAP selection provides an $y=f(x)+\epsilon$0 approximation using incremental Cholesky updates (Moss et al., 2022).

The same uncertainty-reduction principle appears in sensor placement. In Gaussian regression, selecting sensor locations $y=f(x)+\epsilon$1 is equivalent to selecting inducing inputs because the locations at which one samples are the same locations at which a sparse GP would place inducing inputs to minimize posterior uncertainty. Two standard objectives are the Gaussian mutual information

$y=f(x)+\epsilon$2

and the integrated variance

$y=f(x)+\epsilon$3

MI acts on a log-det of posterior covariance, whereas IVAR acts on the integrated trace; in Gaussian models both target reduction of posterior uncertainty, and the AM–GM inequality links the two viewpoints (Jakkala et al., 2023).

Common criteria can be organized as follows.

Criterion	Objective	Uncertainty target
CVR / greedy maximum variance	$y=f(x)+\epsilon$4	Pointwise posterior variance
Integrated variance reduction	$y=f(x)+\epsilon$5	Average posterior variance
MI / DPP	$y=f(x)+\epsilon$6	Global information about $y=f(x)+\epsilon$7
wIMSE	minimize weighted integrated local variance around $y=f(x)+\epsilon$8	Local predictive variance
ENT-DPP	$y=f(x)+\epsilon$9	Global uncertainty and uncertainty in $\epsilon\sim \mathcal{N}(0,\sigma_n^2)$0
IVR infilling for quantile regression	$\epsilon\sim \mathcal{N}(0,\sigma_n^2)$1	Conditional-prior variance

A persistent misconception is that variance-based placement is synonymous with space-filling design. The literature does use k-means centroids, uniformly spread points, and related coverage heuristics as baselines, but the variance-based formulations above are posterior criteria: they are driven by predictive variance, entropy, log-det objectives, or integrated residual covariance, not by geometry alone (Moss et al., 2022, Jakkala et al., 2023).

3. Differentiable continuous, discrete, and local formulations

A major development is the use of differentiable sparse-GP objectives to optimize inducing locations directly. In sensor placement, the inducing set $\epsilon\sim \mathcal{N}(0,\sigma_n^2)$2 is identified with the sensor set, and one optimizes the collapsed VFE bound with respect to $\epsilon\sim \mathcal{N}(0,\sigma_n^2)$3 on an unlabeled proxy design $\epsilon\sim \mathcal{N}(0,\sigma_n^2)$4 while setting $\epsilon\sim \mathcal{N}(0,\sigma_n^2)$5:

$\epsilon\sim \mathcal{N}(0,\sigma_n^2)$6

where $\epsilon\sim \mathcal{N}(0,\sigma_n^2)$7. The log-det term prefers diverse, nonredundant inducing locations, while the trace term is minus the sum of conditional variances and therefore a Monte Carlo estimate of integrated posterior variance. Because $\epsilon\sim \mathcal{N}(0,\sigma_n^2)$8 depends smoothly on $\epsilon\sim \mathcal{N}(0,\sigma_n^2)$9 through $Z$0 and $Z$1, gradient descent can be used in continuous domains, with per-iteration cost $Z$2. In discrete settings, the same formulation yields either a greedy incremental rule or a continuous optimization followed by assignment to the nearest candidate set (Jakkala et al., 2023).

Local sparse GP methodology sharpens this idea by making placement transductive. In locally induced Gaussian processes, for each prediction point $Z$3 one first forms a neighborhood $Z$4 and then constructs a local inducing set $Z$5. The key placement criterion is weighted integrated MSE,

$Z$6

which minimizes integrated posterior variance near $Z$7 rather than globally. For the squared-exponential kernel, a closed form is derived, and gradients are available for L-BFGS-B optimization. The paper emphasizes a pathology of singleton-target placement: if one greedily minimizes only $Z$8, inducing points “pile up” at $Z$9. Weighting and integrating around $u=f(Z)$0 mitigates this degeneracy and stabilizes local hyperparameter estimation (Cole et al., 2020).

The local framework also introduces a cascade of reusable designs. A single optimized template can be shifted to different $u=f(Z)$1 values, and space-filling templates such as qNorm approximate the wIMSE pattern at low cost. This suggests that variance-based placement is not tied to a single global inducing set: it can be made local, transductive, and amortized across many prediction sites when the application is large-scale surrogate modeling rather than global posterior approximation (Cole et al., 2020).

4. Bayesian inference over inducing sets and adaptive cardinality

A different line of work reframes variance-based placement as Bayesian inference over the inducing set itself. In this formulation, both the subset $u=f(Z)$2 and its cardinality $u=f(Z)$3 are random. The model augments the standard sparse-GP hierarchy with a prior over sets,

$u=f(Z)$4

and introduces a variational point process over a finite candidate set $u=f(Z)$5,

$u=f(Z)$6

The resulting objective becomes

$u=f(Z)$7

where $u=f(Z)$8 is the standard SVGP ELBO for subset $u=f(Z)$9. The conditional GP terms cancel exactly as in SVGP, and the KL term is analytically tractable under the Poisson point-process construction (Uhrenholt et al., 2020).

This formulation does not discard the classical variance argument; it embeds it into the ELBO. Adding an inducing point changes $K_{ZZ}$0 and the variational covariance, typically reducing variances and tightening the fit, thereby increasing $K_{ZZ}$1. The REINFORCE estimator drives $K_{ZZ}$2 upward when inclusion of $K_{ZZ}$3 consistently yields larger ELBO values, while the prior penalizes large $K_{ZZ}$4 through the quadratic term. The learned mean number of inducing points is $K_{ZZ}$5, so the method learns not only where to place points but how many to keep (Uhrenholt et al., 2020).

Empirically, the method reports that as observation noise, kernel smoothness, or input clustering increase, the exact GP posterior becomes smoother or more noise-dominated, the marginal value of extra inducing points decreases, and the adaptive procedure lowers $K_{ZZ}$6. The same construction extends layer-wise to deep Gaussian processes and integrates naturally with GP-LVMs, where the active inducing set changes as the latent representation evolves. A plausible implication is that variance-based placement need not be posed as a fixed-budget combinatorial problem; it can be treated as posterior inference over both informativeness and model complexity (Uhrenholt et al., 2020).

5. Goal-aligned placement in Bayesian optimization and sequential quantile regression

Pure global variance reduction can be misaligned with decision-making tasks. In high-throughput Bayesian optimization, the relevant objective is not uniform fidelity across the domain but accurate modeling where the maximizer likely lies. The ENT-DPP method therefore replaces the purely global criterion with

$K_{ZZ}$7

where $K_{ZZ}$8. When $K_{ZZ}$9, the criterion recovers the global variance/DPP objective; when $K_{XZ}$0, it focuses purely on information about the maximum value. A tractable lower bound leads to a reweighted DPP kernel

$K_{XZ}$1

with weights determined by pointwise approximations to $K_{XZ}$2. The resulting greedy MAP procedure has the same $K_{XZ}$3 complexity as CVR and is inserted into the BO loop after each batch to reselect inducing points from the enlarged dataset. In experiments with noisy $K_{XZ}$4D Shekel, $K_{XZ}$5D Michalewicz, and $K_{XZ}$6D Ackley under a total budget $K_{XZ}$7 over $K_{XZ}$8 iterations with batches of $K_{XZ}$9, ENT-DPP consistently achieves the lowest simple regret across all tasks and inducing budgets $O(NM^2)$0 (Moss et al., 2022).

Sequential sparse GP quantile regression introduces a different but related decomposition. Under an asymmetric Laplace likelihood and a Laplace approximation to the posterior over inducing variables, the predictive variance splits additively into conditional-prior and posterior-induced components,

$O(NM^2)$1

with

$O(NM^2)$2

and

$O(NM^2)$3

Adding an inducing input changes only the conditional-prior term, so the infilling rule maximizes the integrated reduction

$O(NM^2)$4

where

$O(NM^2)$5

By contrast, additional data reduce only $O(NM^2)$6, so data acquisition is driven by posterior-induced variance through rejection sampling. The switching rule compares integrated $O(NM^2)$7 and integrated $O(NM^2)$8 and chooses either inducing-input infilling or new data acquisition accordingly (Nicolas et al., 30 Jun 2026).

Together, these two lines of work show that variance-based placement can be made explicitly task-aligned. Rather than treating all uncertainty as interchangeable, they distinguish uncertainty about the maximizer in BO or the structural projection residual in quantile regression. This suggests that the central design problem is often not whether to reduce variance, but which variance component is operationally relevant (Moss et al., 2022, Nicolas et al., 30 Jun 2026).

6. Pathologies, limitations, and recurrent empirical patterns

Several recurring limitations appear across the literature. Global VFE optimization with respect to inducing locations is often multimodal and cubic in $O(NM^2)$9; direct targeting of a single prediction site can cause inducing-point pile-up; and candidate-restricted methods depend on the coverage of the observed set. In ENT-DPP, the information term $f:X\to\mathbb{R}$00 is approximated through a Gaussian approximation to the distribution of $f:X\to\mathbb{R}$01, which can be coarse when the posterior over the maximum is highly non-Gaussian or in very high dimensions. In probabilistic point-process selection, the inducing set is sampled from a fixed candidate pool and optimized with score-function gradients, so continuous location moves are deferred to pre- or post-training phases. In sequential quantile regression, the sparse framework does not remove the kernel and high-dimensional challenges intrinsic to GP modeling (Moss et al., 2022, Uhrenholt et al., 2020, Nicolas et al., 30 Jun 2026).

The empirical record nevertheless shows a stable pattern. In sensor placement, the differentiable sparse-GP objective produces placements on par with or better than MI-based baselines in terms of MI and reconstruction quality, while being up to $f:X\to\mathbb{R}$02 faster on temperature data, $f:X\to\mathbb{R}$03 faster on precipitation, and $f:X\to\mathbb{R}$04 faster on soil and salinity. In local surrogate modeling, variance-based wIMSE and its template approximations improve the accuracy–computational efficiency frontier, and template reuse can deliver approximately $f:X\to\mathbb{R}$05 speedup with similar accuracy in the Herbie’s tooth example. In high-throughput BO, reweighting the DPP criterion by information about $f:X\to\mathbb{R}$06 yields consistently lower simple regret than CVR, k-means centroids, or uniformly spread points. In probabilistic inducing-point selection, higher noise, smoother kernels, and clustered inputs systematically shift the preferred number of points downward (Jakkala et al., 2023, Cole et al., 2020, Moss et al., 2022, Uhrenholt et al., 2020).

A broader interpretation is that variance-based inducing-input placement is best understood as a family of uncertainty-allocation principles rather than a single algorithmic recipe. Some methods reduce global posterior variance, some reduce weighted local variance, some learn the number of inducing points through a Bayesian prior on sets, and some isolate task-relevant uncertainty components such as uncertainty in $f:X\to\mathbb{R}$07 or the conditional-prior residual. What they share is the use of inducing locations as the principal control surface for balancing fidelity and cost in sparse GP inference (Moss et al., 2022, Uhrenholt et al., 2020, Jakkala et al., 2023, Cole et al., 2020, Nicolas et al., 30 Jun 2026).