Interdomain Inducing Points in Sparse GPs

• Interdomain inducing points are a generalization of conventional GP inducing variables defined as linear functionals of latent functions.
• They enable efficient variational sparse-GP inference by incorporating transformations such as integrals, derivatives, and convolutions into the covariance structure.
• This approach offers modular scalability for multioutput, online, and deep GP models while maintaining Gaussianity and tractable computations.

Searching arXiv for recent and foundational papers on interdomain inducing points. Interdomain inducing points are a generalization of sparse Gaussian process inducing points in which the inducing variables are not restricted to point evaluations of a latent function at input locations, but are instead defined by linear transformations of that latent function. In the standard sparse approximation one writes $\mathbf{u}=f(Z)=\{f(\mathbf z_m)\}_{m=1}^M$; in the interdomain setting one replaces these input-domain variables by quantities such as derivatives, integrals, convolutions, spectral coefficients, or other linear functionals, while retaining Gaussianity and the standard variational sparse-GP machinery (Wilk et al., 2020). The same idea appears in several later forms: inducing variables tied to latent processes in multioutput models, inducing representations for inter-domain observations, time-dependent projection memories in online GPs, and, in a broader analogical sense, learned inducing-point bottlenecks in attention architectures such as SPIN and IPOT (Wu et al., 2021, Chen et al., 12 Feb 2025, Rastogi et al., 2022, Lee et al., 2023).

1. Formal definition and covariance structure

The defining distinction is between input-domain inducing variables and interdomain inducing variables. In the standard sparse-GP construction, the inducing variables are function values at inducing locations,

$\mathbf{u} = f(Z) = \{f(\mathbf z_m)\}_{m=1}^M.$

Interdomain inducing variables generalize this by applying a linear operator $L$ to the latent function. A generic observation operator is

$$L_n f(\cdot) = \int f(\mathbf x)\,w_n(\mathbf x)\,d\mathbf x,$$

with derivative observations as another special case,

$$L_n f(\cdot)=\frac{\partial f}{\partial x_d}(\mathbf x_n),$$

and direct point observations recovered by $L_n f(\cdot)=f(\mathbf x_n)$ (Wilk et al., 2020).

Because $L$ is linear and the latent process is Gaussian, the transformed variables remain jointly Gaussian with the latent function. The core covariance identities are

$$\mathrm{Cov}[L_i f(\cdot), f(\mathbf x')] = L_i\,k(\cdot,\mathbf x'),$$

and

$$\mathrm{Cov}[L_i f(\cdot), L_j f(\cdot)] = L_i\,k(\cdot,\cdot')\,L_j'.$$

For integral operators these become explicit kernel transforms,

$$\mathrm{Cov}[L_i f(\cdot), f(\mathbf x')] = \int k(\mathbf x,\mathbf x')\,w_i(\mathbf x)\,d\mathbf x,$$

$\mathbf{u} = f(Z) = \{f(\mathbf z_m)\}_{m=1}^M.$0

These formulas are the central mathematical object behind interdomain approximations: only the covariance blocks change, while the sparse variational framework itself is unchanged (Wilk et al., 2020).

A related but more specialized construction appears in activated deep Gaussian processes, where inducing variables are defined as RKHS inner products,

$\mathbf{u} = f(Z) = \{f(\mathbf z_m)\}_{m=1}^M.$1

By the reproducing property,

$\mathbf{u} = f(Z) = \{f(\mathbf z_m)\}_{m=1}^M.$2

so the basis function in the sparse posterior mean is exactly the inducing function itself. In that formulation, activation-shaped inducing functions become interdomain inducing features (Dutordoir et al., 2021).

2. Variational sparse-GP formulation

Interdomain inducing points fit into the standard sparse variational GP family without changing the basic ELBO. The approximate posterior is written as

$\mathbf{u} = f(Z) = \{f(\mathbf z_m)\}_{m=1}^M.$3

and the resulting posterior process is

$\mathbf{u} = f(Z) = \{f(\mathbf z_m)\}_{m=1}^M.$4

The ELBO remains

$\mathbf{u} = f(Z) = \{f(\mathbf z_m)\}_{m=1}^M.$5

This invariance of the variational objective is one of the main reasons interdomain inducing variables are useful: they alter representation and computational structure without requiring a new inference principle (Wilk et al., 2020).

The KL term is still computed only over the finite inducing variables,

$\mathbf{u} = f(Z) = \{f(\mathbf z_m)\}_{m=1}^M.$6

and whitening can again simplify the objective. In HIP-GP, whitening converts the inducing-variable prior to a standard normal and yields

$\mathbf{u} = f(Z) = \{f(\mathbf z_m)\}_{m=1}^M.$7

removing the problematic $\mathbf{u} = f(Z) = \{f(\mathbf z_m)\}_{m=1}^M.$8 term from the KL (Wu et al., 2021).

This unchanged variational skeleton is the key technical reason interdomain constructions are modular. They can be introduced through the definitions of $\mathbf{u} = f(Z) = \{f(\mathbf z_m)\}_{m=1}^M.$9 and $L$0, or through specialized covariance operators and conditionals, rather than by redesigning the optimization problem itself (Wilk et al., 2020).

3. Inter-domain observations and hierarchical inducing-point inference

A distinct but closely related setting is the inter-domain observation problem, where the GP realization and the noisy observations lie on different domains. HIP-GP formulates this as

$L$1

with $L$2 a linear operator such as integration, differentiation, or the identity (Wu et al., 2021).

HIP-GP places inducing variables in the latent domain,

$L$3

rather than in the transformed observation domain. The joint prior over an inter-domain observation $L$4 and the inducing variables is

$L$5

which gives the conditional

$L$6

A practical consequence is that the method avoids forming the full transformed-domain covariance $L$7; only the diagonal transformed variances $L$8 are needed per datapoint (Wu et al., 2021).

The main scalability mechanism is structural rather than probabilistic. HIP-GP assumes a stationary kernel and inducing points on a fixed evenly spaced grid, so that $L$9 has hierarchical Toeplitz structure. Via circulant embedding and FFT-based matrix-vector multiplies, whitening-related computations such as

$$L_n f(\cdot) = \int f(\mathbf x)\,w_n(\mathbf x)\,d\mathbf x,$$0

can be performed in

$$L_n f(\cdot) = \int f(\mathbf x)\,w_n(\mathbf x)\,d\mathbf x,$$1

The method is designed to scale to millions of inducing points, but the assumptions are restrictive: linear inter-domain mapping, stationary kernel, fixed grid, and low-dimensional input space (Wu et al., 2021).

This setup clarifies an important point about terminology. Interdomain inducing points and inter-domain observations are not identical notions. The former concerns the representation chosen for sparse approximation; the latter concerns the relation between latent process and observation operator. In HIP-GP they are coupled: the observations are inter-domain, and the inducing variables are placed in the latent domain to make scalable inference possible (Wu et al., 2021).

4. Time-dependent interdomain variables and recurrent memory

Online HiPPO Sparse Variational Gaussian Processes extend the interdomain idea by replacing fixed inducing locations with time-dependent projection coefficients. In this model, standard inducing variables $$L_n f(\cdot) = \int f(\mathbf x)\,w_n(\mathbf x)\,d\mathbf x,$$2 are replaced by integral transforms

$$L_n f(\cdot) = \int f(\mathbf x)\,w_n(\mathbf x)\,d\mathbf x,$$3

and the basis functions themselves vary with time through the HiPPO construction (Chen et al., 12 Feb 2025).

HiPPO maintains a memory state $$L_n f(\cdot) = \int f(\mathbf x)\,w_n(\mathbf x)\,d\mathbf x,$$4 representing the history of a signal $$L_n f(\cdot) = \int f(\mathbf x)\,w_n(\mathbf x)\,d\mathbf x,$$5 over $$L_n f(\cdot) = \int f(\mathbf x)\,w_n(\mathbf x)\,d\mathbf x,$$6 using time-dependent orthonormal polynomial basis functions $$L_n f(\cdot) = \int f(\mathbf x)\,w_n(\mathbf x)\,d\mathbf x,$$7 and a measure $$L_n f(\cdot) = \int f(\mathbf x)\,w_n(\mathbf x)\,d\mathbf x,$$8 satisfying

$$L_n f(\cdot) = \int f(\mathbf x)\,w_n(\mathbf x)\,d\mathbf x,$$9

The projection coefficients are

$$L_n f(\cdot)=\frac{\partial f}{\partial x_d}(\mathbf x_n),$$0

Embedding this into a GP yields interdomain inducing variables

$$L_n f(\cdot)=\frac{\partial f}{\partial x_d}(\mathbf x_n),$$1

so the inducing variables become a compressed summary of process history rather than pointwise latent values (Chen et al., 12 Feb 2025).

The memory interpretation is explicit. For HiPPO-LegS, the paper uses

$$L_n f(\cdot)=\frac{\partial f}{\partial x_d}(\mathbf x_n),$$2

with scaled Legendre polynomials on $$L_n f(\cdot)=\frac{\partial f}{\partial x_d}(\mathbf x_n),$$3, and the finite coefficient vector approximates the past signal via

$$L_n f(\cdot)=\frac{\partial f}{\partial x_d}(\mathbf x_n),$$4

The dynamics admit a recurrent update,

$$L_n f(\cdot)=\frac{\partial f}{\partial x_d}(\mathbf x_n),$$5

which is the basis of the model’s online memory mechanism (Chen et al., 12 Feb 2025).

Kernel blocks also evolve recurrently. The cross-covariance satisfies

$$L_n f(\cdot)=\frac{\partial f}{\partial x_d}(\mathbf x_n),$$6

with ODE update

$$L_n f(\cdot)=\frac{\partial f}{\partial x_d}(\mathbf x_n),$$7

For stationary kernels, the inducing covariance is approximated online using random Fourier features. The paper’s stated advantages are memory preservation, online efficiency, and computational efficiency relative to vanilla OSGPR (Chen et al., 12 Feb 2025).

5. Multioutput, convolutional, Fourier, and neural-network interpretations

Interdomain inducing variables are especially useful in multioutput GPs because they can be aligned with latent model structure. In the linear model of coregionalization,

$$L_n f(\cdot)=\frac{\partial f}{\partial x_d}(\mathbf x_n),$$8

choosing inducing variables as point values of the latent processes $$L_n f(\cdot)=\frac{\partial f}{\partial x_d}(\mathbf x_n),$$9 rather than of the observed outputs makes the prior covariance $L_n f(\cdot)=f(\mathbf x_n)$0 block diagonal across latent processes. The same framework covers convolution processes, inducing patches for convolutional GPs, variational Fourier features, and other structured inducing-variable choices implemented through GPflow’s unified abstractions for Kuu, Kuf, conditional, and prior_kl (Wilk et al., 2020).

The Fourier-feature case shows the computational motive clearly. Variational Fourier features are interdomain inducing variables in the spectral domain and can produce a low-rank-plus-diagonal inducing covariance,

$L_n f(\cdot)=f(\mathbf x_n)$1

This is not merely a representational variant; it can materially change the algebraic cost profile of inference (Wilk et al., 2020).

A more unusual reinterpretation appears in deep Gaussian processes. There, each inducing variable is defined by an RKHS inner product,

$L_n f(\cdot)=f(\mathbf x_n)$2

and the sparse posterior mean of a layer takes the form

$L_n f(\cdot)=f(\mathbf x_n)$3

Since $L_n f(\cdot)=f(\mathbf x_n)$4, choosing inducing functions shaped like activations turns the layer mean into a neural-network computation. If $L_n f(\cdot)=f(\mathbf x_n)$5 and $L_n f(\cdot)=f(\mathbf x_n)$6, then

$L_n f(\cdot)=f(\mathbf x_n)$7

which is exactly a fully connected neural-network layer (Dutordoir et al., 2021).

That correspondence is conditional on kernel-feature compatibility. The paper develops the construction using zonal kernels on the unit hypersphere, especially the first-order Arc Cosine kernel, and notes several caveats: GP samples are not generally in the RKHS, some kernel-activation pairs are incompatible, and truncating the spherical-harmonic expansion can inflate uncertainty and limit posterior expressivity (Dutordoir et al., 2021). Interdomain inducing points are therefore not only a compression device; they can also be a mechanism for matching sparse-GP approximations to latent algebraic structure, spectral structure, or neural-network nonlinearities.

6. Broader analogies, non-classical uses, and common misconceptions

In a strict GP sense, “interdomain inducing point” refers to inducing variables that are not ordinary point evaluations in the input domain. Several recent neural architectures use the language or logic of inducing points more loosely. SPIN, for example, is described as borrowing the logic of inducing-point approximations but applying it in a learned attention-based, interdomain way over a training set rather than as a classical kernel approximation. It learns a semi-parametric map

$L_n f(\cdot)=f(\mathbf x_n)$8

compresses a large dataset into a smaller learned set of inducing points $L_n f(\cdot)=f(\mathbf x_n)$9, and uses a datapoint-to-inducing-point cross-attention layer

$L$0

to construct that memory. The dominant dataset-dependent term for the key $L$1 operation is

$L$2

which is linear in dataset size $L$3 for fixed $L$4 (Rastogi et al., 2022).

IPOT uses a related inducing-point bottleneck in operator learning. Its encoder, processor, and decoder are

$L$5

$L$6

with computational complexity

$L$7

The paper explicitly presents this as being in the same broad spirit as interdomain inducing points: observations are mapped into a smaller latent domain, processed there, and decoded back to arbitrary output queries (Lee et al., 2023).

These broader usages should be distinguished from standard sparse-GP terminology. The high-throughput Bayesian optimization papers on ENT-DPP and inducing point allocation explicitly use inducing points in the standard sparse-GP sense of point evaluations or pseudo-observations in input space, and state that they do not develop an interdomain inducing-variable framework in the technical sense (Moss et al., 2022, Moss et al., 2023). Likewise, probabilistic subset selection of inducing points places a point-process prior over subsets of candidate input locations and is described as conceptually compatible with interdomain extensions, but it is not itself an interdomain inducing-point method (Uhrenholt et al., 2020).

A common misconception is therefore to treat any nonstandard inducing-point strategy as “interdomain.” The literature represented here supports a sharper distinction. In the technical GP sense, interdomain inducing points are linear functionals of the latent GP, chosen so that the sparse approximation lives in a transformed representation space. Broader neural uses preserve the compression logic and sometimes the cross-domain intuition, but they generally do not preserve the probabilistic semantics of interdomain inducing variables.