---
title: Interdomain Inducing Points in Sparse GPs
url: https://www.emergentmind.com/topics/interdomain-inducing-point
type: topic
---

# Interdomain Inducing Points in Sparse GPs

Searching arXiv for recent and foundational papers on interdomain inducing points.
Interdomain inducing points are a generalization of sparse Gaussian process inducing points in which the inducing variables are not restricted to point evaluations of a latent function at input locations, but are instead defined by linear transformations of that latent function. In the standard sparse approximation one writes $\mathbf{u}=f(Z)=\{f(\mathbf z_m)\}_{m=1}^M$; in the interdomain setting one replaces these input-domain variables by quantities such as derivatives, integrals, convolutions, spectral coefficients, or other linear functionals, while retaining Gaussianity and the standard variational sparse-GP machinery [2003.01115]. The same idea appears in several later forms: inducing variables tied to latent processes in multioutput models, inducing representations for inter-domain observations, time-dependent projection memories in online GPs, and, in a broader analogical sense, learned inducing-point bottlenecks in attention architectures such as SPIN and IPOT [2103.00393][2502.08736][2205.11718][2312.10975].

## 1. Formal definition and covariance structure

The defining distinction is between input-domain inducing variables and interdomain inducing variables. In the standard sparse-GP construction, the inducing variables are function values at inducing locations,
\[
\mathbf{u} = f(Z) = \{f(\mathbf z_m)\}_{m=1}^M.
\]
Interdomain inducing variables generalize this by applying a linear operator $L$ to the latent function. A generic observation operator is
\[
L_n f(\cdot) = \int f(\mathbf x)\,w_n(\mathbf x)\,d\mathbf x,
\]
with derivative observations as another special case,
\[
L_n f(\cdot)=\frac{\partial f}{\partial x_d}(\mathbf x_n),
\]
and direct point observations recovered by $L_n f(\cdot)=f(\mathbf x_n)$ [2003.01115].

Because $L$ is linear and the latent process is Gaussian, the transformed variables remain jointly Gaussian with the latent function. The core covariance identities are
\[
\mathrm{Cov}[L_i f(\cdot), f(\mathbf x')] = L_i\,k(\cdot,\mathbf x'),
\]
and
\[
\mathrm{Cov}[L_i f(\cdot), L_j f(\cdot)] = L_i\,k(\cdot,\cdot')\,L_j'.
\]
For integral operators these become explicit kernel transforms,
\[
\mathrm{Cov}[L_i f(\cdot), f(\mathbf x')] = \int k(\mathbf x,\mathbf x')\,w_i(\mathbf x)\,d\mathbf x,
\]
\[
\mathrm{Cov}[L_i f(\cdot), L_j f(\cdot)] = \iint k(\mathbf x,\mathbf x')\,w_i(\mathbf x)\,w_j(\mathbf x')\,d\mathbf x\,d\mathbf x' .
\]
These formulas are the central mathematical object behind interdomain approximations: only the covariance blocks change, while the sparse variational framework itself is unchanged [2003.01115].

A related but more specialized construction appears in activated deep Gaussian processes, where inducing variables are defined as RKHS inner products,
\[
u_m=\langle f(\cdot), g_m(\cdot)\rangle_{\mathcal H}.
\]
By the reproducing property,
\[
\operatorname{Cov}(u_m,f(x)) = g_m(x),
\]
so the basis function in the sparse posterior mean is exactly the inducing function itself. In that formulation, activation-shaped inducing functions become interdomain inducing features [2105.04504].

## 2. Variational sparse-GP formulation

Interdomain inducing points fit into the standard sparse variational GP family without changing the basic ELBO. The approximate posterior is written as
\[
q(f(\cdot)) = p(f(\cdot)\mid \mathbf u)\,q(\mathbf u), \qquad q(\mathbf u)=\mathcal N(\mathbf u;\mathbf m,S),
\]
and the resulting posterior process is
\[
q(f(\cdot)) = \mathcal{GP}\!\left( K_{\cdot u}K_{uu}^{-1}\mathbf m,\; k(\cdot,\cdot)-K_{\cdot u}K_{uu}^{-1}(K_{uu}-S)K_{uu}^{-1}K_{u\cdot} \right).
\]
The ELBO remains
\[
L = \sum_{n=1}^N \mathbb E_{q(f(\mathbf x_n))}[\log p(y_n\mid f(\mathbf x_n))] - \mathrm{KL}(q(\mathbf u)\|p(\mathbf u)).
\]
This invariance of the variational objective is one of the main reasons interdomain inducing variables are useful: they alter representation and computational structure without requiring a new inference principle [2003.01115].

The KL term is still computed only over the finite inducing variables,
\[
\mathrm{KL}(q(\mathbf u)\|p(\mathbf u)) = -\tfrac12 \log|S| -\tfrac{M}{2} +\tfrac12 \log|K_{uu}| +\tfrac12 \mathrm{Tr}\!\left(K_{uu}^{-1}(S+\mathbf m\mathbf m^\top)\right),
\]
and whitening can again simplify the objective. In HIP-GP, whitening converts the inducing-variable prior to a standard normal and yields
\[
b' = \frac{1}{2}\left(\operatorname{tr}(S) + m^\top m - \log|S| - M\right),
\]
removing the problematic $\log|K_{u,u}|$ term from the KL [2103.00393].

This unchanged variational skeleton is the key technical reason interdomain constructions are modular. They can be introduced through the definitions of $K_{uu}$ and $K_{uf}$, or through specialized covariance operators and conditionals, rather than by redesigning the optimization problem itself [2003.01115].

## 3. Inter-domain observations and hierarchical inducing-point inference

A distinct but closely related setting is the inter-domain observation problem, where the GP realization and the noisy observations lie on different domains. HIP-GP formulates this as
\[
\rho \sim GP(0, k_\theta(\cdot,\cdot)), \qquad \rho^* = \mathcal{L}\circ \rho, \qquad y_n \mid \rho^*(x_n) \sim \mathcal{N}(\rho^*(x_n), \sigma_n^2),
\]
with $\mathcal L$ a linear operator such as integration, differentiation, or the identity [2103.00393].

HIP-GP places inducing variables in the latent domain,
\[
u = \rho(\bar{x}),
\]
rather than in the transformed observation domain. The joint prior over an inter-domain observation $\rho_n^*$ and the inducing variables is
\[
\begin{pmatrix} \rho_n^* \\ u \end{pmatrix} \sim \mathcal{N}\!\left( 0, \begin{pmatrix} k^{**}_{n,n} & k^*_{n,u} \\ k^*_{u,n} & K_{u,u} \end{pmatrix} \right),
\]
which gives the conditional
\[
p(\rho_n^* \mid u) = \mathcal{N}\!\Big( \rho_n^* \,\Big|\, k^*_{n,u}K_{u,u}^{-1}u,\; k^{**}_{n,n} - k^*_{n,u}K_{u,u}^{-1}k^*_{u,n} \Big).
\]
A practical consequence is that the method avoids forming the full transformed-domain covariance $K^{**}_{N,N}$; only the diagonal transformed variances $k^{**}_{n,n}$ are needed per datapoint [2103.00393].

The main scalability mechanism is structural rather than probabilistic. HIP-GP assumes a stationary kernel and inducing points on a fixed evenly spaced grid, so that $K_{u,u}$ has hierarchical Toeplitz structure. Via circulant embedding and FFT-based matrix-vector multiplies, whitening-related computations such as
\[
k_n = R^\top K_{u,u}^{-1}k^*_{u,n}
\]
can be performed in
\[
O(M\log M)\text{ time}, \qquad O(M)\text{ space}.
\]
The method is designed to scale to millions of inducing points, but the assumptions are restrictive: linear inter-domain mapping, stationary kernel, fixed grid, and low-dimensional input space [2103.00393].

This setup clarifies an important point about terminology. Interdomain inducing points and inter-domain observations are not identical notions. The former concerns the representation chosen for sparse approximation; the latter concerns the relation between latent process and observation operator. In HIP-GP they are coupled: the observations are inter-domain, and the inducing variables are placed in the latent domain to make scalable inference possible [2103.00393].

## 4. Time-dependent interdomain variables and recurrent memory

Online HiPPO Sparse Variational Gaussian Processes extend the interdomain idea by replacing fixed inducing locations with time-dependent projection coefficients. In this model, standard inducing variables $u_m=f(z_m)$ are replaced by integral transforms
\[
u_m := \int f(x)\phi_m(x)\,\mathrm{d}x,
\]
and the basis functions themselves vary with time through the HiPPO construction [2502.08736].

HiPPO maintains a memory state $c(t)\in\mathbb{R}^M$ representing the history of a signal $y(t)$ over $(-\infty,t]$ using time-dependent orthonormal polynomial basis functions $g_n^{(t)}(s)$ and a measure $\omega^{(t)}(s)$ satisfying
\[
\int_{-\infty}^t g_m^{(t)}(s)g_n^{(t)}(s)\omega^{(t)}(s)\,\mathrm{d}s = \delta_{mn}.
\]
The projection coefficients are
\[
c_n(t) = \int_{-\infty}^t y(s)g_n^{(t)}(s)\omega^{(t)}(s)\,\mathrm{d}s.
\]
Embedding this into a GP yields interdomain inducing variables
\[
u_{m}^{(t)} = \int f(x)\phi_{m}^{(t)}(x)\,\mathrm{d}x, \qquad \phi_{m}^{(t)}(x)=g_{m}^{(t)}(x)\,\omega^{(t)}(x),
\]
so the inducing variables become a compressed summary of process history rather than pointwise latent values [2502.08736].

The memory interpretation is explicit. For HiPPO-LegS, the paper uses
\[
\omega^{(t)}(s) = \frac{1}{t}\mathbf{1}_{[0,t]}(s),
\]
with scaled Legendre polynomials on $[0,t]$, and the finite coefficient vector approximates the past signal via
\[
y(s) \approx \sum_{n=0}^{M-1} c_n(t)g_n(s).
\]
The dynamics admit a recurrent update,
\[
\frac{\mathrm{d}}{\mathrm{d}t}\mathbf{c}(t)=\mathbf{A}(t)\mathbf{c}(t)+\mathbf{B}(t)y(t),
\qquad
\mathbf{c}_t = \mathbf{A}_t\mathbf{c}_{t-1}+\mathbf{B}_t y_t,
\]
which is the basis of the model’s online memory mechanism [2502.08736].

Kernel blocks also evolve recurrently. The cross-covariance satisfies
\[
\left[\mathbf{K}_{\mathbf{fu}^{(t)}}\right]_{nm} = \int k(x_n,x)\phi_m^{(t)}(x)\,\mathrm{d}x,
\]
with ODE update
\[
\frac{\mathrm{d}}{\mathrm{d}t}\left[\mathbf{K}_{\mathbf{fu}^{(t)}}\right]_{n,:}
=
\mathbf{A}(t)\left[\mathbf{K}_{\mathbf{fu}^{(t)}}\right]_{n,:} + \mathbf{B}(t)k(x_n,t).
\]
For stationary kernels, the inducing covariance is approximated online using random Fourier features. The paper’s stated advantages are memory preservation, online efficiency, and computational efficiency relative to vanilla OSGPR [2502.08736].

## 5. Multioutput, convolutional, Fourier, and neural-network interpretations

Interdomain inducing variables are especially useful in multioutput GPs because they can be aligned with latent model structure. In the linear model of coregionalization,
\[
g_\ell(\cdot)\sim \mathcal{GP}(0,k_\ell),\qquad f(\mathbf x)=Wg(\mathbf x),
\]
choosing inducing variables as point values of the latent processes $g_\ell$ rather than of the observed outputs makes the prior covariance $K_{uu}$ block diagonal across latent processes. The same framework covers convolution processes, inducing patches for convolutional GPs, variational Fourier features, and other structured inducing-variable choices implemented through GPflow’s unified abstractions for `Kuu`, `Kuf`, `conditional`, and `prior_kl` [2003.01115].

The Fourier-feature case shows the computational motive clearly. Variational Fourier features are interdomain inducing variables in the spectral domain and can produce a low-rank-plus-diagonal inducing covariance,
\[
K_{uu} = \mathrm{diag}(\alpha) + \beta\beta^\top.
\]
This is not merely a representational variant; it can materially change the algebraic cost profile of inference [2003.01115].

A more unusual reinterpretation appears in deep Gaussian processes. There, each inducing variable is defined by an RKHS inner product,
\[
u_m=\langle f(\cdot), g_m(\cdot)\rangle_{\mathcal H},
\]
and the sparse posterior mean of a layer takes the form
\[
\mathbb E_q[f_\ell(x)] = B_\ell^\top c_{u_\ell}(x).
\]
Since $c_{u_m}(x)=g_m(x)$, choosing inducing functions shaped like activations turns the layer mean into a neural-network computation. If $c_{u_\ell}(x)=\sigma(W_\ell x)$ and $B_\ell=V_\ell$, then
\[
\mathbb E_q[f_\ell(x)] = V_\ell^\top \sigma(W_\ell x),
\]
which is exactly a fully connected neural-network layer [2105.04504].

That correspondence is conditional on kernel-feature compatibility. The paper develops the construction using zonal kernels on the unit hypersphere, especially the first-order Arc Cosine kernel, and notes several caveats: GP samples are not generally in the RKHS, some kernel-activation pairs are incompatible, and truncating the spherical-harmonic expansion can inflate uncertainty and limit posterior expressivity [2105.04504]. Interdomain inducing points are therefore not only a compression device; they can also be a mechanism for matching sparse-GP approximations to latent algebraic structure, spectral structure, or neural-network nonlinearities.

## 6. Broader analogies, non-classical uses, and common misconceptions

In a strict GP sense, “interdomain inducing point” refers to inducing variables that are not ordinary point evaluations in the input domain. Several recent neural architectures use the language or logic of inducing points more loosely. SPIN, for example, is described as borrowing the logic of inducing-point approximations but applying it in a learned attention-based, interdomain way over a training set rather than as a classical kernel approximation. It learns a semi-parametric map
\[
y = f_\theta(x \mid D),
\]
compresses a large dataset into a smaller learned set of inducing points $H$, and uses a datapoint-to-inducing-point cross-attention layer
\[
XABD(H_D, H_A') = \mathrm{fold}\big(MAB(\mathrm{unfold}(H_D), \mathrm{unfold}(H_A'))\big)
\]
to construct that memory. The dominant dataset-dependent term for the key $XABD$ operation is
\[
O(nhfe),
\]
which is linear in dataset size $n$ for fixed $h,f,e$ [2205.11718].

IPOT uses a related inducing-point bottleneck in operator learning. Its encoder, processor, and decoder are
\[
Z_0 = \mathcal{G}_{enc}(a) = \textsl{Attention}(Z_q, a, a),\qquad
Z_{l+1} = \mathcal{G}_l(Z_l) = \textsl{Attention}(Z_l, Z_l, Z_l),
\]
\[
\tilde{u} = \mathcal{G}_{dec}(Y, Z_L) = \textsl{Attention}(Y, Z_L, Z_L),
\]
with computational complexity
\[
O(n_x n_z d + L n_z^2 d + n_z n_y d).
\]
The paper explicitly presents this as being in the same broad spirit as interdomain inducing points: observations are mapped into a smaller latent domain, processed there, and decoded back to arbitrary output queries [2312.10975].

These broader usages should be distinguished from standard sparse-GP terminology. The high-throughput Bayesian optimization papers on ENT-DPP and inducing point allocation explicitly use inducing points in the standard sparse-GP sense of point evaluations or pseudo-observations in input space, and state that they do not develop an interdomain inducing-variable framework in the technical sense [2206.02437][2301.10123]. Likewise, probabilistic subset selection of inducing points places a point-process prior over subsets of candidate input locations and is described as conceptually compatible with interdomain extensions, but it is not itself an interdomain inducing-point method [2010.09370].

A common misconception is therefore to treat any nonstandard inducing-point strategy as “interdomain.” The literature represented here supports a sharper distinction. In the technical GP sense, interdomain inducing points are linear functionals of the latent GP, chosen so that the sparse approximation lives in a transformed representation space. Broader neural uses preserve the compression logic and sometimes the cross-domain intuition, but they generally do not preserve the probabilistic semantics of interdomain inducing variables.

Source: https://www.emergentmind.com/topics/interdomain-inducing-point