Nonparametric GP Convolution Models

• Nonparametric GP Convolution Models (GPCM/CGPCM) are defined by convolving latent noise processes with data-driven impulse responses, enabling flexible modeling of diverse time series.
• They leverage GP priors and variational inference to learn spectral properties and structural complexity directly from data, accommodating both smooth and rough signals.
• Extensions such as causal constraints and multi-output formulations facilitate scalable inference and application across physical, financial, and biomedical domains.

Nonparametric Gaussian Process Convolution Models (GPCM/CGPCM) refer to a class of models in which stochastic processes are constructed by convolving latent noise processes with unknown, data-driven impulse responses, both of which are given nonparametric Gaussian process (GP) priors. This paradigm generalizes classic process convolutions by integrating nonparametric kernel learning and variational inference, enabling flexible modeling of complex, potentially causal, and multi-output time series and signals. The nonparametric approach allows the model's spectral and structural complexity to be directly learned from data, yielding a spectrum of behaviors including smooth, rough, and causally-constrained processes, and supports practical scalable inference via sparse approximations and advanced sampling techniques (Yue et al., 2019, Bruinsma et al., 2022, Bruinsma et al., 2018, McDonald et al., 2022).

1. Generative Construction and Model Family

The original Gaussian Process Convolution Model (GPCM) defines output processes $f(t)$ as convolutions of latent white noise $x(t)$ with an unknown filter $h$, both equipped with independent GP priors: $$f(t) = \int_{-\infty}^{+\infty} h(t - \tau) x(\tau) d\tau \qquad x \sim \mathcal{GP}(0, \delta(t-t')), \quad h \sim \mathcal{GP}(0, k_h)$$ Observations $y_i$ are typically modeled as $y_i = f(t_i) + \epsilon_i$, $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$.

The causal GPCM (CGPCM) adds a causality constraint by restricting the filter support: $$f(t) = \int_{-\infty}^{t} h(t - \tau) x(\tau) d\tau \qquad h(\tau) = 0 \ \text{for} \ \tau < 0$$ This representation is also known as moving-average construction. The covariance of the observed process remains nonparametric: $$k_f(t, t') = \sigma_x^2 \int_{-\infty}^\infty k_h(t-u, t'-u) du \quad \text{(acausal)}$$

$$k_f(t, t') = \sigma_x^2 \int_{0}^{\min(t, t')} k_h(t-u, t'-u) du \quad \text{(causal)}$$

This approach generalizes to the multivariate (multi-output) setting, notably in the Multivariate Gaussian Convolution Process (MGCP) or multi-output GPCM: $x(t)$0 with $x(t)$1 denoting independent latent GPs, and $x(t)$2 process/output-specific smoothing kernels (Yue et al., 2019).

2. Spectral and Smoothness Properties

The power spectral density (PSD) of the GPCM features rich nonparametric structure: $x(t)$3 where $x(t)$4 is the Fourier transform of $x(t)$5; for white-noise $x(t)$6, $x(t)$7. For GPCM with a decaying-EQ prior on $x(t)$8, $x(t)$9 decays super-polynomially, constraining $h$0 to almost surely be infinitely differentiable (“very smooth” paths) (Bruinsma et al., 2022, Bruinsma et al., 2018). In contrast, the CGPCM—by imposing causality—produces processes $h$1 whose regularity depends on the filter’s value at the origin: if $h$2, $h$3 is almost surely nowhere differentiable, resembling Brownian motion increments on small scales (see CGPCM Proposition 3.2 in (Bruinsma et al., 2022)). This generalization facilitates explicit modeling of non-smooth or rough signals, addressing key limitations of the standard GPCM.

Variants such as the Rough GPCM (RGPCM) further expand this family by convolving non-smooth latent processes (e.g., OU processes) with modulated white-noise filters, yielding models that can approximate fractional Ornstein–Uhlenbeck behavior with Hurst exponent $h$4, and capturing a wider range of rough and fractal-like structure (Bruinsma et al., 2022).

3. Inference: Variational Approximations and Sampling

Given the infinite-dimensionality and nonparametric priors, inference in GPCM/CGPCM relies heavily on variational approximations with inducing variables, and sometimes direct sampling approaches. The typical variational scheme augments the model with:

• Inducing variables $h$5 for the filter GP
• Inducing variables $h$6 for the latent process

The variational posterior is factorized as: $h$7 The Evidence Lower Bound (ELBO) is given by: $h$8 Optimal forms for $h$9 and $$f(t) = \int_{-\infty}^{+\infty} h(t - \tau) x(\tau) d\tau \qquad x \sim \mathcal{GP}(0, \delta(t-t')), \quad h \sim \mathcal{GP}(0, k_h)$$0 are Gaussian, updated by closed-form expressions derived from the structure of the model (Bruinsma et al., 2018, Bruinsma et al., 2022).

Recent advances include structured mean-field (SMF) approximations and fast Gibbs samplers that directly alternate conditionals $$f(t) = \int_{-\infty}^{+\infty} h(t - \tau) x(\tau) d\tau \qquad x \sim \mathcal{GP}(0, \delta(t-t')), \quad h \sim \mathcal{GP}(0, k_h)$$1, $$f(t) = \int_{-\infty}^{+\infty} h(t - \tau) x(\tau) d\tau \qquad x \sim \mathcal{GP}(0, \delta(t-t')), \quad h \sim \mathcal{GP}(0, k_h)$$2, each Gaussian, efficiently mixing towards the optimal solution: $$f(t) = \int_{-\infty}^{+\infty} h(t - \tau) x(\tau) d\tau \qquad x \sim \mathcal{GP}(0, \delta(t-t')), \quad h \sim \mathcal{GP}(0, k_h)$$3 This structured scheme overcomes calibration and mean-field limitations of classical approaches (Bruinsma et al., 2022).

For deep and scalable nonparametric convolutions, the use of interdomain inducing variables and pathwise functional sampling (via Matheron’s rule) further accelerates inference, allowing for minibatch training and large-scale datasets (McDonald et al., 2022).

4. Multi-Output, Deep, and Causal Extensions

The MGCP generalizes single-output GPCM to $$f(t) = \int_{-\infty}^{+\infty} h(t - \tau) x(\tau) d\tau \qquad x \sim \mathcal{GP}(0, \delta(t-t')), \quad h \sim \mathcal{GP}(0, k_h)$$4 outputs by convolving $$f(t) = \int_{-\infty}^{+\infty} h(t - \tau) x(\tau) d\tau \qquad x \sim \mathcal{GP}(0, \delta(t-t')), \quad h \sim \mathcal{GP}(0, k_h)$$5 shared latent GPs with $$f(t) = \int_{-\infty}^{+\infty} h(t - \tau) x(\tau) d\tau \qquad x \sim \mathcal{GP}(0, \delta(t-t')), \quad h \sim \mathcal{GP}(0, k_h)$$6 smoothing kernels $$f(t) = \int_{-\infty}^{+\infty} h(t - \tau) x(\tau) d\tau \qquad x \sim \mathcal{GP}(0, \delta(t-t')), \quad h \sim \mathcal{GP}(0, k_h)$$7: $$f(t) = \int_{-\infty}^{+\infty} h(t - \tau) x(\tau) d\tau \qquad x \sim \mathcal{GP}(0, \delta(t-t')), \quad h \sim \mathcal{GP}(0, k_h)$$8 Joint covariances between outputs are given by weighted sums of double integrals over products of kernel and latent GP covariances (Yue et al., 2019). Closed-form solutions (for Gaussian kernels) are: $$f(t) = \int_{-\infty}^{+\infty} h(t - \tau) x(\tau) d\tau \qquad x \sim \mathcal{GP}(0, \delta(t-t')), \quad h \sim \mathcal{GP}(0, k_h)$$9 where $y_i$0.

Deep nonparametric GP convolution architectures (NP-DGP) stack multiple convolutional GP layers, each defined by latent $y_i$1 and kernel $y_i$2, resulting in models capable of approximating highly structured, hierarchical signal properties. Each layer’s convolutional kernel is inferred nonparametrically, and variational inference is accomplished using doubly stochastic gradient methods and functional sampling (McDonald et al., 2022).

Causality is incorporated by restricting the support of convolutional kernels or filters to $y_i$3, enforcing the constraint that only past and present inputs influence outputs, which is critical for physical and dynamical systems (Bruinsma et al., 2018, Bruinsma et al., 2022, Yue et al., 2019).

5. Computational Complexity and Scalability

Standard formulations require manipulation of $y_i$4 dense covariance matrices (with $y_i$5 the number of observations), incurring $y_i$6 time and $y_i$7 memory. Inducing variable approximations reduce this to $y_i$8 (with $y_i$9) for shallow models, and further accelerations are available via interdomain approaches:

• Functional sampling via random Fourier features reduces the cost for sampling to $y_i = f(t_i) + \epsilon_i$0 (where $y_i = f(t_i) + \epsilon_i$1 is the feature basis size) (McDonald et al., 2022).
• Deep convolutional GP models with $y_i = f(t_i) + \epsilon_i$2 layers, $y_i = f(t_i) + \epsilon_i$3 latent processes, and $y_i = f(t_i) + \epsilon_i$4 outputs have per-iteration complexity $y_i = f(t_i) + \epsilon_i$5 for generic multi-output, or $y_i = f(t_i) + \epsilon_i$6 if the kernel GP is shared over outputs (McDonald et al., 2022).

The MGCP–Cox joint model further integrates time-to-event data, with its ELBO—including the expected Cox likelihood—computed efficiently using predictive mean/traces, reducing the overall inference and storage demands while retaining nonparametric flexibility (Yue et al., 2019).

6. Empirical Findings and Application Domains

Empirical studies across synthetic spectral mixtures, financial signals (e.g. VIX, crude oil prices), and environmental time series substantiate the flexible modeling capacity of GPCM/CGPCM:

• The GPCM under mean-field inference tends to underfit high-frequency structure, while structured inference and causal/rough extensions recover sharp, interpretable impulse responses, improved uncertainty calibration, and superior prediction error and likelihood metrics (Bruinsma et al., 2022).
• The CGPCM’s enforcement of causality leads to substantive improvements in predictive root mean square error (RMSE) and marginal likelihoods relative to the acausal GPCM (e.g., for sunspot and environmental time series, RMSE reduced by 10%, average predictive log-likelihood up by 0.8 nats/year) (Bruinsma et al., 2018).
• Deep NP-GPCM outperforms both classic GP and standard deep GP baselines on large-scale regression problems, especially with multi-output and high-dimensional data (McDonald et al., 2022).
• Application domains encompass physical and engineering system identification, financial volatility modeling (rough/causal structure), turbulence, and biomedical signal processing.

7. Open Challenges and Future Directions

Scalability, multivariate modeling, and inference flexibility remain active areas of research:

• Efficient GPU implementations for large $y_i = f(t_i) + \epsilon_i$7 and high-dimensional inputs are underexplored. Automated selection of inducing point locations and numbers, optimization of quadrature/sampling strategies, and the extension to nonstationary or non-Gaussian observation models are prominent research directions (Bruinsma et al., 2022).
• The computation of bivariate Gaussian CDFs required by certain causal models and fast hyperparameter gradients in structured inference schemes represent computational bottlenecks.
• Further theoretical work is motivated by the need to rigorously quantify the class of kernels and spectral densities accessible to GPCM/CGPCM with varying choices of kernel priors and latent process structure, especially as model depth increases (McDonald et al., 2022).