---
title: Deep Surrogate Models with Gaussian Processes
url: https://www.emergentmind.com/topics/deep-surrogate-models-with-gaussian-processes-gps
type: topic
---

# Deep Surrogate Models with Gaussian Processes

Deep surrogate models with Gaussian Processes (GPs) denote a class of surrogate modeling frameworks in which multiple GP layers are composed to capture hierarchical, highly nonlinear, and nonstationary mappings between inputs and outputs in computer experiments and simulation-driven inference tasks. These deep Gaussian process (DGP) surrogates have demonstrated marked advantages in capturing multimodal, heteroscedastic, or abrupt behavior, often with superior uncertainty quantification and active learning potential compared to conventional single-layer GPs. Their development is closely linked to the requirements of Bayesian optimization, likelihood-free inference, large-scale simulation campaigns, and structured prediction in scientific computing.

## 1. Mathematical Foundations and Model Construction

The core principle of DGP surrogates is the functional composition of GPs across several layers. Consider a model of depth $L$ mapping $x \in \mathbb{R}^d$ to output $y$:

$$
f(x) = f_L(f_{L-1}(\cdots f_1(x) \cdots))
$$

where each $f_\ell$ is a GP defined on the output space of the previous layer, typically with its own stationary kernel $k^{(\ell)}(\cdot,\cdot)$. This recursive architecture produces a highly flexible prior over functions whose induced marginal is typically non-Gaussian, capable of modeling nonstationary, heteroscedastic, or multimodal behaviors via the nonlinear interaction of layerwise GPs [2305.19242].

Layer architectures vary. The standard approach utilizes independent GPs per output dimension at each layer, combined with RBF or ARD kernels. There are also specialized variants:

- **Latent variable DGPs:** Augment inputs at each layer with stochastic latent variables, e.g., $w \sim \mathcal{N}(0,1)$, to induce additional marginal flexibility and facilitate multimodal surrogates, as in “LV-2GP” for Bayesian optimization in likelihood-free inference (BOLFI) [2006.10571].
- **Locally linear projections:** Employed in Deep Jump Gaussian Processes (DJGP) for piecewise continuous functions, where a region-specific $W_j$ (local projection) is learned by a GP prior to extract local active subspaces, followed by a “jump” GP for regime-sensitive modeling [2510.21974].
- **Multi-output hierarchies:** The Deep Intrinsic Coregionalization Model GP (deepICMGP) embeds coregionalization matrices within each layer so as to model both nonlinear warpings and output-wise dependencies, critical for vector-structured outputs [2508.16434].

## 2. Variational Inference, MCMC, and Optimization

Exact Bayesian inference in DGPs is analytically intractable owing to the non-Gaussian, high-dimensional latent structure. Two main inference paradigms dominate:

- **Sparse variational inference (SVI):** For each layer $\ell$, $m_\ell$ inducing points $Z^{(\ell)}$, with associated function values $u^{(\ell)}$, are introduced; the variational posterior posits $q(u^{(\ell)}) = \mathcal{N}(m^{(\ell)}, S^{(\ell)})$. Doubly-stochastic variational methods (DSVI) sample through the composition using the reparameterization trick, optimizing an ELBO comprising expected log-likelihood and KL divergences per layer [1905.03350, 2411.14690, 2011.00415, 2112.05888].

- **MCMC with Vecchia/ESS acceleration:** For small- to moderate-$n$ problems or where full UQ is essential, MCMC can sample layerwise latent variables and hyperparameters, leveraging the Vecchia approximation to reduce cubic operations—forming block-sparse Cholesky factorizations and univariate conditionals for each row. Elliptical slice sampling (ESS) enables efficient proposals for latent GPs under Gaussian priors [2204.02904, 2012.08015, 2305.19242].

Optimization schedules in SVI typically alternate natural-gradient updates (for variational means/variances of the final GP) with Adam or SGD steps on hyperparameters and inducing inputs [1905.03350, 2411.14690]. For quantile-conditioning in multimodal settings, IWVI (importance-weighted VI) is adopted for tighter bounds [2006.10571].

## 3. Predictive Uncertainty and Hierarchical Propagation

The predictive posterior of a DGP does not remain Gaussian due to nonlinear propagation through random layers. At test input $x^*$:

1. For SVI/Doubly-stochastic inference: Multiple ($S$) draws are propagated through each layer, approximating the posterior predictive mean and variance as empirical estimates over these draws.
2. For fully Bayesian (MCMC) approaches: Predictive samples are averaged over MCMC draws, often with latent layer and hyperparameter samples, employing the law of total variance [2204.02904, 2012.08015].

Through this composition, hierarchical uncertainty accumulates: uncertainty in the latent GPs at lower layers amplifies or transforms through each subsequent GP. In particular, DGP marginals can become strongly non-Gaussian—exhibiting multimodal, heavy-tailed, or skewed distributions. To maintain principled acquisition strategies for active learning or Bayesian Optimization, quantile-conditioning on the predictive samples, rather than relying on mean-variance approximations, has been shown to yield superior exploration-exploitation trade-offs in multimodal or skewed targets [2006.10571].

## 4. Surrogate Integration for Optimization and Inverse Problems

DGP surrogates are particularly prominent in:

- **Bayesian Optimization (BO):** DGPs replace stationary GPs as the BO surrogate, with tailored acquisition rules such as quantile-conditioned lower confidence bounds (LCB) or expected improvement computed from DGP posterior samples [2006.10571, 1905.03350].
- **Likelihood-Free Inference (BOLFI):** DGP surrogates capture multimodal posteriors and irregular discrepancy landscapes, reducing simulator calls required for accurate posterior approximation (e.g., Wasserstein distance benchmarks on BDM, TE1–3, NW tasks) [2006.10571].
- **Active design and sequential experimental design:** DGPs, via information-based acquisition functions (e.g., ALC), direct simulator evaluation to non-uniformly cover regions of epistemic uncertainty, outperforming stationary GPs especially on regime-changing domains [2012.08015, 2508.16434].
- **High-dimensional, piecewise, and categorical surrogates:** DJGP provides reliable UQ and accuracy on discontinuous targets in high-$D$, and “deep” GP surrogates with warping layers extend to categorical or binary outputs [2510.21974, 2501.14946].

## 5. Scalability and Algorithmic Efficiency

Traditional DGPs incur $O(n^3)$ costs per layer due to covariance inversions. Several strategies have been introduced to enhance computational tractability:

| Method                   | Computational Complexity               | Scale Feasible                |
|--------------------------|----------------------------------------|-------------------------------|
| Inducing-point VI        | $O(L n m^2)$ (with $m \ll n$ per layer)| Large ($n\simeq 10^6$)        |
| Vecchia MCMC/ESS         | $O(L n m^2)$ with $m$ fixed            | $n \simeq 10^5$               |
| Hierarchical Expansion (DTMGP) | $O(\text{polylog}\,M)$ forward/backward | $d \sim 10$, $M$ large     |

- **Vecchia Approximation:** Enabling exact or MCMC-based inference for $n\gg 10^4$, as in the deepgp R package, with empirical performance matching full DGP and outpacing inducing-point VI in RMSE, CRPS, and time [2204.02904, 2112.05888].
- **Sparse tensor Markov expansion:** DTMGP leverages Markov structure for hierarchical expansions, so that forward/inverse operations scale only as $O(\text{polylog}\,M)$, permitting rapid training and prediction for high-dimensional computer models [2112.05888].
- **Mini-batch SVI and cross-layer parameter sharing:** Used in large-scale BO, simulation, or categorical surrogate settings [2411.14690, 2501.14946].

## 6. Empirical Benchmarks and Case Studies

Extensive experimental validation confirms the superiority of DGP surrogates in nonstationary and multimodal regimes:

- **Likelihood-free inference:** DGP surrogates outperform GPs in bimodal (TE2, NW) settings with median scaled Wasserstein distance as low as $1.6$, compared to GP at $2.65$ [2006.10571].
- **Bayesian optimization tasks:** Two- and three-layer DGPs reach lower regret and sub-optimality (e.g., Trid-10d, Hartmann-6d, aerospace booster), achieving solutions with typically half the number of function calls as stationary GP surrogates [1905.03350].
- **Large-scale surrogates (satellite drag, COMPAS BBH):** Vecchia-approximate DGPs scale to $n=10^5$ with UQ and RMSE outperforming variational DGP and local GPs [2204.02904, 2411.14690].
- **Piecewise and discontinuous problems:** DJGP outperforms global DGP and preprojected JGP in both RMSE and CRPS across real and synthetic high-$D$ benchmarks [2510.21974].
- **Multi-output modeling:** deepICMGP achieves top-three performance for RMSE and CRPS across 8 synthetic and industrial benchmarks, with superior multivariate uncertainty quantification [2508.16434].

## 7. Application Considerations and Current Limitations

While DGP surrogates offer enhanced flexibility and modeling fidelity, several caveats persist:

- **Inference complexity:** MCMC-based DGPs offer superior UQ at higher computational cost; variational methods may under-quantify uncertainty when multimodal posteriors are present [2305.19242].
- **Model selection:** Overly deep or flexible DGPs can overfit small data regimes unless regularizing priors or smoothness penalties are used [2411.14690].
- **Scalability:** While inducing-point, Vecchia, and sparse-tensor approaches mitigate cubic bottlenecks, there is still a tradeoff between computational cost, fidelity, and uncertainty propagation, particularly as dataset size and output dimension increase [2112.05888, 2204.02904].

Emerging trends include integrating gradient observations into DGP frameworks for enhanced surrogate fidelity (critical in scientific computation), rigorous treatment of high-dimensional and multi-task regimes, and unified modeling for categorical or structured outputs using warping extensions [2512.18066, 2501.14946, 2508.16434].

## References

- "Likelihood-Free Inference with Deep Gaussian Processes" [2006.10571]
- "Bayesian Optimization using Deep Gaussian Processes" [1905.03350]
- "Vecchia-approximated Deep Gaussian Processes for Computer Experiments" [2204.02904]
- "Deep Jump Gaussian Processes for Surrogate Modeling of High-Dimensional Piecewise Continuous Functions" [2510.21974]
- "Deep Gaussian Processes with Gradients" [2512.18066]
- "Modernizing full posterior inference for surrogate modeling of categorical-output simulation experiments" [2501.14946]
- "Deep Intrinsic Coregionalization Multi-Output Gaussian Process Surrogate with Active Learning" [2508.16434]
- "Nonstationary Gaussian Process Surrogates" [2305.19242]
- "Deep Gaussian Process Emulation and Uncertainty Quantification for Large Computer Experiments" [2411.14690]
- "A Sparse Expansion For Deep Gaussian Processes" [2112.05888]
- "Inter-domain Deep Gaussian Processes" [2011.00415]
- "Active Learning for Deep Gaussian Process Surrogates" [2012.08015]
- "Gaussian process regression + deep neural network autoencoder for probabilistic surrogate modeling in nonlinear mechanics of solids" [2407.10732]

Source: https://www.emergentmind.com/topics/deep-surrogate-models-with-gaussian-processes-gps