--- title: Gaussian Processes in Approximate Bayesian Computation url: https://www.emergentmind.com/topics/gaussian-processes-with-approximate-bayesian-computation type: topic --- # Gaussian Processes in Approximate Bayesian Computation Gaussian Processes (GPs) combined with Approximate Bayesian Computation (ABC) constitute an advanced suite of methodologies for likelihood-free Bayesian inference, targeting simulation-based models where explicit evaluation of the likelihood \( p(y \mid \theta) \) is either intractable or computationally prohibitive. By constructing surrogate models for key quantities such as the discrepancy, likelihood, or summary statistics, GPs enable dramatic reductions in simulator calls, provide uncertainty quantification, and facilitate automatic acquisition and decision strategies in ABC pipelines. ## 1. Fundamentals of Gaussian Processes in ABC A Gaussian Process is a nonparametric prior over functions, characterized by mean \( m(\theta) \) and kernel \( k(\theta, \theta') \). Within ABC, GPs are most commonly deployed in three ways: - **Discrepancy Surrogates**: Modeling the distance between simulator outputs and observed data, \( \Delta(\theta) = d(S(\text{sim}(\theta)), S_{obs}) \), as a GP, allows for probabilistic predictions of ABC acceptance and rejection [1610.06462][1910.06121][2404.08898]. - **Synthetic Likelihood Surrogates**: Modeling the unknown log-likelihood \( \ell(\theta) = \log \Pr(d(s(\text{sim}(\theta)), s(y)) \le \varepsilon \mid \theta) \) via a GP for accelerated ABC-MCMC [1401.1436][1401.2838][2104.03942]. - **Trajectory and Derivative Modeling**: In ODE/DDE parameter inference, a GP fitted to noisy trajectories permits direct matching of empirical derivatives to instantaneous model vector fields, bypassing costly numerical integration [1507.05117]. The ABC posterior, with a kernel threshold \( \varepsilon \), is typically expressed as \( \pi_{ABC}(\theta) \propto \pi(\theta) \Pr[\Delta(\theta) \le \varepsilon] \), where the probability term can be efficiently approximated or analytically integrated via the GP surrogate. ## 2. GP Surrogates for Discrepancy and Synthetic Likelihood ### Discrepancy Modeling Given observations \( \{(\theta_i, \delta_i)\} \), the GP provides for any new input \( \theta^* \): \[ \delta(\theta^*) \sim \mathcal{N}(\mu^*(\theta^*), \sigma^{2,*}(\theta^*)) \] where predictive mean and variance use standard GP regression formulas. The acceptance probability under the GP is [1610.06462]: \[ U(\theta^*) = \Phi\left(\frac{\varepsilon - \mu^*(\theta^*)}{\sqrt{\sigma^{2,*}(\theta^*) + \sigma_n^2}}\right) \] This enables either weighted prior sampling or analytical calculation of the model-based ABC posterior \( \pi_{ABC}(\theta) \propto \pi(\theta) U(\theta) \) [1910.06121][1704.00520]. ### Synthetic Log-Likelihood GP For the likelihood-free case, surrogates for summary statistics or log-likelihood allow direct emulation of the MH accept step [1401.1436][2104.03942][1401.2838]: - Place a GP prior on the simulated summary statistics or directly on the log-likelihood. - Use GP posterior variance to construct acceptance probabilities or error-aware thresholds. - Implement sequential or batch strategies to update the GP, targeting acquisition of informative simulation evaluations. ### Derivative Matching in DE Models By fitting a GP to observed time-series data, the empirical derivatives \( v^d(t) \) are inferred; model vector fields \( f(\hat{x}(t), \theta) \) are matched in ABC by a “derivative-space distance”: \[ \Delta(v^d, v^s) = \sum_{i=1}^L \| v^d(t_i) - f(\hat{x}(t_i), \theta) \|^2 \] This obviates trajectory simulation for each candidate \( \theta \), yielding orders-of-magnitude computational gains [1507.05117]. ## 3. Acquisition, Batch Design, and Early Rejection Strategies ### Sequential and Batch Acquisition Rules Efficient experimental design, exploiting GP surrogate uncertainty, is critical: - **Expected Integrated Variance (EIV)**: Select new \( \theta \) to minimize posterior variance of the ABC estimator [1704.00520][1910.06121]. - **Batch Bayesian Experimental Design**: Parallelize expensive simulator evaluations via greedy construction or loss minimization for batches \( \{\theta^*_1, ..., \theta^*_b\} \), enabling linear wall-time speedup [1910.06121]. - **Early Rejection ejMCMC**: GP-predicted quantile bounds \( h_a(\theta) \) on the discrepancy enable up-front rejection of hopeless proposals, yielding 20–80% fewer simulator calls versus standard ABC-MCMC [2404.08898]. #### Algorithmic Schematic (ejMCMC [2404.08898]): ```python for iteration in range(N): propose theta_star ~ q(. | theta_n) stage 1: if GP_predicted_lower_quantile(Delta(theta_star)) > threshold: early reject else: simulate x_star ~ p(. | theta_star) standard ABC-MCMC accept/reject ``` ### Adaptive Confidence and Error Control in ABC-MCMC - GPS-ABC uses Monte Carlo samples from the GP posterior to construct acceptance probabilities, proceeding only when estimated error \( E \le \xi \) is achieved [1401.2838]. - Experimental design is adaptive, with GP hyperparameters tuned upon addition of new simulation data. ## 4. Kernel Selection and Model Comparison GP kernel choice critically impacts reconstruction fidelity. Zhang et al. [2304.03911] systematically compare RBF, Cauchy, and Matérn (\( \nu = 5/2, 7/2, 9/2 \)) kernels in cosmological GP reconstruction under two inference schemes: - **ABC Rejection** gave moderate to strong Bayes-factor preference for Matérn 5/2 over RBF across datasets (CC, SNIa, GRB), but nested sampling of the exact log-marginal likelihood showed inconclusive evidence and sometimes reversed the ranking. - **Interpretation**: ABC-based selection is sensitive to the chosen discrepancy metric, potentially exaggerating differences; fully marginalized model evidences may be less discriminative. Kernel selection strategies extend to model-based GP surrogate ABC: cross-validation and expected utility maximization are used to automate kernel choice [1610.06462]. ## 5. Applications: Differential Equations, Genetics, Cosmology, Densities ### Parameter Inference in Dynamical Models - GP-ABC-SMC achieves reliable ODE/DDE parameter posteriors in <30 s, versus 10²–10⁶ s for standard ABC, with only minor loss of credible interval precision [1507.05117]. - Biochemical network, delay blowfly, and gene-transfer models demonstrate high simulation savings, accurate marginal recovery, and robust posterior-predictive checks [1507.05117][2404.08898][1610.06462]. ### Hierarchical Density Modeling via GP Priors Functional regression ABC with hierarchical GP priors allows nonparametric estimation of grouped densities, learning shrinkage relations and correcting KDE bias robustly [1410.8276]. ### Cosmological and Population-Genetic Inference ABC-GP surrogates reconstruct cosmic histories from chronometer, SNIa, and GRB datasets, allowing rigorous comparison of kernel choices for GP reconstructions [2304.03911]. In genetics, GP surrogates accelerate ABC on species divergence and horizontal gene transfer scenarios, yielding 10–100× simulation savings [1401.1436][1610.06462]. ## 6. Computational Performance, Scalability, and Limitations GP surrogate ABC methods scale cubically in the number of training points (O(\( t^3 \))) but this overhead is negligible relative to expensive simulator calls when \( t \lesssim 1000 \). Batch acquisition and parallel evaluation further reduce wall-clock times [1910.06121]. Limitations include: - Breakdown for highly multimodal or discontinuous response surfaces [1401.1436][1610.06462]. - Need for careful monitoring of surrogate fit and acquisition function efficacy. - Failure to recover multimodal ABC posteriors when relying entirely on GP surrogates (see bimodal examples in [2404.08898]). - Hyperparameter uncertainty is typically ignored—uncertainty bands may be under-covering [1910.06121]. ## 7. Theoretical Guarantees and Practical Guidelines Detailed balance and posterior consistency are maintained in GP-augmented Metropolis-Hastings schemes under regularity conditions [2104.03942][2404.08898][1401.2838]. Total variation error in stationary distributions can be controlled via per-step error thresholds [1401.2838]. Practical guidelines include: - Dimensionality reduction in summary statistics to \( q \lesssim 10 \) for GP tractability [1401.2838][1410.8276]. - Use of variance-based versus quantile-based acquisition functions depending on the uncertainty learning objective [1704.00520][1910.06121]. - Regular hyperparameter updates and use of multi-output GP formulations for correlated summaries [1401.2838]. - Cross-validation and utility maximization for kernel and surrogate model selection [1610.06462][2304.03911]. --- **References**: [1507.05117], [1401.1436], [1401.2838], [1610.06462], [1410.8276], [1704.00520], [2104.03942], [1910.06121], [2404.08898], [2304.03911] Source: https://www.emergentmind.com/topics/gaussian-processes-with-approximate-bayesian-computation