Fast and Provably Accurate Sequential Designs using Hilbert Space Gaussian Processes

Abstract: Gaussian processes are widely used for accurate emulation of unknown surfaces in sequential design of expensive simulation experiments. Integrated mean squared error (IMSE) is an effective acquisition function for sequential designs based on Gaussian processes. However, existing approaches struggle with its implementation because the required integrals often lack closed-form expressions for most kernel functions. We propose a novel and computationally efficient Hilbert space Gaussian process approximation for the IMSE acquisition function, where a truncated eigenbasis representation of the integral enables closed-form evaluation. We establish sharp global non-asymptotic bounds for both the approximation error of isotropic kernels and the resulting error in the acquisition function. In a series of numerical experiments with $γ$-stabilizing, the proposed method achieves substantially lower prediction error and reduced computation time compared to existing benchmarks. These results demonstrate that the proposed Hilbert space Gaussian process framework provides an accurate and computationally efficient approach for Gaussian process based sequential design.

• The paper proposes an HSGP-based IMSE surrogate to overcome the challenge of intractable kernel integrals in sequential GP designs.
• It leverages truncated sine basis expansions for stationary GPs, providing analytic closed forms and non-asymptotic error bounds.
• Empirical results demonstrate improved surrogate fidelity, reduced RMSE, and lower computational costs compared to traditional methods.

Fast and Provably Accurate Sequential Designs using Hilbert Space Gaussian Processes

Overview and Motivation

Gaussian process (GP) surrogates are central to sequential design in emulation of computationally expensive simulators. Among the acquisition criteria for sequential design, Integrated Mean Squared Error (IMSE) is particularly relevant for global surrogate accuracy. However, direct implementation of IMSE is computationally challenging for general kernels: the key kernel integrals involved generally lack analytic expressions beyond certain special cases. This paper proposes a closed-form, efficient approximation for IMSE-based sequential design leveraging the Hilbert Space Gaussian Process (HSGP) representation, offers detailed non-asymptotic error analyses, and validates its approach both theoretically and empirically (2604.20414).

HSGP-Based IMSE: Methodology

The HSGP method approximates stationary GPs on bounded domains via truncated eigenfunction expansions of the Laplace operator. Specifically, for domain $\Omega = (-B, B)^d$, the authors leverage a sine basis on an extended domain $(-L, L)^d$, yielding kernel approximations:

$$\hat k_m(x, x') = \sum_{j \in [m]^d} S\left(\frac{\pi j}{2L}\right) \phi_j(x) \phi_j(x')$$

where $S(\cdot)$ is the spectral density of the target kernel. The crucial observation is that all integrals involving sums of these basis functions over hyperrectangles have analytic closed forms, allowing the IMSE acquisition to be approximated by a quadratic form in an HSGP feature space of manageable (tunable) rank.

This leads to a surrogate acquisition

$\widehat{\imse}_m(t) = \frac{h^T(t)WG_dWh(t)}{P_{N,\eta}^2(t)+\eta}$

where $h(t)$, $W$, $G_d$, and the normalization term all exploit the analytic structure of the basis (see main text, Eq. (HSGP-IMSE_acq)). Such a structure supports efficient computation, avoids expensive quadrature, and generalizes flexibly to any kernel with closed-form spectral density (including Matérn, Gaussian, and generalized Wendland).

Theoretical Analysis: Convergence and Error Rates

The authors provide explicit global, non-asymptotic error bounds for both kernel approximation and IMSE-level acquisition error. For the kernel, the error decomposes into an exponential-decay “aliasing” term in $L$ and a “truncation” term decaying exponentially (Gaussian) or polynomially (Matérn, GW) in $m$. For IMSE, these converge accordingly under fill distance $(-L, L)^d$0:

• For Gaussian kernels (infinite smoothness): exponential decay in both $(-L, L)^d$1 and $(-L, L)^d$2.
• For Matérn kernels (finite smoothness $(-L, L)^d$3): exponential decay in $(-L, L)^d$4, polynomial in $(-L, L)^d$5.

Moreover, the analysis precisely quantifies the impact of $(-L, L)^d$6-stabilization (i.e., restricting candidate points a minimum distance from existing designs), which enforces quasi-uniformity and guarantees a denominator lower bound for numerical stability and theoretical control.

Numerical Experiments

Empirical studies are extensive and target both surrogate fidelity (IMSE profile reproduction) and end-to-end sequential design performance (measured by RMSE, mean posterior variance, and computational time).

Surrogate Fidelity

The one- and two-dimensional acquisition functions $(-L, L)^d$7 and $(-L, L)^d$8 are nearly indistinguishable for standard kernels—demonstrating that the HSGP surrogate matches the exact acquisition surface both pointwise and in its global geometry, including the localization of maxima and geometry of level-sets (see below).

Figure 1: Generalized Wendland kernel IMSE curves in 1D, showing near-perfect overlap between true IMSE and HSGP-approximated IMSE.

Sequential Simulation Results

For both 1D and 2D design tasks, sequential sampling using the HSGP-derived IMSE acquisition (HSGP-IMSE) consistently achieves:

• Lower test RMSE than IMSPE (hetGP) and LHS baselines,
• Faster reduction in mean posterior variance,
• Comparable or lower cumulative computational cost.

Representative figures:

Figure 2: Root mean squared error (RMSE) over sequential iterations for different acquisition baselines using the Matérn-3/2 kernel.

Figure 3: HSGP-IMSE posterior mean vs. true function with 95% confidence intervals, demonstrating accurate surrogate recovery in 1D.

For non-standard kernels (generalized Wendland), HSGP-IMSE considerably outperforms systematic space-filling sampling with similar computational overhead.

Figure 4: RMSE for generalized Wendland kernel in 2D: HSGP-IMSE outperforms LHS baseline for emulation accuracy.

The method’s time complexity analysis shows that, under typical tuning (number of basis functions $(-L, L)^d$9), HSGP-IMSE achieves computational costs $$\hat k_m(x, x') = \sum_{j \in [m]^d} S\left(\frac{\pi j}{2L}\right) \phi_j(x) \phi_j(x')$$0 per step (excluding covariance matrix updates), improving over direct quadrature approaches and scaling more gracefully with design size.

Claims and Practical Implications

The authors rigorously prove that for all shift-invariant kernels with closed-form spectral density, their surrogate achieves

• Globally sharp non-asymptotic error bounds at the kernel and acquisition levels,
• Exponentially decaying error rates for Gaussian; polynomial for Matérn,
• Acquisition-level error convergence matching the theoretical optimal $$\hat k_m(x, x') = \sum_{j \in [m]^d} S\left(\frac{\pi j}{2L}\right) \phi_j(x) \phi_j(x')$$1 for Matérn-$$\hat k_m(x, x') = \sum_{j \in [m]^d} S\left(\frac{\pi j}{2L}\right) \phi_j(x) \phi_j(x')$$2 (given tuning),
• Provable quasi-uniform design point placement under $$\hat k_m(x, x') = \sum_{j \in [m]^d} S\left(\frac{\pi j}{2L}\right) \phi_j(x) \phi_j(x')$$3-stabilization.

The framework is strictly more general than existing IMSE/IMSPE implementations (which are confined to analytic cases: Gaussian and half-integer Matérn). The demonstrated empirical improvements—in both surrogate fidelity and end-to-end learning efficiency—underscore the benefit of this approach for high-dimensional and/or nonstandard kernel problems in scientific design, simulation, and Bayesian experimental design.

Future Directions

The authors suggest several lines for further research:

• Exploration of alternative boundary conditions and bases for HSGP approximations, which may further reduce edge artifacts,
• Extension to other kernel-based integration tasks (kernel mean embeddings, Bayesian quadrature, spatial data fusion),
• Investigation of adaptive parameter schemes for the basis size and padding in high-dimensional spaces.

Conclusion

This paper presents a significant advance in practical and theoretical methodology for GP-based sequential design using global accuracy criteria. The HSGP-IMSE surrogate removes the bottleneck of intractable kernel integrals, generalizes to arbitrary spectral kernels, and provides rigorous—and tight—error controls for acquisition design. Empirical evaluations confirm that HSGP-IMSE matches or exceeds prior IMSE/IMSPE methods in accuracy and efficiency, while broadening kernel flexibility and offering theoretically justified parameter choices.

This framework is likely to become central in the design of sequential experiments for computer models, generalized surrogate modeling, and related applications in statistical machine learning.

---

Figures Referenced

Figure 1: Generalized Wendland (1D) IMSE comparison.

Figure 2: RMSE (Matérn).

Figure 3: HSGP-IMSE posterior mean vs. true function.

Figure 4: RMSE (GW).