Surrogate Modeling & Acquisition Functions

Updated 29 October 2025

Surrogate modeling is a technique that approximates expensive objective functions using data-driven models for efficient decision-making.
Acquisition functions balance exploration and exploitation by quantifying the value of sampling to guide optimal candidate selection.
These methods are critical in applications like engineering design and hyperparameter tuning, significantly enhancing sample efficiency.

Surrogate modeling and acquisition functions are foundational concepts in computational optimization, machine learning-guided experimentation, and simulation-based science. Surrogate models provide data-driven approximations of expensive or unavailable objective functions, enabling efficient decision-making in design, control, or resource allocation. Acquisition functions operationalize the trade-off between exploitation (sampling where surrogates predict favorable outcomes) and exploration (sampling where uncertainty is high or the surrogate is likely to be inaccurate), serving as selection policies to determine the next sample locations. These elements are critical in workflows such as Bayesian optimization, adaptive experimental design, active learning, sequential simulation, and meta-optimization.

1. Fundamental Roles and Mathematical Formulation

Surrogate models serve as computationally inexpensive proxies for complex or expensive-to-evaluate functions $g(\mathbf{x})$ , such as simulations, physical processes, or physical experiments. Let $\hat{g}(\mathbf{x})$ denote the surrogate, trained from data $\mathcal{D} = \{ (\mathbf{x}_i, y_i) \}_{i=1}^n$ where $y_i = g(\mathbf{x}_i) + \epsilon_i$ , with $\epsilon_i$ stochastic observation noise.

The acquisition function $\mathrm{Ac}(\mathbf{x})$ quantifies the value of sampling $\mathbf{x}$ next. In Bayesian settings, acquisition functions are expressed as expected utilities over the surrogate posterior: $\mathrm{Ac}(\mathbf{x}) = \mathbb{E}_{y \sim p(y|\mathbf{x}, \mathcal{D})}\big[ u(y; \tau) \big]$ where $u(y; \tau)$ is a utility function (e.g., improvement over incumbent, entropy reduction), and $p(y|\mathbf{x}, \mathcal{D})$ is the surrogate's predictive posterior. Examples:

Probability of Improvement (PI): $u^{PI}(y; \tau) = \mathbb{I}(y > \tau)$
Expected Improvement (EI): $u^{EI}(y; \tau) = \max(0, y-\tau)$
Lower Confidence Bound (LCB): $\mathrm{LCB}(\mathbf{x}) = \mu(\mathbf{x}) - \kappa \sigma(\mathbf{x})$

Acquisition function maximization is typically a nonconvex, potentially high-dimensional optimization problem. The optimal sample $\mathbf{x}_{n+1}$ is selected as

$\mathbf{x}_{n+1} = \arg\max_{\mathbf{x}} \mathrm{Ac}(\mathbf{x})$

2. Surrogate Models: Classes and Properties

A variety of surrogate model classes are used, each with distinct statistical and computational properties.

Gaussian Processes (GPs): Provide analytic expressions for mean and variance, supporting closed-form acquisition function computation for common choices (e.g., PI, EI). GPs are commonly used in low-to-moderate dimensions and for continuous covariates due to their flexibility and tractable posteriors (Naveiro et al., 19 Jan 2024, Pawar et al., 2021, Benjamins et al., 2022, Merchán et al., 2020).
Random Forests and Non-differentiable Models: Offer scalability and model non-smooth or high-dimensional functions. Transfer learning for non-differentiable surrogates (e.g., random forests) may rely on evolutionary strategies rather than gradient-based adaptation, given the non-smooth objective landscape (Pan et al., 23 Jan 2025).
Ensemble Models (EGP): Use mixtures of GPs or other model classes to adaptively capture complex function structures, avoid dependence on a single kernel, and improve empirical performance; acquisition can be performed via Thompson sampling (Lu et al., 2022).
Bayesian Neural Networks, Sparse Regressors: Appropriate for large-scale or high-dimensional problems; SBBO enables use of Bayesian surrogates where only posterior samples—not analytic densities—are accessible (Naveiro et al., 19 Jan 2024).
Kernel Interpolation (RKHS): Used when surrogates must be highly accurate and offer pointwise error bounds, as in dynamical system simulation; enables rigorous error analysis and adaptive sample placement (Cangelosi et al., 4 Sep 2025).

A surrogate's suitability strongly affects sample efficiency and final surrogate accuracy, sometimes more than the sampling or acquisition strategy itself (Lämmle et al., 2023).

3. Acquisition Function Design: Principles, Variants, and Extensions

The acquisition function encapsulates sequential decision-making by quantifying the value of information provided by sampling a candidate $\mathbf{x}$ , balancing:

Exploration: Selecting points with high predictive uncertainty or large model error.
Exploitation: Selecting points where the surrogate predicts favorable (e.g., optimal) outcomes.
Cost-awareness: Adjusting for variable cost of evaluating candidate points (Daningburg et al., 2022).

Several classes of acquisition functions—in addition to the classical PI and EI—have emerged:

Information-based (Entropy, Mutual Information): Select samples that maximally reduce uncertainty about the objective or a derived property (e.g., level set, optimum location) (Letham et al., 2022, Chen et al., 28 Apr 2025).
Look-ahead/Non-myopic Acquisition: FigBO incorporates future impact on global uncertainty by adding a term quantifying reduction in posterior variance over the domain, with an adaptively decaying weighting to ensure asymptotic convergence properties (Chen et al., 28 Apr 2025).
Batch and Multi-objective Acquisitions: Multi-objective ensembles treat several acquisition functions (e.g., EI, PI, LCB variants) as objectives in a multi-objective bootstrapping scheme, producing diverse candidate batch sets (e.g., via NSGA-II) (Chen et al., 2022).
Gradient- and Local Feature-Enhanced Functions: GUESS uses predictive uncertainty for exploration and higher-order local curvature (via Taylor/exploitation terms) for targeting non-linear regions in the global fit context (Lämmle et al., 2023).
Cost-sensitive Acquisition: Model cost-aware acquisition functions select points to maximize uncertainty reduction per unit cost, achieving large reductions in computational expense for given surrogate accuracy (Daningburg et al., 2022).
Strategy Ensembles and Adaptive Schedules: Alternating between, or weighting, multiple acquisition functions (or schedules such as EI-then-PI) often outperforms any fixed choice, especially as different landscapes favor different exploitation/exploration trade-offs (Benjamins et al., 2022, Merchán et al., 2020, Chen et al., 2022).

4. Optimization of Acquisition Functions: Computational Considerations

Efficiently maximizing acquisition functions is crucial for scalable surrogate-based optimization. Challenges include nonconvexity, high effective dimension, and multi-modality, especially for batch, non-myopic, or high-dimensional settings (Zhao et al., 2023, Wilson et al., 2018, Wilson et al., 2017, Kim et al., 2019).

Gradient-Based Optimization via MC/Reparameterization: Many acquisition functions can be expressed as integrals over the surrogate predictive posterior. If the surrogate is differentiable, the reparameterization trick permits unbiased gradient estimation and fast gradient-based optimization, even in parallel acquisition contexts (Wilson et al., 2018, Wilson et al., 2017). This is critical for high-dimensional and batch queries.
Global vs. Local Optimization: While global optimization of the acquisition is desirable for theoretical guarantees, practical implementations often use multi-start local optimizers (e.g., L-BFGS), yielding near-global performance with significantly reduced computational cost; theoretical regret bounds between local and global optimizers are quantified (Kim et al., 2019, Pawar et al., 2021).
Initialization Strategies: In high-dimensional domains, the initialization of the acquisition function optimizer greatly influences the quality of candidate solutions. Ensemble and history-driven initializations (e.g., via CMA-ES, GA, or sampling informed by earlier iterations) outperform traditional random initialization, which suffers from insufficient candidate coverage as dimension increases (Zhao et al., 2023).
Batch and Submodular Optimization: For submodular acquisition functions (e.g., EI, UCB), greedy maximization yields solutions with guarantees on proximity to the optimal batch utility, supporting scalable and parallel query generation (Wilson et al., 2018).

5. Adaptation, Transfer, and Meta-Learning in Surrogate Modeling

Increasing emphasis is placed on transferability, adaptability, and meta-learning—learning or leveraging knowledge across related tasks to reduce data and computation cost in new domains.

Transfer via Domain Affine Transformation: Pretrained surrogates can be adapted to new but affine-transformed domains by learning appropriate domain transformations; for non-differentiable surrogates, CMA-ES or similar evolutionary strategies enable adaptation without derivatives by optimizing empirical loss over a small set of transfer samples (Pan et al., 23 Jan 2025).
Sensitivity-Driven Adaptive Refinement: In simulation-based design or large-scale dynamical systems, adaptive refinement procedures select new surrogate training points by combining solution sensitivity with surrogate error bounds, efficiently reducing uncertainty where it matters for quantities of interest (Cangelosi et al., 4 Sep 2025).
Simulation-Based and Likelihood-Free Acquisition: Acquisition function construction and maximization can be performed via simulation or classification, freeing BO from the need for analytic predictive densities or utility expectations and allowing surrogate flexibility; for example, simulation-based BO (SBBO) enables use of arbitrary Bayesian models equipped with posterior sampling (Naveiro et al., 19 Jan 2024), and likelihood-free BO recasts acquisition computation as a weighted classification problem (Song et al., 2022).
Multi-output and Multi-objective Surrogate Models: Multi-surrogate models, which build one surrogate per objective in multi-objective BO, address deficiencies in scalarization-based single-surrogate (mono-surrogate) approaches—improving Pareto coverage and validity of acquisition function computation, although at increased computational expense (Chugh, 2022, Svendsen et al., 2019).

6. Applications and Empirical Evaluations

Surrogate modeling and refined acquisition strategies have demonstrated advantages across synthetic and real-world applications, including:

Engineering design and simulation: Trajectory optimization, vehicle dynamics, robotic control, remote sensing, and gravitational wave modeling (Svendsen et al., 2019, Pan et al., 23 Jan 2025, Daningburg et al., 2022).
Combinatorial and mixed-variable optimization: BO approaches generalize beyond GPs via SBBO and similar frameworks, allowing effective search over discrete, categorical, or hybrid spaces (Naveiro et al., 19 Jan 2024).
Hyperparameter optimization and neural architecture search: Surrogate ensemble methods, adaptive acquisition, and automatic selector heuristics improve convergence in ML pipeline tuning (Lu et al., 2022, Merchán et al., 2020).
Level set estimation and active experimental design: In tasks with binary or categorical outcomes, analytic look-ahead acquisition functions based on closed-form posteriors are essential for impact and sample efficiency (Letham et al., 2022).
Global surrogate modeling (not just optimization): Acquisition strategies that consider both unexplored and non-linear regions (e.g., GUESS) outperform space-filling or variance-only designs even as dimensionality increases (Lämmle et al., 2023).

Performance generally demonstrates that adaptive, sensitivity-driven, exploration-exploitation balanced, or transfer/meta-learned surrogate and acquisition strategies yield substantial sample and computational efficiency improvements, subject to appropriate model choice, correct handling of cost, and rigorous optimization of acquisition functions.

7. Current Challenges and Future Directions

Despite progress, several open directions persist:

High-dimensional settings: The curse of dimensionality continues to pose challenges for both surrogate accuracy and acquisition function maximization. Hybrid or embedded subspace methods, as well as advanced initializations and meta-modeling, are active research areas (Zhao et al., 2023).
Hybrid and cost-aware schemes: Developed acquisition functions weighing information gain per unit simulation or experimental cost have proven effective in domains with preferential or highly heterogeneous costs, but require accurate expense models (Daningburg et al., 2022).
Automated and dynamic selection: Adaptive, ensemble, or schedule-based selection of surrogate models and acquisition functions accommodates varying landscape structures and phase transitions in optimization, demanding continued theoretical and algorithmic development (Benjamins et al., 2022, Merchán et al., 2020, Chen et al., 2022).
Complex objectives and black-box simulators: Surrogate and acquisition strategies compatible with likelihood-free, composite, or simulator-defined models enable application to broader classes of scientific and engineering tasks (Song et al., 2022, Naveiro et al., 19 Jan 2024).
Transfer/meta-learning: Scaling beyond per-instance adaptation—toward systematic capture and deployment of cross-task structure—remains an area for further impact (Pan et al., 23 Jan 2025, Cangelosi et al., 4 Sep 2025).

Surrogate modeling and acquisition functions thus provide an extensible and mathematically grounded framework for sample-efficient decision-making in data- and resource-constrained optimization, learning, and simulation. Continued advances in model flexibility, acquisition formulation, optimization algorithms, and adaptive/refined learning schemes are progressively broadening their scope and impact across scientific, engineering, and data-driven disciplines.