Sequential Bayesian Optimization Process

Updated 29 June 2026

Sequential Bayesian Optimization is a method that iteratively updates a probabilistic surrogate model, typically a Gaussian Process, to guide experimental selection.
It employs acquisition functions like mutual information and expected improvement to balance exploration, exploitation, and resource constraints.
Modern implementations integrate belief-MDP formulations with online planning techniques such as MCTS and rollout policies to enhance sample efficiency and decision accuracy.

Sequential Bayesian Optimization (SBO) refers to the repeated, data-driven selection of experimental settings or actions so as to efficiently maximize a reward (e.g., utility, information gain) or minimize uncertainty, using a probabilistic surrogate model and an acquisition function. In its canonical form, SBO iteratively constructs and updates a statistical belief (typically a Gaussian Process) over an unknown objective, uses this belief to select the most promising candidate for evaluation, incorporates the new data, and repeats until a stopping criterion is met. Modern sequential BO frameworks rigorously embed uncertainty–cost trade-offs, resource constraints, and domain knowledge via Markov decision processes, information-theoretic objectives, rollout/dynamic programming, and scalable approximations.

1. Belief-State MDP and Surrogate Model Construction

At the heart of the SBO process is a belief Markov Decision Process (belief-MDP), where the state at time $t$ encodes the current knowledge:

State $b_t$ : Includes the agent's location $v_t$ , remaining resource budget $\Delta e_t$ , and the posterior GP belief over the unknown field $f$ , $X_V\sim GP(m(x),k(x,x'))$ , conditioned on all measurements to date.
Action Space $\mathcal{A}$ : Encompasses both movement actions ( $a_{\mathrm{move}}\in$ neighbors of $v_t$ ) and sensing modal actions ( $a_{\mathrm{sense}}\in S$ ) at the current location.
Transition Model: Deterministically updates the state, deducting movement or sensing costs from the budget, updating the agent's position, and, when sensing is performed, adding a noisy observation and updating the GP via standard closed-form posterior equations:

$b_t$ 0

with $b_t$ 1 and $b_t$ 2 denoting kernel matrices at observed and candidate locations.

Resource Constraints: SBO strictly enforces finite total budget $b_t$ 3 and per-action movement $b_t$ 4 and sensing costs $b_t$ 5, terminating episodes upon budget exhaustion or goal achievement (Ott et al., 2022).

This MDP formalism generalizes the setting to resource-constrained, movement- and action-aware sequential optimization, accommodating complex objectives in, e.g., informative path planning with multimodal sensing.

2. Information-Theoretic Acquisition Functions

At each step, SBO uses an acquisition (information) function to quantify the expected utility of candidate actions.

Mutual Information Gain: The canonical acquisition in resource-constrained path planning takes the form:

$b_t$ 6

an efficient surrogate for

$b_t$ 7

denoting the reduction in the GP posterior covariance from performing action $b_t$ 8. Instantaneous reward is thus directly tied to information-theoretic reduction in model uncertainty (Ott et al., 2022).

Expected Improvement (EI) and Extensions: In many domain-agnostic SBO applications, EI and generalized versions (Probability of Improvement, Knowledge Gradient, q-Expected Hypervolume Improvement for multi-objective cases) are used:

$b_t$ 9

with adaptations supporting batch selection and constraints (Asru et al., 5 Apr 2025, Feliot et al., 2015).

3. Online Sequential Planning: Lookahead, MCTS, and Rollout

Selecting the next query entails solving a lookahead planning problem under model and resource constraints.

Monte Carlo Tree Search with Double Progressive Widening (MCTS-DPW): At each SBO step, Monte Carlo simulations estimate the expected long-term sum of task and information rewards for possible action sequences from the current belief state. Actions are selected using an upper-confidence bound (UCB) criterion:

$v_t$ 0

subject to expansion rules that limit exploration of the vast action space (Ott et al., 2022).

Rollout and Dynamic Programming: Sequential BO is rigorously connected to finite-horizon stochastic control by embedding acquisition-driven greedy heuristics ("base policies") in a one-step lookahead (rollout) policy:

$v_t$ 1

where $v_t$ 2 is the cost-to-go under the greedy policy, and $v_t$ 3 is the belief update. This strategy provably improves upon the myopic base policy and reduces sample complexity in practice (Bertsekas, 2022).

POMDP Formulation and Continuous Trajectories: In continuous domains (e.g., UAV environment monitoring), SBO is realized as a POMDP solved by variants of MCTS with trajectory parameterization, balancing exploration of uncertain regions (via $v_t$ 4) and exploitation of known high-gradient areas (via $v_t$ 5) (Morere et al., 2017).

4. Sparse and Scalable Surrogate Modeling

Conventional GP surrogates require $v_t$ 6 computation, limiting SBO scalability.

Sparse Spectrum GP with Entropy Regularization: Efficient O( $v_t$ 7) surrogates, such as the sparse spectrum GP (SSGP), are regularized with entropy of the global maximizer distribution to mitigate overconfidence. Maximizing log-marginal likelihood plus log-entropy of the distribution over optimizer locations retains both fit and uncertainty necessary for efficient SBO. SSGP with entropy regularization matches or outperforms full GPs in ill-conditioned or large-data regimens (Yang et al., 2019).
SMC and Particle-Based Approximations: Sequential Monte Carlo (SMC) methods provide fully Bayesian posterior updates over hyperparameters and acquisition-driven queries, enabling both robust uncertainty quantification and efficient optimization in complex, multi-objective or constrained domains (Benassi et al., 2011, Feliot et al., 2015).

Surrogate Method	Complexity per Update	Uncertainty Calibration
Full GP	$v_t$ 8	Accurate, data-dependent
Sparse Spectrum GP	$v_t$ 9	Requires entropy regularization
SMC-based	Linear in $\Delta e_t$ 0 (particles)	Posterior sampling, fully Bayesian

5. Adaptive Bayesian Experimental Design and Multi-Objective Extensions

SBO generalizes to broader experimental design and multi-criterion optimization.

Multi-Objective and Constrained SBO: Sophisticated acquisition rules, such as expected (hypervolume) improvement under extended Pareto-dominance, and corresponding SMC estimation schemes, are core for constrained and multi-objective SBO. This allows identification of Pareto fronts in high-dimensional objective–constraint spaces, extending classical improvement criteria (Feliot et al., 2015, Asru et al., 5 Apr 2025).
Sequential Bayesian Experimental Design: When the objective is parameter inference, SBO is formulated as a dynamic program maximizing expected cumulative information gain (e.g., sum of mutual information or KL-divergences), solved by approximate dynamic programming, with value function regression and backward induction to handle intractable, high-dimensional state spaces (Huan et al., 2016, Iollo et al., 2024).
Batch versus Sequential Decision Trade-offs: Time-aware SBO frameworks compare the efficiency of strictly sequential (fully adaptive) to batch or manifold (low-dimensional) sampling policies, quantifying regime boundaries via dimensionless experimental time metrics. Such analyses guide optimal strategy selection in self-driving labs or manufacturing pipelines (Slautin et al., 8 Feb 2026).

6. Domain-Specific Tailoring and Practical Implementation

Informative Path Planning with Multimodal Sensing: SBO is employed for agents operating in physical environments, integrating movement, sensing action selection, cost constraints, and GP belief updates. Online MCTS-based planners, informed by mutual information acquisition, enable simultaneous reasoning over movement and sensing, consistently outperforming earlier approaches in reward and environment reconstruction accuracy (Ott et al., 2022).
High-Dimensional and Functional Optimization: Sequential subspace, random embedding, or functional Bayesian optimization methods project the task to low-dimensional random subspaces, carrying out inner-loop GP optimization and re-centering after each block. Effective-dimension analysis provides regret bounds and convergence guarantees for these procedures (Shilton et al., 2020).
Sequential Quadratic Programming with Bayesian Optimization: In high-dimensional, constrained black-box settings, joint second-order GPs estimate values and derivatives, and sequential quadratic programming subproblems are solved under model uncertainty (SOCP formulations), followed by line-search via constrained Thompson sampling. Strong performance is documented in moderate- to high-dimensional engineering problems (Brunzema et al., 3 Feb 2026).

7. Theoretical Properties, Guarantees, and Empirical Validation

Regret and Convergence Guarantees: In both unconstrained and constrained settings, non-myopic policies (rollout/MCTS) exhibit provable improvements over greedy/myopic approaches, reducing regret constants and function evaluations by 10–30% in empirical benchmarks (Bertsekas, 2022). Fully Bayesian SMC-based approaches maintain uncertainty calibration and deliver competitive or superior regret profiles relative to UCB, PI, or EI (Bijl et al., 2016, Yang et al., 2019, Benassi et al., 2011).
Consistency and Accuracy: Theoretical results characterize the consistency of particle approximations for both posterior inference and design selection, including convergence rates for the estimation of expected information gain and optimal design policies in sequential Bayesian experimental design (Iollo et al., 2024).
Sample Efficiency and Resource Utilization: SBO methods consistently reduce required experiment counts relative to classical design-of-experiments methodology, supporting adaptive, sample-efficient exploration of infeasible, high-cost, or otherwise intractable problem domains (Asru et al., 5 Apr 2025, Li et al., 2024).

In summary, the sequential Bayesian optimization process fuses probabilistic surrogate learning, information-driven action selection, and online planning to efficiently solve high-cost, complex optimization and experimental design tasks. Contemporary approaches generalize the classic surrogate+acquisition paradigm by integrating belief-MDP models, robust scalable surrogate surrogates (sparse or SMC-based), advanced acquisition criteria (mutual information, Pareto hypervolume improvement), and online dynamic programming, establishing SBO as a core methodology for resource-constrained, adaptive optimal decision-making across scientific and engineering disciplines (Ott et al., 2022, Bertsekas, 2022, Feliot et al., 2015, Benassi et al., 2011, Asru et al., 5 Apr 2025).