Stochastic Control for Global Optimization

• Stochastic control for global optimization is a framework that reformulates the search process as a control problem with adaptive variance and policy-guided exploration.
• It employs Monte Carlo integration, surrogate models, and occupation-measure techniques to handle noisy, expensive evaluations in high-dimensional spaces.
• The approach rigorously balances exploration and exploitation, offering certified convergence guarantees and scalable performance in complex optimization scenarios.

A stochastic control framework for global optimization recasts the search for global minimizers as a stochastic optimal control problem, employing probabilistic models, adaptive variance control, and policy-gradient or occupation-measure techniques to balance exploration and exploitation efficiently. Such frameworks are particularly effective when objective function evaluations are noisy, high-dimensional, or only available via expensive black-box simulation. Recent research advances have unified adaptive variance strategies, Monte Carlo-based surrogate modeling, semidefinite programming relaxations, and particle filtering to deliver statistically robust global search capabilities with rigorous convergence guarantees and explicit scalability considerations (Carraro et al., 2019, Qiu, 3 Jan 2026, Holtorf et al., 2022).

1. Core Principles and Problem Formulation

Global optimization under stochastic control formulations seeks to minimize an expected objective or integral: $$J(d)=\int_{\Omega} \phi(d,x) w(x)\,dx,\quad d\in S\subset\mathbb{R}^n$$ where $\phi$ is a computationally expensive black-box integrand, $x\in\mathbb{R}^{n_x}$ is random, and $J$ is typically multimodal. In this context, objective evaluations may be inherently noisy (e.g., due to sampling, simulation, or measurement uncertainty), and the goal is to recover a global minimizer or global minimum of $J(\cdot)$ using as few expensive calls to $\phi$ as possible.

The stochastic control perspective reformulates the global optimization problem as one of determining an optimal control (policy, variance schedule, or occupation measure) that directs the search trajectory or sampling allocation. For example, in adaptive stochastic EGO (Efficient Global Optimization), the variance of Monte Carlo Integration (MCI) errors is controlled adaptively to balance the cost of evaluations with the information gained in the surrogate modeling step (Carraro et al., 2019).

2. Monte Carlo Integration with Adaptive Variance Selection

Monte Carlo Integration is often used to approximate the intractable objective $J(d)$. For a fixed candidate $d$: $$\bar{J}(d) = \frac{1}{n_r}\sum_{i=1}^{n_r} \phi(d, x^{(i)})$$ with sample $x^{(i)} \sim w(x)$. The variance of this estimate, $\bar{\sigma}^2(d)$, can be controlled by adjusting the replication number $n_r$. Importantly, the variance is not merely a nuisance: it can be leveraged as a control variable within the optimization framework.

By defining a target variance $\sigma_{\text{target}}^2$ and iteratively adjusting $n_r$ so that $\bar{\sigma}^2(d) \approx \sigma_{\text{target}}^2$, one embeds variance control directly into the optimization algorithm. The Stochastic Kriging (SK) metamodel uses these statistics to construct a surrogate of $J(\cdot)$, accommodating for the uncertainty in function estimates via the controlled variance.

An adaptive variance-selection law,

$$\sigma_\text{adapt}^2 = \sigma_\text{target}^2 \exp[-g(n, n_{\text{close}})]$$

modulates the level of effort (i.e., cost) at each candidate according to the "closeness" of existing samples (exploitation) versus the need to explore new regions (exploration). This leads to efficient automatic trade-offs, typically outperforming schemes with fixed variance targets by reducing unnecessary costly sampling in unpromising regions (Carraro et al., 2019).

3. Stochastic Control Optimizations: Dynamic Programming and HJB Structures

Global optimization problems can equivalently be viewed as regularized stochastic control problems: $$V^\varepsilon(0,x_0) = \inf_\theta \mathbb{E}\left[ G\left(X_1^{0,x_0;\theta}\right) + \frac{\varepsilon}{2} \int_0^1 |\theta_s|^2 ds \right]$$ where $$X_t^{0,x_0; \theta} = x_0 + \int_0^t \theta_s ds + W_t$$ is the controlled search trajectory and $\varepsilon$ is a penalty for large controls.

The dynamic programming principle yields a Hamilton-Jacobi-Bellman (HJB) PDE: $$-\partial_t V - \frac{1}{2}\Delta V + \frac{1}{2\varepsilon} |\nabla_x V|^2 = 0, \quad V(1, x) = G(x)$$ which, via the Cole-Hopf transform and Feynman-Kac formula, admits an efficient probabilistic representation and tractable Monte Carlo-based algorithms. As $\varepsilon \downarrow 0$, the stochastic control value function approaches the global minimum of $G$ with explicit rate $O(\ln(1/\varepsilon))$ (Qiu, 3 Jan 2026).

Extensions to the Wasserstein space of probability measures and mean-field optimal control introduce Lions-differentiability and master HJB equations, enabling population-based search strategies and interaction models.

4. Surrogate Modeling, Adaptive Exploration–Exploitation, and Policy Structures

Stochastic control frameworks exploit surrogate models (e.g., Stochastic Kriging, occupation measures) that integrate information about function values, local gradients, and uncertainty. Adaptive infill criteria (such as Augmented Expected Improvement) drive selection of new evaluation points by balancing the predicted benefit against the uncertainty and cost of sampling.

Algorithmic realization (e.g., adaptive sEGO) typically involves an EGO loop:

1. Fit SK metamodel using current evaluations and their variances.
2. Maximize AEI over search space to select infill candidate $q$.
3. Adjust the MCI variance target at $q$ based on local sampling density.
4. Evaluate $J(q)$ using enough samples to reach $\sigma^2_\text{adapt}$.
5. Augment model and repeat (Carraro et al., 2019).

Policy-gradient frameworks for stochastic control provide alternative optimization flows with provable global convergence. In such methods, the control policy (e.g., parameterized as $u(t, x; \theta)$) is updated via continuous-time gradient descent on the cost-to-go function, and local optimality is assessed using HJB or FBSDE equations (Zhou et al., 2023).

5. Occupation-Measure, Semidefinite, and Restart-Synchronous Approaches

Global optimization via stochastic control also includes occupation-measure and moment-SDP relaxations. Here, the controlled stochastic process is encoded by its occupation measure on space-time and control space; the global optimum of the original control problem is lower-bounded by the solution of a sequence of SDPs (Moment-Sum-of-Squares hierarchy) (Holtorf et al., 2022).

A space–time partition into local occupation measures enables fine-grained, scalable, and monotonic SDP relaxations where accuracy can be improved by either increasing the polynomial degree or refining the partition. Each SDP provides a certified global lower bound, and empirical studies demonstrate improved efficiency and accuracy compared to global high-degree SDPs.

Multi-start stochastic record-value frameworks (e.g., DMSS/RDMSS) use probabilistic stopping rules and record-process statistics to control restarts, offering convergence guarantees even when only black-box access is available. These frameworks optimize the trade-off between local search progress (exploitation) and the need to escape to new regions (exploration), guided by explicit estimates of the improvement rate and probability of missing the global minimum (Rele et al., 2024).

6. Empirical Performance and Computational Characteristics

Empirical studies demonstrate substantial efficiency gains for adaptive stochastic control frameworks relative to fixed-variance, purely exploitative, or uninformed global search:

• Adaptive sEGO consistently reduces the number of expensive MCI calls by 20–50% and improves both solution quality and robustness compared to fixed-target or multi-start schemes.
• In high-dimensional settings (e.g., 6D–10D, or with >50 random variables), adaptive variance control maintains solution quality and reduces budget exhaustion as compared to non-adaptive approaches (Carraro et al., 2019).
• Space–time partitioned local occupation-measure SDPs approach global optimality gaps of 1–2% in substantially less time than global SDPs, enabling tractability for large, semialgebraically-constrained problems (Holtorf et al., 2022).

Performance gains arise from real-time adjustment of the variance, sampling effort, or restart timing, as dictated by the local geometry of the objective and the distribution of acquired information.

7. Limitations and Practical Considerations

While stochastic control frameworks for global optimization offer significant theoretical and empirical advantages, several limitations persist:

• The complexity of stochastic kriging or occupation-measure models may grow rapidly with the dimension of the design space, leading to increased inference and optimization time.
• The tuning of adaptive parameters (e.g., clustering radius, decay rates, partitioning schemes) can significantly impact practical performance and may require problem-specific calibration.
• In settings with very high design dimension, the efficiency of surrogate-based approaches (such as SK in sEGO) may degrade, necessitating alternative or hybrid strategies.

A plausible implication is that future work may focus on scalable surrogates, distributed implementation of occupation-measure relaxations, and principled parameter adaptation schemes to further expand the tractability of these methods for large-scale and highly uncertain global optimization problems.

---

References:

(Carraro et al., 2019) Monte Carlo Integration with adaptive variance selection for improved stochastic Efficient Global Optimization (Qiu, 3 Jan 2026) Stochastic Control Methods for Optimization (Holtorf et al., 2022) Stochastic Optimal Control via Local Occupation Measures (Zhou et al., 2023) A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guarantee (Rele et al., 2024) A Stochastic Record-Value Approach to Global Simulation Optimization