Efficient Bayesian Optimal Experimental Design for Expensive Computational Models over Finite Design Sets

Abstract: Bayesian calibration is a powerful framework for identifying parameters in complex and large-scale computational models. However, when there is insufficient or poorly suited data for the calibration process, significant uncertainty about the identified parameters remains. This uncertainty can hinder effective decision-making and understanding of the system. In many applications, the experiment that generates the data can be influenced by design variables, such as sensor placements, loading conditions, or test configurations. Often, however, these design variables can not be chosen arbitrarily but only from a finite set of possible experimental designs. We propose an adaptive algorithm for Bayesian optimal experimental design over a finite design set, specifically tailored for applications involving expensive computational models. The method integrates an accelerated nested Monte Carlo estimator that reuses parameter samples to reduce model evaluations. Additionally, it employs common random numbers and Rao--Blackwellization to reduce variance in pairwise expected information gain comparisons. To support decision-making within the algorithm, we estimate the probability that one design outperforms another using bootstrap sampling. Starting with small sample sizes, the algorithm iteratively eliminates inferior designs based on these probabilistic comparisons, allocating additional computational effort only to promising candidates until a single design remains. The resulting approach achieves high reliability at substantially reduced computational cost, making it well-suited for large-scale engineering applications.

• The paper introduces Leave-out Reused Nested Monte Carlo, Rao–Blackwellized estimators, common random numbers, and adaptive design elimination to rank finite experimental designs with fewer costly model evaluations.
• The paper reports up to two orders of magnitude fewer forward-model evaluations while maintaining near-perfect selection accuracy across synthetic, pharmacokinetic, lung, and viscoplastic material examples.
• The paper shows that favorable bias direction can improve design ranking, although its guarantees remain empirical and the method assumes a finite design set with cheaply sampled observation noise.

Problem setting and motivation

Bayesian optimal experimental design (BOED) seeks the experimental design $\xi$ that maximizes the expected information gain (EIG), i.e., the expected entropy reduction from prior to posterior over uncertain model parameters. For expensive computational models—finite element or multiphysics simulations where each forward evaluation is costly—the EIG is doubly intractable: it requires a high-dimensional integral over both parameters and observations, with an embedded evidence term $p(y \mid \xi)$ that itself demands many likelihood evaluations. Existing remedies each carry drawbacks: Laplace-based estimators are biased under local Gaussian assumptions and require Hessians unavailable for legacy black-box codes; nested Monte Carlo (NMC) is consistent but converges slowly, with mean squared error rates of $\mathcal{O}(C^{-2/3})$ at cost $C = NM$; multilevel debiasing schemes achieve $\mathcal{O}(C^{-1/2})$ but add complexity; and gradient-based optimization requires model derivatives with respect to design variables.

The paper targets a setting in which the design space $\Xi$ is finite and unordered—a fixed library of sensor locations, ventilation protocols, or loading configurations—so optimization reduces to identifying the most informative element of a small candidate set. The authors argue, correctly, that this reframing changes the computational objective: one does not need accurate absolute EIG estimates, only reliable identification of the sign of pairwise EIG differences.

Estimator design: reuse, leave-out, and Rao–Blackwellization

The methodological core is a family of accelerated NMC estimators. Starting from standard NMC, the authors adopt sample reuse (RNMC), in which the outer-loop parameter samples $\Theta$ are reused to approximate the evidence, reducing model evaluations by a factor of $M$. Their key modification is the Leave-out RNMC (LRNMC): the current outer-loop sample $\theta_i$ is excluded from the inner evidence estimate. This matters because including $\theta_i$ evaluates the likelihood at the very parameter that generated $p(y \mid \xi)$0, which on average inflates the inner sum and introduces a large negative bias term beyond the positive bias of standard NMC. Excluding it restores the bias profile of plain NMC while retaining the factor-of-$p(y \mid \xi)$1 cost reduction.

A second layer exploits the assumed observation model $p(y \mid \xi)$2: for fixed $p(y \mid \xi)$3, the only randomness in the pointwise mutual information comes from the noise $p(y \mid \xi)$4, which can be sampled without re-evaluating the model. Averaging over $p(y \mid \xi)$5 noise draws yields a Rao–Blackwellized PMI estimator (ANMC) whose variance decreases monotonically in $p(y \mid \xi)$6; the authors use $p(y \mid \xi)$7 throughout, observing diminishing returns beyond that.

The paper's most consequential empirical claim concerns bias directionality. Using a synthetic lognormal problem with $p(y \mid \xi)$8 replications, they show that ANMC's bias is strictly positive and increases superlinearly with true EIG, whereas RNMC's bias is strictly negative and decreases with EIG. Since the bias of ANMC/LRNMC/NMC amplifies differences between competing designs while RNMC compresses them, the "wrong" bias of RNMC actively hinders design selection even though all estimators are asymptotically consistent. This is a notable inversion of the usual view that lower bias is uniformly preferable: here a positively biased estimator is advantageous for ranking.

Adaptive design elimination

For selection, the authors contrast a static design choice (SDC) baseline—all designs evaluated at equal budget—with adaptive design elimination (ADE). ADE augments a shared parameter set by $p(y \mid \xi)$9 samples per iteration, computes Rao–Blackwellized PMI scores for all surviving designs using common random numbers (CRN), designates the current best as reference, and eliminates any competitor whose bootstrap-estimated probability of inferiority exceeds $\mathcal{O}(C^{-2/3})$0 in $\mathcal{O}(C^{-2/3})$1 consecutive iterations, with $\mathcal{O}(C^{-2/3})$2 bootstrap replicates. The procedure terminates when one design remains, automatically determining the required computational budget—an explicit answer to SDC's inability to know a priori how many evaluations suffice.

Variance control in the pairwise comparisons comes from three sources: CRN across designs, Rao–Blackwellization, and the paired-difference structure fed to the bootstrap, which avoids distributional assumptions about the difference statistic.

Numerical evidence

Four examples support the claims, evaluated over 1000 runs each:

Example	Dimension	Designs	Key result
Synthetic linear-Gaussian	100	10	Closed-form EIG; top-two gap small (8.485 vs. 8.082)
Pharmacokinetic model	3	15	Three sampling-schedule families
Human lung model	6	4	Reduced-dimensional ventilated lung
Viscoplastic material model	7	12	FEM, Student-$\mathcal{O}(C^{-2/3})$3 noise

In the synthetic case, moving from NMC to ANMC reduces the evaluations needed for 100% accuracy by roughly two orders of magnitude, and ADE with ANMC achieves another order-of-magnitude reduction over SDC with RNMC. ADE identifies the correct design in all runs for ANMC and LRNMC and in 99.8% of runs for RNMC—the latter deficit attributed directly to its unfavorable bias characteristics. The pharmacokinetic example reproduces the same ordering once CRN is enabled; notably, without CRN, RNMC slightly outperforms ANMC and LRNMC due to its lower variance, indicating that CRN is what makes the bias argument decisive. In the lung example, ADE attains 100% accuracy for all three accelerated estimators, though relative gains over SDC shrink because early elimination has less room with only four candidates. The viscoplastic example demonstrates applicability to non-Gaussian, design-dependent observation models and a genuine black-box FEM code (4C orchestrated via QUEENS); ADE with ANMC or LRNMC needs about 2.5× fewer evaluations than SDC with CRN, and about four times fewer than SDC with RNMC.

An interesting substantive finding emerges in the viscoplastic study: although higher displacement rates amplify viscoplastic effects per measurement, the longer observation horizon at low displacement rates dominates, so the slowest-rate design $\mathcal{O}(C^{-2/3})$4 attains the highest EIG (26.185 under ANMC versus 5.353 for the fastest). This contradicts the intuitive expectation that aggressive loading is more informative and illustrates the diagnostic value of EIG computation itself.

Limitations and open questions

Several caveats are stated or implicit. The favorable bias behavior of ANMC is established empirically on a single synthetic problem and argued heuristically via concavity of the logarithm and the Jensen gap; no general theorem guarantees that ANMC's bias amplifies EIG differences across arbitrary models, so the reliability of elimination decisions rests partly on this unproven regularity condition. The estimator remains biased for finite $\mathcal{O}(C^{-2/3})$5, and the bootstrap probability is computed from biased paired differences, meaning the nominal threshold $\mathcal{O}(C^{-2/3})$6 does not correspond to a calibrated frequentist guarantee on selection error. The lung and viscoplastic experiments use precomputed sample pools drawn without replacement within runs but shared across runs, so the reported run accuracies involve slight dependencies. The method also presumes the noise model is cheap to sample relative to the forward model and that the design set is genuinely finite; extension to continuous design spaces or goal-oriented utility functions (e.g., posterior information about derived quantities of interest rather than parameters themselves) is left open, as is treatment of nuisance uncertainty.

Conclusion

This paper contributes a practical BOED pipeline for finite design sets and expensive simulators, combining sample-reuse NMC with a leave-out correction, Rao–Blackwellization over cheap noise draws, common random numbers, and bootstrap-driven adaptive elimination. Its central empirical results—up to two orders of magnitude reduction in forward model evaluations at near-perfect selection accuracy across four problems, including a high-dimensional synthetic case and two engineering-scale applications—are supported by a clear account of why bias directionality, not merely variance, governs ranking reliability. The framework's dependence on empirically observed bias behavior, and its restriction to finite design spaces, define the boundaries within which these efficiency gains hold.