Gibbs Expected Information Gain

• Gibbs Expected Information Gain is a robust criterion for optimal experimental design that replaces traditional likelihood functions with flexible loss functions.
• It quantifies information gain via the expected Kullback–Leibler divergence between the prior and a Gibbs posterior, enabling better handling of model misspecification.
• Computational strategies such as nested Monte Carlo and Laplace approximations support its practical application, though high-dimensional designs pose challenges.

Gibbs Expected Information Gain (Gibbs EIG) is a criterion for optimal experimental design rooted in generalised Bayesian (Gibbs) inference, which replaces the conventional likelihood function with a user-specified loss. By evaluating the expected Kullback–Leibler (KL) divergence between the prior and the Gibbs posterior, Gibbs EIG provides a robust metric for design selection that is less sensitive to model misspecification than classical Bayesian approaches. The framework accommodates arbitrary loss functions and relaxes the requirement for a fully specified statistical model of the data-generating process.

1. Foundations of Gibbs Inference and Gibbs EIG

Gibbs inference considers a parameter $\theta \in \Theta$, a prior density $\pi(\theta)$, and a loss function $\ell(\theta; y, d)$, where $y$ are data under design $d$. The unnormalised Gibbs posterior is

$$\pi_G(\theta|y,d) \propto \exp[-w \ell(\theta; y, d)] \pi(\theta)$$

with calibration (or “temperature”) parameter $w > 0$ controlling loss influence. In the special case where $\ell$ is the self-information loss $\ell_{SI}(\theta; y, d) = -\log p(y|\theta, d)$ and $w = 1$, the Gibbs posterior recovers the standard Bayesian posterior.

Given this generalisation, Gibbs EIG is defined as the pseudo-expectation of the KL divergence from the prior to the Gibbs posterior: $$U(d) = \mathbb{E}_{y|d}[\operatorname{KL}(\pi_G(\cdot|y,d) \| \pi(\cdot))]$$ Explicitly,

$$U(d) = \int p_G(y|d) \left[ \int \pi_G(\theta|y,d) \log \frac{\pi_G(\theta|y,d)}{\pi(\theta)} d\theta \right] dy$$

where $$p_G(y|d) = \int \exp[-w\ell(\theta; y, d)] \pi(\theta) d\theta$$ is the marginal generalised likelihood.

2. Information-Theoretic Framework and Formal Properties

The information-theoretic formulation of Gibbs EIG generalises Lindley's classical expected information gain: $$\operatorname{EIG}_{\text{Bayes}}(\xi) = \mathbb{E}_{y \sim p(y|\xi)}[\operatorname{KL}(p(\theta|y,\xi)\|\pi(\theta))]$$ By contrast, Gibbs EIG replaces the likelihood-based posterior $p(\theta|y,\xi)$ and marginal $p(y|\xi)$ with their Gibbs-generalised counterparts, introducing the notion of “pseudo-joint” and pseudo-random variables: $$\pi(\theta|y,\xi) = \frac{\exp[-\omega \ell_\theta(\xi, y)] \pi(\theta)}{Z(\xi, y)}, \quad Z(\xi, y) = \int \exp[-\omega \ell_\theta(\xi, y)] \pi(\theta) d\theta$$

$$\tilde{\pi}(y|\xi) = \int \exp[-\omega \ell_\theta(\xi, y)] \pi(\theta) d\theta$$

Pseudo-expectation is defined relative to $\tilde{\pi}(y|\xi)$. Gibbs EIG is the pseudo-mutual information between $\theta$ and the pseudo-random variable $\tilde{Y}|\xi$: $$\operatorname{EIG}_{\text{Gibbs}}(\xi) = \tilde{\mathbb{E}}_{\pi(\theta, y|\xi)}[\log\{\pi(\theta, y|\xi)/[\pi(\theta)\tilde{\pi}(y|\xi)]\}] = \tilde{\mathbb{E}}_{\tilde{\pi}(y|\xi)}[\operatorname{KL}(\pi(\theta|y,\xi)\|\pi(\theta))]$$

3. Distinction From Classical Bayesian EIG and Robustness Characteristics

Classic EIG depends entirely on the likelihood $p(y|\theta,\xi)$ for both design selection and posterior update. This dependence presents vulnerability to model misspecification, leading to unreliable information gain landscapes and suboptimal design concentration in regions poorly reflecting the target phenomenon.

Gibbs EIG, by employing an arbitrary loss $\ell_\theta(\xi, y)$, grants two robustness properties:

• Inference robustness: The posterior down-weights outlier or model-incongruent observations through the loss.
• Design robustness: The acquisition function computes pseudo-mutual information using the same loss, explicitly mitigating overcommitment to a possibly flawed model.

Empirical results—such as source localization in 2D with outlier contamination—exhibit that classical EIG leads to pathological clustering in uninformative regions, whereas Gibbs EIG distributes queries across the search space and achieves more effective parameter targeting.

4. Computational Strategies for Gibbs EIG Estimation

The estimation of Gibbs EIG is typically accomplished via a nested Monte Carlo (NMC) approach. The estimator structure parallels that used for classical EIG, but each step replaces the likelihood with evaluations of the generalised loss. The procedure for a candidate design $\xi$ involves:

• Drawing prior samples $\theta_i \sim \pi(\theta)$ and data $y_i \sim p(y|\theta_i, \xi)$.
• Estimating the marginal generalised likelihood for each $y_i$ via inner Monte Carlo over $\theta$:

$$w_{ij} = \exp[-\omega \ell_{\theta_{ij}}(\xi, y_i)],\quad \widehat{\tilde{\pi}}(y_i|\xi) = \frac{1}{M_\text{inner}}\sum_j w_{ij}$$

• Computing the importance weight:

$$Z_i = \frac{\exp[-\omega \ell_{\theta_i}(\xi, y_i)]}{p(y_i|\theta_i, \xi)}$$

with self-normalisation so $\sum_i Z_i = 1$.

• Evaluating utility contributions:

$$u_i = \left[-\omega \ell_{\theta_i}(\xi, y_i) - \log \widehat{\tilde{\pi}}(y_i|\xi)\right] \cdot Z_i$$

• Aggregating:

$\widehat{U}_\text{Gibbs}(\xi) = \sum_i u_i$

Design optimisation follows by grid search for discrete $\xi$, Bayesian optimisation for continuous domains, or gradient-based strategies if $\ell$ is differentiable. In practice, computational expense is dominated by the nested sampling ($O(NM)$ per design).

5. Closed-Form and Approximate Solutions in Standard Models

In models where loss and prior yield tractable structure, closed-form or efficient approximations for Gibbs EIG are available:

• Linear regression with squared-error loss: For $\ell(\beta; y, d) = \|y - X\beta\|^2$ under uniform prior and $w = 1/(2\sigma^2)$, the Gibbs posterior is Gaussian, and utilities recover modified $A$- and $D$-optimality criteria:
  • Negative-squared-error utility: $$U_{NSE}(d) = -\sigma^2 \operatorname{tr}[(X^TX)^{-1}]$$
  • Shannon information utility: $U_{SH}(d) = \frac{1}{2}\log|X^TX| - (p/2)$ [constant in $n-\operatorname{rank}(X)$]
• Count data with quasi-Poisson loss: Designer negative-binomial/Poisson GLM models can be handled by normal (Laplace) approximations to the Gibbs posterior and Monte Carlo averaging.
• General case: When closed-form expressions are unattainable, the normal approximation around the Gibbs posterior mode enables Laplace-type estimators for utilities.

The Approximate Coordinate Exchange (ACE) algorithm is recommended for high-dimensional design optimisation, iteratively fitting a Gaussian-process emulator along coordinates to efficiently converge.

6. Practical Implications and Illustrative Applications

Empirical studies demonstrate the practical efficacy of Gibbs EIG in several settings:

• Linear regression with heavy-tailed outliers: Standard BOED/EIG results in repeated querying at extremes and inferior posterior RMSE/MMD/NLL, while Gibbs EIG (using weighted score-matching loss) distributes points more broadly and reduces RMSE and NLL by 2–4$\times$.
• Pharmacokinetics paper: In the presence of noise misspecification (e.g., Student–$t$ versus Gaussian), classic BOED is misled in sampling time selection, but GBOED achieves lower predictive error.
• 2D location-finding with outliers: Classic BOED clusters at uninformative hot-spots; GBOED using robust score matching spreads queries and identifies true signal sources reliably.

Applications extend to any resource-constrained sequential data collection where model misspecification or adversarial contamination is relevant (biology, imaging, psychometrics, sensor placement), as well as to settings with intractable or heavy-tailed likelihoods.

7. Limitations, Open Problems, and Outlook

Principal limitations of Gibbs EIG include:

• Computational intensity: Nested Monte Carlo estimation incurs substantial cost, particularly for large budgets or high-dimensional designs. Alternative estimators (variational, low-variance methods) may mitigate this burden.
• Tuning requirements: Selection of the loss function $\ell$ and temperature parameter $w$ (or $\omega$) is context-dependent; no universally optimal specification currently exists.
• Design optimisation challenges: For high-dimensional $\xi$, myopic (single-step-ahead) optimisation may be suboptimal. Non-myopic or amortised policy approaches are potential directions.
• Variance of importance weights: If the proposal $p(y|\theta,\xi)$ is a poor fit to the generalised likelihood, weights may have high variance and reduce estimator stability.

A plausible implication is that, while the Gibbs framework enhances robustness and flexibility in experimental design, practical implementation in large-scale or complex domains may require methodological innovations in estimation and design search.

---

In summary, Gibbs Expected Information Gain extends the classical information-theoretic paradigm of Bayesian experimental design to generalised Bayesian (Gibbs) inference by substituting likelihood-based updates with loss-based updates in both posterior and acquisition function. This results in robust design selection and inference mechanisms that are less sensitive to model misspecification, with broad applicability in modern data-collection scenarios and established synergy with both standard and robust statistical modeling approaches (Overstall et al., 2023, Barlas et al., 10 Nov 2025).