---
title: Bayesian Optimization with Preferential Feedback
url: https://www.emergentmind.com/topics/bayesian-optimization-with-preferential-feedback
type: topic
---

# Bayesian Optimization with Preferential Feedback

Searching arXiv for recent and foundational papers on preferential Bayesian optimization.
Bayesian optimization with preferential feedback is the branch of Bayesian optimization in which the latent objective cannot be queried as a scalar and is instead accessed through comparative judgments such as pairwise preferences, rankings, or winner-only feedback. In its canonical form, an unknown utility \(f:\mathcal X\to\mathbb R\) is optimized by repeatedly querying two candidates \(x,x'\in\mathcal X\) and observing a binary outcome \(y\in\{0,1\}\) with \(P[y=1\mid x,x']=\sigma(f(x)-f(x'))\), where \(\sigma\) is a known strictly increasing link such as the logistic or probit function [2604.25025]. This setting extends Bayesian optimization to applications in which expert judgment, human perception, laboratory comparison, or tacit design criteria are available while reliable scalar labels are not [1704.03651].

## 1. Formal problem setting and feedback models

The standard formulation assumes a compact search domain \(\mathcal X\subset\mathbb R^d\) and a latent utility function \(f\). Some works pose the task as maximization of \(f\), while others use a latent cost and seek \(\theta^*=\arg\min_{\theta\in\Theta} f(\theta)\); the two conventions differ only by sign [2512.01754]. In the pairwise setting, a duel \((x,x')\) returns the preference of one point over the other, and the latent comparison function is the utility difference
\[
h(x,x')=f(x)-f(x').
\]
The most common likelihoods are the logistic Bradley–Terry–Luce model and the Thurstone–Mosteller probit model [1704.03651].

Several objective notions coexist. One line of work measures performance by cumulative regret under preferential queries, for example
\[
R(T)=\sum_{t=1}^{T}\bigl[P[x_*\succ x_t]+P[x_*\succ x'_t]-1\bigr],
\]
where \(x_*\in\arg\max_x f(x)\) [2604.25025]. Another writes the pairwise regret as
\[
r_t := [P(x^*\succ x_t)+P(x^*\succ x'_t)-1]/2,\qquad R(T)=\sum_{t=1}^T r_t,
\]
and shows that this is equivalent, up to constants, to the usual utility regret [2505.23673]. Earlier preferential BO papers also used a Copeland-style objective: with soft-Copeland score
\[
C(x)=\frac{1}{\mathrm{Vol}(\mathcal X)}\int_{x'\in\mathcal X}\pi_f([x,x'])\,dx',
\]
the terminal recommendation is the Condorcet winner \(x_c=\arg\max_x C(x)\) under the final posterior [1704.03651].

Preferential feedback is not limited to binary duels. Batch and setwise variants ask for the best element of a set of size \(q\), inducing multinomial or winner-only likelihoods [2003.11435, 2303.15746]. Projective preferential queries ask for the minimizer along a one-dimensional projection \(P_{\boldsymbol\xi}(\mathbf x)\), effectively encoding infinitely many pairwise preferences along a line [2002.03113]. Consecutive preferential BO introduces a three-outcome model \(R_t\in\{+1,0,-1\}\) with a Just-Noticeable Difference threshold \(\delta>0\), so that small utility differences yield explicit indifference [2511.05163]. Contextual preferential BO conditions the latent utility on a context variable \(c\), using comparisons of \((x_t,c_t)\) against \((x_{t-1},c_{t-1})\) rather than static alternatives [2512.09481].

## 2. Gaussian-process surrogates and posterior inference

The dominant modeling approach places a Gaussian-process prior either on the latent utility \(f\) itself or directly on a duel function. In the original preferential BO formulation, a zero-mean GP prior is imposed on the duel reward \(f([x,x'])\), with Bernoulli likelihood and approximate GP-classification inference [1704.03651]. A more structured construction starts from a base kernel \(k\) on \(\mathcal X\), assumes \(f\in H_k\) with \(\|f\|_{H_k}\le B\), and then observes that the comparison function \(h(x,x')=f(x)-f(x')\) lies in the RKHS induced by the dueling kernel
\[
k_{\mathrm{duel}}\bigl((x,x'),(u,u')\bigr)=k(x,u)+k(x',u')-k(x,u')-k(x',u)
\]
[2604.25025]. This construction explicitly propagates information across pairs and underlies several recent regret analyses.

Because preference likelihoods are non-Gaussian, exact posterior inference is usually intractable. Laplace approximation and expectation propagation were already standard in early PBO and remain common in later variants [1704.03651, 2512.01754]. Preferential batch BO combines GP priors with EP or variational inference for batch-comparison likelihoods [2003.11435]. Contextual PBO derives a Laplace-approximate Gaussian posterior over stacked latent utility values \(f\in\mathbb R^{2t}\) [2512.09481]. CPBO follows the same non-Gaussian GP-preference template for the objective while maintaining a separate GP for inequality constraints [2505.10954].

An important refinement concerns the geometry of the true posterior. Under probit preference models, the exact posterior is a unified skew-normal or skew-Gaussian distribution rather than a Gaussian one. A detailed comparison of approximations reports that Gaussian approximations can yield inaccurate predictive duel probabilities, motivating improved MCMC-based estimation via Gibbs sampling and low-variance Monte Carlo estimators [2302.01513]. This observation is significant because many acquisition rules depend not only on marginal posterior means and variances, but on the quality of predicted win probabilities between candidates.

Beyond standard GP surrogates, some extensions alter the latent representation itself. BO with abstract properties augments the BO input with latent property predictors \(\hat\omega_i(x)\) learned from expert pairwise preferences, then selects between a control arm and a human-augmented arm by predictive likelihood [2402.17343]. Asynchronous expert-feedback BO models the latent preference function \(r(x)\) with a neural network \(r_\theta(x)\), uses a Bradley–Terry likelihood, and forms a Gaussian Laplace approximation in parameter space for non-blocking integration with a standard BO loop [2406.06459]. PABBO replaces per-iteration GP inference entirely by a meta-learned transformer neural process that amortizes both the surrogate and the acquisition rule [2503.00924].

## 3. Acquisition functions and sequential decision rules

Acquisition design in preferential BO differs from scalar-feedback BO because the algorithm must choose comparisons rather than single points. The original framework introduced pure exploration, Copeland Expected Improvement, and Dueling Thompson Sampling. In Dueling Thompson Sampling, a posterior function sample induces a soft-Copeland score used to select one point, and posterior uncertainty then selects a challenger [1704.03651]. This established the now-standard two-stage structure of incumbent selection plus challenger selection.

A later decision-theoretic line replaces Copeland-based surrogates with the expected utility of the best option. For a query \(X=(x_1,\dots,x_q)\), qEUBO is
\[
qEUBO_n(X)=E_n\bigl[\max_i f(x_i)\bigr],
\]
optimized by Monte Carlo sample-average approximation [2303.15746]. In the pairwise special case, EUBO admits a closed form under a Gaussian posterior over \(f(x^{(i)})-f(x^{(j)})\), and this closed form becomes the basis of the constrained acquisition EUBOC when multiplied by a feasibility probability [2505.10954].

Thompson-style policies have also been adapted to the duel-kernel setting. Preferential-Feedback Thompson Sampling maintains a GP surrogate over \(h(\cdot,\cdot)\), draws two independent posterior samples \(h^{(1)},h^{(2)}\sim GP(\mu_t,v_t^2k_t)\), anchors both at a fixed \(x_0\), and sets
\[
x_t=\arg\max_x h^{(1)}(x,x_0),\qquad x'_t=\arg\max_x h^{(2)}(x,x_0).
\]
Because of anchor invariance, the choice of \(x_0\) does not affect the outcome [2604.25025]. This yields a particularly symmetric preferential analogue of standard GP-TS.

Other acquisition families are tailored to specific constraints or interfaces. POP-BO constructs a likelihood-ratio confidence set \(\mathcal B_f^t\) and applies optimism:
\[
x_t\in\arg\max_{x\in\mathcal X}\max_{f\in\mathcal B_f^t}(f(x)-f(x'_t)),
\]
with \(x'_t\) typically set to the previous point [2402.05367]. Projective preferential BO defines expected improvement over the maximum value attained along a projection and also introduces preferential coordinate descent as a restricted-query alternative [2002.03113]. Consecutive PBO adapts a mutual-information criterion to the case in which each new comparison must involve a previously produced candidate and the human may return indifference [2511.05163]. Contextual PBO conditions expected improvement on the observed context \(c_t\) and uses the previous day’s utility as the reference level [2512.09481].

These acquisition developments also clarify a recurrent misconception: preferential BO is not a single policy, but a family of sequential design rules whose behavior depends strongly on how the query set is parameterized, how posterior uncertainty is propagated through a preference model, and whether the method seeks information about the global maximizer, the local incumbent, or the next most informative challenger.

## 4. Regret theory and decision-theoretic guarantees

Theoretical analysis of preferential BO has recently moved from heuristic empirical comparison to finite-time regret guarantees. POP-BO derives confidence sets from likelihood-ratio arguments and obtains an information-theoretic cumulative-regret bound
\[
R_T = O\!\Bigl(\sqrt{\beta_T\,T\,\gamma_T^{ff'}}\Bigr),
\]
described as a first-of-its-kind for preferential BO, together with a reporting rule \(\widehat x\) satisfying a high-probability convergence rate [2402.05367]. The technical novelty lies in converting preference-only observations into duel-wise confidence bands of the form
\[
\bigl|(\hat f(x)-\hat f(x'))-(f(x)-f(x'))\bigr|\le C\,\sigma_t^{ff'}(x,x').
\]

A second line, BOHF under the Bradley–Terry–Luce model, establishes near-optimal regret bounds of order
\[
\tilde{\mathcal O}\bigl(\sqrt{\Gamma(T)T}\bigr),
\]
where \(\Gamma(T)\) is the maximum information gain of the dueling kernel [2505.23673]. The same work states that, for common kernels, the order-optimal sample complexities of conventional BO are recovered, so that the same number of preferential samples as scalar-valued samples is sufficient to find a nearly optimal solution. This challenges the view that reduced feedback must automatically imply worse order-wise sample complexity.

Preferential-Feedback Thompson Sampling sharpens this picture in a TS-specific setting. Under \(\|f\|_{H_k}\le B\) and with an exploration scale \(v_t\) on the order of the confidence-width multiplier \(B_t(\delta)=O(B+\sqrt{\gamma_t+\ln(1/\delta)})\), PF-TS satisfies
\[
R(T)=O\Bigl(B_T(\delta)\sqrt{T\,\gamma_T\,\ln T}\Bigr)=\widetilde O(\sqrt{T\,\gamma_T}),
\]
matching the finite-time rate of standard TS for scalar-feedback BO [2604.25025]. The proof exploits a separability fact: any GP sample under the duel kernel can be written almost surely as \(h(x,x')=f_h(x)-f_h(x')\), so maximizing \(h(\cdot,x_0)\) is equivalent to maximizing a sampled scalar function.

Decision-theoretic analysis has produced complementary guarantees. qEUBO is one-step Bayes optimal in the noise-free setting and thus equivalent to knowledge gradient; under noisy responses it enjoys an additive constant approximation guarantee to the one-step Bayes-optimal policy [2303.15746]. The same paper proves that Bayesian simple regret under qEUBO converges to zero at rate \(o(1/n)\), while qEI adapted from standard BO can fail to converge to zero simple regret. This result is important because qEI is frequently used as a convenient baseline in preferential settings even though its asymptotic behavior can be qualitatively wrong.

## 5. Structured and generalized preferential BO

Preferential BO has expanded well beyond the basic duel model. Preferential batch BO allows full rankings or batch-winner feedback for a set of \(q\) points and introduces batch analogues of expected improvement and Thompson sampling [2003.11435]. Projective Preferential Bayesian Optimization replaces pairwise duels by line minimizers \(Q(\boldsymbol\xi,\mathbf x)=\alpha^*\), arguing that one projective query is informationally much richer than a single duel and reporting convergence on Hartmann (6D), Levy (10D), and Ackley (20D) where pairwise PBO variants stagnated [2002.03113]. MultiBO generalizes pairwise input to \(K\)-choice image preference queries with a multinomial-logit GP and a Dynamic Balanced Subspace acquisition around a prompt-derived anchor image [2602.02388].

Several variants address operational constraints. Consecutive PBO models production cost \(c_p\), evaluation cost \(c_e\), and a Just-Noticeable Difference threshold \(\delta\), selecting new configurations under a mutual-information objective when only consecutive comparisons are feasible [2511.05163]. CPBO adds inequality constraints \(g_k(x)\ge 0\) through a separate GP and proposes EUBOC, which multiplies EUBO by the probability that both candidates satisfy the constraint [2505.10954]. Contextual PBO introduces a GP prior on \(U(x,c)\) to optimize time-varying utilities under exogenous context \(c\), using pairwise comparisons across days rather than within-day static duels [2512.09481]. BO with abstract properties augments the design input with latent preference-based property estimates, while asynchronous expert-feedback BO uses a separate preference-learning thread so that the BO loop never blocks on expert responses [2402.17343, 2406.06459].

Scalability has motivated two additional directions. PABBO fully amortizes preferential BO by meta-learning both the surrogate and the acquisition function with a transformer neural process trained by reinforcement learning, reporting several orders of magnitude lower latency than GP-based strategies [2503.00924]. Local Preferential Bayesian Optimization transfers trust-region and derivative-informed local search to the preference setting; TuRPBO adapts TuRBO-style region updates, while GIPBO and PrefSQP use first- and second-order derivatives of the Laplace-approximated GP posterior to form local ascent directions [2606.02351].

The same logic has also been exported to RLHF. Bayesian preference inference with Laplace posterior approximation over reward-model parameters can be embedded into the RLHF loop, where a Dueling Thompson Sampling–inspired acquisition chooses informative preference queries before PPO updates the policy [2511.04286]. This suggests that preferential BO and RLHF are not disjoint topics but partially overlapping formulations of active preference elicitation.

## 6. Applications and empirical evidence

Reported applications span human-, laboratory-, and expert-in-the-loop design, scientific discovery, recommender-style tasks, and interactive creative systems [2604.25025, 1704.03651]. The empirical picture is heterogeneous because tasks differ in dimensionality, feedback modality, and whether scalar observations are available for comparison, but several studies provide concrete demonstrations.

| Domain | Setting | Reported outcome |
|---|---|---|
| Catalyst design | PF-TS on the OCx24 dataset | Lowest instantaneous regret among BOHF methods; outperforms vanilla GP-TS for \(\phi\ge 3\) on a fixed budget [2604.25025] |
| Robotic commissioning | Human expert provides binary duels on pushing behavior | PBO can tune the controller to the expert’s satisfaction purely from duels, needing \(\approx 15\) comparisons; \(J_{PBO}\) matched \(\approx 90\%\) of duels vs. \(\approx 70\%\) for the expert cost \(J\) [2512.01754] |
| Building climate control | Contextual PBO on BOPTEST | Improvement of up to \(23\%\) in utility over a two-month simulation period [2512.09481] |
| High-moisture extrusion | Consecutive comparisons with indifference | Three human operators converged to high-quality recipes in \(25\)–\(30\) iterations [2511.05163] |
| Personalized image generation | Multi-choice human feedback with \(K\) candidates | Human study reported \(70.8\%\) win-rate and MOS \(\approx 3.58\) [2602.02388] |

Synthetic and benchmark studies reinforce these application results. PF-TS achieves lower cumulative regret than POP-BO and MR-LPF on a one-dimensional Ackley function, is competitive with MaxMinLCB, and shows faster decline in regret on LCBench/MNIST hyperparameter tuning up to \(T=200\) [2604.25025]. qEUBO outperforms qEI, qNEI, qTS, MPES, and random across 6-d Ackley, 7-d Alpine1, 6-d Hartmann, car-cab design, sushi preference, and particle animation [2303.15746]. Local PBO reports especially strong results in high-dimensional and steep landscapes, including MuJoCo policy-search tasks [2606.02351]. PCPBO uses a bi-level combination of preferential BO and physics-based simulation to estimate physically feasible food arrangements under qualitative domain rules [2209.10602]. Interactive melody composition combines preference GP modeling with a reduced MusicVAE latent space to guide users toward desired melodies using only best-of-set choices [2010.03190].

A recurrent empirical theme is that preference-based surrogates can capture criteria that domain experts struggle to formalize as a scalar objective. In the robotic commissioning study, the learned posterior mean \(J_{PBO}(\theta):=\mu(\theta)\) agreed with the human’s binary choices more often than a hand-designed cost function and then accelerated standard BO when reused as a scalar surrogate [2512.01754]. A plausible implication is that preferential BO is often valuable not only as an optimizer, but also as a mechanism for distilling tacit objectives into reusable latent utility models.

## 7. Practical limitations and research directions

Despite its breadth, preferential BO remains constrained by inference cost, optimization difficulty, and human factors. Several papers note cubic scaling in the number of comparisons: exact or approximate posterior logistic-regression inference can be costly in \(O(t^3)\) per round, continuous domains require discretization or global acquisition optimization, and GP-based PBO scales poorly to dimensions above \(10\) in some settings [2604.25025, 2512.01754]. Gaussian approximation can misestimate duel probabilities, while exact skew-GP inference is more accurate but slower [2302.01513]. Human experts also face cognitive burden: repeated duels can be tiring, fixed acquisition rules may stick to the boundary, and binary comparisons discard information that could be captured by rankings, ratings, or indifference labels [2512.01754, 2511.05163].

Current research addresses these limitations along several axes. Cost-aware methods mix production and evaluation costs [2511.05163]. Structured feedback models incorporate one-out-of-many queries, top-\(k\) feedback, contextual covariates, or active anchor selection [2604.25025]. High-dimensional methods localize the search or amortize inference [2606.02351, 2503.00924]. Constraint-aware variants search feasible regions directly [2505.10954]. Asynchronous schemes support non-blocking expert interaction in self-driving labs [2406.06459]. RLHF-style formulations import preferential acquisition into large-scale preference learning [2511.04286].

The field therefore no longer consists only of “GP classification on duels.” It now includes duel-kernel formulations with finite-time regret guarantees, decision-theoretic acquisitions with Bayes-optimality results, structured-query models that go beyond pairwise feedback, and system-level adaptations for robotics, scientific design, creative interaction, and policy learning. Taken together, these developments frame Bayesian optimization with preferential feedback as a general methodology for optimizing latent utilities when judgments are comparative, scalar objectives are unavailable or inadequate, and the query interface itself is part of the design problem.

Source: https://www.emergentmind.com/topics/bayesian-optimization-with-preferential-feedback