Bayesian Optimization with Preferential Feedback

• The paper introduces preferential Bayesian optimization, which optimizes latent utility functions using pairwise or setwise comparative feedback instead of scalar evaluations.
• It employs Gaussian-process surrogates and advanced posterior inference methods to propagate uncertainty and model non-Gaussian likelihoods effectively.
• The methodology finds practical application in areas such as robotics, climate control, and RLHF, while addressing challenges like scalability and cognitive load.

Searching arXiv for recent and foundational papers on preferential Bayesian optimization. Bayesian optimization with preferential feedback is the branch of Bayesian optimization in which the latent objective cannot be queried as a scalar and is instead accessed through comparative judgments such as pairwise preferences, rankings, or winner-only feedback. In its canonical form, an unknown utility $f:\mathcal X\to\mathbb R$ is optimized by repeatedly querying two candidates $x,x'\in\mathcal X$ and observing a binary outcome $y\in\{0,1\}$ with $P[y=1\mid x,x']=\sigma(f(x)-f(x'))$, where $\sigma$ is a known strictly increasing link such as the logistic or probit function (Lazzaro et al., 27 Apr 2026). This setting extends Bayesian optimization to applications in which expert judgment, human perception, laboratory comparison, or tacit design criteria are available while reliable scalar labels are not (Gonzalez et al., 2017).

1. Formal problem setting and feedback models

The standard formulation assumes a compact search domain $\mathcal X\subset\mathbb R^d$ and a latent utility function $f$. Some works pose the task as maximization of $f$, while others use a latent cost and seek $\theta^*=\arg\min_{\theta\in\Theta} f(\theta)$; the two conventions differ only by sign (Witte et al., 1 Dec 2025). In the pairwise setting, a duel $(x,x')$ returns the preference of one point over the other, and the latent comparison function is the utility difference

$x,x'\in\mathcal X$0

The most common likelihoods are the logistic Bradley–Terry–Luce model and the Thurstone–Mosteller probit model (Gonzalez et al., 2017).

Several objective notions coexist. One line of work measures performance by cumulative regret under preferential queries, for example

$x,x'\in\mathcal X$1

where $x,x'\in\mathcal X$2 (Lazzaro et al., 27 Apr 2026). Another writes the pairwise regret as

$x,x'\in\mathcal X$3

and shows that this is equivalent, up to constants, to the usual utility regret (Kayal et al., 29 May 2025). Earlier preferential BO papers also used a Copeland-style objective: with soft-Copeland score

$x,x'\in\mathcal X$4

the terminal recommendation is the Condorcet winner $x,x'\in\mathcal X$5 under the final posterior (Gonzalez et al., 2017).

Preferential feedback is not limited to binary duels. Batch and setwise variants ask for the best element of a set of size $x,x'\in\mathcal X$6, inducing multinomial or winner-only likelihoods (Siivola et al., 2020, Astudillo et al., 2023). Projective preferential queries ask for the minimizer along a one-dimensional projection $x,x'\in\mathcal X$7, effectively encoding infinitely many pairwise preferences along a line (Mikkola et al., 2020). Consecutive preferential BO introduces a three-outcome model $x,x'\in\mathcal X$8 with a Just-Noticeable Difference threshold $x,x'\in\mathcal X$9, so that small utility differences yield explicit indifference (Erarslan et al., 7 Nov 2025). Contextual preferential BO conditions the latent utility on a context variable $y\in\{0,1\}$0, using comparisons of $y\in\{0,1\}$1 against $y\in\{0,1\}$2 rather than static alternatives (Wang et al., 10 Dec 2025).

2. Gaussian-process surrogates and posterior inference

The dominant modeling approach places a Gaussian-process prior either on the latent utility $y\in\{0,1\}$3 itself or directly on a duel function. In the original preferential BO formulation, a zero-mean GP prior is imposed on the duel reward $y\in\{0,1\}$4, with Bernoulli likelihood and approximate GP-classification inference (Gonzalez et al., 2017). A more structured construction starts from a base kernel $y\in\{0,1\}$5 on $y\in\{0,1\}$6, assumes $y\in\{0,1\}$7 with $y\in\{0,1\}$8, and then observes that the comparison function $y\in\{0,1\}$9 lies in the RKHS induced by the dueling kernel

$P[y=1\mid x,x']=\sigma(f(x)-f(x'))$0

(Lazzaro et al., 27 Apr 2026). This construction explicitly propagates information across pairs and underlies several recent regret analyses.

Because preference likelihoods are non-Gaussian, exact posterior inference is usually intractable. Laplace approximation and expectation propagation were already standard in early PBO and remain common in later variants (Gonzalez et al., 2017, Witte et al., 1 Dec 2025). Preferential batch BO combines GP priors with EP or variational inference for batch-comparison likelihoods (Siivola et al., 2020). Contextual PBO derives a Laplace-approximate Gaussian posterior over stacked latent utility values $P[y=1\mid x,x']=\sigma(f(x)-f(x'))$1 (Wang et al., 10 Dec 2025). CPBO follows the same non-Gaussian GP-preference template for the objective while maintaining a separate GP for inequality constraints (Iwai et al., 16 May 2025).

An important refinement concerns the geometry of the true posterior. Under probit preference models, the exact posterior is a unified skew-normal or skew-Gaussian distribution rather than a Gaussian one. A detailed comparison of approximations reports that Gaussian approximations can yield inaccurate predictive duel probabilities, motivating improved MCMC-based estimation via Gibbs sampling and low-variance Monte Carlo estimators (Takeno et al., 2023). This observation is significant because many acquisition rules depend not only on marginal posterior means and variances, but on the quality of predicted win probabilities between candidates.

Beyond standard GP surrogates, some extensions alter the latent representation itself. BO with abstract properties augments the BO input with latent property predictors $P[y=1\mid x,x']=\sigma(f(x)-f(x'))$2 learned from expert pairwise preferences, then selects between a control arm and a human-augmented arm by predictive likelihood (V et al., 2024). Asynchronous expert-feedback BO models the latent preference function $P[y=1\mid x,x']=\sigma(f(x)-f(x'))$3 with a neural network $P[y=1\mid x,x']=\sigma(f(x)-f(x'))$4, uses a Bradley–Terry likelihood, and forms a Gaussian Laplace approximation in parameter space for non-blocking integration with a standard BO loop (Kristiadi et al., 2024). PABBO replaces per-iteration GP inference entirely by a meta-learned transformer neural process that amortizes both the surrogate and the acquisition rule (Zhang et al., 2 Mar 2025).

3. Acquisition functions and sequential decision rules

Acquisition design in preferential BO differs from scalar-feedback BO because the algorithm must choose comparisons rather than single points. The original framework introduced pure exploration, Copeland Expected Improvement, and Dueling Thompson Sampling. In Dueling Thompson Sampling, a posterior function sample induces a soft-Copeland score used to select one point, and posterior uncertainty then selects a challenger (Gonzalez et al., 2017). This established the now-standard two-stage structure of incumbent selection plus challenger selection.

A later decision-theoretic line replaces Copeland-based surrogates with the expected utility of the best option. For a query $P[y=1\mid x,x']=\sigma(f(x)-f(x'))$5, qEUBO is

$P[y=1\mid x,x']=\sigma(f(x)-f(x'))$6

optimized by Monte Carlo sample-average approximation (Astudillo et al., 2023). In the pairwise special case, EUBO admits a closed form under a Gaussian posterior over $P[y=1\mid x,x']=\sigma(f(x)-f(x'))$7, and this closed form becomes the basis of the constrained acquisition EUBOC when multiplied by a feasibility probability (Iwai et al., 16 May 2025).

Thompson-style policies have also been adapted to the duel-kernel setting. Preferential-Feedback Thompson Sampling maintains a GP surrogate over $P[y=1\mid x,x']=\sigma(f(x)-f(x'))$8, draws two independent posterior samples $P[y=1\mid x,x']=\sigma(f(x)-f(x'))$9, anchors both at a fixed $\sigma$0, and sets

$\sigma$1

Because of anchor invariance, the choice of $\sigma$2 does not affect the outcome (Lazzaro et al., 27 Apr 2026). This yields a particularly symmetric preferential analogue of standard GP-TS.

Other acquisition families are tailored to specific constraints or interfaces. POP-BO constructs a likelihood-ratio confidence set $\sigma$3 and applies optimism: $\sigma$4 with $\sigma$5 typically set to the previous point (Xu et al., 2024). Projective preferential BO defines expected improvement over the maximum value attained along a projection and also introduces preferential coordinate descent as a restricted-query alternative (Mikkola et al., 2020). Consecutive PBO adapts a mutual-information criterion to the case in which each new comparison must involve a previously produced candidate and the human may return indifference (Erarslan et al., 7 Nov 2025). Contextual PBO conditions expected improvement on the observed context $\sigma$6 and uses the previous day’s utility as the reference level (Wang et al., 10 Dec 2025).

These acquisition developments also clarify a recurrent misconception: preferential BO is not a single policy, but a family of sequential design rules whose behavior depends strongly on how the query set is parameterized, how posterior uncertainty is propagated through a preference model, and whether the method seeks information about the global maximizer, the local incumbent, or the next most informative challenger.

4. Regret theory and decision-theoretic guarantees

Theoretical analysis of preferential BO has recently moved from heuristic empirical comparison to finite-time regret guarantees. POP-BO derives confidence sets from likelihood-ratio arguments and obtains an information-theoretic cumulative-regret bound

$\sigma$7

described as a first-of-its-kind for preferential BO, together with a reporting rule $\sigma$8 satisfying a high-probability convergence rate (Xu et al., 2024). The technical novelty lies in converting preference-only observations into duel-wise confidence bands of the form

$\sigma$9

A second line, BOHF under the Bradley–Terry–Luce model, establishes near-optimal regret bounds of order

$\mathcal X\subset\mathbb R^d$0

where $\mathcal X\subset\mathbb R^d$1 is the maximum information gain of the dueling kernel (Kayal et al., 29 May 2025). The same work states that, for common kernels, the order-optimal sample complexities of conventional BO are recovered, so that the same number of preferential samples as scalar-valued samples is sufficient to find a nearly optimal solution. This challenges the view that reduced feedback must automatically imply worse order-wise sample complexity.

Preferential-Feedback Thompson Sampling sharpens this picture in a TS-specific setting. Under $\mathcal X\subset\mathbb R^d$2 and with an exploration scale $\mathcal X\subset\mathbb R^d$3 on the order of the confidence-width multiplier $\mathcal X\subset\mathbb R^d$4, PF-TS satisfies

$\mathcal X\subset\mathbb R^d$5

matching the finite-time rate of standard TS for scalar-feedback BO (Lazzaro et al., 27 Apr 2026). The proof exploits a separability fact: any GP sample under the duel kernel can be written almost surely as $\mathcal X\subset\mathbb R^d$6, so maximizing $\mathcal X\subset\mathbb R^d$7 is equivalent to maximizing a sampled scalar function.

Decision-theoretic analysis has produced complementary guarantees. qEUBO is one-step Bayes optimal in the noise-free setting and thus equivalent to knowledge gradient; under noisy responses it enjoys an additive constant approximation guarantee to the one-step Bayes-optimal policy (Astudillo et al., 2023). The same paper proves that Bayesian simple regret under qEUBO converges to zero at rate $\mathcal X\subset\mathbb R^d$8, while qEI adapted from standard BO can fail to converge to zero simple regret. This result is important because qEI is frequently used as a convenient baseline in preferential settings even though its asymptotic behavior can be qualitatively wrong.

5. Structured and generalized preferential BO

Preferential BO has expanded well beyond the basic duel model. Preferential batch BO allows full rankings or batch-winner feedback for a set of $\mathcal X\subset\mathbb R^d$9 points and introduces batch analogues of expected improvement and Thompson sampling (Siivola et al., 2020). Projective Preferential Bayesian Optimization replaces pairwise duels by line minimizers $f$0, arguing that one projective query is informationally much richer than a single duel and reporting convergence on Hartmann (6D), Levy (10D), and Ackley (20D) where pairwise PBO variants stagnated (Mikkola et al., 2020). MultiBO generalizes pairwise input to $f$1-choice image preference queries with a multinomial-logit GP and a Dynamic Balanced Subspace acquisition around a prompt-derived anchor image (Rajagopalan et al., 2 Feb 2026).

Several variants address operational constraints. Consecutive PBO models production cost $f$2, evaluation cost $f$3, and a Just-Noticeable Difference threshold $f$4, selecting new configurations under a mutual-information objective when only consecutive comparisons are feasible (Erarslan et al., 7 Nov 2025). CPBO adds inequality constraints $f$5 through a separate GP and proposes EUBOC, which multiplies EUBO by the probability that both candidates satisfy the constraint (Iwai et al., 16 May 2025). Contextual PBO introduces a GP prior on $f$6 to optimize time-varying utilities under exogenous context $f$7, using pairwise comparisons across days rather than within-day static duels (Wang et al., 10 Dec 2025). BO with abstract properties augments the design input with latent preference-based property estimates, while asynchronous expert-feedback BO uses a separate preference-learning thread so that the BO loop never blocks on expert responses (V et al., 2024, Kristiadi et al., 2024).

Scalability has motivated two additional directions. PABBO fully amortizes preferential BO by meta-learning both the surrogate and the acquisition function with a transformer neural process trained by reinforcement learning, reporting several orders of magnitude lower latency than GP-based strategies (Zhang et al., 2 Mar 2025). Local Preferential Bayesian Optimization transfers trust-region and derivative-informed local search to the preference setting; TuRPBO adapts TuRBO-style region updates, while GIPBO and PrefSQP use first- and second-order derivatives of the Laplace-approximated GP posterior to form local ascent directions (Menn et al., 1 Jun 2026).

The same logic has also been exported to RLHF. Bayesian preference inference with Laplace posterior approximation over reward-model parameters can be embedded into the RLHF loop, where a Dueling Thompson Sampling–inspired acquisition chooses informative preference queries before PPO updates the policy (Cercola et al., 6 Nov 2025). This suggests that preferential BO and RLHF are not disjoint topics but partially overlapping formulations of active preference elicitation.

6. Applications and empirical evidence

Reported applications span human-, laboratory-, and expert-in-the-loop design, scientific discovery, recommender-style tasks, and interactive creative systems (Lazzaro et al., 27 Apr 2026, Gonzalez et al., 2017). The empirical picture is heterogeneous because tasks differ in dimensionality, feedback modality, and whether scalar observations are available for comparison, but several studies provide concrete demonstrations.

Domain	Setting	Reported outcome
Catalyst design	PF-TS on the OCx24 dataset	Lowest instantaneous regret among BOHF methods; outperforms vanilla GP-TS for $f$8 on a fixed budget (Lazzaro et al., 27 Apr 2026)
Robotic commissioning	Human expert provides binary duels on pushing behavior	PBO can tune the controller to the expert’s satisfaction purely from duels, needing $f$9 comparisons; $f$0 matched $f$1 of duels vs. $f$2 for the expert cost $f$3 (Witte et al., 1 Dec 2025)
Building climate control	Contextual PBO on BOPTEST	Improvement of up to $f$4 in utility over a two-month simulation period (Wang et al., 10 Dec 2025)
High-moisture extrusion	Consecutive comparisons with indifference	Three human operators converged to high-quality recipes in $f$5–$f$6 iterations (Erarslan et al., 7 Nov 2025)
Personalized image generation	Multi-choice human feedback with $f$7 candidates	Human study reported $f$8 win-rate and MOS $f$9 (Rajagopalan et al., 2 Feb 2026)

Synthetic and benchmark studies reinforce these application results. PF-TS achieves lower cumulative regret than POP-BO and MR-LPF on a one-dimensional Ackley function, is competitive with MaxMinLCB, and shows faster decline in regret on LCBench/MNIST hyperparameter tuning up to $\theta^*=\arg\min_{\theta\in\Theta} f(\theta)$0 (Lazzaro et al., 27 Apr 2026). qEUBO outperforms qEI, qNEI, qTS, MPES, and random across 6-d Ackley, 7-d Alpine1, 6-d Hartmann, car-cab design, sushi preference, and particle animation (Astudillo et al., 2023). Local PBO reports especially strong results in high-dimensional and steep landscapes, including MuJoCo policy-search tasks (Menn et al., 1 Jun 2026). PCPBO uses a bi-level combination of preferential BO and physics-based simulation to estimate physically feasible food arrangements under qualitative domain rules (Kwon et al., 2022). Interactive melody composition combines preference GP modeling with a reduced MusicVAE latent space to guide users toward desired melodies using only best-of-set choices (Zhou et al., 2020).

A recurrent empirical theme is that preference-based surrogates can capture criteria that domain experts struggle to formalize as a scalar objective. In the robotic commissioning study, the learned posterior mean $\theta^*=\arg\min_{\theta\in\Theta} f(\theta)$1 agreed with the human’s binary choices more often than a hand-designed cost function and then accelerated standard BO when reused as a scalar surrogate (Witte et al., 1 Dec 2025). A plausible implication is that preferential BO is often valuable not only as an optimizer, but also as a mechanism for distilling tacit objectives into reusable latent utility models.

7. Practical limitations and research directions

Despite its breadth, preferential BO remains constrained by inference cost, optimization difficulty, and human factors. Several papers note cubic scaling in the number of comparisons: exact or approximate posterior logistic-regression inference can be costly in $\theta^*=\arg\min_{\theta\in\Theta} f(\theta)$2 per round, continuous domains require discretization or global acquisition optimization, and GP-based PBO scales poorly to dimensions above $\theta^*=\arg\min_{\theta\in\Theta} f(\theta)$3 in some settings (Lazzaro et al., 27 Apr 2026, Witte et al., 1 Dec 2025). Gaussian approximation can misestimate duel probabilities, while exact skew-GP inference is more accurate but slower (Takeno et al., 2023). Human experts also face cognitive burden: repeated duels can be tiring, fixed acquisition rules may stick to the boundary, and binary comparisons discard information that could be captured by rankings, ratings, or indifference labels (Witte et al., 1 Dec 2025, Erarslan et al., 7 Nov 2025).

Current research addresses these limitations along several axes. Cost-aware methods mix production and evaluation costs (Erarslan et al., 7 Nov 2025). Structured feedback models incorporate one-out-of-many queries, top-$\theta^*=\arg\min_{\theta\in\Theta} f(\theta)$4 feedback, contextual covariates, or active anchor selection (Lazzaro et al., 27 Apr 2026). High-dimensional methods localize the search or amortize inference (Menn et al., 1 Jun 2026, Zhang et al., 2 Mar 2025). Constraint-aware variants search feasible regions directly (Iwai et al., 16 May 2025). Asynchronous schemes support non-blocking expert interaction in self-driving labs (Kristiadi et al., 2024). RLHF-style formulations import preferential acquisition into large-scale preference learning (Cercola et al., 6 Nov 2025).

The field therefore no longer consists only of “GP classification on duels.” It now includes duel-kernel formulations with finite-time regret guarantees, decision-theoretic acquisitions with Bayes-optimality results, structured-query models that go beyond pairwise feedback, and system-level adaptations for robotics, scientific design, creative interaction, and policy learning. Taken together, these developments frame Bayesian optimization with preferential feedback as a general methodology for optimizing latent utilities when judgments are comparative, scalar objectives are unavailable or inadequate, and the query interface itself is part of the design problem.