A Finite Time Analysis of Thompson Sampling for Bayesian Optimization with Preferential Feedback

Abstract: Preference feedback, in the form of pairwise comparisons rather than scalar scores, has seen increasing use in applications such as human-, laboratory-, and expert-in-the-loop design, as well as scientific discovery. We propose a Thompson Sampling (TS) approach to Bayesian optimization with preferential feedback that models comparisons using a monotone link on latent utility differences and leverages the dueling kernel induced by a base kernel. We provide a finite-time analysis showing that the performance of the proposed method matches that of standard TS for conventional Bayesian optimization with scalar feedback. The analysis exploits the anchor invariance of TS for challenger selection and introduces a double-TS pairing variant. We also demonstrate the performance of the method on both synthetic and real-world examples.

• The paper introduces PF-TS, a Thompson Sampling adaptation achieving finite-time regret matching scalar-feedback BO despite weak (1-bit) preference signals.
• It employs a dueling kernel and Gaussian Process surrogate with kernel ridge regression, ensuring symmetric pair selection and efficient computation.
• Empirical tests on synthetic and real-world tasks validate PF-TS's robustness and cost-effectiveness in scenarios with abundant preference data.

Finite-Time Regret Analysis of Thompson Sampling for Bayesian Optimization with Preferential Feedback

Problem Setting and Motivation

The paper investigates the canonical Bayesian Optimization (BO) setup under preferential feedback, where an agent optimizes an unknown, expensive-to-evaluate objective but only receives feedback in the form of pairwise comparisons between candidates, not calibrated scalar values. This setting is increasingly relevant for experiment design, human-in-the-loop evaluation, and scientific discovery scenarios where absolute measurements are noisy, expensive, or unavailable, yet reliable pairwise judgments can be elicited. The formal model assumes the Bradley-Terry-Luce mechanism, i.e., preference probabilities modeled via logistic links of latent utility differences. Regret is quantified by the cumulative shortfall relative to the optimal action across queried pairs.

Algorithmic Contribution: Preferential Feedback Thompson Sampling (PF-TS)

The core methodological contribution is a variant of Thompson Sampling tailored to BO with preference feedback (BOHF). PF-TS maintains a Gaussian Process (GP) surrogate over pairwise utility differences, derived via a "dueling kernel" from a base kernel, and uses kernel ridge regression with regularized logistic loss for inference. At each round, the agent draws two independent Thompson samples from the GP posterior and selects two actions using anchor-independent single-argument maximizations—thereby preserving symmetry and sequentiality. The formulation leverages the property that the comparison function decomposes into latent utilities, resulting in anchor invariance for challenger selection and action selection efficiency.

Notably, the algorithm avoids anchor bias and computational overhead associated with alternative acquisition rules like UCB—where action selection couples both arguments and breaks symmetry. PF-TS is thus well-suited for practical preference elicitation, e.g., from human judges or via LLM oracle feedback, as neither candidate in the pair is privileged.

Theoretical Regret Analysis

A principal result is the establishment of finite-time regret bounds for PF-TS, showing that despite operating under strictly weaker (1-bit) feedback compared to scalar BO, the cumulative regret matches the best-known rates for Thompson Sampling in conventional BO. The regret bound is

$$R(T) = \widetilde{O}(B_T(\delta) \sqrt{T \cdot \mathcal{I}(T)}),$$

where $B_T(\delta)$ is a confidence-width multiplier and $\mathcal{I}(T)$ denotes maximum information gain under the dueling kernel. For standard kernels (e.g., linear, Matérn, RBF), $\mathcal{I}(T)$ admits well-characterized growth rates. The proof crucially exploits the separability (anchor-independence) property, information-theoretic bounds, and anti-concentration arguments that guarantee sufficient exploration.

This result closes the gap between scalar and preference feedback settings for sequential BO, as previous approaches generally required batching or fixed-action asymmetry to attain competitive guarantees. The regret matching is robust to kernel choice; for the dueling kernel, eigenvalues scale identically to the base kernel—the theoretical complexity does not increase compared to scalar feedback BO.

Empirical Evaluation

PF-TS is validated across synthetic (Ackley) and real-world (Open Catalyst Experiments 2024, OCx24) objectives. For the Ackley function, PF-TS delivers lower cumulative and instantaneous regret compared to recent BOHF baselines (MR-LPF, MaxMinLCB, POP-BO) at horizons of 300 iterations. On the OCx24 catalyst design task, PF-TS achieves the lowest instantaneous regret and substantially reduced cumulative regret relative to BOHF competitors over 800 iterations, while remaining competitive with MaxMinLCB. Application to hyperparameter optimization and RKHS synthetic functions further confirms robustness.

A cost-adjusted comparison demonstrates that when preference queries are significantly cheaper than scalar measurements, PF-TS outperforms standard scalar-feedback BO on practical cost budgets. This has strong implications for settings where high-throughput preference feedback—e.g., from LLM oracles or human evaluators—can be generated at scale.

Practical and Theoretical Implications

PF-TS's symmetry in pair selection and full sequentiality address critical requirements in experimental workflows, human/AI-in-the-loop design, and model alignment via preference judgements. The finite-time matching of scalar-feedback regret orders means the method can be deployed for real-world optimization tasks with sample-efficiency guarantees, even given only weak feedback. The anchor-independence property delivers computational and statistical efficiency, enabling practical policy design in domains where pairwise evaluation is the dominant cost.

From a theoretical perspective, the analysis techniques—particularly the handling of link nonlinearity, posterior uncertainty propagation, and anchor-independence reduction—may inform future algorithmic and analytical developments in preference-based learning. The information-theoretic regret matching indicates that preference Bayesian optimization can be, in a complexity sense, as efficient as full-information BO.

Outlook and Future Directions

PF-TS opens avenues for hybrid optimization strategies that combine scalar and preferential feedback, more flexible "one-out-of-many" preference settings, and active preference data collection for large-scale LLM alignment. Scaling experiments with AI-judged preference oracles, especially for fine-tuning foundation models, are a promising direction. Further theoretical exploration of regret lower bounds, robustness to model misspecification, and the interplay between feedback cost and optimization horizon will be key.

Conclusion

This work introduces PF-TS, a Thompson Sampling approach for Bayesian Optimization with preferential feedback, providing finite-time regret guarantees matching scalar-feedback BO, and demonstrating competitive empirical performance on synthetic and real-world optimization tasks. PF-TS is particularly impactful in settings where preference queries are accessible and cost-effective, enabling principled, efficient optimization under weak feedback regimes. The methodology and analysis contribute both to the practical deployment of BOHF and to the foundational understanding of preference-driven sequential optimization.