Preference-based optimization from noisy pairwise comparisons

Abstract: In interactive systems, feedback is often provided in the form of preference between queried options rather than precise scores, which motivates optimization methods to learn from such comparisons. In this work, we propose a preference-based optimization algorithm that relies on noisy two-point comparisons. At each iteration, the algorithm employs a uniform-sphere perturbation to generate a perturbed action and queries the resulting loss comparison to estimate a descent direction. We demonstrate that, under standard smoothness and bounded variance assumptions, the algorithm converges to a stationary point when the smoothing and step size parameters are properly chosen. Numerical experiments on an LQG system demonstrate the effectiveness of the preference-based optimization algorithm with comparison feedback.