---
title: Preference-Based Optimization Algorithm
url: https://www.emergentmind.com/topics/preference-based-optimization-algorithm
type: topic
---

# Preference-Based Optimization Algorithm

A preference-based optimization algorithm is an iterative procedure for optimizing an unknown or inaccessible objective function using only preference information—typically pairwise, ordinal judgments—between candidate solutions rather than direct numerical evaluations. Such algorithms are increasingly critical in domains where the cost function is latent, multi-faceted (e.g., human comfort, aesthetic, or multi-criteria tradeoff), or costly/impossible to quantify explicitly. They exploit preference feedback—potentially involving gradated levels and certainty scoring—to construct surrogates of the latent utility landscape, and leverage exploration-exploitation tradeoffs to efficiently query new candidate solutions.

## 1. Formulation of Preference-Based Optimization Problems

In the general setting, one seeks the minimizer (or maximizer) of a latent objective $f(x)$ over a feasible set, often under black-box constraints, but $f(x)$ is not directly available. Instead, one obtains preference outcomes $p(x_1, x_2)$, which may be:
- discrete ordinal scales (e.g., a 5-point Likert: “much better”, “slightly better”, “as good as”, etc.) possibly coupled with a certainty level $c(x_1, x_2)$ quantized on a multi-level scale,
- noisy binary pairwise comparisons, informing only $x_1 \succ x_2$ (preferred) or $x_2 \succ x_1$.

The problem is then re-cast as seeking
\[
x^* = \underset{x}{\arg\min}\,f(x)
\]
with the only access being preference queries and outcomes (possibly with multiple grades and certainty weights). For $x^*$, one requires
\[
p(x^*, x) \le 0\quad\forall x\quad\text{(all other feasible points are not preferred over $x^*$)}
\]
[2302.14630].

## 2. Surrogate Construction and Preference Encoding

Preference-based optimization algorithms universally rely on building a surrogate function $\widehat{f}$ mapping the decision space to approximate the “hidden” utility $f$, exploiting observed preferences as constraints. Prominent approaches are:

- **Radial Basis Function (RBF) Surrogates**: Constructed as
  \[
  \widehat{f}(x) = \sum_{i=1}^N \beta_i \phi(\gamma\Vert x-x_i\Vert^2)
  \]
  with $\phi(t) = 1/(1+t^2)$ (inverse-quadratic RBF), $\gamma > 0$ shape parameter, and weights $\beta$ estimated via a convex program that enforces surrogate differences to fall within bands consistent with observed preferences and their certainty scores. Multi-level outcomes and certainty produce multiple interval constraints per query, and associated slack penalties scale with certainty [2302.14630].

- **Gaussian Process (GP) Surrogates**: Place a prior over $f$ and fit it using ordinal likelihoods, e.g., Thurstone–Mosteller probit models for pairwise comparisons:
  \[
  P(x \succ x'|f) = \Phi\left(\frac{f(x)-f(x')}{\sqrt{2}\sigma}\right)
  \]
  where $\Phi$ is the standard normal CDF and $\sigma^2$ models preference noise [1909.12316].

- **Piecewise Affine Surrogates (PWA)**: For mixed variable problems, construct surrogates as maxima over affine regions, subject to MILP-feasible constraints [2302.04686].

Surrogate fitting incorporates slack variables for inconsistency/noise and certainty-weighted penalties, enabling robust modeling in the presence of uncertain or pluralistic feedback [2302.14630].

## 3. Acquisition and Exploration–Exploitation Strategies

The acquisition function determines which new candidate $x$ to propose for preference querying, balancing regions deemed desirable by the surrogate and unexplored areas. Typical forms are:

- **Surrogate–Exploration blends**:
  \[
  a(x) = \frac{\widehat{f}(x)}{\Delta\widehat{F} - \alpha_N z(x)}
  \]
  where $z(x)$ is, for instance, an arctangent of the reciprocal sum of inverse fourth powers of distances to previous points (zero at prior sampled locations), and $\alpha_N$ is an adaptive exploration weight responsive to improvement in search [2302.14630]. Other schemes use min–max normalized surrogate and exploration terms with an adjustable bi-criteria trade-off parameter $\delta \in [0,1]$ [2202.01468, 2202.01125].

- **Purely gradient-free pairwise SGD**: Construct a gradient estimator from comparison feedback by uniform-sphere perturbation, e.g.,
  \[
  g_t = (d/\delta_t) Y_t u_t
  \]
  where $Y_t$ is the one-bit comparison result and $u_t$ is sampled from the unit sphere, updating $x$ via gradient descent. Convergence to stationary points is established under standard smoothness and variance assumptions [2512.18511].

Acquisition rules often adaptively trade off surrogate exploitation against exploration, cycling between global and local search [2202.01468, 2202.01125]. For multi-objective or combinatorial problems, batch acquisition and scalarization (e.g., augmented Chebyshev) are employed for Pareto front exploration [2406.14699].

## 4. Handling Multi-Level Outcomes, Certainty, and Constraints

Modern Preference-Based Optimization algorithms generalize simple binary comparison by incorporating richer human judgment:
- **Multi-level outcomes**: Each query produces a set of preference grades (e.g., from a Likert scale), inducing multiple band constraints per preference in surrogate fitting. Certainty is encoded as weight factors in the convex program, such that high-certainty judgments create tight surrogate bands while uncertain outcomes impose looser constraints [2302.14630].
- **Unknown constraints**: Surrogates of feasibility and satisfaction are learned (e.g., via inverse-distance weighting from binary labels), with penalties for infeasible or unsatisfactory points in the acquisition function to guide safe exploration [2106.05639].

For mixed categorical and numerical variables, piecewise affine surrogates enable feasible optimization via mixed-integer programming [2302.04686].

## 5. Theoretical Guarantees and Empirical Performance

- **Global convergence** is established in frameworks where exploration functions guarantee infinitely many pure-exploration steps, ensuring sample density and convergence to the global optimum under continuity conditions [2202.01468, 2202.01125].
- **Convergence rates** for pairwise-SGD methods are $\mathcal{O}(T^{-1/2})$ under standard smoothness and stochasticity [2512.18511].
- Surrogate-based methods can leverage certainty and multi-level outcomes to reduce worst-/average-case distances to the true optimum by $20$–$50\%$ compared to binary-only baselines across benchmark problems [2302.14630].
- Incorporation of uncertainty weighting and advanced constraint surrogates leads to robust near-optimal solution finding in practical engineering tasks, e.g., exoskeleton gait personalization and MPC controller calibration, with high data-efficiency [1909.12316, 2106.05639].

Preference-based optimization is demonstrably effective for black-box engineering calibration, multi-objective design, and human-in-the-loop tasks where scalar reward is inaccessible.

## 6. Algorithmic Structure and Practical Implementation

A prototypical preference-based optimization loop includes:
1. Initialization: Random or space-filling design to generate initial candidates; initial preferences collected.
2. Surrogate update: Solve a convex program or probabilistic model fitting that encodes the observed preferences (possibly with multi-level, certainty, and constraint labels).
3. Acquisition optimization: Solve for the next candidate via minimization of the acquisition function blending surrogate and exploration terms.
4. Querying: Present candidate(s) for preference judgment, possibly collecting multi-level/certainty feedback.
5. Update sample set and exploration weights; possibly adapt surrogate hyperparameters (e.g., via LOOCV).
6. Repeat until budget exhausted; return best sample as estimated optimum.

Pseudocode reflecting these steps appears in [2302.14630, 2202.01468, 2202.01125].

## 7. Comparison to Alternative and State-of-the-Art Methods

Preference-based methods generalize and often improve upon pure black-box optimization processes by:
- Handling settings where direct objective evaluation is impossible or unreliable.
- Flexibly encoding human judgment, including variable intensity and certainty.
- Avoiding pitfalls of reward hacking, scalarization bias, or ill-posed numerical criteria.
- Enabling robust performance under noise and uncertainty via adaptive slack penalization and weighted fitting [2302.14630].
- Outperforming binary-only preference frameworks in practical settings and achieving significant improvements in sample efficiency and convergence to preferred solutions.

AmPL, gMRS, GLISp-r, CoSpar, and their variants are characterized by lightweight surrogate fitting via convex optimization, straightforward implementation for engineering workflows, and modular integration of multi-level preference structures and uncertainty weighting [2302.14630, 1909.12316, 2202.01468, 2202.01125].

Source: https://www.emergentmind.com/topics/preference-based-optimization-algorithm