---
title: Freeform Preference Learning (FPL) Overview
url: https://www.emergentmind.com/topics/freeform-preference-learning-fpl
type: topic
---

# Freeform Preference Learning (FPL) Overview

Freeform Preference Learning (FPL) is a principled framework for aligning generative models and policy learners to preference data, expressing model quality in terms of a “win rate” under specified sampling processes. The core premise is that preference learning should maximize the probability that generated outputs are selected as preferable under a fixed distribution of comparisons, encompassing both classical win-rate maximization and generalizations that admit multi-dimensional, language-conditioned, and human-in-the-loop feedback. The FPL paradigm subsumes and critically distinguishes between algorithmic families, theoretical evaluation criteria, and practical deployment challenges in settings ranging from large language models to robotic manipulation [2502.10505][2606.32027].

## 1. Foundations: Win Rate as the Central Metric

FPL regards the only meaningful evaluation of a generator (e.g., conditional distribution $p(y|x)$) against human or oracle preferences as its win rate in pairwise comparison. Given access to a sampling distribution over triplets $(x, y_0, y_1)$ and a binary preference label $l \in \{0,1\}$ indicating whether $y_1$ is preferred to $y_0$ under a preference classifier fixed by the data-collection environment, evaluation is defined by the $h$-win-rate functional:
$$
\text{WinRate}_h[p(y|x); p_0] = \mathbb{E}_{x \sim p(x)} \mathbb{E}_{y \sim p(y|x)} \mathbb{E}_{y_0 \sim p_0(y_0|x)} [ h(p(l=1|x, y_0, y)) ]
$$
where $h$ is any strictly increasing scalar transformation. The uniqueness theorem for win rate proves that any scalar evaluation, subject to preference-consistency and prevalence-consistency, must have this triple-expectation form. Empirical estimation employs Monte Carlo sampling over the relevant triplets and preference queries [2502.10505].

## 2. Algorithmic Design: Win-Rate Optimization and Its Variants

All FPL algorithms are naturally categorized into Win-Rate Optimization (WRO) and non-WRO methods. WRO directly maximizes the expected (possibly transformed) win rate:
$$
\max_{p(\cdot|\cdot)} \mathbb{E}_{x,y,y_0} [ h(p(l=1|x, y_0, y)) ]
$$
Policy-gradient methods, including score-function estimators, implement this black-box optimization. KL-regularized WRO (RLHF-style) and game-theoretic WRO (Nash HF) are two broad classes:
- **KL-regularized WRO:** Introduces a KL penalty to a reference distribution $p_{\text{ref}}$,
  $$
  \max_p \mathbb{E}_{x,y,y_0}[h(p(l=1|x,y_0,y))] - \beta \cdot \text{KL}(p(y|x) \| p_{\text{ref}}(y|x))
  $$
  Under Bradley–Terry assumptions, and $h=\text{logit}$, this reduces to the standard RLHF-PPO loop.
- **WRO Games (NLHF):** Co-optimize $p$ and $p_0$ each maximizing its win rate against the other, by treating the optimization as an antisymmetric, possibly constant-sum game.

Novel WRO variants arise by altering $h$, sampling $y_0$ dynamically, or bypassing Bradley–Terry and using learned classifiers [2502.10505].

## 3. Theoretical Guarantees and Limitations

Any WRO objective equivalent to a win-rate function, $O_h(p)$, uniquely possesses two theoretical guarantees:
1. **Win-rate correspondence:** $O_h(p_1) > O_h(p_2)$ implies $\text{WinRate}_h[p_1] > \text{WinRate}_h[p_2]$. Optimization progress in WRO directly translates to improved metric.
2. **Win-rate consistency:** The global optimum $p^*$ achieves the maximal win rate possible with respect to the anchor; no “surrogate gap” exists.

Non-WRO approaches fail one or both properties. For instance:
- **DPO (Direct Preference Optimization):** Minimizes
  $$
  L_{\text{DPO}}(\theta) = -\mathbb{E}_{(x,y_0,y_1) \sim q} [\log p_\theta(l|x, y_0, y_1)]
  $$
  but does not aggregate queries and draws using correct prevalence, and can descend in DPO loss while resulting in strictly worse win rate.
- **Supervised Fine-Tuning (SFT) on preferred samples:** Yields limited win-rate improvement
  $$
  \text{WinRate}[p_0; p_{\text{SFT}}] = \frac{1}{2} + 2 \mathbb{E}_x \big[ \operatorname{Var}_{y \sim p_0}[ \mathbb{E}_{y_0}[p(l=1|x,y_0,y)] ] \big ] < 1
  $$
  and cannot reach perfect win rate except in degenerate settings.

These results demonstrate that surrogate objectives, without careful correction, can fail to improve true preference alignment [2502.10505].

## 4. Methodological Innovations in Multi-Axis and Language-Conditioned FPL

Recent advancements extend FPL to multi-dimensional and language-specified feedback, particularly in robotic manipulation [2606.32027]. Here, human annotators provide, for each trajectory pair, a set of natural-language “preference axes” $\ell_k$ (e.g., speed, carefulness) and axis-specific binary preferences $y_k$. The reward model $R_\theta(\tau, \ell)$, based on prefix-sum decomposition of multimodal transformer outputs, learns to localize behavior quality along these axes:
$$
P(\tau_i \succ \tau_j | \ell_k) = \sigma(R_\theta(\tau_i, \ell_k) - R_\theta(\tau_j, \ell_k))
$$
$$
R_\theta(\tau, \ell) = \sum_{t=1}^T g_\theta(o_{1:t}, \ell)
$$
Leveraging a pretrained vision-language model for feature backbone and conditional prompting, this framework enables open-vocabulary, densely-supervised, compositional reward learning. Policy optimization is performed via reward-conditioned behavior cloning, where the policy is prompted with normalized per-axis returns and trained to maximize action likelihood over diverse, axis-labeled rollouts.

## 5. Empirical Assessments and Emergent Properties

Empirical studies validate the efficacy of FPL over classical baselines. In four real-world and two simulated long-horizon robotic manipulation tasks, FPL achieves average real-world success rates of 75%, representing a 38 percentage point improvement over the next best baseline (Filtered BC: 37%). Simulated object rearrangement reports 84% for FPL versus 79% for single-preference training; throughput in bimodal tasks is 1.19 vs. 0.73 trajectories/s for FPL and single-preference, respectively. Key emergent properties include:
- **Dense credit assignment:** Axis-conditioned rewards induce localized supervision spikes aligned to task-relevant events, without requiring subtask segmentation.
- **Compositionality:** The policy exhibits combinatorial generalization to axis-intersections absent in demonstrations.
- **Test-time steerability:** A single policy can be navigated across axes at deployment by modifying the textual prompt alone, eliminating the need for retraining.
- **Organic axis evolution:** As policy performance increases, annotator-specified axes shift toward finer granularities without rubric redesign [2606.32027].

## 6. Algorithmic Implementation and Best Practices

The iterative FPL training procedure for robotic settings is formalized as follows:

```python
# Algorithm 1: Freeform Preference Learning (FPL)
for n in range(N):
    # 1. Collect new rollouts using current policy
    # 2. Aggregate trajectories into dataset D
    # 3. Query annotators for freeform axes and preferences on sampled pairs
    # 4. Update preference buffer P
    # 5. Train reward model θ on P using L_{FPL-Rew}
    # 6. Select representative axes L
    # 7. Compute normalized returns for trajectories and axes
    # 8. Train policy π_n via reward-conditioned behavior cloning on D
# Return final policy π_N
```

Best practices emphasize:
- Always evaluating and checkpointing via win rate or calibrated surrogates rather than surrogate losses (e.g., DPO loss).
- For non-WRO methods, incorporating online sampling and checkpoint selection by win rate.
- Ensuring diversity in generation and filtering to maximize the variance of average preference probabilities.
- For WRO, deploying variance reduction, large batch sizes, and exploring alternative surrogates for $h$ to enhance gradient signal quality.
- Combining methodological elements (e.g., initial SFT, followed by WRO-KL) to achieve stability and alignment.

## 7. Open Problems and Future Directions

Current FPL algorithms do not simultaneously deliver theoretical alignment (win-rate consistency) and practical ease of optimization. Promising research avenues include:
- Construction of differentiable win-rate surrogates with low-variance gradients, motivated by advances in H-consistency and adversarial surrogate loss theory.
- Development of game-theoretic WRO algorithms (e.g., mirror descent, regret minimization) with convergence guarantees in non-constant-sum scenarios.
- Integration of richer feedback, such as freeform natural language axes, to support continuous, scalable reward learning from diverse human input.

FPL thus articulates a unified, theoretically-grounded, and practical paradigm for preference-based alignment across both discrete and continuous domains, with win rate as its core organizing principle [2502.10505][2606.32027].

Source: https://www.emergentmind.com/topics/freeform-preference-learning-fpl