Freeform Preference Learning (FPL) Overview

• Freeform Preference Learning (FPL) is a framework that defines model quality through win rate maximization in pairwise preference comparisons.
• It employs techniques like policy-gradient optimization, KL regularization, and game-theoretic approaches to align outputs with human feedback, including multi-axis and language-conditioned inputs.
• Empirical results in robotic manipulation and simulation demonstrate FPL’s success by significantly improving win rates and policy performance compared to traditional methods.

Freeform Preference Learning (FPL) is a principled framework for aligning generative models and policy learners to preference data, expressing model quality in terms of a “win rate” under specified sampling processes. The core premise is that preference learning should maximize the probability that generated outputs are selected as preferable under a fixed distribution of comparisons, encompassing both classical win-rate maximization and generalizations that admit multi-dimensional, language-conditioned, and human-in-the-loop feedback. The FPL paradigm subsumes and critically distinguishes between algorithmic families, theoretical evaluation criteria, and practical deployment challenges in settings ranging from LLMs to robotic manipulation (Zhang et al., 14 Feb 2025, Torne et al., 30 Jun 2026).

1. Foundations: Win Rate as the Central Metric

FPL regards the only meaningful evaluation of a generator (e.g., conditional distribution $p(y|x)$) against human or oracle preferences as its win rate in pairwise comparison. Given access to a sampling distribution over triplets $(x, y_0, y_1)$ and a binary preference label $l \in \{0,1\}$ indicating whether $y_1$ is preferred to $y_0$ under a preference classifier fixed by the data-collection environment, evaluation is defined by the $h$-win-rate functional:

$$\text{WinRate}_h[p(y|x); p_0] = \mathbb{E}_{x \sim p(x)} \mathbb{E}_{y \sim p(y|x)} \mathbb{E}_{y_0 \sim p_0(y_0|x)} [ h(p(l=1|x, y_0, y)) ]$$

where $h$ is any strictly increasing scalar transformation. The uniqueness theorem for win rate proves that any scalar evaluation, subject to preference-consistency and prevalence-consistency, must have this triple-expectation form. Empirical estimation employs Monte Carlo sampling over the relevant triplets and preference queries (Zhang et al., 14 Feb 2025).

2. Algorithmic Design: Win-Rate Optimization and Its Variants

All FPL algorithms are naturally categorized into Win-Rate Optimization (WRO) and non-WRO methods. WRO directly maximizes the expected (possibly transformed) win rate:

$$\max_{p(\cdot|\cdot)} \mathbb{E}_{x,y,y_0} [ h(p(l=1|x, y_0, y)) ]$$

Policy-gradient methods, including score-function estimators, implement this black-box optimization. KL-regularized WRO (RLHF-style) and game-theoretic WRO (Nash HF) are two broad classes:

• KL-regularized WRO: Introduces a KL penalty to a reference distribution $p_{\text{ref}}$,

$(x, y_0, y_1)$0

Under Bradley–Terry assumptions, and $(x, y_0, y_1)$1, this reduces to the standard RLHF-PPO loop.

• WRO Games (NLHF): Co-optimize $(x, y_0, y_1)$2 and $(x, y_0, y_1)$3 each maximizing its win rate against the other, by treating the optimization as an antisymmetric, possibly constant-sum game.

Novel WRO variants arise by altering $(x, y_0, y_1)$4, sampling $(x, y_0, y_1)$5 dynamically, or bypassing Bradley–Terry and using learned classifiers (Zhang et al., 14 Feb 2025).

3. Theoretical Guarantees and Limitations

Any WRO objective equivalent to a win-rate function, $(x, y_0, y_1)$6, uniquely possesses two theoretical guarantees:

1. Win-rate correspondence: $(x, y_0, y_1)$7 implies $(x, y_0, y_1)$8. Optimization progress in WRO directly translates to improved metric.
2. Win-rate consistency: The global optimum $(x, y_0, y_1)$9 achieves the maximal win rate possible with respect to the anchor; no “surrogate gap” exists.

Non-WRO approaches fail one or both properties. For instance:

• DPO (Direct Preference Optimization): Minimizes

$l \in \{0,1\}$0

but does not aggregate queries and draws using correct prevalence, and can descend in DPO loss while resulting in strictly worse win rate.

• Supervised Fine-Tuning (SFT) on preferred samples: Yields limited win-rate improvement

$l \in \{0,1\}$1

and cannot reach perfect win rate except in degenerate settings.

These results demonstrate that surrogate objectives, without careful correction, can fail to improve true preference alignment (Zhang et al., 14 Feb 2025).

4. Methodological Innovations in Multi-Axis and Language-Conditioned FPL

Recent advancements extend FPL to multi-dimensional and language-specified feedback, particularly in robotic manipulation (Torne et al., 30 Jun 2026). Here, human annotators provide, for each trajectory pair, a set of natural-language “preference axes” $l \in \{0,1\}$2 (e.g., speed, carefulness) and axis-specific binary preferences $l \in \{0,1\}$3. The reward model $l \in \{0,1\}$4, based on prefix-sum decomposition of multimodal transformer outputs, learns to localize behavior quality along these axes:

$l \in \{0,1\}$5

$l \in \{0,1\}$6

Leveraging a pretrained vision-LLM for feature backbone and conditional prompting, this framework enables open-vocabulary, densely-supervised, compositional reward learning. Policy optimization is performed via reward-conditioned behavior cloning, where the policy is prompted with normalized per-axis returns and trained to maximize action likelihood over diverse, axis-labeled rollouts.

5. Empirical Assessments and Emergent Properties

Empirical studies validate the efficacy of FPL over classical baselines. In four real-world and two simulated long-horizon robotic manipulation tasks, FPL achieves average real-world success rates of 75%, representing a 38 percentage point improvement over the next best baseline (Filtered BC: 37%). Simulated object rearrangement reports 84% for FPL versus 79% for single-preference training; throughput in bimodal tasks is 1.19 vs. 0.73 trajectories/s for FPL and single-preference, respectively. Key emergent properties include:

• Dense credit assignment: Axis-conditioned rewards induce localized supervision spikes aligned to task-relevant events, without requiring subtask segmentation.
• Compositionality: The policy exhibits combinatorial generalization to axis-intersections absent in demonstrations.
• Test-time steerability: A single policy can be navigated across axes at deployment by modifying the textual prompt alone, eliminating the need for retraining.
• Organic axis evolution: As policy performance increases, annotator-specified axes shift toward finer granularities without rubric redesign (Torne et al., 30 Jun 2026).

6. Algorithmic Implementation and Best Practices

The iterative FPL training procedure for robotic settings is formalized as follows:

$l \in \{0,1\}$8

Best practices emphasize:

• Always evaluating and checkpointing via win rate or calibrated surrogates rather than surrogate losses (e.g., DPO loss).
• For non-WRO methods, incorporating online sampling and checkpoint selection by win rate.
• Ensuring diversity in generation and filtering to maximize the variance of average preference probabilities.
• For WRO, deploying variance reduction, large batch sizes, and exploring alternative surrogates for $l \in \{0,1\}$7 to enhance gradient signal quality.
• Combining methodological elements (e.g., initial SFT, followed by WRO-KL) to achieve stability and alignment.

7. Open Problems and Future Directions

Current FPL algorithms do not simultaneously deliver theoretical alignment (win-rate consistency) and practical ease of optimization. Promising research avenues include:

• Construction of differentiable win-rate surrogates with low-variance gradients, motivated by advances in H-consistency and adversarial surrogate loss theory.
• Development of game-theoretic WRO algorithms (e.g., mirror descent, regret minimization) with convergence guarantees in non-constant-sum scenarios.
• Integration of richer feedback, such as freeform natural language axes, to support continuous, scalable reward learning from diverse human input.

FPL thus articulates a unified, theoretically-grounded, and practical paradigm for preference-based alignment across both discrete and continuous domains, with win rate as its core organizing principle (Zhang et al., 14 Feb 2025, Torne et al., 30 Jun 2026).