Online Inverse Preference Learning

• Online inverse preference learning is a sequential framework that infers latent utility and preference structures from real-time behavioral feedback such as pairwise comparisons and improved choices.
• It leverages methodologies like the preference perceptron, inverse optimization, and Bayesian updating to adapt models as the acting policy and data distribution evolve.
• Applications span from recommender systems and web search to RLHF and multi-agent reinforcement learning, providing regret guarantees and improved decision-making under non-scalar feedback.

Online inverse preference learning denotes sequential inference of latent utility, reward, or preference structure from behavior that arrives during interaction. Instead of assuming access to numeric rewards, it uses improved outputs, pairwise comparisons, revealed choices, or trajectory preferences, and it updates the preference model online while the data distribution may itself evolve because the acting policy changes over time (Shivaswamy et al., 2011, Dong et al., 2018, Ye et al., 2024, Bui et al., 26 Sep 2025).

1. Problem formulations and feedback regimes

A unifying feature of the area is that the learner observes behavior that is informative about preferences but not a directly revealed scalar objective. In one early structured formulation, round $t$ begins with a context $x_t\in X$, the learner predicts an object $y_t\in Y$, and the user returns an improved object $y_t'\in Y$ that would have provided higher utility under an unknown $U(x,y)$. Preference informativeness is formalized by the $\alpha$-informative condition

$$U(x_t,y_t')-U(x_t,y_t)\;\ge\;\alpha\big(U(x_t,y_t^*)-U(x_t,y_t)\big)-\xi_t,$$

where $y_t^*=\arg\max_{y\in Y}U(x_t,y)$ and $\xi_t\ge 0$ is slack (Shivaswamy et al., 2011).

A second formulation uses contextual pairwise comparisons. In the contextual Bradley–Terry–Luce model, each item $i$ has a parameter $x_t\in X$0, the context is $x_t\in X$1, and the comparison probability is

$x_t\in X$2

The action at time $x_t\in X$3 is the choice of a pair $x_t\in X$4, and the learning problem is to infer the latent contextual scores $x_t\in X$5 while simultaneously making near-optimal online decisions (Lu et al., 27 Apr 2025).

A third line of work replaces explicit preference models with a general preference oracle. In that setting there exists a function $x_t\in X$6, queried through

$x_t\in X$7

and no scalar reward function or Bradley–Terry structure is assumed. This explicitly allows intransitive preferences and motivates online minimax formulations rather than reward recovery as an intermediate step (Ye et al., 2024).

A fourth formulation comes from revealed preferences and inverse optimization. There the learner observes a signal $x_t\in X$8 and a decision $x_t\in X$9 produced by an unknown parameter $y_t\in Y$0, and defines loss as distance to the optimal set or efficient set induced by the forward model. In generalized inverse optimization this takes the form

$y_t\in Y$1

while in revealed-preference online convex optimization the forward problem is

$y_t\in Y$2

with the goal of learning the hidden parameter from the observed choice sequence (Dong et al., 2018, Chen et al., 2020).

Trajectory-based variants extend the same idea to sequential control. Online inverse reinforcement learning estimates a latent reward from trajectories observed in real time, sometimes under uncertain dynamics or sparse terminal feedback, and online inverse preference learning for a learning agent asks a predictor to infer the reward function being optimized by an agent whose policy is itself evolving online rather than approximately optimal from the outset (Self et al., 2020, Sadek et al., 9 May 2026).

Regime	Online observation	Latent object
Structured preference feedback	$y_t\in Y$3	utility $y_t\in Y$4
Contextual pairwise comparison	$y_t\in Y$5	item parameters $y_t\in Y$6
General preference oracle	$y_t\in Y$7	preference function $y_t\in Y$8
Revealed preference / inverse optimization	$y_t\in Y$9	objective parameter $y_t'\in Y$0
Trajectory preference / online IRL	trajectories or actions over time	reward or action-value structure

2. Preference representations and latent models

The simplest and most influential representation is a linear utility or reward model. In structured online preference feedback,

$y_t'\in Y$1

with bounded features $y_t'\in Y$2 and prediction by

$y_t'\in Y$3

For web search, the feature map is DCG-inspired over the top 5 positions, so inference reduces to sorting documents by scores $y_t'\in Y$4 (Shivaswamy et al., 2011).

Bayesian formulations enrich the latent object beyond a single reward vector. In the model of ignorant and inconsistent agents, the latent state is $y_t'\in Y$5, where $y_t'\in Y$6 is belief over states, $y_t'\in Y$7 is deterministic utility, $y_t'\in Y$8 encodes whether the agent is Naive or Sophisticated under hyperbolic discounting, $y_t'\in Y$9 is the hyperbolic discount parameter, and $U(x,y)$0 is the softmax noise parameter. The posterior is

$U(x,y)$1

and the same likelihood structure is described as immediately compatible with online Bayesian updating in principle (Evans et al., 2015). A related Bayesian preference-elicitation view of inverse reinforcement learning places a prior on reward functions and policies, with softmax action likelihood

$U(x,y)$2

so preference inference becomes posterior updating over reward and bounded-rationality parameters (Rothkopf et al., 2011).

In ranking and recommender settings, latent utility is often heterogeneous and partially observed. A recent ranking model writes

$U(x,y)$3

with interpretable item attributes, item fixed effects, and a low-rank user–item factorization, while pairwise preference follows a logistic model

$U(x,y)$4

Because comparisons are only observed when both items are observable, pair observability is modeled as $U(x,y)$5 and corrected by inverse-probability weighting (Chen et al., 18 Feb 2026).

Modern RL and MARL variants often represent preferences as implicit rewards rather than explicit utilities. In sparse-reward cooperative MARL, the implicit global reward is defined through an inverse soft Bellman operator,

$U(x,y)$6

and is decomposed into local rewards

$U(x,y)$7

This representation is then used to construct dual global and local advantage streams for CTDE PPO (Bui et al., 26 Sep 2025).

A different response to the limits of scalarization appears in preference feature preservation for LLM alignment. There the latent preference is represented as a 5-dimensional feature vector over predefined sub-features for style, tone, harmlessness, background knowledge, and informativeness, rather than a single scalar reward. This explicitly targets the claim that binary pairwise labels can collapse multifaceted human preferences into mostly preferred features (Kim et al., 6 Jun 2025).

3. Online update mechanisms and learning algorithms

Direct online updates often use the difference between preferred and non-preferred objects. The Preference Perceptron initializes $U(x,y)$8 and applies

$U(x,y)$9

with no step size by default. In structured spaces, the per-round cost is inference plus an $\alpha$0 update in the feature dimension (Shivaswamy et al., 2011).

Inverse-optimization approaches replace explicit gradient steps with implicit proximal updates. Generalized inverse optimization uses

$\alpha$1

or, more generally, a Bregman-divergence version with optional regularizer $\alpha$2. Inverse multiobjective optimization uses the same proximal form with the efficient-set surrogate loss $\alpha$3, and a practical implementation rewrites the update via KKT conditions and binary selection variables over sampled scalarization weights (Dong et al., 2018, Dong et al., 2020).

Online convex optimization yields a parallel family of first-order algorithms. For dynamically revealed preferences, Mirror Descent updates

$\alpha$4

where for the new convex loss the subgradient is $\alpha$5. On the simplex, this produces an entropic update; in Euclidean geometry it reduces to projection after a gradient step (Chen et al., 2020).

Contextual human-feedback models integrate estimation and decision-making. The two-stage algorithm for contextual online uncertainty-aware preference learning begins with an $\alpha$6-greedy exploration stage: at time $\alpha$7, with probability $\alpha$8 it samples a random edge in the comparison graph and otherwise exploits by choosing the current best item and comparing it with a uniformly chosen neighbor. After $\alpha$9, it switches to pure exploitation and updates a regularized MLE with $$U(x_t,y_t')-U(x_t,y_t)\;\ge\;\alpha\big(U(x_t,y_t^*)-U(x_t,y_t)\big)-\xi_t,$$0 (Lu et al., 27 Apr 2025).

Online RLHF introduces further algorithmic patterns. Under a general preference oracle, one algorithm fits a maximum-likelihood estimate $$U(x_t,y_t')-U(x_t,y_t)\;\ge\;\alpha\big(U(x_t,y_t^*)-U(x_t,y_t)\big)-\xi_t,$$1 from past comparisons, computes the main agent by solving a reverse-KL-regularized Nash equilibrium against $$U(x_t,y_t')-U(x_t,y_t)\;\ge\;\alpha\big(U(x_t,y_t^*)-U(x_t,y_t)\big)-\xi_t,$$2, and selects an enhancer policy by maximizing an empirical information ratio. OPTune instead reduces online cost by regenerating only the bottom-$$U(x_t,y_t')-U(x_t,y_t)\;\ge\;\alpha\big(U(x_t,y_t^*)-U(x_t,y_t)\big)-\xi_t,$$3 prompts ranked by current reward and optimizing a weighted DPO loss

$$U(x_t,y_t')-U(x_t,y_t)\;\ge\;\alpha\big(U(x_t,y_t^*)-U(x_t,y_t)\big)-\xi_t,$$4

where the weight $$U(x_t,y_t')-U(x_t,y_t)\;\ge\;\alpha\big(U(x_t,y_t^*)-U(x_t,y_t)\big)-\xi_t,$$5 is a sigmoid of the reward gap. PFP inserts an additional online step between data collection and DPO: it adjusts predicted feature distributions to match an offline target distribution and conditions generation on system prompts synthesized from those preference features (Ye et al., 2024, Chen et al., 2024, Kim et al., 6 Jun 2025).

When the observed agent is itself learning, the update rule may be purely count-based. For Boltzmann-convergent learners, the averaging strategy estimates empirical action probabilities and then inverts the softmax map to obtain a $$U(x_t,y_t')-U(x_t,y_t)\;\ge\;\alpha\big(U(x_t,y_t^*)-U(x_t,y_t)\big)-\xi_t,$$6-normalized reward or action-value estimate. In the stateless case,

$$U(x_t,y_t')-U(x_t,y_t)\;\ge\;\alpha\big(U(x_t,y_t^*)-U(x_t,y_t)\big)-\xi_t,$$7

and there is an analogous statewise construction for $$U(x_t,y_t')-U(x_t,y_t)\;\ge\;\alpha\big(U(x_t,y_t^*)-U(x_t,y_t)\big)-\xi_t,$$8 (Sadek et al., 9 May 2026).

4. Guarantees: regret, consistency, asymptotics, and impossibility

The classical structured preference-feedback guarantee is an average regret bound. Under $$U(x_t,y_t')-U(x_t,y_t)\;\ge\;\alpha\big(U(x_t,y_t^*)-U(x_t,y_t)\big)-\xi_t,$$9-informative feedback with slack $y_t^*=\arg\max_{y\in Y}U(x_t,y)$0,

$y_t^*=\arg\max_{y\in Y}U(x_t,y)$1

Noise-free feedback therefore gives average regret vanishing at rate $y_t^*=\arg\max_{y\in Y}U(x_t,y)$2 (Shivaswamy et al., 2011).

Implicit inverse-optimization methods establish comparable online rates. Generalized inverse optimization proves

$y_t^*=\arg\max_{y\in Y}U(x_t,y)$3

together with a risk-consistency result under finite second moment assumptions; under correct model and additive noise, the average loss converges in probability to the noise variance. Online inverse optimization for linear objectives over arbitrary feasible sets gives vanishing average total error at rate $y_t^*=\arg\max_{y\in Y}U(x_t,y)$4, with separate objective-error and solution-error interpretations (Dong et al., 2018, Bärmann et al., 2018).

Contextual human-feedback models add statistical inference to regret analysis. The two-stage contextual BTL framework proves a Stage I estimation rate

$y_t^*=\arg\max_{y\in Y}U(x_t,y)$5

a Stage II rate

$y_t^*=\arg\max_{y\in Y}U(x_t,y)$6

a regret bound with leading term $y_t^*=\arg\max_{y\in Y}U(x_t,y)$7, and asymptotic normality of a debiased estimator despite adaptive sampling (Lu et al., 27 Apr 2025).

General-preference RLHF obtains finite-sample online guarantees in terms of the eluder coefficient. With prescribed $y_t^*=\arg\max_{y\in Y}U(x_t,y)$8, $y_t^*=\arg\max_{y\in Y}U(x_t,y)$9, $\xi_t\ge 0$0, and $\xi_t\ge 0$1, there exists an iteration $\xi_t\ge 0$2 such that the returned main-agent policy is $\xi_t\ge 0$3-approximate Nash with probability at least $\xi_t\ge 0$4. This replaces reward-model consistency with preference-oracle equilibrium consistency (Ye et al., 2024).

Trajectory-based OIPL in MARL proves a different type of identifiability. If preferences are generated by Bradley–Terry with Gumbel noise and all trajectory pairs are observed with coverage $\xi_t\ge 0$5, maximizing the BT likelihood over $\xi_t\ge 0$6 recovers an implicit reward in an equivalence class

$\xi_t\ge 0$7

which preserves optimal policies (Bui et al., 26 Sep 2025).

Not all guarantees are positive. For the problem of learning the preferences of a learning agent, if the learner is only assumed to be no-regret, then cumulative $\xi_t\ge 0$8, $\xi_t\ge 0$9, and KL errors for full reward recovery are $i$0 in the worst case. What remains possible is sublinear best-response tracking: a stateless previous-action predictor satisfies

$i$1

and the stateful version yields

$i$2

This sharply separates recoverability of optimal actions from recoverability of full preferences (Sadek et al., 9 May 2026).

5. Relations to neighboring areas and recurrent points of confusion

Online inverse preference learning is adjacent to, but not identical with, inverse reinforcement learning. Inverse reinforcement learning usually infers a reward that rationalizes expert behavior, whereas several OIPL formulations learn directly from pairwise comparisons, improved objects, or revealed decisions without requiring expert demonstrations. The distinction is explicit in both the general preference-oracle RLHF framework and the MARL sparse-reward setting, where online preference collection is interleaved with policy optimization rather than performed on a fixed offline dataset (Ye et al., 2024, Bui et al., 26 Sep 2025).

It is also distinct from dueling bandits, although the overlap is substantial. Preference-based multi-armed bandits study online selection of pairs of alternatives under stochastic comparison feedback and target objects such as Condorcet, Borda, Copeland, or Von Neumann winners. That literature provides regret and PAC guarantees under transitivity assumptions, but it does not require a structured forward model or a recovered utility representation; OIPL often does (Bengs et al., 2018).

A recurrent misunderstanding is that observed behavior must be approximately optimal for preference inference to be meaningful. Several papers reject that premise. One constructs generative models with false beliefs and hyperbolic time inconsistency, showing that detours, temptation, and belief errors can be evidence about preferences rather than mere noise. Another proves that if the observed agent is still learning online, full reward recovery may be impossible under no-regret behavior even though best-response prediction remains feasible (Evans et al., 2015, Sadek et al., 9 May 2026).

Another recurrent misunderstanding is that online preference learning always reduces to scalar rewards and transitive rankings. The general preference oracle explicitly allows intransitive preferences; the preference-feature-preservation framework argues that binary pairwise labels collapse multifaceted preferences into mostly preferred features; and exposure-corrected ranking models show that even standard logistic pairwise formulations can be biased by which comparisons become observable (Ye et al., 2024, Kim et al., 6 Jun 2025, Chen et al., 18 Feb 2026).

Exploration is likewise heterogeneous across the literature. Some early algorithms are greedy and rely on informativeness assumptions about received feedback; others insert explicit exploration through $i$3-greedy pair selection, enhancer policies, entropy regularization, Gumbel perturbations, or off-trajectory querying of an estimated controller (Shivaswamy et al., 2011, Lu et al., 27 Apr 2025, Self et al., 2020, Roy et al., 7 May 2026). This suggests that “online” in OIPL refers to sequential updating, not to a single canonical exploration strategy.

6. Applications and contemporary extensions

Web search and recommendation were among the earliest concrete targets. In a Yahoo! Learning to Rank experiment, the Preference Perceptron used a DCG-inspired top-5 feature map, achieved utility regret approaching zero under noise-free $i$4-informative feedback, and, under noisy relevance-label feedback, substantially outperformed a periodically retrained ranking SVM baseline in both utility and DCG* regret. The reported runtime was approximately 30 minutes for the Preference Perceptron versus approximately 20 hours for the SVM approach (Shivaswamy et al., 2011).

Online recommendation has remained a central use case. InvRec casts recommendation as an MDP in VirtualTB, learns an implicit reward through adversarial IRL rather than manual reward design, and reports better CTR than IRecGAN, PGCR, GAUM, and KGRL. The environment uses 11 static demographic attributes encoded into an 88-dimensional binary vector, a 3-dimensional dynamic-interest vector, and pages of 10 recommended items per step (Chen et al., 2020).

Preference learning from observed rankings extends OIPL to selection-biased transactional data. In an online wine-retailer application with 311,089 transactions and 23,721 customers, a model combining attributes, item effects, and latent factors improves out-of-sample recommendation relative to a popularity benchmark. Reported gains include Recall@10 of 0.0908 versus 0.0763, Recall@20 of 0.1428 versus 0.1241, Recall@40 of 0.2140 versus 0.1986, Precision@1 of 0.0407 versus 0.0299, and Precision@10 of 0.0297 versus 0.0245, with particularly strong gains for previously unconsumed categories (Chen et al., 18 Feb 2026).

Sparse-reward MARL has produced a trajectory-level version of OIPL. Preference-Guided Learning for Sparse-Reward MARL converts terminal outcomes into online trajectory preferences, learns dense implicit global and local rewards, and uses dual advantage streams for centralized critic and decentralized actors. On SMACv2, SparseMAPPO often fails, whereas IMAP-Rule and IMAP-LLM achieve sizable gains; on MAMuJoCo, IMAP-Rule outperforms all baselines across tasks, including HalfCheetah with approximately 3854 versus approximately 2479 for SL-MAPPO and approximately 1824 for SparseMAPPO (Bui et al., 26 Sep 2025).

LLM alignment has become a major application domain. OPTune reduces online preference-tuning cost by selectively regenerating only low-reward prompts and weighting DPO by reward-gap utility, yielding 1.27–1.56x faster training than fully online DPO while maintaining or exceeding performance at reduced regeneration ratios. PFP adds an explicit multi-attribute preference layer and reports AlpacaEval 2.0 length-controlled win rate 15.24 for PFP versus 13.13 for Iterative DPO and 14.23 for SPA, together with MT-Bench 6.88 versus 6.53 and 6.56, respectively (Chen et al., 2024, Kim et al., 6 Jun 2025).

Recent extensions also demonstrate that OIPL is not confined to action selection or ranking. Prefer formulates personalized review summarization as online learning over latent aspect weights, updates those weights by entropic online mirror descent,

$i$5

and reports that online preference learning improves evidence alignment and preference alignment in controlled simulations on Amazon Reviews’23 All_Beauty while stochastic extraction aids recovery under preference drift (Roy et al., 7 May 2026).

Across these applications, the topic has expanded from structured online learning with improved objects to a broader class of sequential inverse problems in which preferences are latent, feedback is partial or biased, and interaction itself changes what can be inferred. A plausible implication is that “online inverse preference learning” is best understood not as a single algorithmic template but as a family of sequential estimation problems linked by a common objective: recovering actionable preference structure from interaction without relying on directly observed rewards.