---
title: Learning from Human Feedback (LHF)
url: https://www.emergentmind.com/topics/learning-from-human-feedback-lhf
type: topic
---

# Learning from Human Feedback (LHF)

Searching arXiv for recent and foundational papers on Learning from Human Feedback to ground the article.
Learning from Human Feedback (LHF) is a family of interactive, human-in-the-loop training methods that replace or complement engineered objectives with signals derived from people. Reinforcement Learning from Human Feedback (RLHF) is its most prominent instantiation, but the term also covers settings that use pairwise comparisons, ratings, critiques, demonstrations, corrections, and natural-language feedback; one recent study explicitly subsumes both RLHF and Direct Preference Optimization (DPO) under LHF [2312.14925] [2508.01930]. Across the literature, LHF is used to steer systems toward “helpfulness,” “honesty,” and “harmlessness,” along with related dimensions such as safety, quality, groundedness, correctness, engagingness, and moral acceptability, especially where no clear programmatic reward exists [2310.07629].

## 1. Scope, neighboring paradigms, and feedback modalities

Within the broader alignment literature, RLHF is commonly presented as a generalization of preference-based reinforcement learning. The survey literature distinguishes it from inverse reinforcement learning, which infers rewards from demonstrations, and from imitation learning, which directly mimics demonstrations; RLHF instead elicits direct human preferences and commonly updates the learned objective during training [2312.14925]. In large language models, this distinction became operationally important because supervised fine-tuning can teach instruction following, while LHF supplies preference information that is not directly recoverable from demonstrations alone [2312.14925].

The space of feedback signals is broader than pairwise preference labels. The literature surveyed for RLHF includes pairwise comparisons and rankings, scalar ratings, critiques and corrections, inter-temporal feedback, proxy reward design, social or implicit feedback, and natural-language feedback [2312.14925]. A systems paper on RLHF-Blender formalizes this breadth through a unified encoding over targets, relations, content, intention, expression, and metadata, so that demonstrations, rankings, comparisons, natural-language instructions, and descriptive feature-level supervision can be processed within one interface and one reward-learning stack [2308.04332].

This broader framing matters because different modalities expose different information. Pairwise comparisons provide relative orderings; ratings supply absolute judgments; critiques and revisions expose reasons for preference; demonstrations and corrections specify desired behavior; and natural-language feedback can make the relevant failure mode explicit. The resulting field is therefore not a single algorithmic recipe but a family of estimation, aggregation, and optimization procedures defined by how human judgments are elicited and operationalized.

## 2. Probabilistic foundations of preference learning

A standard formalization begins with trajectories. In one expertise-oriented treatment, at each timestep \(t\) an agent observes state \(s_t\), takes action \(a_t\), and transitions to \(s_{t+1}\); a \(k\)-step trajectory is \(\sigma=((s_1,a_1),\ldots,(s_k,a_k))\), and the ground-truth reward is \(r(\sigma)=\sum_{n=1}^k r(s_n,a_n)\) [2211.06519]. Human feedback is then modeled over paired trajectories.

The canonical stochastic choice model is Bradley–Terry/Thurstone-like. If a teacher has rationality parameter \(\beta \ge 0\), the probability of preferring \(\sigma_1\) over \(\sigma_2\) is
\[
P[\sigma_1 \succ \sigma_2 \mid r,\beta]
=
\frac{\exp(\beta r(\sigma_1))}
{\exp(\beta r(\sigma_1))+\exp(\beta r(\sigma_2))}
=
\sigma\!\big(\beta[r(\sigma_1)-r(\sigma_2)]\big),
\]
with \(\sigma\) denoting the logistic function [2211.06519]. In the survey literature this same structure appears as a Bradley–Terry/Luce preference model over returns \(R_\theta(\tau)\), with reward-model training given by a cross-entropy loss on preferred and dispreferred trajectories [2312.14925].

For reward-model learning from labeled comparisons \(D=\{(\sigma_i^+,\sigma_i^-,\mu_i)\}\), a standard objective is logistic cross-entropy:
\[
\mathcal{L}_{\text{pref}}(\phi)
=
-\sum_i
\Big[
\mu_i(+)\log \sigma(r_\phi(\sigma_i^+)-r_\phi(\sigma_i^-))
+
\mu_i(-)\log \sigma(r_\phi(\sigma_i^-)-r_\phi(\sigma_i^+))
\Big].
\]
Once a reward model is learned, policy optimization is often regularized toward a reference policy:
\[
J(\pi)
=
\mathbb{E}_{x\sim D,\;a\sim \pi(\cdot|x)}[r_\phi(x,a)]
-
\beta_{\text{KL}}\,
\mathrm{KL}\!\big(\pi(\cdot|x)\,\|\,\pi_{\text{ref}}(\cdot|x)\big).
\]
This KL-regularized form is central in LLM RLHF and also appears in control-oriented formulations [2211.06519].

A distinct theoretical perspective treats pairwise preference learning as density estimation rather than reward maximization. Under the Luce choice rule,
\[
\mathrm{Prob}(x_A \succ x_B)=\frac{p^*(x_A)}{p^*(x_A)+p^*(x_B)},
\]
and minimizing the binary cross-entropy preference loss yields a reward satisfying
\[
e^{r_\phi(x)} \propto p^*(x).
\]
Under this view, the learned reward recovers the annotator’s implicit preference distribution up to a constant, and model misspecification becomes a mismatch between the assumed and actual generative processes for preferences [2311.14115].

## 3. Standard pipelines and alternative formulations

The canonical LLM RLHF pipeline has four stages: supervised fine-tuning on demonstrations, collection of preference comparisons, reward-model training, and KL-regularized policy optimization, typically with PPO [2312.14925]. This pipeline is widely used because it separates preference elicitation from policy optimization, but it inherits the limitations of scalar reward modeling, including reward overfitting and dependence on the preference model’s assumptions [2312.14925].

Direct optimization methods remove the explicit reinforcement-learning loop. DPO is described as a method that directly optimizes policy parameters from pairwise preference data without an explicit reinforcement-learning loop, and one recent study uses instruction-tuned Llama as a post-LHF model where the instruction tuning uses DPO [2508.01930]. Related direct methods in the survey literature include SLiC-HF, OPPO/DPPO, PRO, and rejection-sampling-based approaches, all of which avoid some of the engineering overhead of PPO-style RLHF while retaining a preference objective [2312.14925].

A more radical alternative is Nash Learning from Human Feedback. NLHF learns a preference model conditioned on two responses and then seeks a policy that is preferred over those generated by any competing policy, thereby defining the Nash equilibrium of the induced preference game [2312.00886]. In the tabular setting, Nash-MD is a mirror-descent algorithm with last-iterate convergence to the regularized Nash equilibrium, and the deep-learning version replaces scalar reward optimization with policy-gradient updates against a learned pairwise preference model [2312.00886]. This construction is motivated by the claim that scalar reward models cannot represent non-transitive human preferences and remain dependent on the response sampling distribution used during training [2312.00886].

LHF also includes natural-language supervision that is richer than rankings. A critique-and-revise formulation fine-tunes an open-source LLM on prompt, initial response, critique, and revision records. With one iteration of revision of ChatGPT responses, the revised responses achieved a 56.6% win rate over the original ones, and this win rate improved to 65.9% after five iterations [2311.14543]. This suggests that, in some settings, textual critiques and revisions can replace the reward-model-plus-RL decomposition with direct supervised alignment on human feedback expressed in language.

## 4. Heterogeneity, expertise, and aggregation of human judgments

A central assumption of much RLHF work is that feedback can be pooled into a single reward model. Several papers challenge that assumption directly. “The Expertise Problem” formalizes a setting with multiple teachers whose reliability depends both on who provides the label and on which part of the task the query probes. In that formulation, the constant rationality parameter \(\beta\) is replaced by a teacher- and query-dependent \(\beta_i(\sigma_1,\sigma_2)\), and some queries are “inter-domain,” with low reliability for all teachers [2211.06519]. Empirically, on cartpole balance, “Max-\(\beta\) + Hybrid” achieved \(794 \pm 196\) final episode reward versus \(396 \pm 338\) for “Unif + Dis,” and on walker walk, “Max-\(\beta\) + Sim” achieved \(950 \pm 21\) versus \(259 \pm 186\) for “Unif + Dis,” directly illustrating the cost of treating all annotators as equally informative [2211.06519].

A broader treatment of heterogeneous preferences introduces two families of solutions. The personalization-based framework learns multiple reward models, either through shared representation learning or via clustering, to trade off the bias induced by pooled heterogeneous preferences against the variance induced by splitting the data across personalized models [2405.00254]. In the representation-learning version, personalized rewards take the form
\[
r_{\omega,\theta_i}(\cdot)=\langle \psi_\omega(\phi(\cdot)), \theta_i\rangle,
\]
with shared representation \(\psi_\omega\) and labeler-specific parameters \(\theta_i\) [2405.00254].

The aggregation-based framework keeps a single-model deployment but changes how preferences are combined. Reward aggregation is parameterized by
\[
Agg_\alpha(\mathbf r)
=
\begin{cases}
\frac{1}{\alpha}\log\!\left(\frac{1}{N}\sum_{i=1}^N e^{\alpha r_i}\right), & \alpha \neq 0,\\[4pt]
\frac{1}{N}\sum_{i=1}^N r_i, & \alpha=0,
\end{cases}
\]
where \(\alpha=0\) yields utilitarian aggregation and \(\alpha\to -\infty\) tends to a Leximin-like rule [2405.00254]. The same paper studies direct aggregation of probabilistic opinions and develops a mechanism-design approach that is dominant-strategy incentive compatible and social-welfare maximizing under KL- or \(\alpha\)-Rényi-based distance functions [2405.00254]. The underlying point is that heterogeneity is not only a statistical nuisance but also a normative problem: different aggregation rules encode different welfare principles.

## 5. Querying, active learning, and multimodal interaction

LHF is increasingly treated as an active data-collection problem rather than a passive annotation stage. RLHF-Blender is a configurable, interactive system that collects diverse feedback types in one interface, encodes them in a unified schema, and plugs them into modular reward-model training and RL pipelines while logging metadata about irrationality, noise, bias, workload, and progress effects [2308.04332]. This tooling perspective makes the data-collection protocol itself a first-class object of study.

Adaptive querying extends that idea algorithmically. Adaptive Feedback Selection learns a penalty function for negative side effects from multiple forms of human feedback by selecting both the queried state and the feedback format. Its second phase uses a cost- and availability-weighted bandit utility,
\[
U(f;t)=\frac{\psi(f)\,V_f}{C(f)}+\sqrt{\frac{\log t}{n_f+\epsilon}},
\]
to choose among Approval, Annotated Approval, Corrections, Annotated Corrections, Ranking, Demo-Action Mismatch, and Gaze [2412.07990]. Across Navigation, Vase, Push, and Freeway, the framework is reported to be sample efficient and to match or outperform non-adaptive baselines on average side-effect penalty while maintaining competitive task cost [2412.07990].

Corrective-feedback systems make human intervention even more direct. Gaussian Process Coach replaces COACH’s RBF parameterization with Gaussian processes for both the policy and the human-feedback model. It uses predictive uncertainty to adapt the learning rate,
\[
r_k=\sigma_p(s_k)+\sigma_h(z_k)+c_r,
\]
and to drive active querying through \(\Delta_k=c_a\sigma_h(z_k)\), yielding better convergence speed, final performance, and robustness to erroneous feedback on OpenAI Gym continuous-control benchmarks [1903.05216].

Multimodal embodied agents require denser and more temporally localized feedback. “Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback” replaces whole-trajectory comparisons with inter-temporal preferences derived from within-episode progress and regress marks. The learned utility \(U(x_{\le t})\) induces a dense per-step reward,
\[
r_t = U(x_{t+1}) - U(x_t),
\]
and training uses 5,104,000 binary marks across 364,690 episodes in a 3D environment [2211.11602]. The resulting BC+RL agent achieved 89% success in live interactive evaluation, representing 93% of the success rate of humans placed in the same role [2211.11602]. This broadens LHF from static preference annotation to continuous human evaluation of multimodal, temporally extended behavior.

## 6. Robustness, misspecification, and contested objectives

Much of the recent literature focuses on the fact that standard Bradley–Terry-based RLHF can be misspecified. A density-estimation analysis calls this “annotator misspecification”: if a model assumes a single Luce-style annotator but the data are generated by a mixture of annotators, the learned policy can collapse to a middle region preferred by neither annotator [2311.14115]. The same paper identifies length bias as a concrete misspecification in autoregressive settings and proposes a length-normalized Luce rule as a correction [2311.14115].

Several works answer misspecification with explicitly robust objectives. Distributionally robust RLHF formulates TV-ball DRO versions of reward modeling, KL-regularized policy optimization, and DPO. On RewardBench, the standard BT reward model averaged about 71.8%, while the robust BT model with TV radius \(0.4\) reached about 74.7%; on the Reasoning subset, the score improved from 65.2% to 76.3% [2503.00539]. Another line introduces contrastive rewards that subtract a prompt-specific baseline from the reward-model score; the stated effects are to penalize reward uncertainty, encourage improvement over baselines, calibrate according to task difficulty, and reduce variance in PPO [2403.07708]. A third paper proposes variance-reduced preference optimization under reward-model misspecification and reports that 77–81% of responses are favored over baselines on the Anthropic Helpful and Harmless dataset [2504.03784].

An information-theoretic strand treats RLHF as exploration under expensive preference queries. Information-Directed Sampling adds a mutual-information term to the episode objective and yields a Bayesian regret upper bound of order
\[
O\!\big(H^{3/2}\sqrt{T\log K(1/T)}\big),
\]
with tabular specialization \(\tilde O(H^2\sqrt{SAT})\) [2502.05434]. This suggests that LHF is not only an estimation problem but also a query-allocation problem in which human feedback should be spent where it is most informative.

The notion of “alignment” itself is contested. One study on lexical overuse treats LHF as including RLHF and DPO and argues that LHF can amplify subtle rater preferences for specific stylistic markers. In its experimental emulation, high-LHF-Score variants were preferred overall at 52.4% versus 47.6%, with \(\chi^2=9.4\) and \(p<0.01\), yet the same paper emphasizes a divergence between LHF workers and some end users, making lexical overuse “a sort of misalignment” rather than a universal preference signal [2508.01930].

Survey work broadens this into a governance critique. Only 9 of 50 “present” papers in one survey reported demographics; roughly 20 workers contributed about 80% of feedback in one helpful/harmless assistant study, and in WebGPT the top 5 contributed about 50% [2310.07629]. The same survey frames five unresolved challenges: preferences and values are not universal, they are inconsistently defined, human feedback is inherently incomplete, operationalizing “good” outputs is difficult, and there is a risk of “the tyranny of crowdworkers” [2310.07629]. This suggests that LHF is not only a technical apparatus for reward learning or policy optimization, but also a social procedure for selecting whose judgments count, how disagreement is represented, and which trade-offs are treated as legitimate.

Source: https://www.emergentmind.com/topics/learning-from-human-feedback-lhf