Learning from Human Feedback (LHF)

• Learning from Human Feedback (LHF) is a family of interactive, human-in-the-loop training methods that integrate diverse human signals to guide AI behavior.
• LHF employs feedback modalities—from pairwise comparisons and scalar ratings to natural language critiques—to optimize reward models and policy performance.
• Frameworks like RLHF, DPO, and Nash Learning address challenges such as model misspecification, heterogeneous preferences, and effective feedback aggregation.

Searching arXiv for recent and foundational papers on Learning from Human Feedback to ground the article. Learning from Human Feedback (LHF) is a family of interactive, human-in-the-loop training methods that replace or complement engineered objectives with signals derived from people. Reinforcement Learning from Human Feedback (RLHF) is its most prominent instantiation, but the term also covers settings that use pairwise comparisons, ratings, critiques, demonstrations, corrections, and natural-language feedback; one recent study explicitly subsumes both RLHF and Direct Preference Optimization (DPO) under LHF (Kaufmann et al., 2023, Juzek et al., 3 Aug 2025). Across the literature, LHF is used to steer systems toward “helpfulness,” “honesty,” and “harmlessness,” along with related dimensions such as safety, quality, groundedness, correctness, engagingness, and moral acceptability, especially where no clear programmatic reward exists (Kirk et al., 2023).

1. Scope, neighboring paradigms, and feedback modalities

Within the broader alignment literature, RLHF is commonly presented as a generalization of preference-based reinforcement learning. The survey literature distinguishes it from inverse reinforcement learning, which infers rewards from demonstrations, and from imitation learning, which directly mimics demonstrations; RLHF instead elicits direct human preferences and commonly updates the learned objective during training (Kaufmann et al., 2023). In LLMs, this distinction became operationally important because supervised fine-tuning can teach instruction following, while LHF supplies preference information that is not directly recoverable from demonstrations alone (Kaufmann et al., 2023).

The space of feedback signals is broader than pairwise preference labels. The literature surveyed for RLHF includes pairwise comparisons and rankings, scalar ratings, critiques and corrections, inter-temporal feedback, proxy reward design, social or implicit feedback, and natural-language feedback (Kaufmann et al., 2023). A systems paper on RLHF-Blender formalizes this breadth through a unified encoding over targets, relations, content, intention, expression, and metadata, so that demonstrations, rankings, comparisons, natural-language instructions, and descriptive feature-level supervision can be processed within one interface and one reward-learning stack (Metz et al., 2023).

This broader framing matters because different modalities expose different information. Pairwise comparisons provide relative orderings; ratings supply absolute judgments; critiques and revisions expose reasons for preference; demonstrations and corrections specify desired behavior; and natural-language feedback can make the relevant failure mode explicit. The resulting field is therefore not a single algorithmic recipe but a family of estimation, aggregation, and optimization procedures defined by how human judgments are elicited and operationalized.

2. Probabilistic foundations of preference learning

A standard formalization begins with trajectories. In one expertise-oriented treatment, at each timestep $t$ an agent observes state $s_t$, takes action $a_t$, and transitions to $s_{t+1}$; a $k$-step trajectory is $\sigma=((s_1,a_1),\ldots,(s_k,a_k))$, and the ground-truth reward is $r(\sigma)=\sum_{n=1}^k r(s_n,a_n)$ (Daniels-Koch et al., 2022). Human feedback is then modeled over paired trajectories.

The canonical stochastic choice model is Bradley–Terry/Thurstone-like. If a teacher has rationality parameter $\beta \ge 0$, the probability of preferring $\sigma_1$ over $\sigma_2$ is

$s_t$0

with $s_t$1 denoting the logistic function (Daniels-Koch et al., 2022). In the survey literature this same structure appears as a Bradley–Terry/Luce preference model over returns $s_t$2, with reward-model training given by a cross-entropy loss on preferred and dispreferred trajectories (Kaufmann et al., 2023).

For reward-model learning from labeled comparisons $s_t$3, a standard objective is logistic cross-entropy: $s_t$4 Once a reward model is learned, policy optimization is often regularized toward a reference policy: $s_t$5 This KL-regularized form is central in LLM RLHF and also appears in control-oriented formulations (Daniels-Koch et al., 2022).

A distinct theoretical perspective treats pairwise preference learning as density estimation rather than reward maximization. Under the Luce choice rule,

$s_t$6

and minimizing the binary cross-entropy preference loss yields a reward satisfying

$s_t$7

Under this view, the learned reward recovers the annotator’s implicit preference distribution up to a constant, and model misspecification becomes a mismatch between the assumed and actual generative processes for preferences (Dumoulin et al., 2023).

3. Standard pipelines and alternative formulations

The canonical LLM RLHF pipeline has four stages: supervised fine-tuning on demonstrations, collection of preference comparisons, reward-model training, and KL-regularized policy optimization, typically with PPO (Kaufmann et al., 2023). This pipeline is widely used because it separates preference elicitation from policy optimization, but it inherits the limitations of scalar reward modeling, including reward overfitting and dependence on the preference model’s assumptions (Kaufmann et al., 2023).

Direct optimization methods remove the explicit reinforcement-learning loop. DPO is described as a method that directly optimizes policy parameters from pairwise preference data without an explicit reinforcement-learning loop, and one recent study uses instruction-tuned Llama as a post-LHF model where the instruction tuning uses DPO (Juzek et al., 3 Aug 2025). Related direct methods in the survey literature include SLiC-HF, OPPO/DPPO, PRO, and rejection-sampling-based approaches, all of which avoid some of the engineering overhead of PPO-style RLHF while retaining a preference objective (Kaufmann et al., 2023).

A more radical alternative is Nash Learning from Human Feedback. NLHF learns a preference model conditioned on two responses and then seeks a policy that is preferred over those generated by any competing policy, thereby defining the Nash equilibrium of the induced preference game (Munos et al., 2023). In the tabular setting, Nash-MD is a mirror-descent algorithm with last-iterate convergence to the regularized Nash equilibrium, and the deep-learning version replaces scalar reward optimization with policy-gradient updates against a learned pairwise preference model (Munos et al., 2023). This construction is motivated by the claim that scalar reward models cannot represent non-transitive human preferences and remain dependent on the response sampling distribution used during training (Munos et al., 2023).

LHF also includes natural-language supervision that is richer than rankings. A critique-and-revise formulation fine-tunes an open-source LLM on prompt, initial response, critique, and revision records. With one iteration of revision of ChatGPT responses, the revised responses achieved a 56.6% win rate over the original ones, and this win rate improved to 65.9% after five iterations (Jin et al., 2023). This suggests that, in some settings, textual critiques and revisions can replace the reward-model-plus-RL decomposition with direct supervised alignment on human feedback expressed in language.

4. Heterogeneity, expertise, and aggregation of human judgments

A central assumption of much RLHF work is that feedback can be pooled into a single reward model. Several papers challenge that assumption directly. “The Expertise Problem” formalizes a setting with multiple teachers whose reliability depends both on who provides the label and on which part of the task the query probes. In that formulation, the constant rationality parameter $s_t$8 is replaced by a teacher- and query-dependent $s_t$9, and some queries are “inter-domain,” with low reliability for all teachers (Daniels-Koch et al., 2022). Empirically, on cartpole balance, “Max-$a_t$0 + Hybrid” achieved $a_t$1 final episode reward versus $a_t$2 for “Unif + Dis,” and on walker walk, “Max-$a_t$3 + Sim” achieved $a_t$4 versus $a_t$5 for “Unif + Dis,” directly illustrating the cost of treating all annotators as equally informative (Daniels-Koch et al., 2022).

A broader treatment of heterogeneous preferences introduces two families of solutions. The personalization-based framework learns multiple reward models, either through shared representation learning or via clustering, to trade off the bias induced by pooled heterogeneous preferences against the variance induced by splitting the data across personalized models (Park et al., 2024). In the representation-learning version, personalized rewards take the form

$a_t$6

with shared representation $a_t$7 and labeler-specific parameters $a_t$8 (Park et al., 2024).

The aggregation-based framework keeps a single-model deployment but changes how preferences are combined. Reward aggregation is parameterized by

$a_t$9

where $s_{t+1}$0 yields utilitarian aggregation and $s_{t+1}$1 tends to a Leximin-like rule (Park et al., 2024). The same paper studies direct aggregation of probabilistic opinions and develops a mechanism-design approach that is dominant-strategy incentive compatible and social-welfare maximizing under KL- or $s_{t+1}$2-Rényi-based distance functions (Park et al., 2024). The underlying point is that heterogeneity is not only a statistical nuisance but also a normative problem: different aggregation rules encode different welfare principles.

5. Querying, active learning, and multimodal interaction

LHF is increasingly treated as an active data-collection problem rather than a passive annotation stage. RLHF-Blender is a configurable, interactive system that collects diverse feedback types in one interface, encodes them in a unified schema, and plugs them into modular reward-model training and RL pipelines while logging metadata about irrationality, noise, bias, workload, and progress effects (Metz et al., 2023). This tooling perspective makes the data-collection protocol itself a first-class object of study.

Adaptive querying extends that idea algorithmically. Adaptive Feedback Selection learns a penalty function for negative side effects from multiple forms of human feedback by selecting both the queried state and the feedback format. Its second phase uses a cost- and availability-weighted bandit utility,

$s_{t+1}$3

to choose among Approval, Annotated Approval, Corrections, Annotated Corrections, Ranking, Demo-Action Mismatch, and Gaze (Anand et al., 2024). Across Navigation, Vase, Push, and Freeway, the framework is reported to be sample efficient and to match or outperform non-adaptive baselines on average side-effect penalty while maintaining competitive task cost (Anand et al., 2024).

Corrective-feedback systems make human intervention even more direct. Gaussian Process Coach replaces COACH’s RBF parameterization with Gaussian processes for both the policy and the human-feedback model. It uses predictive uncertainty to adapt the learning rate,

$s_{t+1}$4

and to drive active querying through $s_{t+1}$5, yielding better convergence speed, final performance, and robustness to erroneous feedback on OpenAI Gym continuous-control benchmarks (Wout et al., 2019).

Multimodal embodied agents require denser and more temporally localized feedback. “Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback” replaces whole-trajectory comparisons with inter-temporal preferences derived from within-episode progress and regress marks. The learned utility $s_{t+1}$6 induces a dense per-step reward,

$s_{t+1}$7

and training uses 5,104,000 binary marks across 364,690 episodes in a 3D environment (Abramson et al., 2022). The resulting BC+RL agent achieved 89% success in live interactive evaluation, representing 93% of the success rate of humans placed in the same role (Abramson et al., 2022). This broadens LHF from static preference annotation to continuous human evaluation of multimodal, temporally extended behavior.

6. Robustness, misspecification, and contested objectives

Much of the recent literature focuses on the fact that standard Bradley–Terry-based RLHF can be misspecified. A density-estimation analysis calls this “annotator misspecification”: if a model assumes a single Luce-style annotator but the data are generated by a mixture of annotators, the learned policy can collapse to a middle region preferred by neither annotator (Dumoulin et al., 2023). The same paper identifies length bias as a concrete misspecification in autoregressive settings and proposes a length-normalized Luce rule as a correction (Dumoulin et al., 2023).

Several works answer misspecification with explicitly robust objectives. Distributionally robust RLHF formulates TV-ball DRO versions of reward modeling, KL-regularized policy optimization, and DPO. On RewardBench, the standard BT reward model averaged about 71.8%, while the robust BT model with TV radius $s_{t+1}$8 reached about 74.7%; on the Reasoning subset, the score improved from 65.2% to 76.3% (Mandal et al., 1 Mar 2025). Another line introduces contrastive rewards that subtract a prompt-specific baseline from the reward-model score; the stated effects are to penalize reward uncertainty, encourage improvement over baselines, calibrate according to task difficulty, and reduce variance in PPO (Shen et al., 2024). A third paper proposes variance-reduced preference optimization under reward-model misspecification and reports that 77–81% of responses are favored over baselines on the Anthropic Helpful and Harmless dataset (Ye et al., 3 Apr 2025).

An information-theoretic strand treats RLHF as exploration under expensive preference queries. Information-Directed Sampling adds a mutual-information term to the episode objective and yields a Bayesian regret upper bound of order

$s_{t+1}$9

with tabular specialization $k$0 (Qi et al., 8 Feb 2025). This suggests that LHF is not only an estimation problem but also a query-allocation problem in which human feedback should be spent where it is most informative.

The notion of “alignment” itself is contested. One study on lexical overuse treats LHF as including RLHF and DPO and argues that LHF can amplify subtle rater preferences for specific stylistic markers. In its experimental emulation, high-LHF-Score variants were preferred overall at 52.4% versus 47.6%, with $k$1 and $k$2, yet the same paper emphasizes a divergence between LHF workers and some end users, making lexical overuse “a sort of misalignment” rather than a universal preference signal (Juzek et al., 3 Aug 2025).

Survey work broadens this into a governance critique. Only 9 of 50 “present” papers in one survey reported demographics; roughly 20 workers contributed about 80% of feedback in one helpful/harmless assistant study, and in WebGPT the top 5 contributed about 50% (Kirk et al., 2023). The same survey frames five unresolved challenges: preferences and values are not universal, they are inconsistently defined, human feedback is inherently incomplete, operationalizing “good” outputs is difficult, and there is a risk of “the tyranny of crowdworkers” (Kirk et al., 2023). This suggests that LHF is not only a technical apparatus for reward learning or policy optimization, but also a social procedure for selecting whose judgments count, how disagreement is represented, and which trade-offs are treated as legitimate.