---
title: In-Context Learning from Human Feedback
url: https://www.emergentmind.com/topics/in-context-learning-from-human-feedback-iclhf
type: topic
---

# In-Context Learning from Human Feedback

In-context learning from human feedback (ICLHF) denotes a family of methods in which a model changes its behavior within an interaction by conditioning on human feedback placed in the context window, rather than relying only on immediate weight updates. In recent arXiv work, this idea appears in several forms: direct robot planning from textual instructions, adjustments, and physical feedback [2508.07606]; in-context reward-program improvement from human preferences over rollout videos [2410.17233]; multi-turn incorporation of natural-language critiques on reasoning tasks, studied in a scalable synthetic proxy setting [2602.16066]; and hybrid systems that use interaction traces to improve future teachability through finetuning or iterative refinement [2402.11450], [2303.16755]. The resulting literature does not describe a single canonical algorithm. Instead, it defines a problem family centered on feedback-conditioned adaptation, trajectory-local revision, and the conversion of conversational feedback into either transient in-context behavior or more persistent capability.

## 1. Scope and variants

The literature distinguishes between strict ICLHF and closely related proxy or hybrid settings. “In-situ Value-aligned Human-Robot Interactions with Physical Constraints” explicitly introduces **In-Context Learning from Human Feedback (ICLHF)** for robot task planning, where feedback comes from direct instructions and from human adjustments made intentionally or unintentionally in daily life, and where physical feedback is also injected as text into the prompt [2508.07606]. “ICPL: Few-shot In-context Preference Learning via LLMs” presents a different but directly relevant formulation in which an LLM iteratively improves executable reward functions by conditioning on accumulated human preference feedback over videos, without parameter-updating the LLM itself [2410.17233]. “Improving Interactive In-Context Learning from Natural Language Feedback” studies a scalable proxy: the feedback provider is typically a simulated teacher model with privileged information, so the work is best viewed as ICL from natural-language feedback that approximates ICLHF in a synthetic setting [2602.16066].

Other papers occupy broader positions in the design space. “Learning to Learn Faster from Human Feedback with Language Model Predictive Control” studies multi-turn human corrections in robot teaching and then consolidates successful interaction trajectories via supervised finetuning, thereby extending short-horizon in-context adaptation into persistent learning [2402.11450]. “Training Language Models with Language Feedback at Scale” uses human-written critiques to generate and select refinements, then finetunes on those refinements; it is therefore a hybrid of in-context critique use and offline imitation learning rather than pure prompt-only adaptation [2303.16755]. “Reinforced Interactive Continual Learning via Real-time Noisy Human Feedback” is even farther from the frozen-model interpretation: it explicitly updates parameters online from streaming human corrections and is best understood as an outer-loop analogue of ICLHF rather than ICLHF in the narrow sense [2505.09925]. By contrast, “Auto-ICL: In-Context Learning without Human Supervision” removes human-authored prompt context through self-generation, but it does not implement human feedback at all and is therefore adjacent rather than direct evidence for ICLHF [2311.09263].

| Formulation | Feedback source | Relation to ICLHF |
|---|---|---|
| ICLHF in robot planning [2508.07606] | Direct instructions, human adjustments, physical feedback | Explicit use of the term |
| ICPL [2410.17233] | Human best/worst video preferences | In-context preference learning |
| RL\(^2\)F [2602.16066] | Teacher-model natural-language critiques | Scalable proxy for human feedback |
| LMPC [2402.11450] | Multi-turn human corrections | Hybrid: in-context plus consolidation |
| ILF [2303.16755] | Human natural-language critiques | Hybrid critique-conditioned refinement |
| RiCL [2505.09925] | Real-time noisy corrective labels | Online continual-learning analogue |
| Auto-ICL [2311.09263] | None; self-generated context | Adjacent, not feedback-based |

This suggests that ICLHF is better understood as a research program than as a single standardized protocol. The common denominator is not a particular optimizer, but the use of human-originating feedback as a behavior-shaping signal inside the interaction itself.

## 2. Formalizations of feedback-conditioned adaptation

A recurring formal move is to treat interactive feedback as part of a sequential decision process rather than as a post hoc supervision label. In RL\(^2\)F, learning from language feedback is formalized as a partially observable Markov decision process
\[
\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, T, R\rangle,
\]
with a student policy \(\pi_S\) and a teacher policy \(\pi_T\). The observation at time \(t\) is the conversation history
\[
o_t = (u_1^T, u_1^S, \dots, u_t^T, u_t^S),
\]
while the state includes both this history and privileged information \(k\),
\[
s_t = (k_t, o_t).
\]
The student sees only the conversation, whereas the teacher sees additional hidden task information such as the ground-truth answer, reference solution, or unit test outputs [2602.16066]. The importance of this formulation is that feedback becomes an in-context information channel under information asymmetry, not merely an offline training label.

LMPC uses a closely related POMDP view for human-robot teaching. The hidden state is the user’s latent intent, the human text input \(o_t\) is a partial observation, the robot code \(a_t\) is the action, and success or failure terminates the episode. Training a decoder-only Transformer to complete full interaction sequences is interpreted as learning a transition or dynamics model of the POMDP, which then supports model predictive control over future dialogue-and-code trajectories [2402.11450]. This links ICLHF to world modeling: a model that can predict how feedback will evolve can choose responses that reduce future corrections.

ICPL shifts the formal object of inference from answers or policies to reward programs. It defines a Markov Decision Process with Preferences
\[
M=\langle\mathcal S, A, \mu,\sigma, \gamma,\rho\rangle,
\]
and a Reward Design Problem with Preferences
\[
P=\langle M,\mathcal R, A_M, \zeta\rangle,
\]
where the goal is to find a reward function \(R \in \mathcal R\) such that the policy \(\pi = A_M(R)\) maximally complies with the preference set \(\zeta\) [2410.17233]. Here, human feedback does not directly update a policy; it steers a prompt-conditioned search over executable reward code.

The robot-planning ICLHF paper introduces a different formalization by replacing scalar reward with text:
\[
R: \mathcal{S} \times \mathcal{A} \to \mathbb{R}
\]
is replaced by
\[
R^*: \mathcal{S} \times \mathcal{A} \to \mathbb{F}^{\text{text}}.
\]
In this view, both human preferences and physical constraints are represented as textual feedback and injected into the LLM’s prompt [2508.07606]. The formal shift is conceptually important: the reward channel is no longer numeric and exogenous, but linguistic and interactional.

## 3. Feedback modalities and algorithmic patterns

The most characteristic feedback modality for ICLHF is free-form natural-language critique. RL\(^2\)F converts single-turn verifiable tasks into multi-turn didactic interactions: the student first attempts the problem, a verifier checks correctness, and if the answer is wrong the teacher produces a natural-language critique or hint that does not reveal the answer, after which the student retries. Because the tasks are verifiable, these interactions can be generated automatically at scale. The paper reports that teacher answer leakage is rare, revealing the solution in only **0.3%** of cases on average in the evaluation setup, and elsewhere notes direct reveal occurs in less than **1%** of cases [2602.16066]. The training objective is RL fine-tuning on multi-turn didactic trajectories, named **Reinforcement Learning with Language Feedback (RL\(^2\)F)**, with sparse reward for eventual correctness.

LMPC uses a different pattern: interaction trajectories are first collected from human-robot teaching, then successful sessions are replayed as sequence-modeling data. The method introduces **LMPC-Rollouts**, trained to predict the remaining portion of a successful chat session, and **LMPC-Skip**, trained to predict only the final successful action. At inference time, LMPC-Rollouts samples **8 rollouts** with non-zero temperature, allows up to **4096 tokens**, chooses among terminated rollouts the one with the fewest predicted chat turns, executes only the first action, and then replans after receiving actual human feedback [2402.11450]. In effect, the model is used both as policy and as predictive simulator of how feedback-conditioned interaction will unfold.

ICPL takes human feedback in a low-bandwidth preference form. Each round, an LLM generates \(K=6\) executable reward functions, PPO trains a policy under each reward, videos of the trained policies are rendered, and a human selects the most preferred and least preferred videos. The prompt for the next round accumulates the good reward code, bad reward code, reward component values during training, reward traces, and LLM-generated textual differences between historical good rewards [2410.17233]. The adaptation is therefore in-context and cumulative, but the optimized object is reward code rather than a textual response.

The robot-planning ICLHF system combines two feedback loops. The upper loop learns and applies human preferences from direct instructions and from human adjustments in daily life, which may be explicit, implicit, intentional, or subconscious. The lower loop handles physical feedback such as execution failures, collisions, and collapses. The LLM planner outputs an execution sequence and a rough goal scene graph; a customized POG-based pose synthesizer instantiates geometry, checks collision and stability properties, and returns physical feedback as text for replanning [2508.07606]. The whole process continues until the generated plan conforms to the physical constraints and human preferences.

Hybrid systems convert in-context feedback into more persistent training targets. ILF proceeds by generating an initial output, collecting human-written language feedback on the most important shortcoming, sampling several critique-conditioned refinements, selecting the refinement that best incorporates the feedback, and finetuning the base model on the selected refinement [2303.16755]. RiCL instead receives streaming model predictions and noisy human corrective labels, partitions the stream into clean and noisy subsets with a **Temporal Consistency-aware Purifier (TCP)**, applies **Interaction-aware Direct Preference Optimization (IPO)** on clean data, and uses **Noise-resistant Contrastive Learning (NCL)** on noisy data [2505.09925]. These methods do not define ICLHF strictly, but they show how interaction traces can be stabilized, filtered, or consolidated once prompt-only adaptation is insufficient.

## 4. Evaluation protocols and empirical findings

A central evaluation principle is that ICLHF should be measured across turns rather than only at the first response. RL\(^2\)F therefore uses **cumulative accuracy**, defined as the percentage of problems correctly solved by a given number of turns. This makes the model’s learning curve over interaction depth the primary object of study. On HardMath2, ARC-AGI, BIG-Bench Extra Hard, and Codeforces, current flagship models still leave “a lot of room for improvement” under multi-turn teacher-guided interaction. For math-trained **Gemini 2.5 Flash**, Table 1 reports out-of-domain teacher-interaction transfer averages of **31.93** for baseline Gemini 2.5 Flash, **31.86** for single-turn RL, and **38.86** for RL\(^2\)F; by task, RL\(^2\)F reaches **23.56** on multi-turn ARC AGI versus **20.47** for both baselines, **37.03** on multi-turn Codeforces versus **32.77/33.33**, and **56.00** on Linguini versus **42.35/42.00**. Table 2 reports broader multi-turn environments with no teacher at inference, where RL\(^2\)F yields an average performance of **51.65**, compared with **46.54** for single-turn RL and **46.92** for the base model, outperforming baselines on **7/10** tasks, including **Maze Navigation** (**87.50** vs **75.00** base) and **Only Connect Wall** (**72.00** vs **53.00** base). The paper also highlights that multi-turn RL fine-tuning lets Gemini 2.5 Flash nearly reach Gemini 2.5 Pro on HardMath2, while single-turn RL only slightly improves interactive learning ability [2602.16066].

LMPC evaluates **teachability**, defined as the average number of human inputs or chat turns before the robot succeeds. On **78 tasks** across **5 robot embodiments**, the abstract reports that the method improves non-expert teaching success rates on unseen tasks by **26.9%** while reducing the average number of human corrections from **2.4** to **1.9**. The detailed test-task table gives **39.4%** success, **2.4** chat turns, and **18.1%** good rating for PaLM 2-S, versus **66.3%**, **1.9**, and **26.5%** for LMPC-Rollouts. The same table shows the characteristic distinction between the two variants: LMPC-Skip has higher **1-turn success** on test tasks (**41.7%** vs **34.8%**), while LMPC-Rollouts has much higher **2+ turn success** (**31.5%** vs **17.8%**). In cross-embodiment transfer to unseen embodiments and APIs, LMPC-Skip improves test embodiments by **+18.6%** and LMPC-Rollouts by **+31.5%** [2402.11450]. These results isolate a specifically feedback-driven effect: modeling future corrective interaction improves later-turn adaptation more than simply predicting the final answer.

ICPL measures human query efficiency in reward learning. With \(K=6\) videos per round and \(N=5\) rounds, the paper defines the total number of human queries as
\[
Q = (K-1)\times 2N - 1 = 49.
\]
At this budget, ICPL substantially outperforms preference-based RL baselines trained tabula rasa from preferences. On Ant, for example, Table 1 reports **0.743** for PrefPPO-49, **4.458** for PrefPPO-1.5k, **4.626** for PrefPPO-15k, **10.86** for Eureka, and **12.04** for ICPL. On Humanoid, the corresponding numbers are **0.457**, **1.329**, **1.317**, **9.059**, and **9.227**. The paper states that ICPL reduces human effort by **at least 30x** relative to PrefPPO for comparable performance, and argues that the improvement curves over iterations are on average monotonic, suggesting preference learning rather than random search [2410.17233].

The household-robot ICLHF paper evaluates three levels of outcome: scene graph quality, action sequence quality, and preference learning/application. In table-tidying experiments without semantic information, ICL yields **12.98** feasibility versus **8.44** without ICL, **0.95** preference learning cosine similarity versus **0.47**, and **85.48** preference application versus **66.67**. With semantic information, the same comparison gives **26.45** versus **10.74** for feasibility, **0.86** versus **0.39** for preference learning, and **93.33** versus **89.02** for preference application [2508.07606]. The paper’s system-level simulation experiments further report that planning with physical constraints only produces feasible but preference-insensitive plans, while planning with preferences only can yield impractical or unexecutable plans; the combined ICLHF loop is designed to satisfy both.

ILF evaluates critique-conditioned refinement directly. On the TL;DR summarization task, refinement without feedback beats the initial summaries at **59.4 ± 2.1%** win rate, refinement with feedback reaches **63.9 ± 2.0%**, and refinement with feedback plus best-of-5 selection reaches **69.1 ± 1.9%**. Human judges rate incorporation of the most important feedback point at **30.8 ± 2.1%** without feedback, **49.6 ± 2.2%** with feedback, **57.4 ± 2.2%** with feedback plus best-of-5, and **74.0 ± 1.9%** for human summaries. In finetuning evaluations on **5K** samples, ILF reaches **31.3 ± 1.7%** win rate versus human-written references, compared with **28.9 ± 1.7%** for finetuning on human summaries; in a separate appendix evaluation, ILF reaches **36.0 ± 1.8%**, OPT-RM best-of-64 FeedME reaches **45.1 ± 1.9%**, and the combined **ILF + OPT-RM (best-of-64)** reaches **50.8 ± 1.9%** [2303.16755]. These numbers show both the usefulness and the incompleteness of critique following.

RiCL evaluates continual adaptation under noisy human-like corrections with **Average Performance (AP)** and **Average Forgotten (AF)**. On TACRED at **20%** noise, RiCL achieves **82.70%** AP and **1.69%** AF, compared with **74.31%** AP and **5.88%** AF for I-LoRA+SL. On FewRel at **20%** noise, RiCL reaches **84.77%** AP and **7.90%** AF, compared with **81.92%** AP and **9.38%** AF for I-LoRA+SL. The ablation study shows that removing TCP gives the largest degradation, while removing IPO mostly harms forgetting rather than immediate AP [2505.09925]. Although this is not prompt-only ICLHF, it quantifies the systems problem of retaining and filtering interactive feedback over long horizons.

## 5. Relation to RLHF, preference learning, and self-generated context

ICLHF is often compared with RLHF, but the two notions are not equivalent. Standard RLHF typically collects preferences, trains a reward model, and optimizes a policy against that reward. Several of the relevant papers depart from this template. ICPL does not learn a neural reward model from scratch; it uses preferences to refine reward code in context and then relies on PPO only after reward synthesis [2410.17233]. The robot-planning ICLHF paper similarly avoids parameter training of the planner and instead uses text-based feedback plus replanning [2508.07606]. LMPC uses supervised sequence modeling and model predictive control rather than reward-model learning or policy-gradient optimization [2402.11450]. ILF frames language feedback as a richer supervision channel than pairwise comparisons and shows theoretically that its refine-select-finetune loop can be viewed as Bayesian inference, but the practical update is imitation learning on selected refinements, not RL [2303.16755].

A second common confusion is to equate ICLHF with generic few-shot prompting. The feedback-centric papers make a sharper distinction. In RL\(^2\)F, the target capability is not mere instruction following, but the ability to revise a latent hypothesis or reasoning trajectory after critique; the paper interprets the gains as improved **in-context plasticity** [2602.16066]. In LMPC, the quantity of interest is teachability under successive corrections rather than one-shot performance [2402.11450]. In robot planning, the upper loop summarizes preference traces and the lower loop integrates physical feedback, so the context is an evolving memory of interaction rather than a static prompt [2508.07606].

The literature also separates feedback-driven adaptation from self-generated prompt construction. Auto-ICL shows that demonstrations, instructions, and reasoning traces can often be generated from the query or from unlabeled related queries, and reports averages of **68.1** in the generating setting and **75.7** in the retrieving setting, compared with **63.9** for Zero-Shot-CoT and **48.3** for Few-Shot [2311.09263]. Yet Auto-ICL explicitly lacks human critiques, preference labels, ranking, or reward modeling. It therefore reduces human-written in-context supervision rather than implementing learning from human feedback. A plausible implication is that self-generated context and ICLHF solve complementary problems: one reduces prompt construction cost, while the other targets interaction-conditioned correction.

Across the papers, verbal critique and preference signals appear complementary rather than mutually exclusive. ILF finds that language feedback improves candidate generation and that comparison-based reward models improve candidate selection; their combination achieves the strongest reported summarization performance [2303.16755]. RiCL similarly converts noisy corrective labels into preference-style supervision through IPO while also maintaining a contrastive channel for uncertain data [2505.09925]. RL\(^2\)F goes further by training on full dialogues, including teacher turns, so that external critique can become an internal capability for self-correction [2602.16066].

## 6. Limitations, controversies, and open problems

The main limitation of current ICLHF evidence is that much of it does not use unrestricted real human feedback at scale. RL\(^2\)F relies primarily on a simulated teacher with privileged information and is therefore a proxy for human feedback rather than a direct human-annotation study [2602.16066]. RiCL simulates human feedback by corrupting ground-truth labels with symmetric noise, so its robustness results concern noisy corrective labels rather than actual online annotator behavior [2505.09925]. ICPL includes real human trials, but much of its quantitative comparison uses proxy preferences induced from human-designed rewards [2410.17233]. This suggests that real-world ICLHF remains underdetermined by current benchmarks.

A second limitation concerns context length, memory, and consolidation. LMPC is motivated precisely by the fact that ordinary in-context learning forgets earlier corrections once they leave the context window, and its persistent gains come from night-time finetuning on successful sessions [2402.11450]. The robot-planning ICLHF system likewise introduces periodic introspection to summarize lower-level preferences into higher-level characteristics when maximum context length is reached [2508.07606]. These mechanisms indicate that prompt-only adaptation may need structured memory or selective consolidation for long interaction histories.

A third issue is robustness to heterogeneous or misleading feedback. ILF shows that even with best-of-5 selection, the most important critique is incorporated only **57.4%** of the time, and that reward or judge selection procedures can be sensitive to feedback style [2303.16755]. RiCL’s TCP exists because blindly updating on all incoming corrections amplifies errors and harms retention [2505.09925]. RL\(^2\)F explicitly notes future concerns about whether increasing adaptability might also increase undesirable properties such as **sycophancy** [2602.16066]. These results complicate the simple intuition that more feedback responsiveness is always beneficial.

A fourth unresolved point is the trade-off between preference satisfaction and external constraints. The household-robot ICLHF system resolves this operationally through iterative replanning with human and physical feedback, but it does not provide a precise scalar trade-off equation between preference and feasibility [2508.07606]. ICPL similarly depends on environment code, human-evaluable videos, and repeated PPO training, which makes it difficult to separate preference inference from task-specific program-search priors [2410.17233]. More generally, several systems rely on strong base-model competence to bootstrap useful interactions at all [2402.11450], [2311.09263].

The literature nevertheless converges on one substantive claim. The relevant question is not only whether models can be trained on feedback data, but whether they can **learn how to learn from language feedback within the context window** [2602.16066]. That formulation places ICLHF at the intersection of in-context adaptation, preference learning, continual interaction, and structured memory. Current work shows multiple viable instantiations of that objective, but it also shows that the distinction between transient prompt-level correction and durable capability remains one of the field’s central technical tensions.

Source: https://www.emergentmind.com/topics/in-context-learning-from-human-feedback-iclhf