In-Context Learning from Human Feedback

• ICLHF is a paradigm that integrates human-originating feedback—such as instructions, adjustments, and critiques—directly into a model's context to enable transient in-context adaptation.
• It encompasses various formulations, including robot planning, reward-program refinement, and multi-turn natural language feedback, each leveraging iterative correction without traditional weight updates.
• Evaluation focuses on multi-turn performance and teachability, highlighting trade-offs between preference satisfaction and practical constraints while driving continuous model improvement.

In-context learning from human feedback (ICLHF) denotes a family of methods in which a model changes its behavior within an interaction by conditioning on human feedback placed in the context window, rather than relying only on immediate weight updates. In recent arXiv work, this idea appears in several forms: direct robot planning from textual instructions, adjustments, and physical feedback (Li et al., 11 Aug 2025); in-context reward-program improvement from human preferences over rollout videos (Yu et al., 2024); multi-turn incorporation of natural-language critiques on reasoning tasks, studied in a scalable synthetic proxy setting (Klissarov et al., 17 Feb 2026); and hybrid systems that use interaction traces to improve future teachability through finetuning or iterative refinement (Liang et al., 2024, Scheurer et al., 2023). The resulting literature does not describe a single canonical algorithm. Instead, it defines a problem family centered on feedback-conditioned adaptation, trajectory-local revision, and the conversion of conversational feedback into either transient in-context behavior or more persistent capability.

1. Scope and variants

The literature distinguishes between strict ICLHF and closely related proxy or hybrid settings. “In-situ Value-aligned Human-Robot Interactions with Physical Constraints” explicitly introduces In-Context Learning from Human Feedback (ICLHF) for robot task planning, where feedback comes from direct instructions and from human adjustments made intentionally or unintentionally in daily life, and where physical feedback is also injected as text into the prompt (Li et al., 11 Aug 2025). “ICPL: Few-shot In-context Preference Learning via LLMs” presents a different but directly relevant formulation in which an LLM iteratively improves executable reward functions by conditioning on accumulated human preference feedback over videos, without parameter-updating the LLM itself (Yu et al., 2024). “Improving Interactive In-Context Learning from Natural Language Feedback” studies a scalable proxy: the feedback provider is typically a simulated teacher model with privileged information, so the work is best viewed as ICL from natural-language feedback that approximates ICLHF in a synthetic setting (Klissarov et al., 17 Feb 2026).

Other papers occupy broader positions in the design space. “Learning to Learn Faster from Human Feedback with LLM Predictive Control” studies multi-turn human corrections in robot teaching and then consolidates successful interaction trajectories via supervised finetuning, thereby extending short-horizon in-context adaptation into persistent learning (Liang et al., 2024). “Training LLMs with Language Feedback at Scale” uses human-written critiques to generate and select refinements, then finetunes on those refinements; it is therefore a hybrid of in-context critique use and offline imitation learning rather than pure prompt-only adaptation (Scheurer et al., 2023). “Reinforced Interactive Continual Learning via Real-time Noisy Human Feedback” is even farther from the frozen-model interpretation: it explicitly updates parameters online from streaming human corrections and is best understood as an outer-loop analogue of ICLHF rather than ICLHF in the narrow sense (Yang et al., 15 May 2025). By contrast, “Auto-ICL: In-Context Learning without Human Supervision” removes human-authored prompt context through self-generation, but it does not implement human feedback at all and is therefore adjacent rather than direct evidence for ICLHF (Yang et al., 2023).

Formulation	Feedback source	Relation to ICLHF
ICLHF in robot planning (Li et al., 11 Aug 2025)	Direct instructions, human adjustments, physical feedback	Explicit use of the term
ICPL (Yu et al., 2024)	Human best/worst video preferences	In-context preference learning
RL$^2$F (Klissarov et al., 17 Feb 2026)	Teacher-model natural-language critiques	Scalable proxy for human feedback
LMPC (Liang et al., 2024)	Multi-turn human corrections	Hybrid: in-context plus consolidation
ILF (Scheurer et al., 2023)	Human natural-language critiques	Hybrid critique-conditioned refinement
RiCL (Yang et al., 15 May 2025)	Real-time noisy corrective labels	Online continual-learning analogue
Auto-ICL (Yang et al., 2023)	None; self-generated context	Adjacent, not feedback-based

This suggests that ICLHF is better understood as a research program than as a single standardized protocol. The common denominator is not a particular optimizer, but the use of human-originating feedback as a behavior-shaping signal inside the interaction itself.

2. Formalizations of feedback-conditioned adaptation

A recurring formal move is to treat interactive feedback as part of a sequential decision process rather than as a post hoc supervision label. In RL$^2$F, learning from language feedback is formalized as a partially observable Markov decision process

$$\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, T, R\rangle,$$

with a student policy $\pi_S$ and a teacher policy $\pi_T$. The observation at time $t$ is the conversation history

$o_t = (u_1^T, u_1^S, \dots, u_t^T, u_t^S),$

while the state includes both this history and privileged information $k$,

$s_t = (k_t, o_t).$

The student sees only the conversation, whereas the teacher sees additional hidden task information such as the ground-truth answer, reference solution, or unit test outputs (Klissarov et al., 17 Feb 2026). The importance of this formulation is that feedback becomes an in-context information channel under information asymmetry, not merely an offline training label.

LMPC uses a closely related POMDP view for human-robot teaching. The hidden state is the user’s latent intent, the human text input $o_t$ is a partial observation, the robot code $^2$0 is the action, and success or failure terminates the episode. Training a decoder-only Transformer to complete full interaction sequences is interpreted as learning a transition or dynamics model of the POMDP, which then supports model predictive control over future dialogue-and-code trajectories (Liang et al., 2024). This links ICLHF to world modeling: a model that can predict how feedback will evolve can choose responses that reduce future corrections.

ICPL shifts the formal object of inference from answers or policies to reward programs. It defines a Markov Decision Process with Preferences

$^2$1

and a Reward Design Problem with Preferences

$^2$2

where the goal is to find a reward function $^2$3 such that the policy $^2$4 maximally complies with the preference set $^2$5 (Yu et al., 2024). Here, human feedback does not directly update a policy; it steers a prompt-conditioned search over executable reward code.

The robot-planning ICLHF paper introduces a different formalization by replacing scalar reward with text: $^2$6 is replaced by

$^2$7

In this view, both human preferences and physical constraints are represented as textual feedback and injected into the LLM’s prompt (Li et al., 11 Aug 2025). The formal shift is conceptually important: the reward channel is no longer numeric and exogenous, but linguistic and interactional.

3. Feedback modalities and algorithmic patterns

The most characteristic feedback modality for ICLHF is free-form natural-language critique. RL$^2$8F converts single-turn verifiable tasks into multi-turn didactic interactions: the student first attempts the problem, a verifier checks correctness, and if the answer is wrong the teacher produces a natural-language critique or hint that does not reveal the answer, after which the student retries. Because the tasks are verifiable, these interactions can be generated automatically at scale. The paper reports that teacher answer leakage is rare, revealing the solution in only 0.3% of cases on average in the evaluation setup, and elsewhere notes direct reveal occurs in less than 1% of cases (Klissarov et al., 17 Feb 2026). The training objective is RL fine-tuning on multi-turn didactic trajectories, named Reinforcement Learning with Language Feedback (RL$^2$9F), with sparse reward for eventual correctness.

LMPC uses a different pattern: interaction trajectories are first collected from human-robot teaching, then successful sessions are replayed as sequence-modeling data. The method introduces LMPC-Rollouts, trained to predict the remaining portion of a successful chat session, and LMPC-Skip, trained to predict only the final successful action. At inference time, LMPC-Rollouts samples 8 rollouts with non-zero temperature, allows up to 4096 tokens, chooses among terminated rollouts the one with the fewest predicted chat turns, executes only the first action, and then replans after receiving actual human feedback (Liang et al., 2024). In effect, the model is used both as policy and as predictive simulator of how feedback-conditioned interaction will unfold.

ICPL takes human feedback in a low-bandwidth preference form. Each round, an LLM generates $$\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, T, R\rangle,$$0 executable reward functions, PPO trains a policy under each reward, videos of the trained policies are rendered, and a human selects the most preferred and least preferred videos. The prompt for the next round accumulates the good reward code, bad reward code, reward component values during training, reward traces, and LLM-generated textual differences between historical good rewards (Yu et al., 2024). The adaptation is therefore in-context and cumulative, but the optimized object is reward code rather than a textual response.

The robot-planning ICLHF system combines two feedback loops. The upper loop learns and applies human preferences from direct instructions and from human adjustments in daily life, which may be explicit, implicit, intentional, or subconscious. The lower loop handles physical feedback such as execution failures, collisions, and collapses. The LLM planner outputs an execution sequence and a rough goal scene graph; a customized POG-based pose synthesizer instantiates geometry, checks collision and stability properties, and returns physical feedback as text for replanning (Li et al., 11 Aug 2025). The whole process continues until the generated plan conforms to the physical constraints and human preferences.

Hybrid systems convert in-context feedback into more persistent training targets. ILF proceeds by generating an initial output, collecting human-written language feedback on the most important shortcoming, sampling several critique-conditioned refinements, selecting the refinement that best incorporates the feedback, and finetuning the base model on the selected refinement (Scheurer et al., 2023). RiCL instead receives streaming model predictions and noisy human corrective labels, partitions the stream into clean and noisy subsets with a Temporal Consistency-aware Purifier (TCP), applies Interaction-aware Direct Preference Optimization (IPO) on clean data, and uses Noise-resistant Contrastive Learning (NCL) on noisy data (Yang et al., 15 May 2025). These methods do not define ICLHF strictly, but they show how interaction traces can be stabilized, filtered, or consolidated once prompt-only adaptation is insufficient.

4. Evaluation protocols and empirical findings

A central evaluation principle is that ICLHF should be measured across turns rather than only at the first response. RL$$\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, T, R\rangle,$$1F therefore uses cumulative accuracy, defined as the percentage of problems correctly solved by a given number of turns. This makes the model’s learning curve over interaction depth the primary object of study. On HardMath2, ARC-AGI, BIG-Bench Extra Hard, and Codeforces, current flagship models still leave “a lot of room for improvement” under multi-turn teacher-guided interaction. For math-trained Gemini 2.5 Flash, Table 1 reports out-of-domain teacher-interaction transfer averages of 31.93 for baseline Gemini 2.5 Flash, 31.86 for single-turn RL, and 38.86 for RL$$\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, T, R\rangle,$$2F; by task, RL$$\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, T, R\rangle,$$3F reaches 23.56 on multi-turn ARC AGI versus 20.47 for both baselines, 37.03 on multi-turn Codeforces versus 32.77/33.33, and 56.00 on Linguini versus 42.35/42.00. Table 2 reports broader multi-turn environments with no teacher at inference, where RL$$\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, T, R\rangle,$$4F yields an average performance of 51.65, compared with 46.54 for single-turn RL and 46.92 for the base model, outperforming baselines on 7/10 tasks, including Maze Navigation (87.50 vs 75.00 base) and Only Connect Wall (72.00 vs 53.00 base). The paper also highlights that multi-turn RL fine-tuning lets Gemini 2.5 Flash nearly reach Gemini 2.5 Pro on HardMath2, while single-turn RL only slightly improves interactive learning ability (Klissarov et al., 17 Feb 2026).

LMPC evaluates teachability, defined as the average number of human inputs or chat turns before the robot succeeds. On 78 tasks across 5 robot embodiments, the abstract reports that the method improves non-expert teaching success rates on unseen tasks by 26.9% while reducing the average number of human corrections from 2.4 to 1.9. The detailed test-task table gives 39.4% success, 2.4 chat turns, and 18.1% good rating for PaLM 2-S, versus 66.3%, 1.9, and 26.5% for LMPC-Rollouts. The same table shows the characteristic distinction between the two variants: LMPC-Skip has higher 1-turn success on test tasks (41.7% vs 34.8%), while LMPC-Rollouts has much higher 2+ turn success (31.5% vs 17.8%). In cross-embodiment transfer to unseen embodiments and APIs, LMPC-Skip improves test embodiments by +18.6% and LMPC-Rollouts by +31.5% (Liang et al., 2024). These results isolate a specifically feedback-driven effect: modeling future corrective interaction improves later-turn adaptation more than simply predicting the final answer.

ICPL measures human query efficiency in reward learning. With $$\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, T, R\rangle,$$5 videos per round and $$\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, T, R\rangle,$$6 rounds, the paper defines the total number of human queries as

$$\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, T, R\rangle,$$7

At this budget, ICPL substantially outperforms preference-based RL baselines trained tabula rasa from preferences. On Ant, for example, Table 1 reports 0.743 for PrefPPO-49, 4.458 for PrefPPO-1.5k, 4.626 for PrefPPO-15k, 10.86 for Eureka, and 12.04 for ICPL. On Humanoid, the corresponding numbers are 0.457, 1.329, 1.317, 9.059, and 9.227. The paper states that ICPL reduces human effort by at least 30x relative to PrefPPO for comparable performance, and argues that the improvement curves over iterations are on average monotonic, suggesting preference learning rather than random search (Yu et al., 2024).

The household-robot ICLHF paper evaluates three levels of outcome: scene graph quality, action sequence quality, and preference learning/application. In table-tidying experiments without semantic information, ICL yields 12.98 feasibility versus 8.44 without ICL, 0.95 preference learning cosine similarity versus 0.47, and 85.48 preference application versus 66.67. With semantic information, the same comparison gives 26.45 versus 10.74 for feasibility, 0.86 versus 0.39 for preference learning, and 93.33 versus 89.02 for preference application (Li et al., 11 Aug 2025). The paper’s system-level simulation experiments further report that planning with physical constraints only produces feasible but preference-insensitive plans, while planning with preferences only can yield impractical or unexecutable plans; the combined ICLHF loop is designed to satisfy both.

ILF evaluates critique-conditioned refinement directly. On the TL;DR summarization task, refinement without feedback beats the initial summaries at 59.4 ± 2.1% win rate, refinement with feedback reaches 63.9 ± 2.0%, and refinement with feedback plus best-of-5 selection reaches 69.1 ± 1.9%. Human judges rate incorporation of the most important feedback point at 30.8 ± 2.1% without feedback, 49.6 ± 2.2% with feedback, 57.4 ± 2.2% with feedback plus best-of-5, and 74.0 ± 1.9% for human summaries. In finetuning evaluations on 5K samples, ILF reaches 31.3 ± 1.7% win rate versus human-written references, compared with 28.9 ± 1.7% for finetuning on human summaries; in a separate appendix evaluation, ILF reaches 36.0 ± 1.8%, OPT-RM best-of-64 FeedME reaches 45.1 ± 1.9%, and the combined ILF + OPT-RM (best-of-64) reaches 50.8 ± 1.9% (Scheurer et al., 2023). These numbers show both the usefulness and the incompleteness of critique following.

RiCL evaluates continual adaptation under noisy human-like corrections with Average Performance (AP) and Average Forgotten (AF). On TACRED at 20% noise, RiCL achieves 82.70% AP and 1.69% AF, compared with 74.31% AP and 5.88% AF for I-LoRA+SL. On FewRel at 20% noise, RiCL reaches 84.77% AP and 7.90% AF, compared with 81.92% AP and 9.38% AF for I-LoRA+SL. The ablation study shows that removing TCP gives the largest degradation, while removing IPO mostly harms forgetting rather than immediate AP (Yang et al., 15 May 2025). Although this is not prompt-only ICLHF, it quantifies the systems problem of retaining and filtering interactive feedback over long horizons.

5. Relation to RLHF, preference learning, and self-generated context

ICLHF is often compared with RLHF, but the two notions are not equivalent. Standard RLHF typically collects preferences, trains a reward model, and optimizes a policy against that reward. Several of the relevant papers depart from this template. ICPL does not learn a neural reward model from scratch; it uses preferences to refine reward code in context and then relies on PPO only after reward synthesis (Yu et al., 2024). The robot-planning ICLHF paper similarly avoids parameter training of the planner and instead uses text-based feedback plus replanning (Li et al., 11 Aug 2025). LMPC uses supervised sequence modeling and model predictive control rather than reward-model learning or policy-gradient optimization (Liang et al., 2024). ILF frames language feedback as a richer supervision channel than pairwise comparisons and shows theoretically that its refine-select-finetune loop can be viewed as Bayesian inference, but the practical update is imitation learning on selected refinements, not RL (Scheurer et al., 2023).

A second common confusion is to equate ICLHF with generic few-shot prompting. The feedback-centric papers make a sharper distinction. In RL$$\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, T, R\rangle,$$8F, the target capability is not mere instruction following, but the ability to revise a latent hypothesis or reasoning trajectory after critique; the paper interprets the gains as improved in-context plasticity (Klissarov et al., 17 Feb 2026). In LMPC, the quantity of interest is teachability under successive corrections rather than one-shot performance (Liang et al., 2024). In robot planning, the upper loop summarizes preference traces and the lower loop integrates physical feedback, so the context is an evolving memory of interaction rather than a static prompt (Li et al., 11 Aug 2025).

The literature also separates feedback-driven adaptation from self-generated prompt construction. Auto-ICL shows that demonstrations, instructions, and reasoning traces can often be generated from the query or from unlabeled related queries, and reports averages of 68.1 in the generating setting and 75.7 in the retrieving setting, compared with 63.9 for Zero-Shot-CoT and 48.3 for Few-Shot (Yang et al., 2023). Yet Auto-ICL explicitly lacks human critiques, preference labels, ranking, or reward modeling. It therefore reduces human-written in-context supervision rather than implementing learning from human feedback. A plausible implication is that self-generated context and ICLHF solve complementary problems: one reduces prompt construction cost, while the other targets interaction-conditioned correction.

Across the papers, verbal critique and preference signals appear complementary rather than mutually exclusive. ILF finds that language feedback improves candidate generation and that comparison-based reward models improve candidate selection; their combination achieves the strongest reported summarization performance (Scheurer et al., 2023). RiCL similarly converts noisy corrective labels into preference-style supervision through IPO while also maintaining a contrastive channel for uncertain data (Yang et al., 15 May 2025). RL$$\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, T, R\rangle,$$9F goes further by training on full dialogues, including teacher turns, so that external critique can become an internal capability for self-correction (Klissarov et al., 17 Feb 2026).

6. Limitations, controversies, and open problems

The main limitation of current ICLHF evidence is that much of it does not use unrestricted real human feedback at scale. RL$\pi_S$0F relies primarily on a simulated teacher with privileged information and is therefore a proxy for human feedback rather than a direct human-annotation study (Klissarov et al., 17 Feb 2026). RiCL simulates human feedback by corrupting ground-truth labels with symmetric noise, so its robustness results concern noisy corrective labels rather than actual online annotator behavior (Yang et al., 15 May 2025). ICPL includes real human trials, but much of its quantitative comparison uses proxy preferences induced from human-designed rewards (Yu et al., 2024). This suggests that real-world ICLHF remains underdetermined by current benchmarks.

A second limitation concerns context length, memory, and consolidation. LMPC is motivated precisely by the fact that ordinary in-context learning forgets earlier corrections once they leave the context window, and its persistent gains come from night-time finetuning on successful sessions (Liang et al., 2024). The robot-planning ICLHF system likewise introduces periodic introspection to summarize lower-level preferences into higher-level characteristics when maximum context length is reached (Li et al., 11 Aug 2025). These mechanisms indicate that prompt-only adaptation may need structured memory or selective consolidation for long interaction histories.

A third issue is robustness to heterogeneous or misleading feedback. ILF shows that even with best-of-5 selection, the most important critique is incorporated only 57.4% of the time, and that reward or judge selection procedures can be sensitive to feedback style (Scheurer et al., 2023). RiCL’s TCP exists because blindly updating on all incoming corrections amplifies errors and harms retention (Yang et al., 15 May 2025). RL$\pi_S$1F explicitly notes future concerns about whether increasing adaptability might also increase undesirable properties such as sycophancy (Klissarov et al., 17 Feb 2026). These results complicate the simple intuition that more feedback responsiveness is always beneficial.

A fourth unresolved point is the trade-off between preference satisfaction and external constraints. The household-robot ICLHF system resolves this operationally through iterative replanning with human and physical feedback, but it does not provide a precise scalar trade-off equation between preference and feasibility (Li et al., 11 Aug 2025). ICPL similarly depends on environment code, human-evaluable videos, and repeated PPO training, which makes it difficult to separate preference inference from task-specific program-search priors (Yu et al., 2024). More generally, several systems rely on strong base-model competence to bootstrap useful interactions at all (Liang et al., 2024, Yang et al., 2023).

The literature nevertheless converges on one substantive claim. The relevant question is not only whether models can be trained on feedback data, but whether they can learn how to learn from language feedback within the context window (Klissarov et al., 17 Feb 2026). That formulation places ICLHF at the intersection of in-context adaptation, preference learning, continual interaction, and structured memory. Current work shows multiple viable instantiations of that objective, but it also shows that the distinction between transient prompt-level correction and durable capability remains one of the field’s central technical tensions.