Few-shot In-Context Preference Learning Using Large Language Models (2410.17233v1)

Abstract: Designing reward functions is a core component of reinforcement learning but can be challenging for truly complex behavior. Reinforcement Learning from Human Feedback (RLHF) has been used to alleviate this challenge by replacing a hand-coded reward function with a reward function learned from preferences. However, it can be exceedingly inefficient to learn these rewards as they are often learned tabula rasa. We investigate whether LLMs can reduce this query inefficiency by converting an iterative series of human preferences into code representing the rewards. We propose In-Context Preference Learning (ICPL), a method that uses the grounding of an LLM to accelerate learning reward functions from preferences. ICPL takes the environment context and task description, synthesizes a set of reward functions, and then repeatedly updates the reward functions using human rankings of videos of the resultant policies. Using synthetic preferences, we demonstrate that ICPL is orders of magnitude more efficient than RLHF and is even competitive with methods that use ground-truth reward functions instead of preferences. Finally, we perform a series of human preference-learning trials and observe that ICPL extends beyond synthetic settings and can work effectively with humans-in-the-loop. Additional information and videos are provided at https://sites.google.com/view/few-shot-icpl/home.

Few-shot In-context Preference Learning Using LLMs

The paper presents a novel method, In-Context Preference Learning (ICPL), aimed at enhancing reinforcement learning (RL) agents by integrating human preferences using LLMs for reward function generation. This approach addresses inefficiencies in traditional Reinforcement Learning from Human Feedback (RLHF) by leveraging the generative capabilities of LLMs to autonomously create reward functions and iteratively refine them based on human feedback.

Core Contributions

1. ICPL Methodology:
   • The paper introduces ICPL, which utilizes LLMs to generate executable reward functions. This process begins with an environment's context and task description which the LLM uses to produce initial reward functions.
   • ICPL iteratively refines these functions through human-in-the-loop feedback, selecting the most and least preferred outcomes from a set of generated videos and utilizing these preferences to guide the LLM in further iterations.
2. Performance and Efficiency:
   • ICPL demonstrates significant improvements in sample efficiency, reducing preference query requirements by orders of magnitude compared to traditional RLHF.
   • Its efficacy is shown across several reinforcement learning benchmarks, highlighting its scalability and robustness.
3. Experimental Validation:
   • Synthetic preference trials illustrate that ICPL achieves over 30 times reduction in preference queries while maintaining performance comparable to or exceeding traditional methods.
   • Human-in-the-loop experiments confirm the practical applicability of ICPL, showing its ability to guide complex tasks with human feedback effectively.

Numerical Results and Analysis

The paper reports strong numerical results demonstrating ICPL’s efficacy. For instance, in tasks using proxy human preferences, ICPL outperforms conventional methods like PrefPPO by achieving higher task scores significantly faster. These improvements are attributed to the method's ability to iteratively fine-tune reward functions directly aligned with human preferences rather than relying on an abstract model of reward feedback.

The paper also details the algorithm's capability to handle diverse and complex environments, reinforcing its robustness. The average reward task score (RTS) improvement over iterations is nearly monotonic, indicating a consistent enhancement in alignment between the learned reward functions and human preferences.

Implications and Future Directions

The introduction of ICPL holds both theoretical and practical implications. Theoretically, it suggests the potential for LLMs to inherently understand and incorporate human preferences directly into RL processes, bypassing the need for extensive manual reward modeling. Practically, ICPL could drastically reduce the cost and time associated with training AI systems to align with human expectations, particularly in complex or subjective tasks such as emulating human-like movements.

Future research directions could explore optimizing the diversity of initial reward functions, further integrating AI with human-centric tasks, and enhancing automatic feedback mechanisms. Additionally, ICPL could benefit from investigating hybrid systems that integrate both human preferences and automated metrics for more comprehensive evaluation criteria.

Conclusion

The paper successfully demonstrates that ICPL can efficiently utilize LLMs for preference learning in reinforcement learning. The methodology not only advances the state-of-the-art in aligning RL agents' behaviors with human expectations but also sets the stage for future innovations in AI driven by nuanced human feedback. The findings validate ICPL as a compelling alternative to traditional RLHF methods, showcasing its potential for broader application across diverse AI domains.