Learning Personalized Agents from Human Feedback
This presentation introduces the PAHF framework, a novel approach to continual personalization in AI agents that addresses fundamental limitations of static personalization methods. The framework uses explicit per-user memory and dual feedback channels—pre-action clarification and post-action correction—to enable agents to adapt efficiently from cold start, handle ambiguous user preferences, and robustly respond to preference drift over time. Through theoretical analysis and empirical evaluation across embodied manipulation and online shopping domains, the work demonstrates that both feedback channels are necessary and complementary for achieving low-regret personalization in interactive settings.Script
What if your AI assistant could actually learn your preferences over time, not just once, but continuously adapting as your needs evolve? This paper tackles the fundamental challenge of making agents that personalize through ongoing interaction rather than static profiles.
Let's start by understanding why existing personalization methods fall short.
Building on that challenge, the authors identify three critical failure modes in current approaches. Static methods based on logs or fixed profiles cannot handle new users, cannot recover from mistakes during interaction, and completely break down when user preferences change over time.
The researchers propose a fundamentally different approach to solve these problems.
Their solution is the PAHF framework, which stands for Personalized Agents from Human Feedback. This approach maintains explicit memory for each user and operates through two complementary feedback channels that work together to minimize personalization errors over time.
This example beautifully illustrates how both feedback channels work in practice. When the agent first encounters Kate, it proactively asks about her drink preference before acting. The next day, when context matters, the agent initially overgeneralizes but learns from correction. Then when Kate's underlying preference shifts entirely, the post-action feedback enables the agent to adapt to this drift without catastrophic failure.
The theoretical analysis reveals why you need both channels. Without the ability to correct after actions, agents accumulate unbounded errors as preferences shift because they have no mechanism to detect drift. Conversely, without proactive clarification, agents must learn every preference through mistakes, leading to unnecessary errors and user frustration during the critical initial interactions.
Looking at the empirical results in the embodied manipulation domain, we see dramatic differences across the four phases of evaluation. In Phase 1, agents with pre-action feedback achieve higher success rates from the start by resolving ambiguity before acting. Then in Phase 3, after preferences drift, only agents with post-action correction can adapt, while pre-action only agents remain stuck with outdated assumptions.
Across two very different domains, embodied scene manipulation and online shopping, the full PAHF system consistently outperforms all baselines. The framework handles both the initial cold-start challenge and the adaptation challenge after preference drift, validating that the dual feedback architecture is necessary for real-world continual personalization.
This work establishes a new foundation for personalized agents that can truly adapt alongside their users. To explore the full theoretical proofs and implementation details, visit EmergentMind.com.
