Learning Personalized Agents from Human Feedback

Abstract: Modern AI agents are powerful but often fail to align with the idiosyncratic, evolving preferences of individual users. Prior approaches typically rely on static datasets, either training implicit preference models on interaction history or encoding user profiles in external memory. However, these approaches struggle with new users and with preferences that change over time. We introduce Personalized Agents from Human Feedback (PAHF), a framework for continual personalization in which agents learn online from live interaction using explicit per-user memory. PAHF operationalizes a three-step loop: (1) seeking pre-action clarification to resolve ambiguity, (2) grounding actions in preferences retrieved from memory, and (3) integrating post-action feedback to update memory when preferences drift. To evaluate this capability, we develop a four-phase protocol and two benchmarks in embodied manipulation and online shopping. These benchmarks quantify an agent's ability to learn initial preferences from scratch and subsequently adapt to persona shifts. Our theoretical analysis and empirical results show that integrating explicit memory with dual feedback channels is critical: PAHF learns substantially faster and consistently outperforms both no-memory and single-channel baselines, reducing initial personalization error and enabling rapid adaptation to preference shifts.

• The paper proposes the PAHF framework that minimizes personalization error via explicit per-user memory and dual feedback channels.
• It introduces a two-phase feedback mechanism—pre-action clarification and post-action correction—to address cold-start issues and preference drift.
• Empirical results in embodied and online shopping domains demonstrate significant improvements in success rates and reduction in cumulative error.

Continual Personalization in Agents: The PAHF Framework

Motivation and Limitations of Static Personalization

The personalization of LLM-based agents remains an unsolved and highly consequential problem, especially as these systems deploy in interactive user-facing applications. Existing implicit and static approaches—whether via log-based fine-tuning, profile retrieval, or role/persona conditioning—suffer from an inability to adapt to cold-start users, correct errors from interaction, or handle preference drift over time. Such methods are structurally incapable of minimizing cumulative personalization error in online settings due to fundamental limitations in coping with partial observability and non-stationarity of user preferences.

Figure 1: Static personalization (top) relies on fixed logs and profiles, whereas continual personalization (bottom) leverages in-loop user feedback, dynamically updating explicit memory to adapt to changing preferences.

The PAHF Loop: Explicit Memory and Dual Feedback Channels

This work proposes the Personalized Agents from Human Feedback (PAHF) framework, operationalizing continual, sample-efficient online personalization via explicit per-user memory coupled with dual human feedback channels.

Formal Problem Setting: At each round $t$, a user exhibits a latent preference state $M_t^*$, issues instruction $I_t$ with observation $O_t$, and the agent relies on its estimate $\hat M_t$ to select action $a_t$. The objective is to learn a policy minimizing cumulative personalization loss, where sources of error include insufficient knowledge ($\hat M_t$ incomplete) and miscalibration ($\hat M_t$ no longer reflects $M_t^*$).

The core loop of PAHF is as follows:

1. Pre-Action Interaction: On ambiguity, the agent either retrieves relevant notes or clarifies directly with the user, updating memory with salient pre-action feedback before acting. This selectively resolves uncertainty, optimal particularly when the agent is aware it does not know.
2. Action Execution: Actions are selected using retrieved and/or freshly acquired context, tightly conditioning decision-making on the latest personalized knowledge.
3. Post-Action Feedback Integration: On error (especially due to drift), the agent incorporates explicit user correction after acting, updating or revising memory to align with the new user profile, regardless of overconfidence.

   Figure 2: Continual personalization requires both pre-action clarification (to resolve ambiguity) and post-action correction (for preference drift), supporting robust adaptation to evolving user states.

Theoretical Analysis: Necessity and Complementarity of Feedback Channels

Theoretical analysis in the paper establishes the necessity for both proactive (pre-action) and reactive (post-action) feedback:

• Without post-action feedback, agents incur $\Omega(T)$ regret under even $K=1$ preference shifts, as they cannot distinguish drift without feedback.
• Without pre-action feedback, initial ambiguity leads to linear error in the ambiguity rate $\gamma$.
• The complementarity result: With at most $K$ switch points (drift), ambiguity rate $\gamma$, and $k$-ary pre-queries, the expected regret is $O\left(K + \gamma T m^{-k}\right)$; with $k = \Theta(\log_m T)$, it is $O(K + \gamma)$. This formalizes that only the union of clarification and correction yields robust, low-regret continual personalization.

Implementation: PAHF Agent System

Agent Architecture: The proposed PAHF agent is instantiable with any LLM (default: GPT-4o), and is agnostic to memory backend (experiments on SQLite and FAISS). Interaction-conforming agentic memory is used, with each human feedback annotated for salience and integrated (summarized/updated) under strict deduplication for per-user notes.

Baselines: Four settings are compared—incrementally isolating the effect of memory and feedback: no memory; pre-action only (clarification, no correction); post-action only (no clarification, reactive correction only); and full PAHF (both channels enabled).

Domains: Evaluation spans (1) embodied scene manipulation—contextualized, physically grounded tasks with highly idiosyncratic personas—and (2) online shopping—feature taxonomy-constrained selection with adversarial attribute distractors and rigorous preference hierarchies, both with systematic preference drift for adaptation benchmarking.

Experimental Protocol and Metrics

A four-phase protocol comprehensively isolates initial personalization, drift, and adaptation:

• Phase 1: Initial online learning from cold start.
• Phase 2: Test of learned memory on novel tasks (no feedback).
• Phase 3: Exposure to drift—profile shift, memory now miscalibrated.
• Phase 4: Post-adaptation evaluation.

Core metrics: Success Rate (SR), Feedback Frequency (FF), and Average Cumulative Personalization Error (ACPE).

Results: Empirical Benefits of Continual Personalization

Learning Curves: Agents with pre-action feedback dramatically reduce initial error under ambiguity, outperforming both post-action only and no-memory baselines in SR/ACPE in Phase 1, as they avoid unnecessary trial-and-error.

Figure 3: Embodied manipulation, Phase 1 (top)—pre-action feedback yields faster success and lower ACPE; Phase 3 (bottom)—post-action and PAHF enable rapid adaptation after drift.

Figure 4: Online shopping, performance mirrors embodied domain—a synergy of both channels is required for robust continual personalization, especially after preference drift (Phase 3).

• Pre-action only: Quickly reduces uncertainty but is brittle to drift; after preferences change, overconfident reliance on stale notes means it stops asking questions, and cannot recover.
• Post-action only: Allows eventual correction after drift but with large early errors—learning occurs only by incurring user corrections.
• PAHF: Highest success rates across all phases; promptly aligns to new users, avoids initial regret, and robustly corrects memory after drift.

Quantitative results (Table summarization): In both domains, PAHF consistently achieves the best Phase 2 and Phase 4 SR (e.g., embodied: 70.5%/68.8%; shopping: 41.3%/70.3%), demonstrating the necessity of dual feedback and explicit memory.

Implications and Future Directions

The PAHF framework directly addresses and theoretically resolves limitations shared by prior works on preference-based RLHF, static persona emulation, and RAG-driven personalization. It establishes a new minimal requirement for continual agent alignment in the online, real-world setting, with explicit architectural modularity (memory backend agnostic) to absorb future improvements in retrieval capability or LLM capacity.

Practically, PAHF scales to both embodied and digital agent scenarios, supporting persistent user modeling and immediate adaptation. Theoretically, it unifies continual learning, co-adaptation, and user-in-the-loop RL paradigms under a single feedback-driven loop. Future research should focus on scalability via hierarchical/structured memory, robust handling of noisy or inconsistent feedback, and enriched clarification strategies that minimize user burden while further reducing regret. Integration with advanced RAG and lifelong reasoning modules is also directly enabled by PAHF’s pluggable architecture.

Conclusion

PAHF demonstrates that robust, continual personalization in modern AI agents is only attainable via explicit memory and the orchestration of both proactive clarification and reactive correction channels. This resolves core failure modes inherent to existing methods, yielding agents that not only personalize efficiently for cold-start users but also adaptively align through persona evolution in live interaction. The theoretical necessity proofs and empirical benchmarks position PAHF as a baseline for future personalized agent research.