Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shopping (2510.07230v1)

Abstract: Simulating step-wise human behavior with LLMs has become an emerging research direction, enabling applications in various practical domains. While prior methods, including prompting, supervised fine-tuning (SFT), and reinforcement learning (RL), have shown promise in modeling step-wise behavior, they primarily learn a population-level policy without conditioning on a user's persona, yielding generic rather than personalized simulations. In this work, we pose a critical question: how can LLM agents better simulate personalized user behavior? We introduce Customer-R1, an RL-based method for personalized, step-wise user behavior simulation in online shopping environments. Our policy is conditioned on an explicit persona, and we optimize next-step rationale and action generation via action correctness reward signals. Experiments on the OPeRA dataset emonstrate that Customer-R1 not only significantly outperforms prompting and SFT-based baselines in next-action prediction tasks, but also better matches users' action distribution, indicating higher fidelity in personalized behavior simulation.

• The paper introduces a personalized simulation framework that integrates explicit user personas with reinforcement learning to model online shopping behavior.
• It employs GRPO and dual reward mechanisms (action and format) to substantially improve next-action prediction accuracy compared to traditional methods.
• Experimental results on the OPeRA dataset demonstrate that including persona data significantly enhances the fidelity of user behavior simulations.

Personalized Simulation of Human Behaviors in Online Shopping

The paper "Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shopping" (2510.07230) presents an innovative approach to simulate personalized human behavior through reinforcement learning (RL) powered by LLMs. This work addresses critical challenges in personalized user behavior simulation, proposing a method that incorporates explicit user personas to enhance the fidelity of online shopping simulations.

Introduction to Personalized User Behavior Simulation

Customer-R1 aims to improve the simulation of user behavior in online shopping environments by combining reinforcement learning with explicit user personas. Traditional approaches generally focus on population-level behaviors and often miss the nuances of individual preferences and styles. Customer-R1 differs by conditioning the simulation on user-specific persona data, allowing for a more faithful representation of how distinct users might behave when browsing or purchasing items online.

The framework is designed to observe past user actions in an online shopping session and predict future actions, leveraging explicit persona information to tailor predictions according to individual tendencies.

Figure 1: User Behavior Simulation in Online Shopping. The model observes a sequence of historical user actions and learns to reason over this behavioral context to predict the user’s next action.

Methodology of Customer-R1

Model Architecture

Customer-R1 uses reinforcement learning to model the sequential nature of shopping actions, with the policy conditioned on a rich set of user persona attributes. The decision-making process is informed by a context that includes historical user interactions and the current state of the shopping environment, as seen through the HTML structure of the page.

Reward Design

The reward system in Customer-R1 is crafted to promote both accuracy and semantic coherence in predicted actions. The reward function incorporates two main components: action rewards for correctness and format rewards ensuring compliance with expected JSON schema structures. This dual-faceted reward approach helps align the model's actions with real-world human behaviors.

Figure 2: Customer-R1 Framework for Simulating User Behavior in Online Shopping. The model observes user history behaviors in a session composed of HTML observations $$o_1, \dots,o_{t-1}..., a session includes real user persona$$P$.

By employing Group Relative Policy Optimization (GRPO), the framework effectively optimizes the action policy, utilizing a combination of context and persona-driven information to predict subsequent user actions accurately.

Experimental Results and Analysis

Dataset and Evaluation Metrics

Experiments were conducted on the OPeRA dataset, which provides a rich set of user interaction logs and persona profiles. The model's performance was evaluated using metrics such as Next Action Generation Accuracy, Action Type F1 Score, Fine-grained Type Accuracy, and Session Outcome F1 Score.

Results

The Customer-R1 framework demonstrated superior performance in next-action prediction tasks, particularly when juxtaposed against traditional baselines relying solely on supervised fine-tuning or prompting techniques. As shown, the inclusion of persona information and the nuanced reward mechanism significantly boosted prediction accuracy and the alignment of simulated actions with real user behavior distributions.

Impact of Persona Information

Incorporating persona data proved crucial in enhancing the personalization of behavior simulations. The ablation studies, which involved removing persona information, highlighted a noticeable drop in prediction performance, underscoring the importance of persona data in accurately modeling user-specific actions.

Figure 3: Fine-grained action distribution. a) Model trained using RL only. b) Model trained using SFT+RL. c) Model trained using SFT+RL without persona.

Concluding Remarks

Customer-R1 introduces a meaningful advancement in the domain of personalized behavior simulation in e-commerce environments. By conditioning reinforcement learning algorithms on detailed user personas, the proposed method achieves a more nuanced understanding and prediction of user actions, leading to better-aligned simulated shopping experiences. This work not only advances the technical methodology for human behavior simulation but also lays the groundwork for future explorations in other domains where personalized user interaction is crucial.

The demonstrated improvements call attention to the potential of personalized simulations driven by RL frameworks, suggesting avenues for further research in incorporating richer persona data and exploring other domains outside online shopping. Future work could involve extending these concepts to broader interaction contexts, such as personalized digital assistants and adaptive user interfaces, to continuously refine and personalize user experiences based on observed behaviors and explicit persona insights.