Shop-R1: RL Framework for Shopping Simulation

• Shop-R1 is a reinforcement learning framework that simulates human shopping behavior by optimizing natural-language rationales and fine-grained action sequences.
• It employs a two-stage process where rationale generation is rewarded via self-certainty and action prediction receives hierarchical, difficulty-aware rewards.
• Empirical results show a 65% boost in exact-match accuracy over standard methods, validating its effectiveness in complex multi-step shopping simulations.

Shop-R1 refers to a reinforcement learning (RL) framework designed to enhance LLMs' (LLMs) ability to simulate human behavior in online shopping environments by optimizing both natural-language rationales and fine-grained action sequences with hierarchical, difficulty-aware rewards. Unlike previous approaches constrained by the reasoning power of synthetic rationale generators or by reward sparsity, Shop-R1 introduces a novel decomposition of the behavior simulation problem, structurally distinct reward signals for each phase, and quantitative evidence of significant performance gains (Zhang et al., 23 Jul 2025).

1. Problem Definition and Motivation

Shop-R1 addresses the task of human behavior simulation in multi-step, web-based shopping sessions. At each time step $t$, the simulated agent receives an observation $c_t$ (web page state) and must output (1) a natural-language rationale $r_t$ describing its reasoning and (2) an action $a_t$ selected from an action space $\mathcal{A}$, incorporating both high-level choices (e.g., "type_and_submit", "click", "terminate") and detailed sub-actions (e.g., specific textual query, button name). The optimization goal is not simply task completion but the high-fidelity emulation of real human decision trajectories, necessitating reward signals that go beyond binary supervision and directly incentivize human-like intermediate steps. This formulation is formalized as learning a conditional policy

$$(r_t, a_t) \sim f_\theta\left(c_{1\ldots t}, a_{1\ldots t-1}, r_{1\ldots t-1}\right),$$

with rewards reflecting both reasoning plausibility and action correctness (Zhang et al., 23 Jul 2025).

2. Two-Stage Decomposition and Reward Assignment

Shop-R1 innovatively decomposes each decision into two stages:

• Rationale Generation: The model is first prompted to generate a natural-language rationale. The quality of $r_t$ is evaluated using an internal self-certainty measure derived from the model's logit distributions across the sequence. This is quantified as the average Kullback–Leibler (KL) divergence from the uniform distribution:

$$s(r_t\mid q_t) = \frac{1}{L\,|V|}\sum_{j=1}^L\sum_{i=1}^{|V|} p_{ij}\,\log\Bigl(\frac{p_{ij}}{1/|V|}\Bigr)$$

where high certainty yields greater reward. Notably, this mechanism circumvents the need for ground-truth human rationales, instead using model confidence as a reward (Zhang et al., 23 Jul 2025).

• Action Prediction: The second phase predicts the next environment action, with a hierarchical partial-credit reward structure:
  • Coarse action type correctness ($R_\mathrm{type}$, +0.3 for correct type),
  • Sub-action attribute presence ($R_\mathrm{attr}$, +0.2 or +0.1 for required slots),
  • Fine-grained textual similarity ($c_t$0) via ROUGE-L, amplified for hard tasks by the DARS (Difficulty-Aware Reward Scaling) factor (default 1000).

The overall reward for $c_t$1 is

$c_t$2

This staged, granular crediting enables smooth learning and avoids degenerate policies, such as always choosing the trivial "terminate" action (Zhang et al., 23 Jul 2025).

3. Reinforcement Learning Algorithm and Policy Optimization

Shop-R1 utilizes an off-policy RL algorithm with additional KL regularization toward a reference (SFT-initialized) policy. The objective is

$c_t$3

where $c_t$4 and $c_t$5 control the rationale-reward and regularization weights, respectively. The optimizer (a PPO variant, "Group Relative Policy Optimization") maximizes a clipped surrogate based on ratio of policy probabilities, maintaining stability. The policy is warm-started from supervised fine-tuning (SFT) on human-labeled sessions for effective formatting and handling of long text fields (Zhang et al., 23 Jul 2025).

4. Empirical Results and Comparative Analysis

Shop-R1 is benchmarked on a proprietary e-commerce corpus (52,000 human sessions with SFT targets), using Qwen-2.5-3B-Instruct as the backbone and compared to several baselines:

Setting	Exact Acc (%)	Type Acc (%)	Type F1 (%)
Zero-shot prompting	0.32	15.33	16.15
RL (Binary)	1.01	6.17	9.92
SFT	16.76	22.25	24.52
SFT + RL (Binary)	16.55	23.74	28.07
Shop-R1	27.72	36.40	31.28

Shop-R1 outperforms supervised and RL (binary-reward) baselines, establishing a 65% relative gain in exact-match accuracy over SFT, and substantial improvements in action-type fidelity and F1 (Zhang et al., 23 Jul 2025).

5. Ablation Studies and Mechanistic Insights

• Removing SFT initialization degrades exact-match to 4.6%, highlighting its necessity for format and output generalization.
• Eliminating the format reward collapses parseable output rate (type F1 ≈ 3–5%, exact ≈ 2.9%), confirming its critical role.
• Excluding DARS or using binary rewards maintains exact-match performance, but type F1 drops to ~11–12%, with the policy degenerating to "terminate" actions.
• Omitting rationale self-certainty yields only minor drop, but leads to observable decline in rationale coherence.

Qualitatively, Shop-R1-trained models produce more detailed, confident rationales and select complex, contextually appropriate actions, while binary-trained models often default to easy or generic behaviors (Zhang et al., 23 Jul 2025).

6. Significance of Hierarchical and Difficulty-Aware Rewarding

The theoretical and empirical properties of Shop-R1's reward hierarchy can be summarized as:

• Hierarchical structure provides stepwise credit assignment, facilitating learning of both the overall structure (correct action type) and details (correct sub-action or text).
• Difficulty-aware scaling (DARS) ensures reward is commensurate with task complexity, which is crucial in avoiding exploitation of reward signals (i.e., resorting to trivial actions).
• Self-certainty for rationales aligns internal model learning dynamics with desired external behaviors. A plausible implication is that improved rationale confidence mediates better sub-action selection.

These mechanisms jointly enable robust policy learning under sparse, compositional objectives that typify human web interaction (Zhang et al., 23 Jul 2025).

7. Broader Implications and Future Directions

Shop-R1 contributes a modular, generalizable framework for aligning LLM-generated, multi-step rationalizing agents with observed human behavior in structured environments. The reward schedule and staged optimization are applicable beyond shopping simulation, potentially extending to educational, navigation, and interactive agent scenarios. Future research may explore adaptive DARS tuning, interpretable rationale validation, and integration with richer multimodal input streams. The explicit reward structures employed prevent degenerate exploitation and improve transfer to novel or complex tasks (Zhang et al., 23 Jul 2025).