---
title: 'ShopGym: E-Commerce Agent Simulation'
url: https://www.emergentmind.com/topics/shopgym
type: topic
---

# ShopGym: E-Commerce Agent Simulation

ShopGym is an overloaded term in recent e-commerce-agent research. In "ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents" [2605.16116], it denotes an integrated framework for constructing e-commerce simulation environments and grounded benchmark tasks, combining ShopArena for environment construction with ShopGuru for task synthesis. In "ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistants" [2601.18225], the structured description explicitly introduces ShopSimulator "hereafter called ShopGym" as a Gym-style environment for training and evaluating RL-driven LLM agents in realistic shopping scenarios. Across these usages, the common theme is a controlled yet realistic substrate for studying shopping agents, but the two systems emphasize different methodological priorities: reproducible simulation and scalable benchmarking in one case, and unified training-and-evaluation support with dialogue, personalization, and reward design in the other.

## 1. Terminological scope and research problem

The central problem shared by both usages is the lack of satisfactory environments for e-commerce web agents. One line of work states that existing methodologies force a tradeoff: live storefronts provide realism but are non-stationary, difficult to inspect, and irreproducible, while hand-built sandbox benchmarks provide control but cover only a narrow range of layouts, catalogs, policies, and interaction patterns [2605.16116]. The other line argues that existing research has yet to provide a unified simulation environment that consistently captures multi-turn dialogue, personalization, and fine-grained product discrimination, and always focuses solely on evaluation benchmarks without training support [2601.18225].

A common misconception is that ShopGym refers to a single benchmark. The record in the cited sources does not support that interpretation. Instead, the term names two related but distinct efforts. One is a framework for constructing sandbox shops and grounded tasks from live seed storefronts [2605.16116]. The other is a Gym-style API and dataset ecosystem centered on product search, dialogue, personalization, and RL-compatible rewards [2601.18225].

| Usage of "ShopGym" | Definition in source | Primary emphasis |
|---|---|---|
| ShopGym [2605.16116] | Integrated framework with ShopArena and ShopGuru | Realistic simulation and scalable benchmarking |
| ShopGym / ShopSimulator [2601.18225] | Gym-style environment for RL-driven LLM agents | Training and evaluation in shopping scenarios |

This suggests that "ShopGym" functions less as a singular canonical artifact than as a convergent label for e-commerce-agent environments that attempt to reconcile realism, control, and scientific comparability.

## 2. ShopGym as an integrated framework: ShopArena and ShopGuru

In the framework introduced in [2605.16116], ShopGym is composed of two tightly integrated layers—ShopArena for environment construction and ShopGuru for task synthesis—designed to bridge realism and experimental control in e-commerce agent evaluation. ShopArena converts live seed storefronts into self-contained sandbox shops through anonymized shop specifications and a staged, validated generation process. ShopGuru then synthesizes benchmark tasks across seven skill categories, grounding each task in the shop's catalog, navigation structure, policies, and interaction affordances.

ShopArena uses a two-phase pipeline. All live-web dependence is confined to an exploration phase, which autonomously crawls one or more seed storefronts and writes a human-readable, anonymized specification \(M\). A separate generation phase consumes \(M\) and only \(M\) to synthesize a runnable sandbox shop [2605.16116]. During exploration, a planner agent first inspects prefetched pages—homepage HTML, sitemap, catalog JSON, and policy pages—and writes a decomposition plan. A fleet of specification agents then executes targeted browser-automation subtasks via Playwright, capturing screenshots and XHR traces and emitting fragments of \(M\). A non-agentic LLM consolidates these fragments into a natural-language design manual, a structured capabilities manifest, and catalog and UI statistics.

The generation phase synthesizes mock product names, descriptions, and images from \(M\). A staged, feature-scoped source-code synthesis pipeline then builds the site in steps: site shell, homepage, collections, product pages, cart, search, and policy pages. Each step is driven by an iteration of an execution agent that edits code and runs build/type checks, together with verifiers consisting of rule-based checks plus a multimodal visual agent that validate functionality and UI. Early termination and a fixed iteration budget ensure reproducibility and debuggability [2605.16116].

ShopGuru operates on top of the sandbox shop. It ingests the sandbox shop’s catalog JSON, navigation graph, filter-and-sort manifest, and policy/page listings. It combines deterministic short-horizon generators, implemented as pure-function scripts, with LLM-authored long-horizon journeys. The short-horizon scripts sample products and collections, check feasibility via per-collection option indices, emit task JSON with natural-language intents plus structured `success_criteria`, and immediately validate all tasks against shop data with rule-based verifiers. For long-horizon tasks, a single LLM such as GPT-5 or Gemini 3 Pro is few-shot prompted with eight human-trace examples and shop context, then a validator-driven polish loop re-audits and refines any hallucinated or infeasible tasks, ensuring zero residual grounding errors before release [2605.16116].

## 3. Task models and skill coverage

The ShopGuru task layer in [2605.16116] covers seven skill categories, grouped into three short-horizon primitives and one long-horizon composite. The categories are: search-exact, search-substitute, browse, filter, shipping policy lookup, returns policy lookup, and end-to-end shopping journeys. The paper further organizes them as Discovery, Filter-Selection, Information-Seeking, and long-horizon E2E journeys.

The grounding sources are explicit. Search-exact and search-substitute use `products.json` with title and handle. The generator samples one active product per `product_type` and emits two variants: "Find product by its exact title" and "Find a semantically similar alternative." Verification ensures the target handle resolves to a real PDP URL [2605.16116]. Browse and filter tasks use `collections.json` plus a per-collection option index derived from the shop’s own products. The browse generator picks a non-generic collection with at least three products and asks the agent to navigate there and add any product. The filter generator pairs the same collection with a real facet \((\mathrm{dim}, \mathrm{value})\) only if at least one product supports it; otherwise it falls back to universal facets such as Vendor or ProductType.

Shipping and returns tasks are grounded in `pages.json` plus default `/policies` endpoints. The generator searches for page handles or titles matching keywords such as "shipping" and "return" and emits a task to find and read the page; `success_criteria` soft-hints URL patterns [2605.16116]. E2E shopping journeys are grounded in full shop context—collections, products, pages, and filter-sort options—together with few-shot examples reflecting real click-stream patterns. The LLM generator mixes navigation, filtering, sorting, policy detours, and cart edits, while a polish loop feeds back tasks that fail any of seven validator rules, including unknown collection, infeasible filter, or hallucinated variant, in order to regenerate only the flagged tasks.

In the Gym-style environment described in [2601.18225], the task model is different. The environment supports both single-turn and multi-turn modes, with and without personalization, and delivers train/test splits, instruction templates, and reward signals to enable SFT and RL research. The task pool contains Single-Turn Non-Pers. (6K train / 600 eval), Multi-Turn Non-Pers. (6K train / 600 eval), Single-Turn Pers. (4.7K train / 600 eval), and Multi-Turn Pers. (4.7K train / 600 eval), for a total of approximately 28K tasks with one-to-one mapping, where each natural-language instruction uniquely identifies one gold product [2601.18225]. Instructions written by human annotators avoid lexical overlap with product titles, and personalized instructions omit demographic cues, which are moved into profile fields.

The two task regimes therefore target different granularities. One emphasizes web-interaction skills grounded in concrete storefront structure; the other emphasizes agentic product search under dialogue, personalization, and purchase selection. A plausible implication is that the former is closer to controlled web-agent evaluation, while the latter is closer to RL-style sequential decision-making over shopping assistance trajectories.

## 4. Structural fidelity, behavioral evaluation, and benchmark methodology

A distinguishing feature of [2605.16116] is its explicit claim that realism should be validated structurally as well as behaviorally. The paper models each site as a directed UI-state graph \(G=(V,E)\), where nodes \(V\) are distinct UI states—page plus widget configurations—and edges \(E\) are user actions such as clicks and filter opens. Three graph-based metrics are used: degree distribution \(p(d)\), clustering coefficient \(C\), and average shortest-path length \(L\).

The formulas are given as follows:
$$
C \;=\; \frac{1}{|V|}\sum_{v\in V}\frac{2\,T(v)}{d(v)\,(d(v)-1)}
$$
where \(d(v)\) is the degree of node \(v\) and \(T(v)\) is the number of triangles through \(v\), and
$$
L \;=\; \frac{1}{|V|\,(|V|-1)}\sum_{i\neq j} d(i,j)
$$
where \(d(i,j)\) is the length of the shortest path from state \(i\) to \(j\) [2605.16116]. These metrics measure interaction branching, the tendency of UI states to form triadic connections, and the average number of interactions needed to navigate between any two states.

Behavioral evaluation in the same work uses six sandbox storefronts—three fully synthetic and three "twin shops" generated with real product data—and 224 tasks per shop, covering all seven skill categories. Three baseline agent architectures are evaluated: GPT-5-mini, Gemini 3 Flash, and GPT-5. Two harnesses are used: BrowserGym with AXTree only, and an internal multimodal harness with AXTree plus screenshots. Judging is performed by GPT-5 as LLM-judge, scoring binary success or fail on each trajectory against `success_criteria` plus a hard rule that the agent must explicitly signal task completion [2605.16116].

Success rates are computed per \((\mathrm{model}, \mathrm{shop}, \mathrm{task\ bundle})\) cell and averaged over three independent rollouts. Short-horizon tasks achieve high pass rates above 80% for all models, whereas long-horizon tasks remain challenging, with GPT-5 at approximately 62%, Gemini 3 Flash approximately 59%, and GPT-5-mini approximately 48% [2605.16116]. The paper further states that synthetic sandbox shops match the difficulty of real shops, showing comparable success rates across twin versus live storefronts.

The live-versus-synthetic correspondence is quantified using the Pearson correlation coefficient \(r\) between agent success rates on live versus twin shops:
$$
r \;=\; \frac{\sum_i (x_i - \bar x)\,(y_i - \bar y)}{\sqrt{\sum_i (x_i - \bar x)^2}\;\sqrt{\sum_i (y_i - \bar y)^2}}
$$
where \(x_i\) and \(y_i\) are success rates on live and twin shops respectively [2605.16116]. A strong positive \(r\), for example greater than 0.8 with \(p<0.01\), is reported as confirmation that synthetic environments faithfully preserve evaluative signal.

## 5. Gym-style API, state design, and reward functions

The environment described in [2601.18225] follows the Gym v0 API with minor extensions for dialogue. The API is episodic and stepwise, using `reset()` and `step(action) → (obs, reward, done, info)`. Observation is a rich text or dictionary describing UI state, user utterance, and user profile, rather than a fixed numerical vector. Actions are textual or structured commands such as `search[...]`, `click[...]`, and `ask_shopper[...]`. Termination occurs when the agent issues `click[buy now]` or reaches a max-step limit of 30–40 steps [2601.18225].

The observation dictionary contains `"ui_state"`, `"user_utterance"`, `"user_profile"` in personalization mode, `"dialogue_history"`, and `"instruction"` in single-turn mode. `"ui_state"` itself contains `search_available`, `clickable_buttons`, and `current_results`, where each result includes fields such as `"id"`, `"title"`, and `"price"`. The action space is a tuple \((\mathrm{action\_type}, \mathrm{content})\), with \(\mathrm{action\_type} \in \{\text{"ask\_shopper"}, \text{"interact\_with\_env"}\}\). If the type is `"ask_shopper"`, the content is a question string; otherwise the content must match one of `search[<keyword string>]` or `click[<button label>]` [2601.18225].

Episode structure is explicit. `reset()` samples one target product and its gold instruction, one user profile if personalized, and initializes an LLM-simulated shopper in "system" role with a hidden goal. At each step \(t\), the agent receives \(obs_t\), returns \(a_t\), the environment updates its UI and, if relevant, the user utterance, and then returns \((obs_{t+1}, reward_t, done, info)\). Product representations include `category_path`, `attributes`, `options`, and `price`. The `"user_profile"` contains demographic fields, long-term preference scores, and historical behavior such as past searches and purchase frequency [2601.18225].

The reward model is one of the most technically specific aspects of this usage of ShopGym. Two final-step reward variants are defined, together with a binary success metric. The loose additive reward is
$$
R_{\rm loose} = R_{\rm cat}\;\times\;
\frac{\;|U_{\rm att}\cap Y_{\rm att}| + |U_{\rm opt}\cap Y_{\rm opt}| + \mathbf{1}[Y_{\rm price}\le U_{\rm price}]\;}
{|U_{\rm att}| + |U_{\rm opt}| + 1}
\quad\in [0,1]
$$
and the strict multiplicative or bottleneck reward is
$$
R_{\rm strict} = R_{\rm cat}\;\times\;
\frac{|U_{\rm att}\cap Y_{\rm att}|}{|U_{\rm att}|}
\;\times\;
\frac{|U_{\rm opt}\cap Y_{\rm opt}|}{|U_{\rm opt}|}
\;\times\;
\mathbf{1}[Y_{\rm price}\le U_{\rm price}]
\quad\in[0,1].
$$
Here, \(U_{\rm att}\), \(U_{\rm opt}\), and \(U_{\rm price}\) are the target’s required attributes, options, and price limit; \(Y_{\rm att}\), \(Y_{\rm opt}\), and \(Y_{\rm price}\) are the purchased product’s values; and \(R_{\rm cat}\) softly measures category-path similarity. The full-success rate \(R_{\rm succ}\) is 1 if and only if all requirements are exactly met, otherwise 0 [2601.18225]. Intermediate rewards are zero by default for all non-terminal steps, though per-click or per-confirmation bonuses can be instrumented externally.

## 6. Empirical findings, reproducibility, and future extension

The empirical conclusions of the two works are complementary. In [2605.16116], the main result is that self-contained, resettable, inspectable, and stable evaluation artifacts can preserve structural properties and agent-evaluation signals relevant to shopping tasks. The paper states that the synthetic shops preserve key structural properties of live storefronts and that agent performance on synthetic shops is positively correlated with performance on live storefronts. It further argues that ShopGym unites the realism of live sites with the repeatability of sandboxes by splitting exploration from generation, with the human-readable specification \(M\) serving as an editing surface so benchmarks can be tweaked without re-crawling or modifying live sites [2605.16116].

In [2601.18225], the main empirical message is that the shopping-assistant problem remains difficult even in a purpose-built environment. The baseline benchmark reports that even the best-performing models achieve less than 40% full-success rate in the hardest multi-turn personalized setting, with substantial gaps between \(R_{\rm loose}\) at roughly 60%, \(R_{\rm strict}\) at roughly 35%, and \(R_{\rm succ}\) at roughly 30% [2601.18225]. Closed-source models tested include GPT-5, OpenAI-o3, GPT-4.1, Claude-4-Sonnet, and Gemini2.5 Pro; open-source models include DeepSeek-R1/V3.1, Kimi-K2, GLM-4.5, GPT-OSS, and the Qwen3 family from 8B to 235B. GPT-5 is reported as best overall at approximately 32.7% \(R_{\rm succ}\).

Training support is another point of divergence. The Gym-style ShopSimulator system explicitly supports supervised fine-tuning and reinforcement learning. It specifies approximately 6K expert trajectories from GPT-4.1 as pseudo-demonstrations for SFT, and RL via GRPO implemented through the ROLL library. The reported findings are that SFT + RL outperforms either alone across all four scenarios, with a 24–40% increase in \(R_{\rm succ}\); that strict reward yields better fine-grained matching than loose reward; and that single-turn settings yield larger gains post-training, while multi-turn remains challenging at roughly 35% \(R_{\rm succ}\) [2601.18225].

For reproducibility and extension, [2605.16116] emphasizes scalable benchmarking, scientific reproducibility through fixed shop snapshots and seed-fixed data generation, and modular extension to gated flows such as login and checkout, multilingual or localization settings, recommendation modules, or post-purchase interactions by enriching \(M\) and downstream generators. It also highlights interpretability and analysis through graph metrics and per-skill suites that can pinpoint strengths and weaknesses such as filter logic or long-horizon planning. A plausible implication is that the two usages of ShopGym occupy adjacent layers of a broader research stack: one constructs realistic and controllable web environments, while the other operationalizes shopping-agent learning and evaluation within a Gym-style interaction loop.

Taken together, the name "ShopGym" marks a developing methodological space in which e-commerce agents are studied under stronger experimental control than live-web evaluation permits, while retaining richer task structure than narrow hand-crafted sandboxes. The precise referent, however, depends on context: either the ShopArena-plus-ShopGuru framework for realistic simulation and scalable benchmarking [2605.16116], or the ShopSimulator-derived Gym-style environment for RL-driven LLM shopping assistants [2601.18225].

Source: https://www.emergentmind.com/topics/shopgym