ShopGym: E-Commerce Agent Simulation

• ShopGym is an integrated e-commerce simulation framework that unites sandbox shop generation with Gym-style APIs for RL-driven agent evaluation.
• It combines components like ShopArena and ShopGuru to synthesize tasks from live storefront data, ensuring reproducibility and realistic benchmarks.
• The framework supports multi-turn dialogue, personalization, and detailed reward models to accurately assess shopping assistant performance.

ShopGym is an overloaded term in recent e-commerce-agent research. In "ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents" (Savadikar et al., 15 May 2026), it denotes an integrated framework for constructing e-commerce simulation environments and grounded benchmark tasks, combining ShopArena for environment construction with ShopGuru for task synthesis. In "ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistants" (Wang et al., 26 Jan 2026), the structured description explicitly introduces ShopSimulator "hereafter called ShopGym" as a Gym-style environment for training and evaluating RL-driven LLM agents in realistic shopping scenarios. Across these usages, the common theme is a controlled yet realistic substrate for studying shopping agents, but the two systems emphasize different methodological priorities: reproducible simulation and scalable benchmarking in one case, and unified training-and-evaluation support with dialogue, personalization, and reward design in the other.

1. Terminological scope and research problem

The central problem shared by both usages is the lack of satisfactory environments for e-commerce web agents. One line of work states that existing methodologies force a tradeoff: live storefronts provide realism but are non-stationary, difficult to inspect, and irreproducible, while hand-built sandbox benchmarks provide control but cover only a narrow range of layouts, catalogs, policies, and interaction patterns (Savadikar et al., 15 May 2026). The other line argues that existing research has yet to provide a unified simulation environment that consistently captures multi-turn dialogue, personalization, and fine-grained product discrimination, and always focuses solely on evaluation benchmarks without training support (Wang et al., 26 Jan 2026).

A common misconception is that ShopGym refers to a single benchmark. The record in the cited sources does not support that interpretation. Instead, the term names two related but distinct efforts. One is a framework for constructing sandbox shops and grounded tasks from live seed storefronts (Savadikar et al., 15 May 2026). The other is a Gym-style API and dataset ecosystem centered on product search, dialogue, personalization, and RL-compatible rewards (Wang et al., 26 Jan 2026).

Usage of "ShopGym"	Definition in source	Primary emphasis
ShopGym (Savadikar et al., 15 May 2026)	Integrated framework with ShopArena and ShopGuru	Realistic simulation and scalable benchmarking
ShopGym / ShopSimulator (Wang et al., 26 Jan 2026)	Gym-style environment for RL-driven LLM agents	Training and evaluation in shopping scenarios

This suggests that "ShopGym" functions less as a singular canonical artifact than as a convergent label for e-commerce-agent environments that attempt to reconcile realism, control, and scientific comparability.

2. ShopGym as an integrated framework: ShopArena and ShopGuru

In the framework introduced in (Savadikar et al., 15 May 2026), ShopGym is composed of two tightly integrated layers—ShopArena for environment construction and ShopGuru for task synthesis—designed to bridge realism and experimental control in e-commerce agent evaluation. ShopArena converts live seed storefronts into self-contained sandbox shops through anonymized shop specifications and a staged, validated generation process. ShopGuru then synthesizes benchmark tasks across seven skill categories, grounding each task in the shop's catalog, navigation structure, policies, and interaction affordances.

ShopArena uses a two-phase pipeline. All live-web dependence is confined to an exploration phase, which autonomously crawls one or more seed storefronts and writes a human-readable, anonymized specification $M$. A separate generation phase consumes $M$ and only $M$ to synthesize a runnable sandbox shop (Savadikar et al., 15 May 2026). During exploration, a planner agent first inspects prefetched pages—homepage HTML, sitemap, catalog JSON, and policy pages—and writes a decomposition plan. A fleet of specification agents then executes targeted browser-automation subtasks via Playwright, capturing screenshots and XHR traces and emitting fragments of $M$. A non-agentic LLM consolidates these fragments into a natural-language design manual, a structured capabilities manifest, and catalog and UI statistics.

The generation phase synthesizes mock product names, descriptions, and images from $M$. A staged, feature-scoped source-code synthesis pipeline then builds the site in steps: site shell, homepage, collections, product pages, cart, search, and policy pages. Each step is driven by an iteration of an execution agent that edits code and runs build/type checks, together with verifiers consisting of rule-based checks plus a multimodal visual agent that validate functionality and UI. Early termination and a fixed iteration budget ensure reproducibility and debuggability (Savadikar et al., 15 May 2026).

ShopGuru operates on top of the sandbox shop. It ingests the sandbox shop’s catalog JSON, navigation graph, filter-and-sort manifest, and policy/page listings. It combines deterministic short-horizon generators, implemented as pure-function scripts, with LLM-authored long-horizon journeys. The short-horizon scripts sample products and collections, check feasibility via per-collection option indices, emit task JSON with natural-language intents plus structured success_criteria, and immediately validate all tasks against shop data with rule-based verifiers. For long-horizon tasks, a single LLM such as GPT-5 or Gemini 3 Pro is few-shot prompted with eight human-trace examples and shop context, then a validator-driven polish loop re-audits and refines any hallucinated or infeasible tasks, ensuring zero residual grounding errors before release (Savadikar et al., 15 May 2026).

3. Task models and skill coverage

The ShopGuru task layer in (Savadikar et al., 15 May 2026) covers seven skill categories, grouped into three short-horizon primitives and one long-horizon composite. The categories are: search-exact, search-substitute, browse, filter, shipping policy lookup, returns policy lookup, and end-to-end shopping journeys. The paper further organizes them as Discovery, Filter-Selection, Information-Seeking, and long-horizon E2E journeys.

The grounding sources are explicit. Search-exact and search-substitute use products.json with title and handle. The generator samples one active product per product_type and emits two variants: "Find product by its exact title" and "Find a semantically similar alternative." Verification ensures the target handle resolves to a real PDP URL (Savadikar et al., 15 May 2026). Browse and filter tasks use collections.json plus a per-collection option index derived from the shop’s own products. The browse generator picks a non-generic collection with at least three products and asks the agent to navigate there and add any product. The filter generator pairs the same collection with a real facet $(\mathrm{dim}, \mathrm{value})$ only if at least one product supports it; otherwise it falls back to universal facets such as Vendor or ProductType.

Shipping and returns tasks are grounded in pages.json plus default /policies endpoints. The generator searches for page handles or titles matching keywords such as "shipping" and "return" and emits a task to find and read the page; success_criteria soft-hints URL patterns (Savadikar et al., 15 May 2026). E2E shopping journeys are grounded in full shop context—collections, products, pages, and filter-sort options—together with few-shot examples reflecting real click-stream patterns. The LLM generator mixes navigation, filtering, sorting, policy detours, and cart edits, while a polish loop feeds back tasks that fail any of seven validator rules, including unknown collection, infeasible filter, or hallucinated variant, in order to regenerate only the flagged tasks.

In the Gym-style environment described in (Wang et al., 26 Jan 2026), the task model is different. The environment supports both single-turn and multi-turn modes, with and without personalization, and delivers train/test splits, instruction templates, and reward signals to enable SFT and RL research. The task pool contains Single-Turn Non-Pers. (6K train / 600 eval), Multi-Turn Non-Pers. (6K train / 600 eval), Single-Turn Pers. (4.7K train / 600 eval), and Multi-Turn Pers. (4.7K train / 600 eval), for a total of approximately 28K tasks with one-to-one mapping, where each natural-language instruction uniquely identifies one gold product (Wang et al., 26 Jan 2026). Instructions written by human annotators avoid lexical overlap with product titles, and personalized instructions omit demographic cues, which are moved into profile fields.

The two task regimes therefore target different granularities. One emphasizes web-interaction skills grounded in concrete storefront structure; the other emphasizes agentic product search under dialogue, personalization, and purchase selection. A plausible implication is that the former is closer to controlled web-agent evaluation, while the latter is closer to RL-style sequential decision-making over shopping assistance trajectories.

4. Structural fidelity, behavioral evaluation, and benchmark methodology

A distinguishing feature of (Savadikar et al., 15 May 2026) is its explicit claim that realism should be validated structurally as well as behaviorally. The paper models each site as a directed UI-state graph $G=(V,E)$, where nodes $V$ are distinct UI states—page plus widget configurations—and edges $E$ are user actions such as clicks and filter opens. Three graph-based metrics are used: degree distribution $p(d)$, clustering coefficient $M$0, and average shortest-path length $M$1.

The formulas are given as follows:

$M$2

where $M$3 is the degree of node $M$4 and $M$5 is the number of triangles through $M$6, and

$M$7

where $M$8 is the length of the shortest path from state $M$9 to $M$0 (Savadikar et al., 15 May 2026). These metrics measure interaction branching, the tendency of UI states to form triadic connections, and the average number of interactions needed to navigate between any two states.

Behavioral evaluation in the same work uses six sandbox storefronts—three fully synthetic and three "twin shops" generated with real product data—and 224 tasks per shop, covering all seven skill categories. Three baseline agent architectures are evaluated: GPT-5-mini, Gemini 3 Flash, and GPT-5. Two harnesses are used: BrowserGym with AXTree only, and an internal multimodal harness with AXTree plus screenshots. Judging is performed by GPT-5 as LLM-judge, scoring binary success or fail on each trajectory against success_criteria plus a hard rule that the agent must explicitly signal task completion (Savadikar et al., 15 May 2026).

Success rates are computed per $M$1 cell and averaged over three independent rollouts. Short-horizon tasks achieve high pass rates above 80% for all models, whereas long-horizon tasks remain challenging, with GPT-5 at approximately 62%, Gemini 3 Flash approximately 59%, and GPT-5-mini approximately 48% (Savadikar et al., 15 May 2026). The paper further states that synthetic sandbox shops match the difficulty of real shops, showing comparable success rates across twin versus live storefronts.

The live-versus-synthetic correspondence is quantified using the Pearson correlation coefficient $M$2 between agent success rates on live versus twin shops:

$M$3

where $M$4 and $M$5 are success rates on live and twin shops respectively (Savadikar et al., 15 May 2026). A strong positive $M$6, for example greater than 0.8 with $M$7, is reported as confirmation that synthetic environments faithfully preserve evaluative signal.

5. Gym-style API, state design, and reward functions

The environment described in (Wang et al., 26 Jan 2026) follows the Gym v0 API with minor extensions for dialogue. The API is episodic and stepwise, using reset() and step(action) → (obs, reward, done, info). Observation is a rich text or dictionary describing UI state, user utterance, and user profile, rather than a fixed numerical vector. Actions are textual or structured commands such as search[...], click[...], and ask_shopper[...]. Termination occurs when the agent issues click[buy now] or reaches a max-step limit of 30–40 steps (Wang et al., 26 Jan 2026).

The observation dictionary contains "ui_state", "user_utterance", "user_profile" in personalization mode, "dialogue_history", and "instruction" in single-turn mode. "ui_state" itself contains search_available, clickable_buttons, and current_results, where each result includes fields such as "id", "title", and "price". The action space is a tuple $M$8, with $M$9. If the type is "ask_shopper", the content is a question string; otherwise the content must match one of search[<keyword string>] or click[<button label>] (Wang et al., 26 Jan 2026).

Episode structure is explicit. reset() samples one target product and its gold instruction, one user profile if personalized, and initializes an LLM-simulated shopper in "system" role with a hidden goal. At each step $M$0, the agent receives $M$1, returns $M$2, the environment updates its UI and, if relevant, the user utterance, and then returns $M$3. Product representations include category_path, attributes, options, and price. The "user_profile" contains demographic fields, long-term preference scores, and historical behavior such as past searches and purchase frequency (Wang et al., 26 Jan 2026).

The reward model is one of the most technically specific aspects of this usage of ShopGym. Two final-step reward variants are defined, together with a binary success metric. The loose additive reward is

$M$4

and the strict multiplicative or bottleneck reward is

$M$5

Here, $M$6, $M$7, and $M$8 are the target’s required attributes, options, and price limit; $M$9, $M$0, and $M$1 are the purchased product’s values; and $M$2 softly measures category-path similarity. The full-success rate $M$3 is 1 if and only if all requirements are exactly met, otherwise 0 (Wang et al., 26 Jan 2026). Intermediate rewards are zero by default for all non-terminal steps, though per-click or per-confirmation bonuses can be instrumented externally.

6. Empirical findings, reproducibility, and future extension

The empirical conclusions of the two works are complementary. In (Savadikar et al., 15 May 2026), the main result is that self-contained, resettable, inspectable, and stable evaluation artifacts can preserve structural properties and agent-evaluation signals relevant to shopping tasks. The paper states that the synthetic shops preserve key structural properties of live storefronts and that agent performance on synthetic shops is positively correlated with performance on live storefronts. It further argues that ShopGym unites the realism of live sites with the repeatability of sandboxes by splitting exploration from generation, with the human-readable specification $M$4 serving as an editing surface so benchmarks can be tweaked without re-crawling or modifying live sites (Savadikar et al., 15 May 2026).

In (Wang et al., 26 Jan 2026), the main empirical message is that the shopping-assistant problem remains difficult even in a purpose-built environment. The baseline benchmark reports that even the best-performing models achieve less than 40% full-success rate in the hardest multi-turn personalized setting, with substantial gaps between $M$5 at roughly 60%, $M$6 at roughly 35%, and $M$7 at roughly 30% (Wang et al., 26 Jan 2026). Closed-source models tested include GPT-5, OpenAI-o3, GPT-4.1, Claude-4-Sonnet, and Gemini2.5 Pro; open-source models include DeepSeek-R1/V3.1, Kimi-K2, GLM-4.5, GPT-OSS, and the Qwen3 family from 8B to 235B. GPT-5 is reported as best overall at approximately 32.7% $M$8.

Training support is another point of divergence. The Gym-style ShopSimulator system explicitly supports supervised fine-tuning and reinforcement learning. It specifies approximately 6K expert trajectories from GPT-4.1 as pseudo-demonstrations for SFT, and RL via GRPO implemented through the ROLL library. The reported findings are that SFT + RL outperforms either alone across all four scenarios, with a 24–40% increase in $M$9; that strict reward yields better fine-grained matching than loose reward; and that single-turn settings yield larger gains post-training, while multi-turn remains challenging at roughly 35% $(\mathrm{dim}, \mathrm{value})$0 (Wang et al., 26 Jan 2026).

For reproducibility and extension, (Savadikar et al., 15 May 2026) emphasizes scalable benchmarking, scientific reproducibility through fixed shop snapshots and seed-fixed data generation, and modular extension to gated flows such as login and checkout, multilingual or localization settings, recommendation modules, or post-purchase interactions by enriching $(\mathrm{dim}, \mathrm{value})$1 and downstream generators. It also highlights interpretability and analysis through graph metrics and per-skill suites that can pinpoint strengths and weaknesses such as filter logic or long-horizon planning. A plausible implication is that the two usages of ShopGym occupy adjacent layers of a broader research stack: one constructs realistic and controllable web environments, while the other operationalizes shopping-agent learning and evaluation within a Gym-style interaction loop.

Taken together, the name "ShopGym" marks a developing methodological space in which e-commerce agents are studied under stronger experimental control than live-web evaluation permits, while retaining richer task structure than narrow hand-crafted sandboxes. The precise referent, however, depends on context: either the ShopArena-plus-ShopGuru framework for realistic simulation and scalable benchmarking (Savadikar et al., 15 May 2026), or the ShopSimulator-derived Gym-style environment for RL-driven LLM shopping assistants (Wang et al., 26 Jan 2026).