ShopArena: E-Commerce Simulation Layer
- ShopArena is an e-commerce simulation layer that converts live storefronts into self-contained, controllable sandbox shops preserving key structural and behavioral properties.
- It employs a two-phase synthesis pipeline using an anonymized specification (M) and stepwise source-code generation to ensure reproducibility, stability, and fidelity.
- The framework enables scalable, grounded evaluations of web agents by replicating navigation, policies, and interaction patterns while removing live-web complexities.
Searching arXiv for the cited papers to ground the article and ensure up-to-date references. ShopArena is the simulation layer of ShopGym, an integrated framework for realistic simulation and scalable benchmarking of e-commerce web agents. Its function is to convert live seed storefronts into self-contained sandbox shops that preserve structurally and behaviorally relevant properties of shopping websites while remaining stable, inspectable, controllable, resettable, and reproducible. Within the ShopGym architecture, ShopArena is responsible for shop synthesis and environment construction, whereas ShopGuru generates grounded benchmark tasks on top of those simulated shops (Savadikar et al., 15 May 2026).
1. System boundary and conceptual role
ShopArena is not the whole ShopGym framework. The paper draws a three-way distinction: ShopGym is the overall system, ShopArena is the shop synthesis and simulation environment, and ShopGuru is the grounded task generator. This distinction is central because it defines ShopArena as an environment-construction pipeline rather than a benchmark suite by itself (Savadikar et al., 15 May 2026).
| Component | Role | Output |
|---|---|---|
| ShopGym | Integrated framework | Simulated shops and benchmark tasks |
| ShopArena | Shop synthesis and simulation environment | Self-contained sandbox shops |
| ShopGuru | Grounded task generator | Evaluation tasks grounded in shop structure |
ShopArena is designed to produce behaviorally aligned approximations rather than exact clones of live storefronts. The generated environments are grounded in real storefront structure, navigation, policies, and interaction patterns, but they use synthetic brand identity, products, descriptions, and imagery. This design choice is directly tied to anonymization and reproducibility. A plausible implication is that fidelity is defined in terms of preserved evaluation signal rather than literal duplication of site content.
2. Methodological motivation
The motivating problem is a tradeoff in e-commerce agent evaluation. Live storefronts provide realism, since they expose genuine catalogs, policies, and UI structure, but they are non-stationary: inventories change, page layouts drift, A/B tests alter behavior, and live services introduce noise. Hand-built sandbox benchmarks offer control and repeatability, but they are costly to construct and typically cover only a narrow range of layouts, catalogs, policies, and interaction patterns (Savadikar et al., 15 May 2026).
ShopArena is introduced as a response to this methodological bottleneck. Its purpose is to provide a scalable pipeline for converting live seed storefronts into high-fidelity sandbox shops that preserve the structural and behavioral signals relevant to shopping tasks while removing the instability of the open web. The resulting middle ground is neither live-web evaluation nor a narrowly handcrafted toy benchmark. Instead, it is a controlled substrate intended for scientific comparison.
The paper emphasizes several target properties of ShopArena-produced environments: they are self-contained, resettable, inspectable, stable, controllable, reproducible, and diverse. Each property has a specific benchmarking function. Self-contained environments avoid dependence on external infrastructure; resettable environments support repeated trials and RL-style training; inspectability supports debugging and auditing; stability removes confounds from site drift; controllability permits calibrated variation; reproducibility supports fair comparisons across agents and time; and diversity reduces overfitting to a single template. This suggests that ShopArena is as much a methodological intervention as a software artifact.
3. Two-phase synthesis pipeline
ShopArena uses a two-phase pipeline with a single intermediate artifact, an anonymized specification document . The first phase explores one or more live seed storefronts; the second phase generates a runnable sandbox shop using only the extracted specification. The paper argues that this decomposition is necessary because long-context, long-horizon generation degrades and is difficult to control or debug (Savadikar et al., 15 May 2026).
In the exploration phase, ShopArena begins with deterministic prefetching of canonical surfaces, including homepage HTML, sitemap, search and cart endpoints, public catalog JSON, and policy pages. A planner agent then creates a decomposition of the storefront into focused subtasks grounded in the prefetched material. For each subtask, a fresh specification agent uses browser automation tools to inspect the live site, collecting evidence such as screenshots, page snapshots, and optionally XHR traces, and writing an anonymized fragment of . The specification agent is instructed to ignore brand names, product names, named individuals, and absolute URLs at write time. A non-agentic LLM then merges the fragments into a single coherent specification. When multiple seed storefronts are used, their outputs are composed into one composite specification.
The specification contains three classes of information: a design manual describing the store’s look and feel; a structured list of storefront attributes such as navigation, filter design, and pagination; and statistics such as number of products, price range, and distribution. The paper presents as the sole interface between live-store exploration and sandbox generation.
In the generation phase, ShopArena reads only , which makes the process anonymous by construction. It first performs synthetic catalog generation, producing collection details, product names, product descriptions, and product images one collection at a time from the design manual and statistics in . It then performs stepwise source-code synthesis in a fixed feature-scoped order: site shell and homepage, collection pages, product detail page, cart, search and filtering, policy pages, and a final integration pass. Each step receives only the relevant portion of , the synthetic catalog, and the current codebase state.
Generation is governed by an execution–verification loop. A coding agent edits the code, lightweight checks are executed, and a separate verifier inspects the result. Verification is split between rule-based checks, which ensure that the site builds, routes work, and outputs are structurally valid, and a multimodal verification agent, which opens the shop locally and checks for visual and functional issues. A final consolidation pass tours canonical routes and repairs remaining defects. The staged structure localizes failures, so a broken cart implementation, for example, can be repaired without regenerating unrelated components.
4. Environment characteristics and representation
The environments produced by ShopArena are meant to preserve the shopping-relevant structure of real storefronts while pruning confounding live-web complexity. The paper characterizes them as self-contained and stable artifacts whose structure can be examined directly through the specification and the generated environment. Because the pipeline can combine multiple seed storefronts, a single synthetic shop can blend structural and behavioral signals from more than one source site (Savadikar et al., 15 May 2026).
Structural fidelity is evaluated along three axes. First, observation and interaction complexity are assessed using the depth of the simplified accessibility tree as a proxy for nesting complexity, following BrowserGym-style reasoning, together with counts of unique elements supporting fill, click, and select-style controls. Second, state-transition complexity is modeled as a directed state-transition graph in which nodes represent UI states and edges represent action-triggered transitions. The paper reports that synthetic shops have a comparable number of nodes to real shops, but fewer edges and lower average out-degree. This reduction is described as expected because sandbox environments intentionally exclude extraneous transitions associated with external links, marketing pages, and auxiliary components. Third, behavioral validation tests whether preserved structure supports comparable agent behavior.
The state-transition representation is qualitative rather than formalized through an explicit graph metric equation in the excerpt. Pages such as / are nodes, and alternate UI configurations such as / : navigation are also represented as nodes when interactive elements are opened. This representation matters because it treats interaction-dependent interface states as benchmark-relevant structure rather than incidental presentation detail.
5. Evaluation protocol and empirical findings
ShopArena is validated both structurally and behaviorally. The behavioral evaluation uses three synthetic sandbox shops in clothing, cookware, and electronics hardware, along with three synthetic twin shops generated from real product data with extra visual verification. Across these six sandbox shops, the framework evaluates 224 generated tasks (Savadikar et al., 15 May 2026).
The agent evaluation uses multiple frontier models—GPT-5-mini, GPT-5, and Gemini 3 Flash—under two harnesses: a BrowserGym-based harness using accessibility trees, and an internal harness using screenshots plus accessibility trees. A GPT-5 judge returns a binary pass/fail verdict after examining the task intent, success criteria, the agent’s memory log, URL trajectory, termination type, and, in the custom harness, screenshots and cart history. The evaluation is repeated across multiple rollouts per task/model/environment cell, and aggregate pass rates are reported.
The main empirical claim is that synthetic shops preserve meaningful behavioral signal from live storefronts. In the twin-shops evaluation, models achieve similar performance on real storefronts and their twin variants. The paper further states that agent performance on synthetic shops positively correlates with performance on live storefronts. This is the key transferability result: success in the sandbox tracks success on the corresponding live site rather than producing arbitrary benchmark behavior.
For fully synthetic sandbox shops, long-horizon task performance remains nontrivial. In one harness, the reported example success rates on complex tasks are 47.9% for GPT-5-mini, 59% for Gemini 3 Flash, and 62.5% for GPT-5. The paper interprets this as evidence that ShopArena, together with ShopGuru, generates tasks that remain challenging even for strong models. A plausible implication is that the benchmark retains difficulty ordering rather than collapsing task structure into shallow pattern matching.
6. Relation to task grounding, scope limits, and adjacent shopping benchmarks
ShopArena supplies the shop data and environment structure that ShopGuru consumes. The paper states that ShopGuru covers seven skill categories spanning short-horizon and long-horizon tasks, grounded in ShopArena outputs such as products.json, collections.json, pages.json, and shop statistics. The categories include exact title search, substitute search, collection browsing, collection filtering, shipping policy lookup, returns policy lookup, and long-horizon end-to-end shopping journeys (Savadikar et al., 15 May 2026).
This grounding is important because task validity depends on actual environment affordances. If a task requires selecting a Color variant, the product must expose that variant; if it references a product and a collection jointly, catalog membership must support the claim. The paper notes that ShopGuru includes a validator to enforce such grounding constraints. ShopArena therefore functions not merely as a renderer of websites but as the substrate that makes grounded benchmark synthesis possible.
The paper is explicit about scope limitations. ShopArena deliberately removes non-stationarity, personalization, checkout, login, payment, and other live-web complexities. It is therefore not presented as a replacement for live-web evaluation. Rather, it is intended as a more controlled benchmark substrate that preserves enough of real shopping structure to support reproducible and inspectable experimentation.
A recurrent misconception is to treat “shopping benchmark” as a single category spanning both web agents and embodied retail agents. The literature block here illustrates a sharper distinction. Sari Sandbox, described as “essentially a ShopArena-style embodied shopping benchmark,” targets a photorealistic 3D retail store with VR-based human interaction, manipulation, barcode scanning, and checkout in a small convenience-store setting, whereas ShopArena targets e-commerce web environments synthesized from live storefronts (Gajo et al., 1 Aug 2025). This suggests that “ShopArena-style” can function descriptively for shopping-task benchmarking more broadly, but in the strict architectural sense of ShopGym, ShopArena denotes the web-shop simulation layer and not a generic shopping benchmark.
In summary, ShopArena is best understood as a specification-driven synthesis pipeline for reproducible e-commerce environments. Its significance lies in the combination of anonymized extraction from live storefronts, staged generation with verification, preserved structural and behavioral signal, and direct compatibility with grounded task generation. Within web-agent research, it occupies the methodological middle ground between unstable live-site evaluation and narrow hand-built sandboxes.