WebMall: Multi-Shop Online Benchmark
- WebMall is a benchmark and simulated environment that tests LLM-based web agents on realistic multi-shop comparison-shopping tasks using heterogeneous, real-world product data.
- It employs four simulated e-commerce sites with 4,421 product offers and 91 cross-shop tasks, emphasizing multi-hop reasoning and efficient memory usage.
- Empirical results show that structured inputs like AX-Tree outperform vision-only approaches, highlighting critical design considerations for agent performance.
WebMall is a multi-shop online shopping benchmark for evaluating LLM-based web agents on comparison-shopping tasks across heterogeneous e-commerce sites (Peeters et al., 18 Aug 2025). It consists of four simulated online shops, authentic product offers sourced from the Common Crawl, and a suite of 91 cross-shop tasks that require search, price comparison, cart operations, checkout, and more advanced reasoning over vague requirements, substitutes, and compatibility. In the broader e-commerce literature, the name also resonates with earlier “virtual mall” system design that gathered several online stores under one roof and with later research that uses “WebMall-style” platforms as a deployment context for multimodal retrieval, attribute extraction, summarization, and recommendation systems. In current usage, however, WebMall primarily denotes a controlled, reproducible benchmark for studying navigation, reasoning, memory, and efficiency in multi-shop web agents (Peeters et al., 18 Aug 2025).
1. Definition and scope
WebMall is a benchmark and simulated environment specifically built to test how well LLM-based web agents can do realistic comparison-shopping across multiple online stores, rather than only finding a single item in a single store (Peeters et al., 18 Aug 2025). Its core research problem is whether web agents can reliably perform cross-shop shopping workflows that involve searching, filtering, comparing prices, reasoning about vague requirements, and completing checkout in complex, heterogeneous environments.
The benchmark contains four simulated online shops, 4,421 real product offers extracted from the October 2024 Common Crawl via schema.org annotations, and 91 tasks across 11 categories, split into basic and advanced shopping tasks (Peeters et al., 18 Aug 2025). The shops are populated with products from three broad product categories—PC components, PC peripherals, and other electronics—and the offers originate from hundreds of distinct real-world shops, which makes titles, descriptions, category structures, and prices substantially heterogeneous.
This design distinguishes WebMall from single-store shopping environments. The tasks explicitly require cross-site aggregation, heterogeneous attribute interpretation, longer trajectories, and multi-step reasoning. A plausible implication is that WebMall is intended not merely as a navigation benchmark but as a testbed for end-to-end shopping behavior under realistic catalog and interface variation.
2. Environment and benchmark construction
WebMall’s environment includes four WooCommerce shops, each using a different free template from the WooCommerce marketplace, plus a separate solution website where agents submit final answers (Peeters et al., 18 Aug 2025). Each shop has product detail pages, shopping cart and checkout functionality, a search bar, a category dropdown with its own category tree, and standard e-commerce UI controls. The shops are Dockerized for local hosting, so the environment can be reproduced exactly.
The product offers come from the WDC Extraction of the October 2024 Common Crawl using schema.org annotations (Peeters et al., 18 Aug 2025). Offers are filtered to ensure the fields name, description, price, and priceCurrency, deduplicated on the combination of those attributes, and restricted to English titles and descriptions using fastText language detection. Schema.org product identifiers such as GTIN and MPN are then used to cluster offers that refer to the same real-world product. Those clusters support tasks such as finding all offers for a named product across shops or identifying the cheapest offer for a given model.
The resulting catalog comprises 4,421 offers: 1,477 PC components, 1,388 PC peripherals, and 1,556 other electronics (Peeters et al., 18 Aug 2025). Titles range from 6 to 264 characters, with median 69 and mean 76.4, while descriptions range from 15 to 14,417 characters, with median 573 and mean approximately 1,059. Each product is imported into WooCommerce using structured fields name, description, price, categories, and image. This heterogeneity is central to the benchmark’s difficulty because agents cannot rely on uniform metadata or templated product language.
An earlier and conceptually distinct use of the term “WebMall” described a 3D virtual shopping mall that gathered several online stores under one roof, supported actions across multiple stores simultaneously such as viewing product availability, and allowed mall-level client authentication across participating stores with a single login (Khalil, 2015). This earlier system used VRML and ASP.NET and emphasized immersive multi-store shopping rather than agent evaluation. The coexistence of these two uses suggests that “WebMall” has served both as a systems concept for multi-store online commerce and, more recently, as a benchmark name for multi-shop agent research.
3. Task structure and evaluation protocol
WebMall defines 91 tasks grouped into 11 categories and divided into a basic task set and an advanced task set (Peeters et al., 18 Aug 2025). The basic categories are Find Specific Product, Find Cheapest Offer, Products Fulfilling Specific Requirements, Add to Cart, and Checkout. The advanced categories are Cheapest Offer with Specific Requirements, Products Satisfying Vague Requirements, Cheapest Offer with Vague Requirements, Find Substitutes, Find Compatible Products, and End-to-End.
Each task is specified by a natural language instruction and either a ground-truth set of solution URLs or a prescribed end condition such as successful checkout (Peeters et al., 18 Aug 2025). Basic tasks test cross-shop search, price comparison, add-to-cart actions, and checkout form completion. Advanced tasks require multi-hop reasoning from vague descriptions to concrete products, domain knowledge such as hardware compatibility, and exhaustive search across multiple heterogeneous shops.
WebMall evaluates predicted sets against ground-truth sets using task-level precision, recall, and F1:
These metrics are macro-averaged across tasks, and completion rate is defined as the percentage of tasks perfectly solved within the 50-step limit (Peeters et al., 18 Aug 2025).
Each episode begins from a defined initial state, typically the solution page or a neutral starting URL, and ends when the agent submits a solution, indicates completion, or reaches the 50-step limit (Peeters et al., 18 Aug 2025). This 50-step cap is an important structural property: it makes trajectory efficiency measurable and turns premature stopping, redundant navigation, and memory failures into explicit benchmark errors rather than informal qualitative weaknesses.
4. Agent interface, observation modalities, and memory
Agents interact with WebMall through the Browsergym/AgentLab framework (Peeters et al., 18 Aug 2025). The observation spaces are AX-Tree, Screenshot (Vision), and AX-Tree + Vision. Vision is implemented using Set-of-Mark prompting, which overlays markers on the screenshot and prompts the multimodal model to localize and act on UI elements visually. The action space includes browser operations such as go_to_page, click, fill_text, and scroll.
AgentLab also supports short-term persistent memory (Peeters et al., 18 Aug 2025). When memory is enabled, the agent can store information across steps, such as the lowest price seen so far, the corresponding URL, or which shops have already been explored. Without memory, the agent sees only the current page observation together with a short action history and previous thoughts within token limits. Memory is especially important for exhaustive search and aggregation, because cross-shop comparison requires maintaining a global view of candidate offers.
The evaluated baselines vary along three axes: observation modality, memory usage, and underlying LLM (Peeters et al., 18 Aug 2025). For each of GPT-4.1 and Claude Sonnet 4, four configurations are tested: AX-Tree, AX-Tree + Memory, AX-Tree + Vision, and Vision only. Architecturally, the agents follow a planner/executor pattern: the model receives instructions, current observation, and optional memory, then outputs a natural language thought and a structured action, after which Browsergym executes the action and returns a new observation.
The results show that AX-Tree is essential and vision alone is insufficient (Peeters et al., 18 Aug 2025). This is a concrete correction to a common misconception that screenshot-based multimodal agents are automatically adequate for realistic shopping tasks. In WebMall, structured accessibility representations remain the decisive input modality for reliable interaction.
5. Empirical findings on current web agents
On the basic task set, the best-performing configurations achieve approximately 75% completion and approximately 87% F1 (Peeters et al., 18 Aug 2025). Specifically, Claude Sonnet 4 with AX-Tree + Vision reaches completion rate 72.92% and F1 77.90%, while GPT-4.1 with AX-Tree + Memory reaches completion rate 75.00% and F1 87.61%. On the advanced task set, the best result reported is Claude Sonnet 4 with AX-Tree at completion rate 53.49% and F1 63.39%.
Per-category results show that checkout and add-to-cart are comparatively tractable (Peeters et al., 18 Aug 2025). GPT-4.1 with AX-Tree, with or without memory, achieves 100% completion rate and F1 on Checkout, and both models with structured inputs reach high Add to Cart scores. Named-product price comparison is also relatively strong, with completion rates up to 90%. By contrast, tasks involving vague requirements, compatibility, and substitutes remain substantially harder. Find Compatible Products often yields completion rates around 40–60% with F1 around 58–60%, while vague requirement tasks often produce materially lower CR and F1 because agents misinterpret requirements, stop before exhaustive search, or confuse subtle attributes.
Vision-only configurations perform markedly worse (Peeters et al., 18 Aug 2025). For example, GPT-4.1 Vision only reaches completion rate 41.67% on basic tasks and 13.95% on advanced tasks, while Claude Vision only reaches 10.42% and 4.65%, respectively. They also take more steps, often fail to locate search bars or clickable elements, and run out of steps without submitting correct solutions. This makes WebMall useful as an empirical stress test for embodied web interaction assumptions that overemphasize screenshot reasoning.
Efficiency measurements reveal a substantial cost and latency burden (Peeters et al., 18 Aug 2025). GPT-4.1 basic tasks require roughly 21–29 steps, cost $0.23–$0.29 per task, and take about 130–176 seconds. GPT-4.1 advanced tasks require roughly 24–33 steps, cost $0.29–$0.40, and take about 159–216 seconds. Claude Sonnet 4 is slower and more expensive, with basic tasks costing $0.67–$1.30 and advanced tasks $1.02–$1.63, while runtimes can extend to 223–447 seconds for basic tasks and 332–536 seconds for advanced tasks. This suggests that current strong agents are still operationally expensive for large-scale deployment in realistic comparison-shopping settings.
6. Position within e-commerce and multimodal systems research
WebMall is positioned explicitly against existing agent benchmarks such as WebShop and ShoppingBench, which simulate a single store, and against live-web benchmarks such as Mind2Web, BrowseComp, and DeepShop, which trade reproducibility for realism (Peeters et al., 18 Aug 2025). Its novelty lies in combining multi-shop comparison-shopping, heterogeneous offers from hundreds of real shops, and reproducible local deployment. This combination turns it into a controlled benchmark for studying long-horizon shopping behavior rather than only page interaction or single-site navigation.
The broader data block places WebMall in a larger technical landscape of e-commerce systems. Multimodal retrieval models such as MIEM construct a single product-level embedding from multiple images and title text and were deployed to improve image search on Shopee, yielding increases in clicks per user and orders per user while reducing index storage (Liu et al., 2023). Multimodal attribute extraction systems use text and images to predict product attributes such as color and material and are deployed to improve catalog structure and faceted search (Comble et al., 2022). Product summarization models such as MMAPS jointly model text, images, and attributes to generate short product summaries (Chen et al., 2023). Re-ranking systems such as ARMMT integrate text and image signals with context-aware fusion and auxiliary ranking-aligned tasks to improve CVR and GMV in search (Xu et al., 2024). A plausible implication is that WebMall can serve not only as an evaluation environment for web agents, but also as a downstream testbed for integrating retrieval, ranking, summarization, and attribute-aware catalog understanding into agentic shopping workflows.
Video-commerce research further clarifies why WebMall’s advanced tasks are difficult. Fashion Focus localizes catalog products in long shopping videos using visual and linguistic features (Zhang et al., 2021), while SGMN uses ASR-guided attention and spatiotemporal graph reasoning for livestreaming product retrieval (Hu et al., 2024). AMPere shows that denoised ASR text improves cross-domain product retrieval among product pages, short videos, and live streams (Zhao et al., 2024). MQMC performs bidirectional microvideo-product retrieval with category-aware multi-queue momentum contrast (Du et al., 2022). These systems address noisy multimodal evidence, ambiguous product matching, and long-horizon retrieval under media heterogeneity—the same broad family of problems that WebMall agents encounter when vague task descriptions, heterogeneous titles, and multi-shop variation must be resolved into concrete actions.
7. Uses, limitations, and research directions
WebMall is publicly released with code and data, including a GitHub repository for the benchmark and task definitions, and a two-command Docker setup that downloads backup files, configures services, and launches the four shops, databases, and Elasticsearch instances (Peeters et al., 18 Aug 2025). This makes it suitable for controlled agent evaluation, prompt ablation, memory design studies, planner comparisons, and cost–performance analysis.
Its limitations are explicit (Peeters et al., 18 Aug 2025). Product coverage is concentrated in electronics and PC hardware; the shops are simulated WooCommerce templates rather than fully dynamic commercial sites; product offers are frozen Common Crawl snapshots with no real-time updates or stock changes; and high-performing agents remain slow and expensive. Reproducibility also depends partly on external API models and pricing that may change over time.
Potential extensions noted in the source include adding more shops and broader product domains, integrating real-time data while preserving reproducibility, incorporating payment systems or budget constraints, adding recommendation logic, promotions, vouchers, or bundles, and developing more advanced memory and planning mechanisms (Peeters et al., 18 Aug 2025). In the surrounding literature, unified multimodal search frameworks such as UniECS handle text, image, and text+image retrieval within one architecture and report online gains in CTR and revenue (Liang et al., 19 Aug 2025), while OneMall unifies product-card, short-video, and live-streaming recommendation with semantic tokenization, transformer generation, and reinforcement learning (Zhang et al., 29 Jan 2026). This suggests that future WebMall-style environments may evolve toward richer multimodal, multi-surface, and end-to-end commercial simulations in which agents must not only navigate shops but also exploit structured retrieval, ranking, and recommendation primitives.
In that sense, WebMall marks a shift in benchmark design from isolated page interaction toward full comparison-shopping behavior. Its empirical results show that current agents can handle a substantial fraction of structured shopping tasks, yet still fail on nearly half of advanced tasks involving vague requirements, compatibility, and exhaustive cross-shop reasoning (Peeters et al., 18 Aug 2025). That gap is precisely what gives the benchmark its continuing research value.
