Amazon-Bench: E-commerce Agent Benchmark

• Amazon-Bench is a functionality-grounded benchmark for evaluating autonomous web agents on live Amazon pages, covering full e-commerce workflows including account management and product interactions.
• It employs an automated pipeline that combines webpage exploration, diversity-based sampling, and LLM-generated queries to ensure realistic task grounding and query generation.
• The evaluation framework distinguishes task success and safety by measuring benign versus harmful failures, offering detailed insights into state-changing operations in e-commerce.

Searching arXiv for the benchmark and closely related e-commerce agent work to ground the article. Amazon-Bench is a functionality-grounded, online benchmark for evaluating autonomous web agents on realistic e-commerce webpages, specifically Amazon.com. It was introduced to address two limitations of prior e-commerce benchmarks: a narrow concentration on product search tasks and the absence of explicit safety evaluation for unintended state-changing actions such as wrong purchases, deleted addresses, or misconfigured gift-card settings (Zhang et al., 18 Aug 2025). The benchmark spans the full range of end-user tasks offered by a large e-commerce platform—including product search, deal discovery, account management, wishlist and store interactions, review checking, and media browsing—and evaluates both task success and safety, distinguishing benign failures from harmful failures (Zhang et al., 18 Aug 2025).

1. Scope, definition, and motivation

Amazon-Bench is defined as a functionality-grounded benchmark in which agents operate on live Amazon pages rather than toy simulators or static snapshots. The benchmark is “online” in the sense that the agent interacts with realistic webpages and receives observations derived from either webpage structure or screenshots. Its central design choice is to ground tasks in actual site functionality, using webpage content and interactive elements such as buttons and check boxes to generate user queries that correspond to operations users can genuinely perform (Zhang et al., 18 Aug 2025).

The motivation follows directly from shortcomings in earlier e-commerce benchmarks. Early offline benchmarks such as WebShop and WebArena, as well as online shopping benchmarks such as WebVoyager and DeepShop, are described as focusing almost entirely on “find this product” or attribute-based product search. Amazon-Bench expands the task space to include account and payment workflows, wishlist management, store following, and related operations. It also introduces explicit safety evaluation, because an agent may achieve a superficially relevant interaction while still making an unintended negative change to user state (Zhang et al., 18 Aug 2025).

This design positions Amazon-Bench as a benchmark for evaluating not only retrieval and navigation competence, but also state-sensitive web interaction. A plausible implication is that it measures a broader notion of agent reliability than benchmarks that grade only whether a target item was found.

2. Task taxonomy and benchmark coverage

Amazon-Bench contains 400 single-turn user queries grouped into six high-level task categories. These categories are intended to cover realistic e-commerce workflows rather than only search behavior (Zhang et al., 18 Aug 2025).

Task category	Examples
Account Management	Add/update delivery addresses; configure gift-card auto-reload; edit payment or contact info
Product Interaction	Add to cart; buy now; multi-item bundles; “Frequently Bought Together” bundles
Product Search	Find products by attributes such as price, rating, color, and size
Deal Search	Filter outlet, Prime Day, or Lightning Deals
Store Interaction	Follow/unfollow brand stores; browse new arrivals in a store
Review Checking & Media Interaction	Show 1-star reviews; filter by customer segment; sort Kindle books by price; list Audible titles

The benchmark’s coverage is notable because it includes workflows with persistent account state and multi-step interaction. Account Management and Product Interaction are especially important in this respect: they expose whether an agent can recognize when an action commits state changes, not merely whether it can navigate to a relevant page. By contrast, Review Checking, Deal Search, and Media Interaction represent less state-changing interactions and therefore support more fine-grained safety analysis (Zhang et al., 18 Aug 2025).

The task taxonomy also reflects the benchmark’s claim to functionality grounding. The categories are not abstract labels imposed independently of the site; rather, they are derived from the functional affordances present on real Amazon pages.

3. Functionality-grounded query generation pipeline

Amazon-Bench generates queries automatically from live pages in a three-stage pipeline: webpage exploration, webpage sampling by functional diversity, and user query generation with refinement (Zhang et al., 18 Aug 2025).

The exploration stage starts at https://www.amazon.com and runs a breadth-first search to depth 3. During this process, the pipeline extracts all links, cleans and enqueues unique URLs, and records each page’s title and URL. Pages are then classified into 10 categories by URL patterns, with examples including "/dp/" → Product Page, "/wishlists/" → User Profile, and "/outlet/deals" → Deal Page (Zhang et al., 18 Aug 2025).

The sampling stage allocates page samples according to a diversity score over interactive element texts within each category. The score is defined as

$$\mathrm{Diversity}(c) = \frac{2}{n(n-1)} \cdot \sum_{1 \le i < j \le n} [1 - \mathrm{sim}(e_i, e_j)]$$

where $e_i$ is the text-embedding, using Sentence-BERT, of all buttons, check boxes, and menus on page $i$, and $\mathrm{sim}(\cdot,\cdot)$ is cosine similarity. A total of $N=100$ sample pages is then allocated proportionally by

$$\mathrm{pages}_i = \max\left(m,\; \left[\frac{\ln(1 + D_i)}{\sum_j \ln(1 + D_j)}\right] \times N\right)$$

with $m=5$ to ensure minimal coverage even for low-diversity categories (Zhang et al., 18 Aug 2025).

In the generation stage, each sampled page’s HTML is converted to an Accessibility Tree (AXTree), retaining only visible structure and interactive elements. An LLM $\mathcal{M}$ is prompted with the AXTree and a query-generation prompt to produce candidate instructions,

$q = \mathcal{M}(w, p_{gen})$

after which an LLM-based post-processing step makes the language natural, and human review filters or lightly edits ambiguous or unnatural queries. The final output is 400 single-turn user queries grounded in real page functionality (Zhang et al., 18 Aug 2025).

This pipeline is significant because it operationalizes “functionality grounding” through the combination of live-page exploration, interactive-element diversity, and AXTree-based prompt construction. It therefore differs from task authoring procedures centered only on products or textual catalog metadata.

4. Evaluation framework: success, safety, and efficiency

Amazon-Bench evaluates agents with an automated framework that measures task success, safety, and efficiency (Zhang et al., 18 Aug 2025).

Task success is measured online with an LLM-as-Judge. Given user query $q$, an agent’s action sequence $e_i$0, and step-by-step screenshots $e_i$1, a separate LLM $e_i$2 produces

$e_i$3

where $e_i$4. The online end-to-end success rate is the fraction of queries judged “Success.” Amazon-Bench also reports offline next-action accuracy, defined at each step $e_i$5 in human-annotated trajectories as the exact match between the agent’s chosen action and the human action (Zhang et al., 18 Aug 2025).

Safety evaluation is based on a tripartite outcome structure. A Benign Failure occurs when the agent fails to complete the query but makes no lasting changes to user state. A Harmful Failure is any unintended negative change, including wrong cart additions, unwanted purchases, deleted addresses, misconfigured settings, or repeated state-changing actions. The harmful failure rate is defined as

$e_i$6

(Zhang et al., 18 Aug 2025).

Efficiency is measured relative to human step counts:

$e_i$7

with lower values indicating that the agent is closer to human-optimal step counts (Zhang et al., 18 Aug 2025).

This evaluation design is distinctive because it does not collapse all unsuccessful trajectories into a single failure category. The distinction between benign and harmful failures makes state corruption part of the benchmark target, which is especially consequential in account and payment workflows.

5. Experimental setting and empirical results

The experimental environment uses live Amazon.com through BrowserGym, with a unified observation consisting of AXTree, query, and history, and an action space containing click, fill, select_option, stop, go_back, goto(url), and hover (Zhang et al., 18 Aug 2025). The agents evaluated online comprise research LLM-based systems—Deepseek-R1, GPT-4o, GPT-o4-mini, Claude-3.7, and GPT-4.1—as well as two off-the-shelf products: WebVoyager, using Claude-3.7, and Nova-Act, described as multimodal with screenshots (Zhang et al., 18 Aug 2025).

Agent	Success rate (%)	Harmful failure rate (%)
WebVoyager	44.0	6.5
Nova-Act	46.3	4.0
Deepseek-R1	42.3	7.8
GPT-4o	49.8	9.0
GPT-o4-mini	51.0	8.5
Claude-3.7	56.5	7.5
GPT-4.1	59.8	4.75

GPT-4.1 achieves the best overall success rate at 59.8%, while Nova-Act has the lowest harmful failure rate at 4.0%. Task-level results show that Store Interaction is the hardest category, with success at or below 39% even for the best models. Account Management and Product Interaction also lag behind pure search and media tasks (Zhang et al., 18 Aug 2025). This pattern suggests that stateful workflows and brand-store operations remain materially more difficult than product retrieval.

Offline next-action accuracy is reported as 39.2 for Deepseek-R1, 41.7 for GPT-4o, 48.8 for GPT-o4-mini, 51.9 for Claude-3.7, and 50.6 for GPT-4.1. These values indicate that even strong end-to-end agents only partially match human action choices step by step (Zhang et al., 18 Aug 2025).

Safety analysis identifies high-risk tasks in Account Management, at up to approximately 8–9% harmful failures, and Product Interaction, at up to approximately 23% for some agents. Review Checking, Deal Search, and Media Interaction are low-risk, with near-zero harmful changes (Zhang et al., 18 Aug 2025). This asymmetry is substantively important: it shows that harmful behavior is concentrated in domains where user state can be modified.

Efficiency and resource usage further differentiate systems. The efficiency ranking is reported as GPT-4.1 < GPT-o4-mini < Claude-3.7 < others, meaning GPT-4.1 is closest to human step counts. Reasoning-heavy models such as GPT-o4-mini and Deepseek use more tokens, while Claude-3.7 uses fewer tokens but has higher latency (Zhang et al., 18 Aug 2025).

The automated judge is partially validated by a human-agreement study: on 120 sampled trajectories, the LLM judge’s success/failure decisions agree with human raters at 92.5% overall (Zhang et al., 18 Aug 2025). Within the benchmark’s own methodology, this supports the use of automated evaluation at scale.

6. Failure modes, limitations, and relation to adjacent benchmarks

The paper includes case studies illustrating both benign and harmful failure modes. In an auto-reload gift card task, Nova-Act produces a benign failure by looping on scrolling for a “Save” button that does not exist; the correct action was to click “Buy Now.” In an “Add one Coach bag” task, an agent produces a harmful failure by clicking “Add to cart” twice—once in search results and again on the product page—thereby adding two items when only one was requested (Zhang et al., 18 Aug 2025). These examples clarify that failure can arise from either perceptual-navigation mismatch or from incorrect commitment semantics on state-changing controls.

The benchmark identifies several implications for agent development. Agents need page-specific knowledge to recognize which buttons commit state changes and which do not. Expanded action and observation spaces, including scrolling and viewport awareness, can reduce blind loops. Safety mechanisms such as “Did you mean to add 2 items?” are proposed as useful additions (Zhang et al., 18 Aug 2025). These are presented as development implications rather than as components of the benchmark itself.

Amazon-Bench also has explicit limitations. It contains only single-turn interactions and does not include multi-turn clarification dialogues. It also omits user-personalization context such as past orders or preferences (Zhang et al., 18 Aug 2025). Potential extensions include multi-turn and mixed-initiative tasks, incorporation of user history to simulate personalized shopping, and expansion beyond Amazon to platforms such as Walmart and eBay (Zhang et al., 18 Aug 2025).

In the emerging literature on shopping agents, Amazon-Bench occupies a distinct position. EComAgentBench later emphasizes long-horizon tasks with distributed hidden intent, in which requirements are split across a visible query, a tool-gated persona, and scripted clarification, and agents must commit to a single product within 100 tool calls (Du et al., 16 Jun 2026). By contrast, Amazon-Bench is centered on functionality-grounded interaction with realistic e-commerce webpages, including account management and store operations, and on explicit safety evaluation of harmful side-effects (Zhang et al., 18 Aug 2025). It should also not be conflated with ARMBench, the “Amazon Robotic Manipulation Benchmark,” which is a warehouse robotics dataset for object segmentation, object identification, and defect detection rather than a web-agent benchmark (Mitash et al., 2023).

Taken together, these distinctions place Amazon-Bench within a broader shift in benchmarking from static product-search evaluation toward richer, stateful, and safety-aware assessments of autonomous agents in real e-commerce environments (Zhang et al., 18 Aug 2025).