---
title: 'ECom-Bench: Multimodal E-Commerce Evaluation'
url: https://www.emergentmind.com/topics/ecom-bench
type: topic
---

# ECom-Bench: Multimodal E-Commerce Evaluation

ECom-Bench is a specialized benchmark framework developed to evaluate large language model (LLM) agents with multimodal capabilities in the context of real-world e-commerce customer support. It addresses critical gaps in prior evaluation suites by incorporating dynamic user simulation, authentic task scenarios grounded in business processes, and explicit tool-use requirements, with a focus on the unique complexities of multimodal, multi-turn customer service dialogues. Since its introduction, ECom-Bench has established itself as the canonical testbed for assessing whether LLM-based agents can resolve intricate e-commerce customer support issues involving text, image, and structured tool interactions [2507.05639].

## 1. Motivation and Scope

ECom-Bench was created to overcome the limitations of traditional rule-based chatbots and earlier web agent benchmarks, which primarily process single-turn, text-only queries and lack capabilities for context retention, multimodal input, and robust invocation of e-commerce-specific tools. The exponential growth of online retail has introduced complex customer queries spanning price protection, product returns, installation scheduling, and documentation, revealing fundamental limitations in existing end-to-end evaluation frameworks. Notably, prior benchmarks such as WebShop, ToolBench, and τ-Bench do not comprehensively cover e-commerce-specific, multi-turn, multimodal scenarios and are unable to measure end-to-end customer service capabilities required in real deployments [2507.05639].

## 2. Dynamic User Simulation Module

ECom-Bench simulates user behavior based on persona vectors derived from hundreds of authentic e-commerce chat logs (home and appliance domains). Each dialogue turn is annotated using LLM-driven chain-of-thought prompting to assign consumer type (e.g., value-oriented, premium-seeking), personality traits (emotion, patience, attention to detail, trust, rights awareness), and behavioral traits (questioning style, tone, pacing). Human experts further curate and refine these personas for realism and diversity.

At the start of each simulation, a persona vector $P\in\mathcal{P}$, comprising discrete and continuous traits (e.g., emotion ∈ {calm, angry}, patience ∈ [0,1]), is sampled. User utterances $U_t$ at turn $t$ are generated conditionally:

$$
\small
\text{prompt} = \text{"Persona: P  |  History: \{A_1, U_1, …, A_{t-1}, U_{t-1}\}  |  Current intent"}
$$

Generation proceeds by iteratively prompting a user-LLM conditioned on $P$, intent templates, and dialogue history until a terminal condition is met (intent stop signal or maximum turns) [2507.05639].

## 3. Task Set Construction and Distribution

The ECom-Bench dataset is sourced directly from real customer service transcripts in the home and appliance product verticals. To ensure privacy and evaluability, personal identifiable information is removed, duplicates collapsed, and complex multi-intent dialogues are segmented into discrete tasks. The final benchmark includes 53 manually verified tasks, with 18 requiring image-based reasoning.

Task scenarios are systematically categorized as follows:

| Scenario Category               | Approximate Share | Modalities       |
|---------------------------------|-------------------|------------------|
| Order status & modifications    | 20%               | text, tool       |
| Price/protection & refunds      | 18%               | text, tool       |
| Returns & exchanges             | 15%               | text, tool       |
| Product inquiry & recommendations| 12%               | text, tool       |
| Installation scheduling         | 10%               | text, tool       |
| Invoice & documentation         | 8%                | text, tool       |
| Multimodal image proof tasks    | 18%               | text, image, tool|

Scenario diversity is quantified as $D = |\mathcal{S}|/|\mathcal{T}| = 7/53 \approx 0.13$ (with $|\mathcal{S}|$ being distinct scenario categories, $|\mathcal{T}|$ the number of tasks). Task difficulty is modeled as a weighted combination of average turns, number of tools invoked, and modality complexity, normalized to $[0,1]$ [2507.05639].

## 4. Evaluation Protocol and Metrics

ECom-Bench introduces the $\text{pass}^k$ metric for robust probabilistic performance measurement:

Given $n$ independent runs per task, $c$ successful runs, and $k\le n$:

$$
\text{pass}^k := \mathbb{E}_{\text{task}}\left[ \frac{C(c,k)}{C(n,k)} \right]
$$

where $C(a,b)$ is the binomial coefficient. $\text{pass}^k$ estimates the probability that an agent can successfully complete the task $k$ times in $n$ trials, capturing robustness under repeated deployment.

Supplementary evaluation metrics include:
- Database state correctness (final state vs. ground-truth).
- Tool-call coverage (invoking all mandatory tools).
- Output keyword checks in natural language replies.
- Human spot-checking for hallucination and inappropriate dialogue tone [2507.05639].

## 5. Agent Evaluation and Benchmark Results

ECom-Bench evaluates both proprietary and open LLM/MLLMs, using the LangGraph simulation framework with a ReAct loop, a maximum of 20 turns or 600 seconds per task, and temperature 0.3. User simulation is powered by Qwen-Max, with image inputs processed by a vision-capable MLLM.

Performance results for leading models are summarized below:

| Model                | $\text{pass}^1$ (%) | $\text{pass}^2$ (%) | $\text{pass}^3$ (%) |
|----------------------|---------------------|---------------------|---------------------|
| GPT-4o               | 44.03               | 26.42               | 16.98               |
| Doubao-1.5-Pro-32k   | 38.99               | 23.90               | 16.98               |
| DeepSeek-V3          | 36.48               | 21.38               | 15.09               |
| Qwen-Max             | 15.09               | 8.81                | 5.66                |
| Moonshot-V1-32k      | 14.47               | 7.55                | 5.66                |
| Doubao-Pro-Vision*   | 20.75               | 13.21               | 11.32               |
| Qwen-VL-Max*         | 8.18                | 4.40                | 3.77                |

Remarkably, even the best-performing model (GPT-4o) achieves only 16.98% at $\text{pass}^3$, illustrating the substantial fragility of current multimodal customer-support agents in complex e-commerce workflows. An analysis of performance as $k$ increases shows rapid reliability degradation; for GPT-4o, $\text{pass}^1 \approx 43.9\%$ falls to $\text{pass}^8 \approx 7.6\%$ [2507.05639].

Failure analysis highlights three dominant error types:
- Wrong Decision: Incorrect tool, object, or flow (50%).
- Wrong Argument: Correct tool, but incorrect parameters (30%).
- Partial Resolution: Only part of the user's intent satisfied (20%).

Most errors occur in the "Action" (database operation) phase, with lesser rates in "Search" (hallucinations) and "Output" (keyword/linguistic failures).

## 6. Limitations and Extension Directions

ECom-Bench's present scope is vertically focused on home appliances and furniture; cross-category generalization is untested. Due to privacy, part of the data is LLM-synthesized, limiting authenticity. The dataset's scale (53 tasks) constrains statistical resolution, while persona-feature design is expert-driven rather than empirically optimized.

Key future extensions:
- Expanding to additional e-commerce verticals (fashion, electronics) and greater scenario variety.
- Inclusion of explicit memory modules for tracking dialogue state.
- Enhancing persona realism via empirical studies (e.g., A/B tests).
- Upscaling the number of task instances and introducing dynamic situations (such as concurrent orders) [2507.05639].

## 7. Significance and Comparative Positioning

ECom-Bench is the first benchmark in e-commerce explicitly targeting end-to-end, persona-driven, multimodal agent performance for customer support. Its joint evaluation of high-fidelity dialogue, tool integration, and multimodal planning distinguishes it from contemporaneous efforts such as EcomBench [2512.08868] (which emphasizes composite QA and business logic), EComAgentBench [2606.17698] (long-horizon shopping with distributed and hidden intent), and OxyEcomBench [2605.13173] (multimodal, multi-role holistic e-commerce understanding), among others. Notably, ECom-Bench's low $\text{pass}^3$ scores expose the difficulty of robust tool-oriented resolution and the current limitations of LLM agents in handling multimodal, rule-driven, stateful customer support in operational e-commerce domains [2507.05639].

Source: https://www.emergentmind.com/topics/ecom-bench