ECom-Bench: Multimodal E-Commerce Evaluation

• ECom-Bench is a benchmark framework designed to evaluate LLM agents with multimodal capabilities in complex, multi-turn e-commerce customer support tasks.
• It employs dynamic user simulations based on authentic chat logs and persona vectors to replicate real-world customer interactions and task diversity.
• The benchmark measures agent performance using pass^k metrics, tool-call coverage, and dialogue state accuracy to assess response robustness in multimodal settings.

ECom-Bench is a specialized benchmark framework developed to evaluate LLM agents with multimodal capabilities in the context of real-world e-commerce customer support. It addresses critical gaps in prior evaluation suites by incorporating dynamic user simulation, authentic task scenarios grounded in business processes, and explicit tool-use requirements, with a focus on the unique complexities of multimodal, multi-turn customer service dialogues. Since its introduction, ECom-Bench has established itself as the canonical testbed for assessing whether LLM-based agents can resolve intricate e-commerce customer support issues involving text, image, and structured tool interactions (Wang et al., 8 Jul 2025).

1. Motivation and Scope

ECom-Bench was created to overcome the limitations of traditional rule-based chatbots and earlier web agent benchmarks, which primarily process single-turn, text-only queries and lack capabilities for context retention, multimodal input, and robust invocation of e-commerce-specific tools. The exponential growth of online retail has introduced complex customer queries spanning price protection, product returns, installation scheduling, and documentation, revealing fundamental limitations in existing end-to-end evaluation frameworks. Notably, prior benchmarks such as WebShop, ToolBench, and τ-Bench do not comprehensively cover e-commerce-specific, multi-turn, multimodal scenarios and are unable to measure end-to-end customer service capabilities required in real deployments (Wang et al., 8 Jul 2025).

2. Dynamic User Simulation Module

ECom-Bench simulates user behavior based on persona vectors derived from hundreds of authentic e-commerce chat logs (home and appliance domains). Each dialogue turn is annotated using LLM-driven chain-of-thought prompting to assign consumer type (e.g., value-oriented, premium-seeking), personality traits (emotion, patience, attention to detail, trust, rights awareness), and behavioral traits (questioning style, tone, pacing). Human experts further curate and refine these personas for realism and diversity.

At the start of each simulation, a persona vector $P\in\mathcal{P}$, comprising discrete and continuous traits (e.g., emotion ∈ {calm, angry}, patience ∈ [0,1]), is sampled. User utterances $U_t$ at turn $t$ are generated conditionally:

$\small \text{prompt} = \text{"Persona: P | History: \{A_1, U_1, …, A_{t-1}, U_{t-1}\} | Current intent"}$

Generation proceeds by iteratively prompting a user-LLM conditioned on $P$, intent templates, and dialogue history until a terminal condition is met (intent stop signal or maximum turns) (Wang et al., 8 Jul 2025).

3. Task Set Construction and Distribution

The ECom-Bench dataset is sourced directly from real customer service transcripts in the home and appliance product verticals. To ensure privacy and evaluability, personal identifiable information is removed, duplicates collapsed, and complex multi-intent dialogues are segmented into discrete tasks. The final benchmark includes 53 manually verified tasks, with 18 requiring image-based reasoning.

Task scenarios are systematically categorized as follows:

Scenario Category	Approximate Share	Modalities
Order status & modifications	20%	text, tool
Price/protection & refunds	18%	text, tool
Returns & exchanges	15%	text, tool
Product inquiry & recommendations	12%	text, tool
Installation scheduling	10%	text, tool
Invoice & documentation	8%	text, tool
Multimodal image proof tasks	18%	text, image, tool

Scenario diversity is quantified as $$D = |\mathcal{S}|/|\mathcal{T}| = 7/53 \approx 0.13$$ (with $|\mathcal{S}|$ being distinct scenario categories, $|\mathcal{T}|$ the number of tasks). Task difficulty is modeled as a weighted combination of average turns, number of tools invoked, and modality complexity, normalized to $[0,1]$ (Wang et al., 8 Jul 2025).

4. Evaluation Protocol and Metrics

ECom-Bench introduces the $\text{pass}^k$ metric for robust probabilistic performance measurement:

Given $U_t$0 independent runs per task, $U_t$1 successful runs, and $U_t$2:

$U_t$3

where $U_t$4 is the binomial coefficient. $U_t$5 estimates the probability that an agent can successfully complete the task $U_t$6 times in $U_t$7 trials, capturing robustness under repeated deployment.

Supplementary evaluation metrics include:

• Database state correctness (final state vs. ground-truth).
• Tool-call coverage (invoking all mandatory tools).
• Output keyword checks in natural language replies.
• Human spot-checking for hallucination and inappropriate dialogue tone (Wang et al., 8 Jul 2025).

5. Agent Evaluation and Benchmark Results

ECom-Bench evaluates both proprietary and open LLM/MLLMs, using the LangGraph simulation framework with a ReAct loop, a maximum of 20 turns or 600 seconds per task, and temperature 0.3. User simulation is powered by Qwen-Max, with image inputs processed by a vision-capable MLLM.

Performance results for leading models are summarized below:

Model	$U_t$8 (%)	$U_t$9 (%)	$t$0 (%)
GPT-4o	44.03	26.42	16.98
Doubao-1.5-Pro-32k	38.99	23.90	16.98
DeepSeek-V3	36.48	21.38	15.09
Qwen-Max	15.09	8.81	5.66
Moonshot-V1-32k	14.47	7.55	5.66
Doubao-Pro-Vision*	20.75	13.21	11.32
Qwen-VL-Max*	8.18	4.40	3.77

Remarkably, even the best-performing model (GPT-4o) achieves only 16.98% at $t$1, illustrating the substantial fragility of current multimodal customer-support agents in complex e-commerce workflows. An analysis of performance as $t$2 increases shows rapid reliability degradation; for GPT-4o, $t$3 falls to $t$4 (Wang et al., 8 Jul 2025).

Failure analysis highlights three dominant error types:

• Wrong Decision: Incorrect tool, object, or flow (50%).
• Wrong Argument: Correct tool, but incorrect parameters (30%).
• Partial Resolution: Only part of the user's intent satisfied (20%).

Most errors occur in the "Action" (database operation) phase, with lesser rates in "Search" (hallucinations) and "Output" (keyword/linguistic failures).

6. Limitations and Extension Directions

ECom-Bench's present scope is vertically focused on home appliances and furniture; cross-category generalization is untested. Due to privacy, part of the data is LLM-synthesized, limiting authenticity. The dataset's scale (53 tasks) constrains statistical resolution, while persona-feature design is expert-driven rather than empirically optimized.

Key future extensions:

• Expanding to additional e-commerce verticals (fashion, electronics) and greater scenario variety.
• Inclusion of explicit memory modules for tracking dialogue state.
• Enhancing persona realism via empirical studies (e.g., A/B tests).
• Upscaling the number of task instances and introducing dynamic situations (such as concurrent orders) (Wang et al., 8 Jul 2025).

7. Significance and Comparative Positioning

ECom-Bench is the first benchmark in e-commerce explicitly targeting end-to-end, persona-driven, multimodal agent performance for customer support. Its joint evaluation of high-fidelity dialogue, tool integration, and multimodal planning distinguishes it from contemporaneous efforts such as EcomBench (Min et al., 9 Dec 2025) (which emphasizes composite QA and business logic), EComAgentBench (Du et al., 16 Jun 2026) (long-horizon shopping with distributed and hidden intent), and OxyEcomBench (Liu et al., 13 May 2026) (multimodal, multi-role holistic e-commerce understanding), among others. Notably, ECom-Bench's low $t$5 scores expose the difficulty of robust tool-oriented resolution and the current limitations of LLM agents in handling multimodal, rule-driven, stateful customer support in operational e-commerce domains (Wang et al., 8 Jul 2025).