EcomScriptBench: E-commerce Script Planning

• EcomScriptBench is a large-scale benchmark that defines and evaluates e-commerce script planning by decomposing user objectives into actionable, product-linked steps.
• It comprises three subtasks—script verification, step-product discrimination, and script-products verification—to assess coherence, feasibility, and product relevance.
• Built using real-world Amazon data and advanced LLM prompting techniques, it offers rigorous metrics to benchmark and enhance multi-step shopping assistance in LLMs.

EcomScriptBench is a large-scale, multi-task benchmark designed to advance research in goal-oriented E-commerce Script Planning, a task domain at the intersection of long-form planning, multi-step action decomposition, and product recommendation. The benchmark provides data and evaluation protocols for assessing LLMs’ (LLMs) abilities to generate coherent, actionable shopping scripts that associate concrete products with user goals, as well as to verify script plausibility and product-step alignment. EcomScriptBench addresses foundational challenges in automating e-commerce assistance, particularly the need for fine-grained product reasoning and complex planning consistency, through a systematically constructed dataset and three rigorously defined evaluation subtasks (Wang et al., 21 May 2025).

1. E-commerce Script Planning Task and Subtask Definitions

E-commerce Script Planning (EcomScript) is defined as the process by which computational models take a user-specified objective $o$ (e.g., “organize a home theater viewing”) and decompose it into a coherent sequence of actionable steps, or a script $S_o = \{s_1, \dots, s_k\}$, where steps requiring purchases are explicitly linked to relevant products. EcomScriptBench formalizes this through three core subtasks:

1. Script Verification (T1).
   • Input: objective $o$ and candidate script $S_o$.
   • Output: binary label $T_1(o, S_o) \in \{0, 1\}$, with $1$ indicating the script is plausible and feasible for $o$.
2. Step-Product Discrimination (T2).
   • Input: objective $o$, step $s_i$, and candidate product $p$.
   • Output: binary label $S_o = \{s_1, \dots, s_k\}$0, indicating if purchasing $S_o = \{s_1, \dots, s_k\}$1 assists execution of $S_o = \{s_1, \dots, s_k\}$2.
3. Script-Products Verification (T3).
   • Input: objective $S_o = \{s_1, \dots, s_k\}$3, script $S_o = \{s_1, \dots, s_k\}$4, and sets of up to three products per step $S_o = \{s_1, \dots, s_k\}$5.
   • Output: binary label $S_o = \{s_1, \dots, s_k\}$6, with $S_o = \{s_1, \dots, s_k\}$7 denoting all products are suitable for their assigned steps and together yield a conflict-free plan.

This decomposition enables modular evaluation of LLM capabilities in script plausibility, fine-grained product-step association, and global multi-product plan consistency (Wang et al., 21 May 2025).

2. Data Construction and Corpus Properties

EcomScriptBench is constructed from real-world e-commerce data, primarily leveraging Amazon product reviews and product metadata. The data generation pipeline includes:

• Product Pool and Review Mining:
  • 10% of unique products sampled per Amazon category, yielding 2.4 million products and 3.7 million associated reviews.
  • Reviews filtered to ensure quality (length > 10 tokens; < 5 hashtags).
• Objective and Script Generation:
  • GPT-4o-mini, prompted with few-shot exemplars, distills user objectives from reviews and generates up to 10-step scripts per objective.
• Purchase Intention Mining:
  • For each product, GPT-4o-mini generates $S_o = \{s_1, \dots, s_k\}$8 purchase intentions $S_o = \{s_1, \dots, s_k\}$9, designed to summarize conceivable user purchase motivations.
• Step-Intention Association:
  • Each script step is flagged by an LLM for product necessity and associated with step-specific product-keywords.
  • Step and intention embeddings are computed using SentenceBERT.
  • Semantic similarity per product:

  $o$0 - Candidate products are filtered by a similarity threshold $o$1 and ranked; top three retained per step.

• Annotation and Gold Labels:

  • For each subtask, 5,000 instances are sampled (total 15,000); crowd annotation via five-annotator majority vote on AMT with expert verification.
  • Inter-annotator agreement: 78% pairwise; Fleiss’ $o$2.
  • Expert acceptance rates: T1 95.5%, T2 96.5%, T3 97.0%.

Dataset statistics:

• 605,229 scripts (avg. 71.5 tokens; 94.0% expert-approved)
• 5.93 million steps (avg. 7.48 tokens/step; 3.02 million with product candidates)
• 2.4 million products with 24 million intentions (avg. 10.27 tokens/intention)
• Coverage of 14 major Amazon categories; product requirements increase in later script steps.

3. Evaluation Protocols and Experimental Setup

EcomScriptBench supports rigorous model evaluation across open and proprietary LLMs and PTLMs. Experimental configuration includes:

• Model Families:
  • PTLMs (zero-shot): RoBERTa-Large, DeBERTa-Large, CAR, CANDLE, VERA-xl (3B), VERA-xxl (11B).
  • Open LLMs (zero-/few-shot): Meta-Llama-3 (8B–405B), Gemma-2 (2B–27B), Phi-3.5-mini (4B), Falcon2-11B, Mistral-7B, Mistral-12B, Mixtral.
  • Proprietary APIs: GPT-4o, GPT-4o-mini (prompting: zero-shot, 5-shot, Chain-of-Thought, Self-Consistency, Self-Reflection).
• Fine-tuning:
  • LoRA adapters (rank=16, $o$3) via LlamaFactory, Adam optimizer ($o$4, batch=8, $o$5 seq len=300, 3 epochs).
  • Intention-injection: sequential fine-tuning on FolkScope+MIND (intention KBs), then EcomScriptBench.
• Prompting:
  • 20 few-shot exemplars per task; appendix of prompt templates.
  • Advanced prompting with Chain-of-Thought, Self-Consistency, and Self-Reflection (Wang et al., 21 May 2025).
• Metrics:
  • Accuracy (Acc), AUC, Macro-F1; baselines of random and majority classifiers.

4. Empirical Results and Model Analyses

Key empirical findings include:

• Task Difficulty:
  • Zero-shot PTLMs yield near-random performance (T1–T3: 52–56% Acc).
  • Zero-shot open LLMs: T1 $o$670–75%, T2 $o$760–68%, T3 $o$860–66%.
  • Fine-tuned LLMs: T1 up to 86%, T2 up to 76%, T3 up to 74%.
  • GPT-4o-family APIs (best few-shot/self-reflection): T1 77.9%, T2 73.9%, T3 72.9%.
• Impact of Intention Knowledge:
  • Adding FolkScope+MIND as a pre-finetuning stage boosts T2 by 5–8 points and T3 by 2–6 points over direct EcomScriptBench fine-tuning.
• Error Analysis (manual, GPT-4o):
  • Incorrect product understanding: 68%
  • Cross-step reasoning conflicts: 27%
  • Internal conflicts/annotation noise: 5%

This demonstrates the difficulty of integrating product-level semantics and maintaining stepwise consistency in multi-product plans.

5. Design Principles and Recommendations for Script Benchmarking

EcomScriptBench synthesizes prior expertise in language specification testing and benchmarks, drawing on lessons from N+1-version differential testing in conformance test design (Park et al., 2021). Core benchmarking design principles include:

• Mechanized Specification Oracle:

Extracting and executing the ECMAScript grammar and semantics enables oracle-based classification of engine versus specification bugs.

• Coverage-Driven Synthesis:

Grammar-based shortest-string and non-recursive seed generation, followed by semantics-guided mutations, maximize coverage with minimal test pool size.

• Assertion and State Capture:

Capturing and asserting over the complete program state (exceptions, variable values, heap and object shape, property attributes, key order) for each synthesized test enables detection of both implementation and specification faults.

• N+1 Execution:

Simultaneous execution on all engines plus the mechanized spec allows cross-validation and accurate localization.

• Statistical Bug Localization:

Use of spectrum-based fault localization (ER1_b) for correlating failures to specific abstract algorithm steps.

• Compactness and Coverage:

~1,700 tests achieve $o$990% semantic coverage; methodology is transferable to script-driven and product-driven scenarios.

A plausible implication is that such methodologies, if applied to future iterations of EcomScriptBench, could enable highly effective core suites for both engine and spec validation, as well as cross-engine benchmarking (Park et al., 2021).

6. Limitations and Prospects for Future Research

EcomScriptBench exhibits several intrinsic limitations:

• Construction Dependency: Heavy reliance on GPT-4o-mini during generation of objectives, scripts, and intentions introduces cost and potential reproducibility issues.
• Conflict Detection: No automated methods yet for detecting multi-product or cross-step plan conflicts; limitations in handling global reasoning.
• Modality Restriction: Dataset and tasks are text-only; images and structured metadata are not considered, limiting real-world product grounding.
• Annotation Constraints: Annotation pipeline includes some irreducible inter-annotator disagreement.

Recommendations for future work include expanding to multimodal benchmarks (integrating images and rich metadata), automating inter-product conflict detection, developing end-to-end generative architectures, exploring on-device LLM deployment, and advancing knowledge-editing for real-time retail assistant applications (Wang et al., 21 May 2025).

7. Significance and Impact

EcomScriptBench establishes the first unified, large-scale benchmark for E-commerce Script Planning, catalyzing research at the nexus of script generation, product retrieval, and actionable assistant planning. Its multi-task formulation bridges isolated research strands in task planning and recommender systems, exposes critical weaknesses in current LLM architectures, and provides systematically curated, expert-verified data across 2.4 million products and 605,229 scripts.

The benchmark sets a new standard for evaluating goal-oriented e-commerce assistance, with challenging subproblems that require joint reasoning over domains, products, and action sequences. By highlighting specific model errors and providing pathways for improvement through injected intention knowledge and advanced prompting/fine-tuning, EcomScriptBench accelerates progress toward LLM-driven, user-centric, multi-step shopping assistants (Wang et al., 21 May 2025).