E-Commerce Cognitive Decision Benchmark

• ECCD-Bench is a domain-specific evaluation framework that rigorously measures AI models' factual, cognitive, and decision-making abilities in e-commerce.
• It employs a knowledge-graph foundation and automated question generation, combined with expert oversight, to mirror real-world customer needs and business logic.
• Empirical results reveal significant gaps in abstract reasoning and consistency among models, underscoring the need for tailored AI training in e-commerce.

The E-Commerce Cognitive Decision Benchmark (ECCD-Bench) is a domain-specific evaluation framework designed to comprehensively and rigorously assess the factual, cognitive, and decision-making capabilities of artificial intelligence models—particularly LLMs—within the complex operating context of online commerce. ECCD-Bench aims to surpass generic NLP or generic “agentic” benchmarks by targeting those decision challenges most critical for e-commerce platforms: accurate reasoning about products, user intentions, constraints, and the implicit relations that underlie real customer needs and business logic.

1. Motivation and Positioning in the Landscape

ECCD-Bench addresses a prominent deficit in existing benchmarks, where the evaluation of LLMs in e-commerce is either superficial (simple factoid QA, basic product retrieval) or insufficiently aligned with high-stakes, domain-specific requirements such as factuality, multi-relational reasoning, scenario awareness, and robustness against hallucination. In conventional e-commerce search and recommendation engines, misalignments between evaluation protocols and production tasks have translated into degraded user experience and real financial losses. ECCD-Bench is explicitly constructed to provide a robust, scalable, and standardized yardstick for factual and cognitive decisions in e-commerce scenarios, leveraging a blend of structured domain knowledge (knowledge graphs), human expertise, and automated data generation (Liu et al., 20 Mar 2025).

2. Underlying Methodology and Benchmark Construction

ECCD-Bench’s technical core is its knowledge-graph-based and human-in-the-loop automated question generation pipeline, which ensures both domain realism and metric reliability. The benchmark construction involves several formally defined and empirically validated steps:

1. Knowledge Graph (KG) Foundation:
   • The dataset is derived from large-scale e-commerce knowledge graphs (e.g., Taobao’s ECKG: 4.8M triples, with entities such as product, function, style, color; and relations such as “has function”, “suitable for”, “similar to”).
   • Each triple is annotated as “true” or “false” by human experts, enabling both positive and negative query construction.
2. Automated Question Generation Pipeline:
   • Relation Templating: LLMs generate candidate natural language templates for each KG relation, which are then edited by experts for semantic clarity and alignment (e.g., “Rapid heating (function) is very important for ____ (category)”).
   • Negative Sampling: A three-stage process: pool selection of plausible but false answers, semantic filtering by embeddings and cosine similarity to control distractor difficulty, and a final LLM-assisted curation step to maximize discrimination power without unfairness.
   • Prompt Assembly: Questions are rendered in multi-choice format, with single blanks and four answer candidates (1 correct, 3 distractors), and accompanied by explicitly crafted instructions.
   • Verification: Low-quality or potential regulatory violations are flagged by LLMs and then manually checked by e-commerce experts.
3. Human Expertise Integration:
   • Human annotators participate in template design, prompt clarification, distractor vetting, and final compliance verification to inject authentic e-commerce expertise at all critical workflow stages.

3. Evaluation Protocols and Cognitive Metrics

ECCD-Bench incorporates a multi-faceted evaluation regime, designed to measure both factual correctness and cognitive discrimination in decision scenarios:

1. Accuracy: Percentage of correct answers in the multiple-choice setup.
2. Inconsistency: Probability that a model’s answer changes when only the distractors (not the correct answer) differ, quantifying reliability and robustness.
3. Cognitive Boundary Analysis: Following SliCK-inspired protocol, the “knowledge state” of each question is classified as Well Known (WK), Somewhat Known (SK), or Unknown (UK) by repeated stochastic sampling at elevated temperature. Granular statistics are reported:

$$\text{SC@k} = \frac{1}{|\mathbf{q}|} \sum_{q_i \in \mathbf{q}} \bigwedge_{j=1}^k I(R^j(q_i) = GT(q_i))$$

$$\text{Recall@k} = \frac{1}{|\mathbf{q}|} \sum_{q_i \in \mathbf{q}} \bigvee_{j=1}^k I(R^j(q_i) = GT(q_i))$$

with - SC = WK proportion - Recall = WK + SK = $1 -$UK

1. Efficiency Metrics: Reports on token usage and average evaluation latency (typically <3 sec), with emphasis on practical cost and throughput gains of the multiple-choice design.

4. Coverage, Question Taxonomy, and Knowledge Types

ECCD-Bench explicitly separates question types according to cognitive load and real-world utility:

• Common Knowledge: User-facing, frequently encountered e-commerce logic (e.g., “Which function is essential for this type of product?”).
• Abstract Knowledge: Higher-order or relational reasoning, such as style similarity or implicit user-group suitability, which is critical for advanced recommendation, product discovery, and prospecting new user needs.

The negative sampling pipeline is engineered to ensure that distractors for both types emulate challenging but non-trivial confusion, avoiding degenerate patterns (e.g., too-easy elimination or semantic implausibility).

5. Empirical Findings and Model Performance

Evaluation of 12 contemporary LLMs—covering both closed-source and open-source families under zero-shot and few-shot regimens—yields several systematic findings:

• Overall difficulty: Even the best-performing models (GPT-4, Qwen2-max) do not surpass 70% accuracy, with most models under 60%. This highlights the sparsity and complexity of e-commerce knowledge in current pretraining regimes.
• Cognitive gap: All models achieve measurably better results on common knowledge versus abstract knowledge, indicating a significant challenge in relational/cognitive decision-making.
• Scaling trends: Size scaling law holds; larger models significantly outperform smaller ones within the same architecture.
• Negative sampling benefit: The adoption of three-phase distractor selection substantially reduces answer inconsistency and stabilizes accuracy statistics.
• Boundary metric insight: Models retain substantial “unknown zones”—questions that are never answered correctly even at high diversity, exposing systematic pretraining blind spots. Fine-tuning or in-context learning boosts SK (somewhat-known) but cannot overcome these deep omissions.
• Computational efficiency: The multiple-choice pipeline achieves substantial cost and time savings, critical for benchmark scaling and routine use in model validation cycles.

6. Significance, Applications, and Future Directions

ECCD-Bench provides a benchmark that is robust, scalable, and attuned to the distinct factual and cognitive requirements of e-commerce decision tasks. Its methodological innovations (e.g., knowledge-graph grounding, robust negative sampling, cognitive-layer question design, expert-in-the-loop dataset curation) establish a reliable protocol for current and future LLM evaluation in production-grade e-commerce deployments. The benchmark is directly applicable for:

• Pretraining/fine-tuning assessment for e-commerce ready LLMs.
• Regression/stress testing after continual learning or retrieval-augmentation updates.
• Vendor/model comparison for enterprise deployment selection.
• Targeted analysis of cognitive and factuality issues in high-value or high-risk product categories.

A plausible implication is that the ECCD-Bench framework can be adapted or extended for other vertical domains where knowledge reliability and high-dimensional decision-making are equally critical. Nevertheless, strong empirical results suggest that even today’s most powerful models require further architectural or data-centric advances to reach expert-level e-commerce cognitive performance.

7. Relationship to Related Benchmarks and Research

ECCD-Bench stands apart from generic QA, dialogue, or open-domain agentic evaluation suites by focusing on grounded, commercially relevant, and cognitively loaded decision tasks in e-commerce. It is similar in spirit, but more domain-targeted, than benchmarks such as ECKGBench (Liu et al., 20 Mar 2025), ShoppingBench (Wang et al., 6 Aug 2025), DeepShop (Lyu et al., 3 Jun 2025), and Amazon-Bench (Zhang et al., 18 Aug 2025). Unlike single-domain or basic search-focused datasets, ECCD-Bench’s integration of structured ontology, negative sampling, and human-in-the-loop quality assurance offers a richer, multi-granular substrate for evaluating LLMs in complex, dynamic, and high-value enterprise domains.

Summary Table: ECCD-Bench Workflow Components

Step	Method / Tool	Role
KG triple extraction	E-commerce KG, human labels	Data grounding
Template generation	LLMs + Human refinement	Question clarity, domain fit
Negative sampling	Embedding + LLM + human	Distractor diversification
Prompt/instruction design	Human + LLM	Evaluation precision
Verification	LLM flagging + expert audit	Filter low-quality/noisy items
Final evaluation	LLM scoring; human analysis	Accuracy / inconsistency metrics

ECCD-Bench’s principled, scalable, and expert-driven methodology sets a high bar for robust evaluation of machine cognition in e-commerce and related high-reliability domains.