SessionIntentBench: E-commerce Intent Benchmark

• SessionIntentBench is a multimodal benchmark that models e-commerce sessions using explicit intention trees to capture inter-session intention shifts.
• It curates data from Amazon sessions and product images with multi-modal attribute extraction and human annotation to create a large-scale, structured intention dataset.
• The benchmark design decomposes evaluation into four subtasks that expose current L(V)LM limitations and motivate architectural and training improvements.

Searching arXiv for the specified paper to ground the article and citation. SessionIntentBench is a sibling multimodal benchmark for understanding inter-session intention shift in e-commerce customer behavior, introduced to address two deficiencies identified in prior work: insufficient information exploitation and reliance on only apparent information such as descriptions and titles. It is built around an explicit intention representation, the intention tree, and evaluates L(V)LMs on four subtasks that probe purchasing likelihood, valued attributes, comparison-based justification, and intention evolution. The benchmark is constructed from Amazon-M2 sessions and Amazon Review product images, and comprises $1{,}952{,}177$ intention entries, $1{,}132{,}145$ session intention trajectories, and $13{,}003{,}664$ available tasks mined using $10{,}905$ sessions (Yang et al., 27 Jul 2025).

1. Conceptual core: the intention tree

The benchmark’s formal object is the intention tree. It is defined as $T=(V,E)$ induced by a customer’s session of length $T$, where $V$ is the set of intention nodes and $E \subseteq V \times V$ is the set of parent$\to$child edges. Each node $v \in V$ is annotated with a triple $1{,}132{,}145$0, where $1{,}132{,}145$1 is the inferred purchase intention at time step $1{,}132{,}145$2, $1{,}132{,}145$3 is the decisive product attribute at $1{,}132{,}145$4, and $1{,}132{,}145$5 is a natural-language comparison between $1{,}132{,}145$6 and $1{,}132{,}145$7 (Yang et al., 27 Jul 2025).

Customer history is written as

$1{,}132{,}145$8

where $1{,}132{,}145$9 is the $13{,}003{,}664$0-th product viewed and $13{,}003{,}664$1 the extracted attribute set. The edge structure links successive intention states: from $13{,}003{,}664$2 to $13{,}003{,}664$3. This formulation makes intention shift explicit rather than treating a browsing session as an undifferentiated click sequence.

Tree growth is generated by branching. If at step $13{,}003{,}664$4 there are $13{,}003{,}664$5 active intention nodes, an LVLM is prompted to generate up to $13{,}003{,}664$6 new intentions per node, so that at step $13{,}003{,}664$7 there are up to $13{,}003{,}664$8 nodes. In the released construction, the branching factor is later specialized to $13{,}003{,}664$9 for $10{,}905$0 and $10{,}905$1 thereafter. This suggests that the benchmark prioritizes broad hypothesis expansion in the early part of a session and deterministic continuation in later steps.

2. Dataset curation pipeline and scale

The curation pipeline proceeds in five stages. Data collection starts from Amazon-M2 sessions and Amazon Review product images, and filters to keep only sessions where every $10{,}905$2 has both text and image, yielding $10{,}905$3 sessions (Yang et al., 27 Jul 2025).

Multi-modal attribute extraction retrieves textual metadata for each $10{,}905$4—including title, price, and color—and its image. GPT-4o-mini is then used to extract a simple category and a set of attribute:value pairs. This stage supplies the structured product-side evidence used later in intention inference.

Customer-intention generation rolls out an intention tree over each of the $10{,}905$5 sessions. With branching factor $10{,}905$6 for $10{,}905$7 and $10{,}905$8 thereafter, the resulting full forest contains total intention entries

$10{,}905$9

that is, $T=(V,E)$0 over the full forest, and approximately $T=(V,E)$1 distinct intention trajectories, defined as root$T=(V,E)$2leaf paths.

Inter-session metadata enrichment adds a comparison $T=(V,E)$3 and a rationale for why $T=(V,E)$4 drives $T=(V,E)$5 at each node. Human annotation then samples $T=(V,E)$6 trajectories and generates $T=(V,E)$7 task-instances across the four subtasks. Each instance is labeled on a 4-way scale for Tasks 1–3 and a 3-way scale for Task 4. The abstract further states that human annotations were used to collect ground-truth label for a subset of the collected data to form an evaluation gold set.

In compact form, the benchmark reports

$T=(V,E)$8

3. Benchmark task design and evaluation protocol

Across all subtasks, the model receives some subset of $T=(V,E)$9 together with selected generated metadata and outputs a discrete score. The four tasks are organized as follows (Yang et al., 27 Jul 2025).

Task	Input	Output
Task 1. Intent-Based Purchasing Likelihood Estimation	$T$0	$T$1
Task 2. Purchasing Likelihood via Valued Attributes	$T$2	$T$3
Task 3. Intention Justification via Comparison	$T$4	$T$5
Task 4. Intention Evolution Modeling	$T$6	$T$7, where $T$8 and $T$9

The training objective is standard cross-entropy:

$V$0

Evaluation reports Accuracy and Macro-F1. Accuracy is defined as

$V$1

For Task 4, binary-classification versions merge labels $V$2 versus $V$3 and report Acc and Macro-F1.

The task suite separates distinct but related reasoning demands. Task 1 conditions on inferred intention, Task 2 on valued attributes, Task 3 on comparison-driven justification, and Task 4 on the exploit–explore structure of intention evolution. This task decomposition makes the benchmark explicitly multi-task rather than a single aggregate prediction problem.

4. Experimental setup and reported results

The benchmark evaluates over 20+ open LLMs and LVLMs, plus GPT-4o and GPT-4o-mini APIs, under zero-shot, few-shot, and fine-tuned regimes (Yang et al., 27 Jul 2025). The reported condensed accuracy table includes Random, Majority vote, Llama-3.1-8B (ZS), Mistral-7B-v0.3 (ZS), LLaVA-vicuna-7B (ZS), Falcon-3-7B (FT on SIB), GPT-4o-mini (ZS), GPT-4o-mini (COT), and GPT-4o (COT).

Representative results show the difficulty of the benchmark. Random obtains $V$4 on Tasks 1–3 and $V$5 on Task 4. Majority vote reaches $V$6 on Task 1, $V$7 on Task 2, $V$8 on Task 3, and $V$9 on Task 4. Among zero-shot open models listed, Llama-3.1-8B records $E \subseteq V \times V$0, $E \subseteq V \times V$1, $E \subseteq V \times V$2, and $E \subseteq V \times V$3 on Tasks 1–4 respectively; Mistral-7B-v0.3 records $E \subseteq V \times V$4, $E \subseteq V \times V$5, $E \subseteq V \times V$6, and $E \subseteq V \times V$7; LLaVA-vicuna-7B records $E \subseteq V \times V$8, $E \subseteq V \times V$9, $\to$0, and $\to$1.

Fine-tuning on SIB does not eliminate the difficulty. Falcon-3-7B (FT on SIB) reports $\to$2 on Task 1, $\to$3 on Task 2, $\to$4 on Task 3, and $\to$5 on Task 4. GPT-4o-mini (ZS) reports $\to$6, $\to$7, $\to$8, and $\to$9, while GPT-4o-mini (COT) reports $v \in V$0, $v \in V$1, $v \in V$2, and $v \in V$3. GPT-4o (COT) reaches $v \in V$4 on Task 1, $v \in V$5 on Task 2, $v \in V$6 on Task 3, and $v \in V$7 on Task 4.

The paper states that extensive experiments on the annotated data confirm that current L(V)LMs fail to capture and utilize intention across the complex session setting. A plausible implication is that explicit intent metadata does not automatically translate into robust session-level reasoning, especially when performance remains close to or below simple baselines on several subtasks. The reported statistical significance result states that, on a held-out test split, paired $v \in V$8-tests between best zero-shot and best fine-tuned scores obtained $v \in V$9 for improvements on Task 4 when using chain-of-thought prompting versus zero-shot.

5. Failure modes and intention injection

A targeted error analysis examines a random sample of 200 GPT-4o (COT) errors. The largest source, $1{,}132{,}145$00, is incorrect understanding of metadata, described as failure to attend to earlier $1{,}132{,}145$01. Another $1{,}132{,}145$02 is attributed to noisy or conflicting human labels. The remaining categories are $1{,}132{,}145$03 from missing salient product features, $1{,}132{,}145$04 from outright hallucinations or irrelevant reasoning, and $1{,}132{,}145$05 from inability to form a coherent long-term intention when $1{,}132{,}145$06 is vague (Yang et al., 27 Jul 2025).

To mitigate these issues, the paper studies intention injection. The procedure first fine-tunes on MIND, described as a $1{,}132{,}145$07 M-entry multimodal intention knowledge base, and then fine-tunes on SIB. Compared to fine-tuning on SIB alone, this two-stage regimen yields average lifts of

$1{,}132{,}145$08

and is used to support the claim that concatenating an external intent embedding $1{,}132{,}145$09 to the model input, or pre-injecting it as adapter layers, measurably helps capture inter-session intention shifts.

This analysis narrows the benchmark’s difficulty to several concrete sources: long-range session context use, metadata reliability, feature completeness, hallucination control, and ambiguity in decisive attributes. It also distinguishes between errors attributable to model reasoning and errors attributable to supervision quality.

6. Recommended extensions and downstream use

The paper recommends several architectural and training directions. Under architectural enhancements, it proposes a dedicated “Intention Encoder” module whose hidden states $1{,}132{,}145$10 are fused, via cross-attention or concatenation, into the base LLM/LVLM. It also recommends hierarchical session models that alternate between “intention summarization” and “product reasoning” layers (Yang et al., 27 Jul 2025).

Under training strategies, the authors recommend multi-task pretraining on MIND + SessionIntentBench to instill a strong inductive bias for intent reasoning, and curriculum learning in which tasks progress from single-item intentions to two items and then to full session. These recommendations are presented as future directions rather than established benchmark results.

Two downstream applications are explicitly suggested. For real-time recommender systems, predicted $1{,}132{,}145$11 and $1{,}132{,}145$12 can be used to rank candidate products $1{,}132{,}145$13 by

$1{,}132{,}145$14

For personalized dialog agents in e-commerce, clarifying questions can be surfaced when

$1{,}132{,}145$15

is high, indicating intention uncertainty.

Taken together, these recommendations position SessionIntentBench not only as an evaluation set but also as a design probe for systems that must integrate session history, multimodal product evidence, comparison structure, and evolving user intention. This suggests that the benchmark’s long-term value may lie as much in diagnosing intent-reasoning bottlenecks as in leaderboard-style model comparison.