SUPERB Schema: Fine-Grained Product Annotation

• SUPERB Schema is a four-level annotation framework that quantifies product relevance for implicit superlative queries using LLM-based reasoning.
• It supports diverse labeling methods—including pointwise, pairwise, listwise, and deliberated prompting—to extract implicit attributes and assign ordinal labels.
• Empirical evaluations demonstrate that methods like listwise re-ranking significantly boost Precision and nDCG metrics, enhancing e-commerce search performance.

The SUPERB schema is a fine-grained, four-level annotation framework for evaluating the relevance and quality of product recommendations in response to implicit superlative queries—those queries where users seek the “best” products using vague, under-specified language (e.g., “best shoes for trail running”) without explicitly articulating the desirable attributes. Designed to leverage the reasoning and world knowledge in LLMs, SUPERB supports the generation of nuanced product relevance labels, enabling recommender and search systems to identify products that excel across the latent dimensions implied by such queries (Dhole et al., 26 Apr 2025).

1. Motivation and Problem Definition

In e-commerce and product search, user queries frequently involve implicit superlative intent, such as “best toy for a 3-year-old who loves dinosaurs” or “best modern fridge for minimalist kitchens.” Unlike explicit superlative queries (e.g., “highest-resolution monitor”), these lack direct references to attributes to be optimized. Standard retrieval and ranking systems, typically dependent on coarse relevance metrics like ESCI or binary labels, are inadequate for capturing the multidimensional and often subjective criteria implicit in user intent. The SUPERB schema was introduced to define a structured, scalable approach for annotating products with respect to such implicit needs—enabling systems to (a) infer relevant latent attributes and (b) perform nuanced, multi-objective assessment of candidate products using LLMs (Dhole et al., 26 Apr 2025).

2. The Four Points of the SUPERB Schema

The SUPERB schema assigns each candidate product an integer label from 0 to 3 based on its degree of excellence with respect to an implicit superlative query, as summarized below:

Label	Definition	Example (for “best running shoes for rocky terrain”)
3 (Overall Best)	Excels across all major, query-relevant dimensions; clear market leader fulfilling the user's need unequivocally	5-star trail shoe, rock-plate sole, superior durability
2 (Almost Best)	Rates highly on most criteria, but with minor deficits (e.g., price, style); near-optimal	Top-tier shoe, but has a higher price or less styling
1 (Relevant Not Best)	Satisfies base relevance but not outstanding in multiple dimensions	Adequate mid-range shoe, average reviews
0 (Not Relevant)	Fails to satisfy core user intent; lacks key features or is off-category	Generic shoe not suitable for rocky terrain

These labels are to be interpreted as an ordinal scale where higher values represent stronger, more evidence-backed matches for the implicit “best” criterion (Dhole et al., 26 Apr 2025).

3. Attribute Generation and Labeling Formalisms

SUPERB operationalizes LLM-based product labeling through several formal structures, supporting both attribute extraction and product assessment:

• Pointwise labeling: Each (query, product) pair is mapped by the LLM to a numeric label with an optional explanation:

$(q, p_1)\;\xrightarrow{M}\; b_1\;+\;E$

where $b_1\in\{0,1,2,3\}$.

• Pairwise labeling: Two products compared jointly for the same query:

$(q, p_1, p_2)\;\xrightarrow{M}\; b_1\,b_2\;+\;E$

• Listwise labeling: N products labeled in a single LLM pass:

$$(q, p_1, \ldots, p_N)\;\xrightarrow{M}\; b_1\,b_2\,\ldots\,b_N\;+\;E$$

• Deliberated prompting (two-stage): First, generate implicit attribute set $a_q$ for query $q$:

$q\;\xrightarrow{M}\;a_q$

then label each product using $a_q$:

$(q, a_q, p_1)\;\xrightarrow{M}\; b_1\;+\;E$

• Listwise re-ranking: Permutation of IDs according to final ranking:

$$(q, p_1, \ldots, p_N)\;\xrightarrow{M}\; r_1\,r_2\,\ldots\,r_N\;+\;E$$

Final sorted order is determined first by descending SUPERB label, then by confidence scores (where available), with original retrieval score (e.g., BM25) as a tiebreaker (Dhole et al., 26 Apr 2025).

4. LLM-Based Annotation Workflow

Construction of a SUPERB-annotated dataset proceeds in a two-stage pipeline:

a. Query Expansion: Starting with 1,825 base queries from the Amazon Shopping Queries set (each with “Exact” ESCI labels), the queries are reformulated into multiple implicit superlative forms using LLM-powered prompting, resulting in 35,651 unique queries.

b. Product Labeling: For each superlative query, products previously marked as “Exact” matches are annotated using four LLM-based techniques:

• Pointwise: Individual product evaluation
• Pairwise: Comparative labeling of two products
• Listwise: Group-wise labeling of $K$ products
• Deliberated (two-stage): Implicit attribute extraction followed by pointwise labeling with explicit attribute guidance

Each annotation includes both the SUPERB label and an LLM-generated rationale. Human evaluation over 107 queries indicated highest inter-annotator agreement for pointwise (66%) and listwise (60%) over pairwise (45%), with deliberated prompting boosting agreement from 75% to 79%. Ultimately, the deliberated pointwise method was employed to produce 29,218 (query, product, label) triplets spanning 2,230 implicit superlative queries (Dhole et al., 26 Apr 2025).

5. Integration into Retrieval and Ranking Systems

With the SUPERB-labeled dataset, several retrieval and re-ranking pipelines were benchmarked:

• BM25 and RM3 baselines served as initial retrieval stages.
• BM25/RM3 + Listwise LLM Re-ranking: Top $K$ candidates presented en masse to LLM for final permutation.
• BM25/RM3 + Deliberated Pointwise Re-ranking: Each product is scored using $q$ and $a_q$; outputs include a confidence score $c_j\in[1,9]$. Sorting occurs first by $b_j$ (label), then $c_j$ (confidence), then BM25.
• Sliding-Window Listwise: For long candidate lists, listwise re-ranking is performed over overlapping windows (e.g., 20 items, stride 10) to handle LLM context limitations.

Performance was evaluated via Precision@K and nDCG@K. Listwise re-ranking registered the largest gains in P@5, P@10, nDCG@5, nDCG@10, and nDCG@20, particularly when initial basal ranking was strong. Deliberated pointwise methods yielded smaller but consistent improvements at reduced computational cost. Sliding-window listwise approaches also outperformed BM25 under long-context scenarios (Dhole et al., 26 Apr 2025).

6. Empirical Evaluation and Observed Limitations

Key experimental insights include:

• Effectiveness: Listwise LLM re-ranking surpasses BM25 by an absolute +5–6% in Precision@5 and +4–5% in nDCG@5 ($p<0.05$). Deliberated pointwise methods provide incremental gains in P@10.
• Limitations: LLM-based ranking demonstrates weaknesses on queries with strong negation (“plates not plastic”) or highly atypical attribute combinations.
• Aesthetic/subjective queries: On queries requiring nuanced style or aesthetic judgments (“modern minimalist kitchen fridge”), LLMs outperform classic retrieval.
• Lexical exact-match queries: Scenarios demanding strict lexical criteria favor baseline IR methods.
• Annotation quality: Human raters preferred deliberated pointwise annotation (agreement 78.9%). Making implicit attributes explicit was found to mitigate annotation bias caused by marketing language.

Collectively, SUPERB enables (1) annotation of product relevance with respect to implicit superlative intent using a unified ordinal scale, (2) exploitation of LLM-generated attribute reasoning for recall and ranking, and (3) empirical advances in recommendation quality over traditional retrieval, especially in multi-dimensional “best” scenarios (Dhole et al., 26 Apr 2025).