MT-Bench-101: Superlative Query Evaluation

• MT-Bench-101 is a comprehensive framework for evaluating and annotating product recommendation systems handling implicit superlative queries.
• It introduces the domain-adapted SUPERB four-point annotation schema and employs pointwise, pairwise, listwise, and deliberated labeling strategies.
• The framework enhances product recommendations by capturing latent attributes, enabling systematic benchmarking and improved ranking in real-world scenarios.

MT-Bench-101 is a comprehensive framework for the fine-grained evaluation and annotation of product recommendation systems in the context of implicit superlative queries—searches where users request the "best" item but do not specify the underlying criteria. The approach addresses the challenge of extracting and operationalizing latent attributes essential for reasoning about "best-ness," leveraging LLMs and introducing the domain-adapted SUPERB four-point annotation schema. This approach enables systematic benchmarking, supervision, and evaluation of retrieval and ranking models on nuanced, multi-objective user queries in real-world e-commerce scenarios (Dhole et al., 26 Apr 2025).

1. Implicit Superlative Queries and Traditional Recall Limitations

A significant portion of user inquiries in e-commerce and other recommendation platforms are implicit superlative queries—searches formulated as "best," "most durable," or "most stylish" without explicitly stating which evidence or attributes should be maximized. Traditional retrieval and ranking strategies such as BM25 with objective filters or standard ESCI-style (Exact, Synonym, Contains, Irrelevant) judgments are effective when superlative criteria are explicit (e.g., "cheapest," "highest rated") but are inadequate when queries require common-sense or specialized domain knowledge. For instance, users seeking "best toy for a 3-year-old girl" omit key safety, material, or usability dimensions, which are fundamental to a holistic ranking but absent from both queries and standard annotation schema (Dhole et al., 26 Apr 2025).

2. The SUPERB Four-Point Annotation Schema

The SUPERB ("SUPErlative with Best relevance annotations") schema operationalizes a novel four-point relevance taxonomy specifically devised for implicit superlative queries. This framework enables the annotation of candidate products by degree rather than in binary or coarse categories:

Label Value	Description	Example Criterion Fulfillment
3	Overall Best	Satisfies all implicit "best" criteria
2	Almost Best	Satisfies most, but not all, critical dimensions
1	Relevant but Not the Best	Meets minimum expectations, surpassed by others
0	Not Relevant	Fails to address implicit needs of the query

This taxonomy reflects real consumer decision-making processes, capturing the multi-objective optimization landscape of "best-ness" and aligning with human judgment granularity as found in purchase scenarios (Dhole et al., 26 Apr 2025).

3. Formal Schema Definitions and LLM Annotation Protocols

To structure annotation and supervision using LLMs, the following formal procedures and notations are established:

• Pointwise Labeling:

$(q, p_i)\;\xrightarrow{\;M\;}\;b_i + E$

For query $q$ and product $p_i$, model $M$ yields label $b_i \in \{0,1,2,3\}$ and an explanation $E$.

• Pairwise Labeling:

$(q, p_i, p_j)\;\xrightarrow{\;M\;}\;(b_i, b_j) + E$

Jointly labels $p_i$ and $p_j$.

• Listwise Labeling:

$$(q, p_1, ..., p_N)\;\xrightarrow{\;M\;}\;(b_1, ..., b_N) + E$$

$N$-item labeling set for direct comparison.

• Deliberated (Two-Step) Labeling:

1. Attribute Generation: $q \xrightarrow{M} a_q$
2. Label Assignment: $(q, a_q, p_i)\xrightarrow{M} b_i + E$ The two-step process first surfaces implicit query attributes $a_q$, then grounds final label decisions in these surfaced dimensions.

• Listwise Ranking (Re-ranking):

$$(q, p_1, ..., p_N)\;\xrightarrow{\;M\;}\;(r_1, ..., r_N) + E$$

Generates a permutation of candidate indices, reflecting an overall ranking.

Prompting strategies are carefully calibrated to direct the LLM to reason over implicit, multi-faceted evidence—preventing reliance on superficial or self-advertised product attributes (Dhole et al., 26 Apr 2025).

4. Annotation Workflow and Dataset Construction

The closed-loop dataset generation and annotation process involves:

1. Query Generation: Seed queries and human-judged "Exact" ESCI items are reformulated into implicit superlative variants using few-shot prompted LLMs (Claude–Sonnet); from 1,825 seeds, 35,651 superlative queries are produced, with those paired to ≥5 "Exact" items retained.
2. Candidate Selection: All products with "Exact" ESCI labels for each superlative query become SUPERB annotation candidates (29,218 query-product pairs over 2,230 queries).
3. Labeling Implementation: Four LLM prompting configurations—pointwise, pairwise, listwise, and deliberated (two-step)—are used, producing the four-point SUPERB labels with accompanying chain-of-thought rationales.
4. Integration into Ranking Pipelines: The labeled dataset supports evaluation of both first-stage retrieval (BM25, RM3) and LLM-driven re-ranking (pointwise, deliberated, and listwise; including confidence scoring and context-aware ranking using sliding windows for large $N$).

5. Empirical Evaluation and Performance Analysis

Multiple pipelines and evaluation metrics substantiate the efficacy of the SUPERB-based supervision and ranking:

• First-stage Retrieval: BM25 and RM3 establish baseline performance.
• Second-stage Re-ranking:
  • Pointwise and Deliberated Pointwise: Each product is scored using explicit attribute prompts; items are sorted by label, confidence ($c\in\{1, \dots, 9\}$), then BM25 score.
  • Listwise and Sliding-Window Listwise: Candidate sets are jointly ranked; for larger contexts, a windowed permutation strategy ensures tractability.
• Metrics: Precision@k (P@5/10/20), nDCG@k, MAP, and Recall@50 assess both local and broad ranking fidelity.

Key findings:

• Listwise re-ranking achieves statistically significant improvements over BM25 (e.g., P@5 from 0.206 to 0.262, nDCG@5 from 0.219 to 0.278, $p<0.05$).
• Deliberated pointwise re-ranking, leveraging explicit attribute grounding, produces annotation quality with 78.9% LLM–human agreement, compared to 75.2% for non-deliberated pointwise, 66.4% for pure pointwise, 60.8% for listwise, and 44.9% for pairwise. This suggests the two-step deliberated approach best aligns LLM and expert judgments.
• Sliding-window listwise methods maintain performance advantage (P@10 increases from 0.154 to ≥0.185; nDCG@50 from 0.279 to ≥0.309) in extended candidate settings.
• LLM-based re-ranking is most effective for queries requiring non-explicit or common-sense criteria, while lexical BM25 matches excel on precise, attribute-specified queries. Query reformulation with surfaced attribute expansions further enhances recall and MAP (Dhole et al., 26 Apr 2025).

6. Practical Applications and Impact

SUPERB and the MT-Bench-101 methodology serve as robust tools for both the evaluation and improvement of multi-stage recommendation and ranking pipelines for implicit superlative queries. The taxonomy and annotation protocols provide the following advances:

• A domain-adapted schema that captures real-world, gradated judgments of "best" for vague or under-specified search needs.
• A comprehensive benchmark and supervision dataset for training and evaluating LLM-based retrieval, especially in scenarios where multi-objective tradeoffs and implicit criteria are prevalent.
• Empirical support for the superiority of two-step deliberated and listwise LLM strategies, providing methodology blueprints for future e-commerce and marketplace deployment.

The practical significance is the facilitation of more accurate, nuanced product recommendations and the establishment of fairer, more human-aligned evaluation pipelines for ranking models—addressing a central limitation in traditional IR for superlative, multi-criteria queries (Dhole et al., 26 Apr 2025).