RecBench+: LLM Recommendation Benchmark

• The paper introduces RecBench+, a benchmark that evaluates LLMs’ ability to generate personalized, context-aware recommendations using natural, open-ended queries.
• It employs a robust query taxonomy—explicit, implicit, and misinformed—that captures real-world ambiguities and diverse user profiles from movie and book domains.
• The dataset facilitates systematic evaluation of factual retrieval and inference through metrics such as precision, recall, and condition match rate.

RecBench+ is a public benchmark specifically designed to evaluate the capabilities of LLMs when acting as interactive, personalized recommendation assistants. Developed in response to the limitations of traditional recommender system (RecSys) datasets, which primarily provide silent user–item interactions or templated prompts, RecBench+ introduces a high-quality corpus of open-ended user queries that reflect the complexity and ambiguity found in real-world recommendation scenarios. The dataset includes approximately 34,500 annotated queries sourced from the MovieLens-1M and Amazon-Book corpora, encompassing diverse query types, explicit and implicit constraints, misleading information, and user profiles with both demographic and interest-based attributes. Each query is paired with a ground-truth set of recommended items, enabling systematic assessment of factual retrieval, inference, and robustness to real-world ambiguities in LLM-powered recommendation systems (Huang et al., 12 Mar 2025).

1. Motivation and Scope

RecBench+ was motivated by two empirical gaps: the absence of free-form, realistic user queries in leading RecSys datasets, and the need to evaluate LLMs’ ability to function beyond static, task-specific prompts. The advance of LLM architectures has shifted the paradigm from item-ranking to providing conversational, context-aware recommendations. However, prior datasets do not test whether LLMs can handle multifaceted, ambiguous, or erroneous user requests, nor do they evaluate resilience to imprecise language. RecBench+ fills this gap by providing a testbed that targets LLMs acting as recommendation assistants, requiring them to process both "hard" conditions (strict attributes) and "soft" preferences (stylistic or affective cues), and to navigate user queries featuring varying levels of specificity, implicitness, or misinformation (Huang et al., 12 Mar 2025).

2. Query Taxonomy and Structure

RecBench+ organizes queries according to a taxonomy that delineates the structural and semantic challenges presented to LLMs. Queries are divided into two broad categories:

• Condition-based Queries: These articulate (directly or indirectly) concrete constraints on recommended items, subdivided into:
  • Explicit Condition Queries: All constraints stated overtly.
  • Implicit Condition Queries: At least one constraint is referenced obliquely, requiring inference.
  • Misinformed Condition Queries: Constraints include factually incorrect elements, necessitating error detection or graceful handling.
• User Profile-based Queries: These depend on user tastes or demographic attributes rather than explicit filters, further split into:
  • Interest-based Queries: Inferred from common co-watch or co-read patterns.
  • Demographics-based Queries: Defined by fixed user attributes such as age, occupation, or gender.

Each query is represented in a structured JSON schema, including fields for query_id, text, type, conditions, preferences, user_profile, ground_truth, difficulty, and domain (movie or book). Queries span a range of difficulties and reflect issues such as under-specified requests and misleading facts (Huang et al., 12 Mar 2025).

3. Data Collection and Annotation Process

The dataset was constructed via a pipeline comprising:

1. Item Knowledge Graph Construction: For movies, features were linked using Wikipedia data for attributes such as director, cast, cinematography, genre, and year. For books, metadata were obtained from the Amazon-Book dataset, including author and category information.
2. Condition Extraction: User histories were mined to extract attribute combinations, e.g., all films by a given director.
3. Query Generation via LLMs: GPT-4o was prompted to convert attribute tuples or profile summaries into natural-language queries, ensuring the generation of explicit, implicit, or misinformed queries as appropriate.
4. Automatic Annotation: Ground-truth recommendation sets were algorithmically determined by matching against the knowledge graph for condition queries or via collaborative filtering and top-popularity thresholds for profile queries. Manual spot-checking was conducted for ~5% of queries to ensure linguistic naturalness.

No fixed train/validation/test splits are imposed, allowing research flexibility; common partitioning approaches include stratification by query type or time-based splits on user histories (Huang et al., 12 Mar 2025).

Data Split	Movie Domain	Book Domain
Explicit	5,000+	5,000+
Implicit	4,000+	4,000+
Misinformed	4,000+	4,000+
Interest-based	~2,500	~2,500
Demographics	279	–

4. Dataset Format and Access

RecBench+ is distributed as a JSON Lines file (.jsonl), with each entry encoding all query metadata in a single line as per the schema described above. Essential fields include the query text, attribution of type and difficulty, conditions or user profile as relevant, ground-truth item IDs, and domain. Example ingestion code in Python is provided:

import json with open("recbench_plus.jsonl") as fin: for line in fin: sample = json.loads(line.strip()) # sample["text"], sample["ground_truth"], …

Comprehensive documentation, example scripts (including PyTorch data loaders), and a BSD-style license are provided. The dataset and related tooling are accessible at https://github.com/jiani-huang/RecBench.git (Huang et al., 12 Mar 2025).

5. Evaluation Protocols and Metrics

The benchmark supports evaluation in two modes: (1) with raw queries only, and (2) with appended user interaction histories. The recommended evaluation metrics encompass:

• Precision: Proportion of recommended items present in the ground-truth set.
• Recall: Proportion of ground-truth items successfully recommended.
• Condition Match Rate (CMR): For condition queries, the fraction of recommended items that satisfy all specified constraints.
• Fail-to-Recommend (FTR): Fraction of queries for which the model outputs no recommendations (higher is preferable only for misinformation detection).

Additional metrics align with standard information retrieval conventions, such as Precision@K, Recall@K, and Mean Reciprocal Rank (MRR):

$$\text{Precision@K} = \frac{1}{K}\sum_{i=1}^{K}\mathbf{1}\{item_i \in \text{ground\_truth}\},\quad \text{Recall@K} = \frac{\lvert \{item_1,\dots,item_K\}\cap \text{ground\_truth}\rvert}{\lvert \text{ground\_truth}\rvert}$$

$$\mathrm{MRR} = \frac{1}{|Q|}\sum_{q=1}^{|Q|}\frac{1}{\mathrm{rank}_q}$$

Seven LLMs were evaluated (GPT-4o, GPT-4o-mini, Llama-3.1-70B-Instruct, Gemini-1.5-Pro, Claude-3.5-Sonnet, DeepSeek-V3, DeepSeek-R1) across both settings. Notable aggregate results for the movie domain include:

Model	Precision	Recall	CMR
GPT-4o	0.157	0.224	0.374
DeepSeek-R1	0.148	0.326	0.414

On profile-based queries, Gemini-1.5-Pro and DeepSeek-R1 achieved highest recall (0.072–0.075) and precision (0.019–0.020) (Huang et al., 12 Mar 2025).

6. Key Insights and Implications

Empirical evaluation with RecBench+ led to several observations:

• LLMs demonstrate initial competence as recommendation assistants, especially when queries are explicit.
• Performance degrades substantially on queries requiring inference (implicit constraints) or fact verification (misinformed queries), highlighting current deficiencies in reasoning and robustness.
• Advanced reasoning models such as DeepSeek-R1 exhibit smaller performance drops between explicit and implicit queries, and are more effective at rejecting recommendations for misinformation.
• Incorporating user history generally increases precision but can decrease CMR, indicating that personalization may conflict with strict attribute conditions.
• LLMs are more reliable for shared interests than for niche preferences or where book metadata present ambiguities (e.g., edition variants).
• Demographics-based queries reflect non-uniform recall, with higher recall for female users and users in certain occupations, likely due to stronger behavioral signals in the training distribution (Huang et al., 12 Mar 2025).

A plausible implication is that future LLM-powered recommendation assistants will need enhanced reasoning, verification, and adaptive personalization strategies to function effectively in real-world, open-ended contexts.

7. Distribution, Licensing, and Citation

RecBench+ is published under a BSD-style license and is intended for unrestricted academic and industry research. Full documentation, scripts for data ingestion, and usage examples are available through the project's GitHub repository: https://github.com/jiani-huang/RecBench. The canonical citation is:

Huang, J., Wang, S., Ning, L., Fan, W., Wang, S., Yin, D., & Li, Q. (2025). Towards Next-Generation Recommender Systems: A Benchmark for Personalized Recommendation Assistant with LLMs. In Proceedings of SIGIR (RecBench+) (Huang et al., 12 Mar 2025).