RecRankerEval: Modular Evaluation Framework

• The paper reproduces RecRanker’s top‑k recommendation pipeline, using a modular evaluation framework to uncover experimental artifacts such as leakage in pointwise methods.
• It decomposes the system into five dimensions—user sampling, base model, LLM backbone, dataset, and instruction tuning—to enable independent analysis of each component.
• Empirical results demonstrate that while pairwise and listwise strategies yield expected performance, the pointwiseFix approach is essential for fair and reproducible LLM evaluation.

Searching arXiv for RecRankerEval and closely related papers to ground the article in current literature. Attempting to retrieve arXiv entries for RecRankerEval, RecRanker, and reciprocal recommendation evaluation. RecRankerEval is a flexible and extensible framework for top-$k$ LLM-based recommendation that was proposed to reproduce the RecRanker pipeline, inspect the reproducibility of its reported gains, and enable fair and comprehensive evaluation of LLM-based recommenders across multiple design choices (Meng et al., 8 Jul 2025). It was introduced in direct response to RecRanker, which instruction-tuned LLMs as rankers for top-$k$ recommendation through adaptive sampling, prompt enhancement, and pointwise, pairwise, and listwise ranking tasks (Luo et al., 2023). In RecRankerEval, the central object of study is not a single ranking model but a modular evaluation environment in which user sampling strategy, initial recommendation model, LLM backbone, dataset selection, and instruction tuning method can be varied independently (Meng et al., 8 Jul 2025).

1. Origins and problem setting

RecRanker introduced an instruction-tuning approach in which sampled user histories and candidate items are converted into prompts for three ranking tasks: pointwise, pairwise, and listwise; it further proposed a hybrid ranking method that ensembles these tasks (Luo et al., 2023). Its training pipeline combined adaptive sampling, position shifting to mitigate position bias, and prompt enhancement with auxiliary information from conventional recommenders. In the original formulation, the goal was to use an LLM as the ranker for top-$k$ recommendations.

RecRankerEval revisits that line of work from a reproducibility and benchmarking perspective. Its stated goals are to reproduce and verify the results of the RecRanker model for top-$k$ recommendation, provide a modular, extensible environment where each of RecRanker’s core components can be swapped, extended or analyzed in isolation, and facilitate fair and comprehensive benchmarking of LLM-based recommenders under different design choices (Meng et al., 8 Jul 2025). A key motivation is that the original RecRanker code release omitted data-processing scripts for user embeddings and prompt files, and required large GPU clusters for full fine-tuning, which made independent verification difficult (Meng et al., 8 Jul 2025).

The framework’s first substantive finding is methodological rather than algorithmic. Its reproduction shows that the pairwise and listwise methods achieve a performance comparable to that reported in the original paper, whereas the pointwise method, although reproducible, appears “abnormally high” because of data leakage from the inclusion of ground-truth information in the prompts (Meng et al., 8 Jul 2025). This positions RecRankerEval as both a reproduction framework and a diagnostic instrument for hidden experimental artifacts.

2. Five-dimensional modular architecture

RecRankerEval is organized around five key dimensions: user sampling strategy, initial recommendation model, LLM backbone, dataset selection, and instruction tuning method (Meng et al., 8 Jul 2025). Figure 1 in the paper is described as sketching these five core components.

Dimension	Options described	Reported impact
User sampling strategy	Importance-aware sampling, clustering-based sampling, DBSCAN, random sampling	DBSCAN generally yields the most stable gains
Initial recommendation model	MF, LightGCN, XSimGCL	Stronger base rankers raise absolute performance
LLM backbone	Llama 2 (7B), Llama 3 (8B), GPT-3.5 Turbo	Llama 3 surpasses Llama 2 after tuning
Dataset selection	MovieLens-100K, MovieLens-1M, BookCrossing, Amazon-Music	BookCrossing underperforms because of timestamp issues
Instruction tuning method	Pointwise, pairwise, listwise, hybrid	After leakage is fixed, listwise is strongest single strategy

The user-sampling module formalizes several alternatives. In importance-aware sampling, each user $u$ has interaction count $q_u$, and the sampling probability is

$p_{u,imp}=\frac{\ln(q_u)}{\sum_{v\in U}\ln(q_v)}.$

In clustering-based sampling, the user interaction vector is embedded in $\mathbb{R}^d$ and K-means is applied through

$$\{k_u\}=\arg\min_{k_u,\mu}\sum_u \|e_u-\mu_{k_u}\|^2,$$

after which $p_{u,clust}\propto |\{v:k_v=k_u\}|$. For DBSCAN, users with fewer than $k$0 neighbors in radius $k$1 are labeled noise, and the remaining users are sampled proportionally to cluster size (Meng et al., 8 Jul 2025).

The initial recommendation model produces a first-pass ranked list $k$2 of candidate items for each user $k$3, which is then inserted into the LLM prompt (Meng et al., 8 Jul 2025). The supported initial rankers are MF, LightGCN, and XSimGCL. The LLM layer distinguishes open, fine-tunable models—Llama 2 (7B) and Llama 3 (8B)—from the closed, API-only zero-shot model GPT-3.5 Turbo (Meng et al., 8 Jul 2025). The instruction-tuning layer preserves the RecRanker task decomposition into pointwise, pairwise, listwise, and hybrid modes.

3. Prompting, learning objectives, and task formulations

RecRankerEval preserves the prompt-centric formulation inherited from RecRanker while turning it into a controlled evaluation axis. Each prompt concatenates four parts: task description, user history, ranking context, and hints from the base ranker (Meng et al., 8 Jul 2025). The implementation details make this concrete: task description may begin with “You are a recommender…”, the history is partitioned into liked versus disliked items, the candidate list comes from $k$4, and the hints depend on the task variant (Meng et al., 8 Jul 2025).

The four instruction-tuning methods are defined as follows. Pointwise predicts a relevance score for each $k$5; pairwise decides preference between two items; listwise reorders a list of $k$6 candidates; and hybrid combines the outputs of all three (Meng et al., 8 Jul 2025). In the original RecRanker formulation, these tasks were realized as natural-language prompts such as “How would the user rate item $k$7?”, “Would the user prefer item $k$8 over item $k$9?”, and “Rank the list $k$0 by user preference” (Luo et al., 2023).

The fine-tuning objective in RecRankerEval is standard sequence-level cross-entropy: $k$1 where $k$2 denotes LLM parameters and $k$3 the prompt-completion pairs (Meng et al., 8 Jul 2025). The paper notes that pointwise, pairwise, and listwise objectives can all be viewed as instantiations of this sequence-level cross-entropy where the label is a numeric score, a “Yes/No” token, or a permutation, respectively.

A central corrective modification is the leakage fix. In the original pointwise prompts, hint scores could directly reveal the true rating or preference. RecRankerEval therefore introduces “pointwiseFix,” in which any hint that directly reveals the ground-truth label is removed (Meng et al., 8 Jul 2025). This correction is essential to the framework’s claim of fair evaluation, because it separates genuine LLM ranking behavior from prompt-induced supervision leakage.

4. Metrics, datasets, and implementation protocol

RecRankerEval evaluates recommendation quality with standard top-$k$4 ranking metrics. Denoting by $k$5 the test users, by $k$6 the top-$k$7 recommendation, and by $k$8 the set of ground-truth relevant items, it uses

$k$9

and

$k$0

It also computes NDCG@$k$1 in standardized fashion (Meng et al., 8 Jul 2025).

The framework provides standard pipelines for MovieLens-100K and MovieLens-1M using the exact splits from the original RecRanker paper, for BookCrossing with $k$2–$k$3 ratings and a $k$4-core filter, and for Amazon-Music with $k$5–$k$6 ratings, a $k$7-core filter, and real timestamps (Meng et al., 8 Jul 2025). In data preprocessing, MovieLens 100K/1M use official splits with 10 historical items per prompt and an $k$8 train/validation/test split by timestamp; BookCrossing applies the same sequential split after randomly assigning timestamps; Amazon-Music uses real timestamps from review data (Meng et al., 8 Jul 2025).

The implementation is notable because it reproduces the pipeline under substantially lighter hardware than the original RecRanker setup. Initial recommenders are configured as MF with latent dimension $k$9, Adam learning rate $u$0, and $u$1 epochs; LightGCN with $u$2 layers, dimension $u$3, learning rate $u$4, and $u$5 epochs; and XSimGCL with the same GCN backbone plus contrastive augmentations (Meng et al., 8 Jul 2025). LLM fine-tuning uses Llama 2-7B or Llama 3-8B-Instruct with LoRA rank $u$6, $u$7, learning rate $u$8, batch size $u$9, gradient accumulation $q_u$0, $q_u$1-step warm-up, and cosine decay on a single RTX A6000 $q_u$2 GB GPU, in contrast to the original RecRanker’s $q_u$3A800 configuration (Meng et al., 8 Jul 2025). Inference uses vLLM decoding with temperature $q_u$4, top-$q_u$5 sampling, and top-$q_u$6 (Meng et al., 8 Jul 2025).

5. Empirical findings and ablation structure

RecRankerEval reports that the original results of RecRanker can be reproduced on ML-100K and ML-1M, as well as the additional Amazon-Music dataset, but not on BookCrossing because the original RecRanker paper lacks timestamp information for that dataset (Meng et al., 8 Jul 2025). This dataset-level asymmetry is a recurring theme in the framework’s analysis.

Its empirical analysis is organized around several research questions. On initial model impact, XSimGCL $q_u$7 LightGCN $q_u$8 MF in absolute performance, but weaker models show larger relative gains from LLM re-ranking (Meng et al., 8 Jul 2025). On user sampling impact, DBSCAN $q_u$9 K-means $p_{u,imp}=\frac{\ln(q_u)}{\sum_{v\in U}\ln(q_v)}.$0 random overall, especially under listwise and hybrid tuning; DBSCAN is described as generally yielding the most stable gains, presumably by excluding outliers and emphasizing dense “intent” groups (Meng et al., 8 Jul 2025). On backbone choice, zero-shot GPT-3.5 and Llama 3 outperform Llama 2 under most variants, and after tuning Llama 3 outperforms Llama 2 by $p_{u,imp}=\frac{\ln(q_u)}{\sum_{v\in U}\ln(q_v)}.$1–$p_{u,imp}=\frac{\ln(q_u)}{\sum_{v\in U}\ln(q_v)}.$2 points in NDCG across both ML-100K and ML-1M (Meng et al., 8 Jul 2025).

The leakage diagnosis is the paper’s most consequential ablation. Table 3 is reported to confirm that listwise and pairwise variants match published numbers, while pointwise “originally appears strongest” but is explained by leakage artifacts (Meng et al., 8 Jul 2025). After removing the offending hints in pointwiseFix, performance drops sharply and aligns with pairwise and listwise results. Once leakage is fixed, listwise emerges as the strongest single strategy, with hybrid close behind in many configurations (Meng et al., 8 Jul 2025).

Cross-dataset generalization further clarifies the role of data quality. On the MovieLens datasets and Amazon-Music, listwise and hybrid variants consistently improve over the base models. On BookCrossing, however, all LLM variants fall below the base ranker, and the paper interprets this as evidence that the lack of real temporal order and the noise introduced by simulated timestamps can severely hamper sequential splits and instruction tuning (Meng et al., 8 Jul 2025). This is one of the clearest cases in which RecRankerEval uses the framework itself to surface a dataset-design limitation rather than merely to compare models.

6. Extensibility, best practices, and broader evaluation context

The framework includes explicit guidelines for extension to new settings. The recommended procedure is to prepare a sequential split of user–item interactions with a $p_{u,imp}=\frac{\ln(q_u)}{\sum_{v\in U}\ln(q_v)}.$3-core filter and real or simulated timestamps; implement or plug in an initial recommender through the standard $p_{u,imp}=\frac{\ln(q_u)}{\sum_{v\in U}\ln(q_v)}.$4 interface; provide user embeddings for clustering or DBSCAN; write prompt templates for pointwise, pairwise, and listwise variants without ground-truth leakage; choose an LLM backbone and apply LoRA or API calls; and run evaluation scripts to compute Precision@$p_{u,imp}=\frac{\ln(q_u)}{\sum_{v\in U}\ln(q_v)}.$5, Recall@$p_{u,imp}=\frac{\ln(q_u)}{\sum_{v\in U}\ln(q_v)}.$6, and NDCG@$p_{u,imp}=\frac{\ln(q_u)}{\sum_{v\in U}\ln(q_v)}.$7 in a standardised fashion (Meng et al., 8 Jul 2025). The paper’s best-practice guidance also recommends using the provided data-processing and prompt-generation scripts, verifying prompt contents for hidden leakage signals, reporting base-model benchmarks alongside LLM variants, and releasing new modules through the framework’s plugin APIs (Meng et al., 8 Jul 2025).

Within the broader ranking-evaluation literature, RecRankerEval’s metric choices are conventional rather than revisionary. It standardizes Precision@$p_{u,imp}=\frac{\ln(q_u)}{\sum_{v\in U}\ln(q_v)}.$8, Recall@$p_{u,imp}=\frac{\ln(q_u)}{\sum_{v\in U}\ln(q_v)}.$9, and NDCG@$\mathbb{R}^d$0, but related work has argued that recall in rankings requires a formal evaluative motivation and can be interpreted through recall-orientation, robustness, and lexicographic evaluation (Diaz et al., 2023). Likewise, rankDCG was proposed to address several shortcomings of Kendall’s $\mathbb{R}^d$1, Average Precision, and nDCG, including handling of ties, lower-bound consistency, and the cost function for user-ranking and recommendation settings (Katerenchuk et al., 2018). This suggests that RecRankerEval’s modular architecture could accommodate richer evaluation modules in future work, although the framework as described is centered on standard top-$\mathbb{R}^d$2 recommender metrics rather than alternative ranking measures.

The framework’s main contribution is therefore methodological. It turns RecRanker from a single paper-specific pipeline into a transparent experimental space where sampling, base ranking, prompting, backbone choice, and tuning objective can be isolated and compared (Meng et al., 8 Jul 2025). At the same time, its findings on pointwise leakage and timestamp sensitivity indicate that reproducibility in LLM-based recommendation depends not only on model capacity but also on prompt hygiene, temporal data quality, and the explicit separation of modular components.