LlamaRec: Two-Stage LLM Ranking

Updated 7 July 2026

LlamaRec is a two-stage framework that combines a fast sequential retriever with an LLM ranker to efficiently refine candidate recommendations.
It employs a prompt-and-verbalizer method that converts user histories and candidate titles into a ranking format based on next-token logits.
Empirical results demonstrate significant gains in NDCG and Recall over conventional methods across benchmarks such as ML-100k, Beauty, and Games.

LlamaRec is a two-stage framework for ranking-based recommendation that combines a small-scale sequential recommender with a LLM. In its original formulation, the first stage retrieves a small candidate set from user interaction history, and the second stage converts the history and candidates into text and uses an LLM to rank the candidates by reading next-token logits rather than generating item titles. The framework was introduced to address two limitations identified in earlier LLM-based recommendation: a reliance on training-free recommendation that heavily depends on pretrained knowledge, and slow inference caused by autoregressive generation (Yue et al., 2023).

1. Concept and problem setting

LlamaRec is situated in sequential recommendation, but its defining feature is the separation of retrieval and semantic ranking. The retriever operates in an ID-based space and is intended to be fast; the ranker operates in a textual space and is intended to exploit the language understanding capabilities of the LLM. This decomposition reflects the observation that conventional recommenders are efficient at pruning the item space, whereas LLMs are better suited to reasoning over item titles and user histories when the candidate set is already small (Yue et al., 2023).

The original problem formulation is explicitly ranking-based rather than generation-based. Instead of asking the LLM to output the next-item title, LlamaRec asks it to select the correct item from a candidate pool by predicting an index letter. This design is central to the framework’s efficiency claims, because ranking is reduced to a single forward pass over a prompt rather than token-by-token title generation.

A common misconception is to treat LlamaRec as an LLM-only recommender. In fact, the framework depends on a first-stage sequential recommender for candidate generation. Another common misconception is to view it as a training-free prompting method. The original work explicitly positions LlamaRec against training-free recommendation and fine-tunes the LLM ranker via instruction-tuning with QLoRA (Yue et al., 2023).

2. Retrieval-and-ranking pipeline

The first stage of LlamaRec uses LRURec, a compact sequential recommender built on linear recurrent units. It is trained with causal next-item prediction over item IDs. At inference, given a user history $[x_1,\ldots,x_T]$ , LRURec computes a predicted feature vector $h_T$ and scores each item $i$ by

$s_i = \langle h_T,\,e_i\rangle,$

where $e_i$ is the learned embedding of item $i$ . The retriever outputs the top- $k$ item IDs, with $k=20$ in the original framework description (Yue et al., 2023).

The second stage uses Llama 2 with 7B parameters as the ranker. The model is quantized via QLoRA so that fewer than $1\%$ of weights are tuned. Retrieved item IDs are mapped to text titles, and the user’s history is likewise represented as titles. A short instruction is prepended, producing a natural-language ranking prompt for the LLM (Yue et al., 2023).

This architecture implements a strict division of labor. LRURec handles large-scale candidate pruning, while the LLM performs local discrimination over a small candidate pool. The original summary characterizes this as using “a small, efficient LRURec to prune millions of items to 20 candidates,” followed by “a prompt-and-verbalizer scheme that turns Llama 2 into a fast ranker without expensive decoding” (Yue et al., 2023).

3. Prompt template and verbalizer mechanism

LlamaRec uses a three-section prompt template consisting of Instruction, Input, and Response. The template is:

$s_i = \langle h_T,\,e_i\rangle,$ 2

During training, the Response line is filled with the correct index letter; at inference, it is left blank so that the LLM predicts the letter corresponding to the correct candidate (Yue et al., 2023).

The decisive technical device is the verbalizer. Each candidate $i\in\{1,\ldots,k\}$ is assigned a unique letter token $h_T$ 0. After one forward pass, the model’s output logits $h_T$ 1 for those fixed letter tokens are extracted, and a probability distribution over candidates is obtained via

$h_T$ 2

Candidates are then ranked by descending $h_T$ 3 (Yue et al., 2023).

The original report states that no additional normalization or temperature calibration was needed in the experiments; straight softmax was sufficient. This design avoids decoding full item titles or ranked lists. It also constrains supervision to a very small label space: the $h_T$ 4 letter tokens, including EOS. The training objective therefore remains close to next-token prediction, which the original description states “retains LLM’s generative abilities” (Yue et al., 2023).

4. Optimization, efficiency, and empirical validation

The ranker is fine-tuned by instruction-tuning with QLoRA, and only the response tokens incur loss. The loss is standard cross-entropy on the correct index letter $h_T$ 5:

$h_T$ 6

This makes LlamaRec a classification-style ranker implemented within a causal language modeling interface (Yue et al., 2023).

The original experiments use three benchmark datasets: ML-100k, Beauty, and Games. Preprocessing applies a 5-core filter, keeps items with titles, and uses leave-one-out splitting. Evaluation reports MRR@5, NDCG@5, Recall@5, and the same metrics at @10. Baselines include NARM, SASRec, BERT4Rec, LRURec, and several LLM-based methods, including P5, PALR, GPT4Rec, RecMind, and POD when available (Yue et al., 2023).

The main reported improvements over the best baseline are summarized below.

Dataset	LRURec NDCG@5 / Recall@5	LlamaRec NDCG@5 / Recall@5
ML-100k	0.0468 / 0.0705	0.0543 / 0.0852
Beauty	0.0435 / 0.0614	0.0450 / 0.0648
Games	0.0640 / 0.0966	0.0714 / 0.1061

On ML-100k, the gains are reported as $h_T$ 7 in NDCG@5 and $h_T$ 8 in Recall@5; on Beauty, $h_T$ 9 and $i$ 0; on Games, $i$ 1 and $i$ 2 (Yue et al., 2023). Within the subset where the true item was retrieved in the top-20 candidate set, the gains are larger; one reported example is Recall@10 on Games, where LlamaRec reaches 0.7560 versus 0.6654 for the comparator, a $i$ 3 gain (Yue et al., 2023).

The efficiency argument is equally central. A generation-based LLM baseline must autoregressively emit titles or lists token by token, with complexity approximately $i$ 4, where $i$ 5 is the average generated length. The verbalizer-based method requires a single forward pass and a lookup of $i$ 6 logits, with complexity approximately $i$ 7 passes independent of title length. On the Beauty dataset with $i$ 8, the reported latency is 56.2 s per query for generation-based ranking with greedy decoding and average title length 20 tokens, versus under 1 s per query for the LlamaRec verbalizer (Yue et al., 2023). On the same dataset, an LLM-only comparison reports PALR at N@10 = 0.0446 and R@10 = 0.0721, POD at N@10 = 0.0443 and R@10 = 0.0688, and LlamaRec at N@10 = 0.0554 and R@10 = 0.0971 (Yue et al., 2023).

5. Subsequent extensions of the framework

A direct extension replaces the Llama 2 ranker with Llama3.1-8b while leaving the two-stage architecture unchanged. The replacement study states that the masking-and-verbalization interface is shared, so no architectural change is required; tokenizer and model loading are swapped, while prompt-template strings, mask-token conventions, negative sampling, data cleaning, sequence length, and candidate count remain unchanged (Choi et al., 26 Jul 2025). Under this controlled setting, the reported average relative improvements over the original LlamaRec are approximately $i$ 9 on ML-100K, $s_i = \langle h_T,\,e_i\rangle,$ 0 on Beauty, and $s_i = \langle h_T,\,e_i\rangle,$ 1 on Games across the six ranker metrics. For NDCG@10 specifically, the reported values are 0.3184 to 0.4142 on ML-100K, 0.4217 to 0.4503 on Beauty, and 0.4169 to 0.4455 on Games (Choi et al., 26 Jul 2025). This suggests that backbone quality alone can materially affect ranking performance without changing the retrieval-and-verbalizer design.

Another extension, LlamaRec-LKG-RAG, inserts a User Preference Module and a KG-RAG retriever between candidate generation and LLM ranking. The method constructs a heterogeneous knowledge graph from user behavior and item metadata, predicts a distribution over relation types for each user, retrieves shortest paths between history items and candidates, scores paths by user preference and relation IDF, and serializes the selected paths into the prompt as a Relations field (Azizi et al., 9 Jun 2025). The ranker still uses a fine-tuned Llama-2-7B with LoRA adapters and still extracts logits for index tokens in a single forward pass. Reported gains over LlamaRec include, on MovieLens-100K, MRR@1 from 0.0214 to 0.0262, NDCG@5 from 0.0467 to 0.0499, and Recall@10 from 0.1065 to 0.1098; on Amazon Beauty, NDCG@10 from 0.0491 to 0.0498 and Recall@10 from 0.0855 to 0.0868 (Azizi et al., 9 Jun 2025). The ablation result is particularly informative: injecting all shortest paths without the preference module degrades performance below vanilla LlamaRec, indicating that structured context must be personalized rather than indiscriminately appended.

Together, these extensions preserve the original LlamaRec core: first-stage candidate generation, prompt-based representation of history and candidates, and single-pass verbalizer-based ranking. What changes is the source of semantic capability—either a stronger foundation model or a richer context construction mechanism.

LlamaRec should be distinguished from Llama4Rec, which is a different framework that combines a conventional recommender and an instruction-tuned LLM through mutual augmentation and adaptive aggregation. Llama4Rec uses LLM-driven data augmentation for the conventional model, prompt augmentation for the LLM, and a user-adaptive gating layer to fuse outputs; it is described as model-agnostic and applies to direct recommendation, sequential recommendation, and rating prediction (Luo et al., 2024). This is conceptually adjacent to LlamaRec but not a renaming of the same method.

The name has also been used in a later multimodal music-recommendation study, where “LlamaRec” denotes a session-based recommender on LastFM-1K that builds multimodal item representations from item ID embeddings, audio embeddings, lyric embeddings, LLM-generated semantic metadata using the MGPHot annotation schema, and listening completion ratios, and then feeds those representations into a LLaMA backbone such as LLaMA-2-13B or LLaMA-3-70B (Kandagatla et al., 28 May 2026). That usage is separate from the original two-stage recommendation framework introduced in 2023.

The broader significance of the original LlamaRec lies in its operational definition of what an LLM-based recommender can be: not a free-form generator of titles, but a ranker over retrieved candidates implemented through prompt construction and a verbalizer. Later work reinforces two implications. First, performance can improve substantially by replacing the underlying LLM while preserving the interface (Choi et al., 26 Jul 2025). Second, additional context can help, but only when it is filtered and structured in a user-specific way rather than appended indiscriminately (Azizi et al., 9 Jun 2025). These developments position LlamaRec less as a fixed model than as a reusable architectural pattern for efficient LLM-based ranking.