Rec-R1: RL-Optimized LLM Recommendation Framework

• Rec-R1 is a reinforcement learning framework that aligns large language model outputs with recommendation systems via closed-loop optimization.
• It employs policy-gradient methods with KL regularization to improve metrics like NDCG and Recall, ensuring efficient, task-specific tuning.
• Rec-R1 preserves the LLM's general capabilities by avoiding catastrophic forgetting while being cost-effective compared to traditional SFT methods.

Rec-R1 is a general reinforcement learning (RL) framework for directly aligning LLMs with user-centric recommendation systems via closed-loop optimization. It departs from conventional prompting and supervised fine-tuning (SFT) by using reward signals from a downstream, black-box recommendation model to optimize LLM text generation for search, recommendation, and related tasks. Rec-R1 robustly improves retrieval/ranking performance while preserving instruction-following and reasoning abilities of the underlying LLM, and achieves this with substantial gains in efficiency and resource economy compared to data distillation or SFT methods (Lin et al., 31 Mar 2025).

1. Mathematical Foundations and Objective

Rec-R1 formalizes recommendation interaction as an episodic, stateless Markov decision process (MDP). The state space $S$ consists of recommendation-relevant user inputs (e.g., natural-language queries for product search or user histories for sequential recommendation). The action space $A$ corresponds to LLM-generated textual outputs—such as rewritten queries, enriched descriptions, or pseudo-reviews—that are input to a downstream retriever or ranker.

For each episode:

• A state $s\sim p(s)$ is sampled from dataset $D$.
• The LLM agent (policy $\pi_\theta$) generates action $a\sim\pi_\theta(a|s)$.
• The fixed recommendation system—treated as a black-box environment—returns a scalar reward $r = f(a|s)$ based on downstream retrieval or ranking quality.
• There are no transitions to future states; each episode is independent.

The canonical optimization target is

$$\max_\theta\, \mathbb{E}_{s\sim p(s), a\sim\pi_\theta(a|s)}\left[f(a|s)\right] - \lambda\,\mathrm{KL}\left[\pi_\theta(\cdot|s)\,||\,\pi_\mathrm{init}(\cdot|s)\right]$$

where $f(a|s)$ is a retrieval/ranking metric (such as Recall@$K$ or NDCG@$K$), and the KL-divergence regularization (with coefficient $\lambda$) penalizes excessive deviation from initial policy $\pi_\mathrm{init}$, mitigating collapse or catastrophic exploration.

2. System Architecture and Training Workflow

Rec-R1 comprises:

• LLM Policy Network: Typically a strong instruction-tuned model (e.g., Qwen-2.5-3B-Instruct), parameterized by $\theta$.
• Retriever/Ranker (fixed environment): This may be a sparse retriever (BM25 via Pyserini), a dense embedding model (e.g., BLAIR + FAISS), or a hybrid/discriminative model (e.g., RoBERTa, SimCSE).

The closed-loop RL training protocol operates as follows:

1. Agent receives input state $s$.
2. Agent samples $n$ candidate texts $\{a_1,\ldots,a_n\}$ from $\pi_\theta(\cdot|s)$ using top-$p$ sampling and controlled temperature.
3. Each $a_i$ is evaluated by the retriever/ranker, yielding a ranked list scored by $f(a_i|s)$ (e.g., NDCG@100).
4. Scalar rewards $r_i=f(a_i|s)$ are fed back to the LLM agent.
5. The policy $\pi_\theta$ is updated via a policy-gradient RL algorithm (e.g., Group Relative Policy Optimization, GRPO; PPO variants are compatible).

High-level pseudocode:

initialize θ ← θ_init (pretrained LLM) for epoch in 1…E: for minibatch S={s₁…s_B} from D: for each s in S: sample n outputs {a₁…a_n} ∼ πθ(·|s) for each aᵢ: rᵢ ← compute_reward(s, aᵢ, GroundTruth) # Compute policy-gradient loss with KL regularization: L = −(1/Bn) ∑_{(s,a,r)∈T} [r · log πθ(a|s)] + λ·KL[πθ(·|s) ∥ π_init(·|s)] θ ← θ − η ∇θ L

Default hyperparameters include learning rate $\eta=10^{-6}$, batch size 256, $n=12$ generations per input, $\text{top-}p=0.95$, temperature $0.6$, KL penalty $\lambda = 0.001$, $E=5$ epochs.

3. Reward Structure and Optimization Signal

Rewards are computed using standard IR and RecSys metrics, enabling task- and method-agnostic evaluation.

• Recall@$K$: $$|\text{Retrieved}_K(s,a)\cap \text{GroundTruth}(s)|/|\text{GroundTruth}(s)|$$
• DCG@$K$: $\sum_{i=1}^K (2^{rel_i}-1)/\log_2(i+1)$; $\text{NDCG}@K = \text{DCG}@K/\text{IDCG}@K$

Training typically uses NDCG@1000 to reduce reward sparsity. During inference, NDCG@100 and Recall@10 are standard.

Reward collection pseudocode:

function compute_reward(s, a, D): results = retrieve(a) # top-K items relevances = [is_in_gt(item, D[s]) for item in results] return ndcg(relevances, K)

4. Comparative Performance and Baseline Analysis

Extensive experiments on product search and sequential recommendation confirm Rec-R1’s empirical superiority:

Task	Baseline	Rec-R1 Variant	NDCG/Recall@K Gain
Product Search (ESCI, NDCG@100)	BM25: ~12–24%	Rec-R1+BM25: 33.9% avg	+21.45 pts
	GPT-4o prompt+BM25: ~23–28%	Rec-R1+BLAIR-LARGE: 31.4% avg	+15.5 pts
Complex Search (Amazon-C4, NDCG@100)	BM25: ~6–9%	Rec-R1+BM25: ~19–20%	+11 pts
Sequential Rec (Amazon Beauty)	SASRec/BLAIR: Recall@10≈3.7%	Rec-R1+BM25: Recall@10=3.53% (+2.23)	(Transductive setting)
	Prompting: near-zero recall	Rec-R1+BM25: Recall@10=6.00% (+4.20)	(Inductive setting)

Rec-R1 consistently improves over zero/few-shot prompting, SFT (upper-bounded by the performance of GPT-4o), and strong discriminative and retrieval baselines.

5. Catastrophic Forgetting and General Capability Preservation

A critical advantage of Rec-R1 over SFT is in maintaining the LLM’s general-purpose abilities:

• On six held-out benchmarks (ESCI, MMLU, IFEval, GSM8K, MBPP, HumanEval), Rec-R1-trained models preserve performance in factual knowledge, code, instruction-following, and mathematical reasoning, while SFT often induces severe degradation (e.g., IFEval: SFT drop of ≈27 points, Rec-R1 gain of +1.9; GSM8K: SFT drop ≈28 points, Rec-R1 improves by +5.7).
• Thus, Rec-R1’s reward-driven RL avoids catastrophic forgetting of core LLM faculties by not biasing generation towards narrow, synthetic SFT data.

6. Efficiency, Practical Considerations, and Ablation Insights

• Cost and Speed: RL training with Rec-R1 (2×A100 GPUs) requires ≈210 s and ≈\$0.48 to match/improve upon SFT (which typically costs ≈\$15.60 and 35 min for SFT pipeline with GPT-4o data).
• Prompt strategy: For dense retrievers such as BLAIR, generic rewriting prompts are non-optimal. RL enables the LLM to autonomously discover effective “review-style” rewriting, boosting NDCG by +8–9 points relative to prompt injection or initial degraded states.
• Reward sparsity: Using NDCG@1000 alleviates reward sparsity and stabilizes RL convergence versus NDCG@100.
• Initialization: Strong instruction-tuned LLMs are essential to bootstrap effective exploration in the large compositional action space.
• Ablations: Training and reward schedule design (e.g., KL penalty presence, batch size, number of rollouts per state) materially impact performance; Rec-R1 is robust provided hyperparameters and initialization are sensible.

7. Significance, Limitations, and Broader Context

Rec-R1 reframes open-ended query rewriting, feature augmentation, and generative item construction as closed-loop RL, unifying LLM optimization with downstream retrieval-based reward. This framework is black-box with respect to the underlying RecSys, making it highly adaptable: any system yielding IR metrics can be attached. The approach avoids excessive reliance on costly synthetic data generation, is cost-effective, and enables continual, task-specific tuning while minimizing the risk of catastrophic forgetting.

Key limitations are inherited from large-scale RL—e.g., sensitivity to reward design, exploration/exploitation stability, and reliance on strong base LLMs for efficient training. Rec-R1 also presumes access to meaningful, deterministic reward signals and a stateless training setup.

Summary Table: Core Rec-R1 Components

Component	Description	Examples
State $s\in S$	Recommendation-relevant input (query or history)	"Find running shoes", user session
Action $a\in A$	Text generated by LLM to guide RecSys/retriever	Rewritten query, enriched metadata, review
Reward $r=f(a|s)$	Scalar score from RecSys in IR metric	NDCG@100, Recall@10
Policy gradient update	RL algorithm with batch sampling and KL regularization	GRPO, PPO variants
Performance metrics	Empirical NDCG, Recall, Generalization on auxiliary LLM tasks	NDCG@100, recall@10, MMLU, GSM8K

Rec-R1 provides a principled, reproducible, and efficient mechanism for directly optimizing LLM outputs for recommendation performance, serving as a foundation for user-centric, continual adaptation of generative models in retrieval and RecSys contexts (Lin et al., 31 Mar 2025).