Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learning (2503.24289v2)

Abstract: We propose Rec-R1, a general reinforcement learning framework that bridges LLMs with recommendation systems through closed-loop optimization. Unlike prompting and supervised fine-tuning (SFT), Rec-R1 directly optimizes LLM generation using feedback from a fixed black-box recommendation model, without relying on synthetic SFT data from proprietary models such as GPT-4o. This avoids the substantial cost and effort required for data distillation. To verify the effectiveness of Rec-R1, we evaluate it on two representative tasks: product search and sequential recommendation. Experimental results demonstrate that Rec-R1 not only consistently outperforms prompting- and SFT-based methods, but also achieves significant gains over strong discriminative baselines, even when used with simple retrievers such as BM25. Moreover, Rec-R1 preserves the general-purpose capabilities of the LLM, unlike SFT, which often impairs instruction-following and reasoning. These findings suggest Rec-R1 as a promising foundation for continual task-specific adaptation without catastrophic forgetting.

Rec-R1 introduces a reinforcement learning (RL) framework designed to align generative LLMs with user-centric recommendation objectives without necessitating supervised fine-tuning (SFT) on potentially costly or proprietary datasets (Lin et al., 31 Mar 2025 ). The core idea is to leverage feedback signals directly from an existing, potentially black-box, recommendation system to guide the LLM's generation process through closed-loop optimization. This contrasts with prevailing methods like few-shot prompting, which may lack task-specific adaptation, and SFT, which often requires curated datasets (sometimes distilled from larger models like GPT-4o) and can impair the LLM's general capabilities.

Methodology: The Rec-R1 Framework

Rec-R1 formalizes the interaction between an LLM and a recommender system within an RL paradigm. The key components are:

1. Policy (π): The LLM acts as the policy network. Given a state representing the user context, the LLM generates an action, typically a textual output like a search query or a sequence of recommended item identifiers. Parameter-efficient fine-tuning techniques (e.g., LoRA) or full fine-tuning can be employed to update the LLM policy based on the RL objective.
2. Environment: The environment encapsulates the user context (e.g., historical interactions, user profile, current query) and the black-box recommendation model. It receives the LLM's generated action and returns a reward signal.
3. State (s): The state typically consists of the current user context, including historical interactions, session information, or explicit user queries, formatted as input to the LLM.
4. Action (a): The action is the textual output generated by the LLM policy π(a|s). In product search, this could be a reformulated query. In sequential recommendation, it could be the predicted next item ID(s) or related textual descriptions.
5. Reward (r): The reward signal is derived from the feedback provided by the fixed, black-box recommendation model. This model evaluates the LLM's generated action (e.g., query, item list) based on its internal scoring or ranking mechanisms relative to the user context. For instance, if the LLM generates a search query, the reward might be based on the relevance score or rank of the target item retrieved by the recommender using that query. If the LLM predicts the next item, the reward could be proportional to the item's score or rank assigned by the black-box model. The objective is typically to maximize the expected cumulative reward: $$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} [\sum_{t=0}^T \gamma^t r_t]$$, where $\tau$ is a trajectory of states, actions, and rewards generated under policy $\pi_\theta$.

The optimization proceeds in a closed loop:

• The LLM generates an action based on the current state.
• The action is evaluated by the black-box recommender, yielding a reward.
• This reward signal is used to update the LLM's parameters via an RL algorithm (e.g., Policy Gradient methods like REINFORCE or proximal policy optimization (PPO)).

A simplified pseudocode representation for a single update step using a policy gradient approach could be:

input_prompt = format_context_for_LLM(C) action = sample_from_LLM(pi_theta, input_prompt) # e.g., generate search query or item ID retrieved_items = R_blackbox.search(query=action) reward = calculate_relevance_reward(retrieved_items, target_item) item_score = R_blackbox.score(user=C.user_id, item=action) reward = transform_score_to_reward(item_score) log_prob_action = calculate_log_prob(pi_theta, input_prompt, action) policy_loss = -log_prob_action * reward optimizer.zero_grad() policy_loss.backward() optimizer.step()

This closed-loop mechanism allows the LLM to directly optimize for the signals provided by the target recommender system, aligning its generative capabilities with specific recommendation objectives.

Experimental Setup and Evaluation

The efficacy of Rec-R1 was assessed on two representative recommendation tasks:

1. Product Search: The LLM's role is to generate effective search queries based on user needs or descriptions. The reward is determined by the performance of these queries when executed by a downstream retrieval system (e.g., BM25 or a neural retriever), measuring how well the retrieved results match the user's intent, often using the rank or score provided by the black-box system for a target item.
2. Sequential Recommendation: The LLM predicts the next item(s) a user might interact with, given their interaction history. The generated item IDs are evaluated by the black-box recommender, and the reward reflects the relevance or rank assigned to the predicted items by that system.

Rec-R1 was compared against several baselines:

• Zero-shot/Few-shot Prompting: Using the base LLM directly with carefully crafted prompts but without any parameter updates.
• Supervised Fine-Tuning (SFT): Fine-tuning the LLM on datasets of (user context, desired output) pairs. The paper emphasizes avoiding SFT based on synthetic data from proprietary models.
• Discriminative Recommenders: Traditional recommendation models that directly predict item scores or rankings (e.g., matrix factorization, neural collaborative filtering).

Evaluation metrics likely included standard recommendation metrics such as NDCG@k, Recall@k, or MRR for assessing the quality of the generated recommendations/queries. Crucially, the evaluation also assessed the preservation of the LLM's general abilities using benchmarks like MMLU (general knowledge), HumanEval (coding), or instruction-following tests.

Results and Analysis

Experimental results reported indicate that Rec-R1 consistently surpasses both prompting-based and SFT-based LLM approaches on the target recommendation tasks. Notably, Rec-R1 demonstrated substantial improvements even over strong discriminative baselines, particularly when integrated with relatively simple retrieval mechanisms like BM25 in the product search task. This suggests that the RL-optimized LLM can effectively enhance or refine the inputs to standard recommenders.

A significant finding is the framework's ability to maintain the LLM's general instruction-following and reasoning capabilities. Unlike SFT, which can lead to catastrophic forgetting and specialize the model too narrowly, Rec-R1's RL approach appears to adapt the LLM for the specific recommendation task without significantly degrading its performance on unrelated tasks. This characteristic positions Rec-R1 as a potential method for continual task-specific adaptation of LLMs.

Implementation Considerations

Deploying Rec-R1 involves several practical considerations:

• Black-Box Reward Interaction: Interfacing with a black-box recommender simplifies integration, as no gradients or internal model details are required. However, this can introduce challenges:
  • Reward Sparsity/Noise: Rewards from the black-box model might be sparse (e.g., only positive for exact matches) or noisy, potentially hindering RL convergence. Reward shaping or using dense feedback might be necessary.
  • Latency: Querying the black-box model for rewards adds latency to the RL training loop, potentially slowing down optimization.
  • Sample Inefficiency: RL methods, especially policy gradient approaches, can be sample inefficient. Efficient exploration strategies and potentially off-policy RL algorithms might be beneficial.
• LLM Training:
  • Parameter Updates: Full fine-tuning an LLM is computationally expensive. Parameter-efficient methods like LoRA or prefix-tuning are likely more practical, reducing memory and compute requirements while still allowing task adaptation.
  • RL Algorithm Choice: While policy gradient methods are common, algorithms like PPO offer more stable training through clipped surrogate objectives and value function estimation. The choice depends on the complexity of the task and the nature of the reward signal.
  • Batching and Parallelism: Training requires generating actions, querying the environment (recommender), and performing RL updates. This process can be parallelized by running multiple environments concurrently to collect trajectories faster.
• Computational Cost: Rec-R1 avoids the cost of creating large SFT datasets, which can involve expensive human annotation or API calls to proprietary models. However, RL training involves repeated inference passes through the LLM and interactions with the recommender system, which can still be computationally intensive, though potentially less so than large-scale SFT data generation and subsequent fine-tuning.
• Scalability: The framework needs to scale with the number of users and items. The primary bottlenecks are LLM inference and interactions with the black-box recommender. Caching recommender scores or using efficient LLM serving infrastructure is important for production deployment.

Conclusion

Rec-R1 presents a reinforcement learning approach to effectively bridge generative LLMs and recommendation systems. By optimizing the LLM directly using feedback from a black-box recommender, it avoids reliance on SFT data and potentially preserves the LLM's general capabilities better than SFT. The reported performance gains over prompting, SFT, and even strong discriminative baselines, particularly in scenarios like product search and sequential recommendation, underscore its potential as a method for adapting LLMs to specific downstream tasks in a targeted and efficient manner. Its ability to work with fixed, black-box components makes it adaptable to various existing recommendation infrastructures.