Retrieval Augmented Conversational Recommendation with Reinforcement Learning

Abstract: LLMs exhibit enhanced capabilities in language understanding and generation. By utilizing their embedded knowledge, LLMs are increasingly used as conversational recommender systems (CRS), achieving improved performance across diverse scenarios. However, existing LLM-based methods rely on pretrained knowledge without external retrieval mechanisms for novel items. Additionally, the lack of a unified corpus poses challenges for integrating retrieval augmentation into CRS. Motivated by these challenges, we present RAR, a novel two-stage retrieval augmented conversational recommendation framework that aligns retrieval and generation to enhance both performance and factuality. To support this framework and provide a unified corpus, we construct a large-scale movie corpus, comprising over 300k movies with rich metadata, such as titles, casts and plot summaries. Leveraging this data, our primary contribution is RAR, the first framework to departs from standard two-stage CRS by dynamically bridging retrieval and generation. First, a retriever model generates candidate items based on user history; in the subsequent stage, an LLM refines the recommendations by incorporating conversational context with retrieved results. In addition, we introduce a novel reinforcement learning (RL) method that leverages LLM feedback to iteratively update the retriever. By creating a collaborative feedback loop that reinforces sampled candidate sets with higher ranking metrics, RAR effectively mitigates the misalignment between the retrieval and generation stages. Furthermore, grounding the LLM in factual metadata allows our RL-driven approach to capture subtle user intentions and generate context-aware recommendations with reduced hallucinations. We validate our approach through extensive experiments on multiple benchmarks, where RAR consistently outperforms state-of-the-art baseline methods.

• The paper presents RAR, a two-stage framework that augments conversational recommendation by integrating an external retriever with RL-driven preference optimization.
• The approach decouples retrieval and LLM-based generation, achieving significant performance gains in NDCG@5 and Recall@5, especially for cold and unpopular items.
• Extensive experiments validate RAR’s superiority over traditional CRS models, offering a scalable method and paving the way for advanced retriever-generator interfaces.

Retrieval-Augmented Conversational Recommendation with Reinforcement Learning: A Technical Essay

Introduction and Motivation

The paper "Retrieval Augmented Conversational Recommendation with Reinforcement Learning" (2604.04457) presents RAR, a two-stage framework designed to address critical bottlenecks in LLM-driven Conversational Recommender Systems (CRS). Recent LLM-based CRS methods excel in language understanding yet remain fundamentally limited by (i) a lack of external retrieval integration for novel or long-tail items, and (ii) persistent misalignment between external candidate retrieval and neural generation. The work directly confronts these issues by proposing a scalable embedding-driven retrieval method that is tightly coupled with RL-based preference optimization, supported by the curation of a large-scale, richly-annotated item corpus.

Retrieval-Augmented CRS Architecture

RAR decouples the CRS process into retrieval and generation. The first stage employs a dedicated retriever model $f_\text{ret}$—specifically, LRURec leveraging linear state space modeling on conversation histories—to yield a candidate set from a pre-indexed, >300k item movie corpus. The second stage utilizes a black-box LLM $f_\text{LLM}$ (e.g., GPT, Gemini, Qwen) to generate the final recommendations, conditioning on conversational history and rich, up-to-date candidate metadata.

Figure 1: RAR's retrieval-augmented conversational recommendation stacks an external retriever for candidate item gathering with an LLM-based generator conditioned on retrieved candidates for final response synthesis.

Unlike traditional approaches reliant on static KG or limited-scale corpora, RAR's architecture enables efficient retrieval augmentation at scale and facilitates rapid adaptation to unseen items without costly LLM retraining. The utilization of a richly-constructed unified movie corpus, spanning comprehensive metadata, ensures coverage for item-level personalization and grounding.

Online On-Policy Preference Optimization

A principal contribution of RAR is its online, on-policy learning loop for retriever optimization. Given that the generator $f_\text{LLM}$ is non-differentiable—and potentially closed-source—RAR deploys RL at the retriever level, using generator feedback based on ranking metrics (NDCG, Recall) as learned reward signals.

The system interleaves three phases per interaction: (i) candidate set sampling from the retriever, (ii) LLM-based ranking and reward computation for each candidate set, and (iii) retriever update via RL policy gradients. The core optimization objectives include Direct Preference Optimization (DPO) and Grouped Relative Policy Optimization (GRPO), with the latter demonstrating higher empirical effectiveness but increased computational cost.

Figure 2: RAR's on-policy preference optimization loop aligns retriever policy with black-box LLM feedback by iteratively sampling candidate sets, collecting ranking-based feedback, and updating the retriever via RL.

The Plackett-Luce model serves to compute selection probabilities over item permutations, and rewards are computed with reference to the generator's output, bridging the feedback gap between discrete retrieval and continuous generation.

Large-Scale Corpus Construction

RAR's underlying corpus consists of 337,731 de-duplicated English-language movie items, each annotated with granular attributes (title, cast, director, genre, plot). The corpus aggregates datasets such as IMDb, MovieLens, Redial, Reddit, and Inspired, facilitating precise entity-level grounding and enabling robust retrieval for both legacy and cold-start items.

This design decisively addresses the lack of unified, item-rich corpora that has hindered scalable embedding-based conversational retrieval, supporting both high-coverage training and efficient deployment.

Experimental Results

Experiments are conducted on multiple CRS datasets: Inspired, Redial, and Reddit. RAR is benchmarked against traditional CRS models (KBRD, KGSF, UniCRS), SFT-based retrieval augmentation schemes, and several advanced sequential recommenders (SASRec, FMLPRec, LRURec). All LLM modules are frozen, isolating retriever updates.

RAR exhibits strict and consistent superiority to all baseline and SFT-finetuned systems, with notable gains on NDCG@5 and Recall@5, confirming the efficacy of end-to-end retrieval-generation alignment. The average performance improvement over the best SFT is 7.7–11.9%, with the largest relative gains concentrated in top-k ranking positions, highlighting the RL-driven retriever's ability to surface contextually salient items for the generator. Notably, GPT-powered RAR achieves peak performance, followed by Gemini and open-source Qwen.

Figure 3: Comparative evaluation showing substantial improvement for DPO-based RAR and gains with increasing retrieved set size. RAR's performance with on-policy RL (DPO) surpasses SimPO and scales robustly with candidate pool.

LRURec is established as the optimal retriever backbone empirically, outperforming FMLPRec and SASRec in all relevant metrics when pretrained and RL-optimized within the RAR loop.

Hybridization with alternative RL approaches (e.g., GRPO) further amplifies ranking performance with improved robustness, at somewhat increased computational cost.

Hyperparameter ablation reveals optimal $\beta$ settings for preference loss and robust monotonic improvements as candidate pool size increases, indicating LLMs' ability to utilize larger, noisier candidate sets without performance degradation.

Addressing Bias and Reliability

A major liability of LLM-based CRS is popularity bias and hallucination, especially on items absent from LLM pretraining. RAR demonstrably mitigates these via explicit retriever grounding: unpopular (cold) items achieve up to 4× improvement in NDCG@10 over LLM-only approaches, and hallucinated entities are reduced to under 1% across all test cases.

Figure 4: RAR's effectiveness against popularity bias, showing N@10 stratified by item popularity. RAR improves retrieval and ranking for unpopular items, narrowing baseline disparity.

Empirical analysis further reveals that explicit reasoning (enabling "thinking" in LLMs) does not guarantee universal performance gains; improvements from such mechanisms are dataset- and model-specific, and often outmatched by the RL-optimized retriever's impact.

Implications and Future Directions

RAR establishes the technical viability of aligning two-stage retrieval-augmented generative CRS via on-policy, sample-efficient RL using only black-box LLM feedback. The approach generalizes across diverse LLM architectures and domains, representing a scalable solution to the persistent disconnect between symbolic retrieval modules and stochastic neural generators.

Practically, RAR's model- and corpus-agnostic design allows rapid adaptation to new domains where both retriever interpretability and external grounding are critical. Theoretically, the work motivates exploration into more advanced retriever-generator interface learning—including differentiable generator proxies, improved surrogate reward formulations, and joint LLM-retriever fine-tuning when allowed.

As RLHF-style online updates become more tractable, further research may focus on fully continuous retriever-generator ecosystems, adaptive prompting based on real-time user feedback, and extending the approach to richer multimodal corpora (vision, audio) for unified recommendation in conversational agents.

Conclusion

RAR delivers a robust, technically principled framework for retrieval-augmented conversational recommendation, effectively integrating external knowledge through efficient and aligned retriever optimization with RL in the presence of black-box LLMs. Experimental evidence across benchmarks confirms substantial and consistent improvements in top-k accuracy, reliability on cold/unpopular items, and significant reduction in hallucination phenomena. The architecture, RL methodology, and corpus design collectively form a foundation for scalable, high-fidelity CRS, paving the way for future advances in retrieval-augmented and control-oriented LLM applications.