Collaborative Large Language Model for Recommender Systems (2311.01343v4)

Abstract: Recently, there has been growing interest in developing the next-generation recommender systems (RSs) based on pretrained LLMs. However, the semantic gap between natural language and recommendation tasks is still not well addressed, leading to multiple issues such as spuriously correlated user/item descriptors, ineffective LLMing on user/item data, inefficient recommendations via auto-regression, etc. In this paper, we propose CLLM4Rec, the first generative RS that tightly integrates the LLM paradigm and ID paradigm of RSs, aiming to address the above challenges simultaneously. We first extend the vocabulary of pretrained LLMs with user/item ID tokens to faithfully model user/item collaborative and content semantics. Accordingly, a novel soft+hard prompting strategy is proposed to effectively learn user/item collaborative/content token embeddings via LLMing on RS-specific corpora, where each document is split into a prompt consisting of heterogeneous soft (user/item) tokens and hard (vocab) tokens and a main text consisting of homogeneous item tokens or vocab tokens to facilitate stable and effective LLMing. In addition, a novel mutual regularization strategy is introduced to encourage CLLM4Rec to capture recommendation-related information from noisy user/item content. Finally, we propose a novel recommendation-oriented finetuning strategy for CLLM4Rec, where an item prediction head with multinomial likelihood is added to the pretrained CLLM4Rec backbone to predict hold-out items based on soft+hard prompts established from masked user-item interaction history, where recommendations of multiple items can be generated efficiently without hallucination. Codes are released at https://github.com/yaochenzhu/LLM4rec.

Overview of "Collaborative LLM for Recommender Systems"

The paper "Collaborative LLM for Recommender Systems" addresses the challenges inherent in merging LLMs with recommender systems (RSs). The authors propose a novel framework, CLLM4Rec, which leverages the strengths of pre-trained LLMs to enhance the accuracy and efficiency of recommendation tasks while addressing existing semantic gaps between natural language processing and recommendation subtasks.

Key Insights and Methodology

The paper highlights several challenges when integrating LLMs into RSs, notably the semantic disconnects between language tasks and recommendation objectives. LLMs, while powerful, may misinterpret or inefficiently model user-item interactions due to their language-centric training paradigms. To bridge these gaps, CLLM4Rec introduces the concept of using user/item ID tokens, extending the vocabulary of pretrained LLMs to inherently accommodate recommendation-specific semantics.

For efficient model understanding and training, the authors devise a soft+hard prompting strategy, segregating user/item tokens (soft prompts) from natural language tokens (hard prompts) during training. This method aids in effectively encoding semantics from interaction data and textual content without overwhelming the model due to token heterogeneity. The training involves a mutually regularized pretraining phase, coupling collaborative filtering with content understanding, thereby mitigating noise capturing from natural language content and preventing model overfitting on sparse interaction data.

Furthermore, CLLM4Rec is refined with a recommendation-oriented finetuning stage to improve recommendation performance and efficiency. Through masked prompting strategies and adaptation of prediction heads for multinomial likelihoods, the model learns to generate recommendations without autoregressive inefficiencies, making it viable for practical deployments.

Experimental Evaluation

The empirical analysis on benchmarks including Amazon product and LinkedIn job recommendation datasets reveals CLLM4Rec's superiority over traditional ID-based RSs and LLM-based counterparts. Notably, CLLM4Rec showed enhanced recall and NDCG metrics, signifying more effective capture and utilization of user-item interaction data. When juxtaposed with established baselines like Multi-VAE and BERT4Rec, CLLM4Rec demonstrated improved performance, especially in scenarios abundant with rich textual content.

Moreover, the model outperforms in real-world settings, as evidenced by experiments on LinkedIn's recommendation dataset. Here, the model's ability to handle large-scale candidate pools and the diversity of user/job features is particularly noteworthy.

Implications and Future Work

CLLM4Rec presents a meaningful advance in the domain of intelligent recommendation systems, showcasing the potential to harness LLMs for improved recommendation outcomes. Its dual-phase training strategy effectively marries collaborative and content-based filtering approaches, further leveraging the innate capabilities of LLMs.

The potential future pathways suggested include further exploration into enhancing the scalability of CLLM4Rec for industrial-scale applications, optimizing computational expense vis-a-vis inference accuracy. Additionally, expanding upon this framework for other domains like conversational recommendations or more complex multi-modal datasets could yield further insights and applications.

Overall, the paper makes a substantive contribution toward integrating LLMs into RSs, providing a robust framework that others in the field can build upon. Researchers venturing into similar territories can draw upon the paper's methodology, results, and insights to scaffold their exploration into the convergence of NLP and RS domains.