Beyond Utility: Evaluating LLM as Recommender

Abstract: With the rapid development of LLMs, recent studies employed LLMs as recommenders to provide personalized information services for distinct users. Despite efforts to improve the accuracy of LLM-based recommendation models, relatively little attention is paid to beyond-utility dimensions. Moreover, there are unique evaluation aspects of LLM-based recommendation models, which have been largely ignored. To bridge this gap, we explore four new evaluation dimensions and propose a multidimensional evaluation framework. The new evaluation dimensions include: 1) history length sensitivity, 2) candidate position bias, 3) generation-involved performance, and 4) hallucinations. All four dimensions have the potential to impact performance, but are largely unnecessary for consideration in traditional systems. Using this multidimensional evaluation framework, along with traditional aspects, we evaluate the performance of seven LLM-based recommenders, with three prompting strategies, comparing them with six traditional models on both ranking and re-ranking tasks on four datasets. We find that LLMs excel at handling tasks with prior knowledge and shorter input histories in the ranking setting, and perform better in the re-ranking setting, beating traditional models across multiple dimensions. However, LLMs exhibit substantial candidate position bias issues, and some models hallucinate non-existent items much more often than others. We intend our evaluation framework and observations to benefit future research on the use of LLMs as recommenders. The code and data are available at https://github.com/JiangDeccc/EvaLLMasRecommender.

• The paper introduces a multidimensional framework that extends traditional recommender evaluations by incorporating LLM-specific dimensions such as history sensitivity, candidate bias, generative performance, and hallucinations.
• The paper finds that LLMs excel in cold-start scenarios and generate explainable user profiles, although they exhibit a notable candidate position bias that impacts accuracy.
• The paper highlights challenges like hallucinations and variable performance with longer user histories, calling for bias mitigation strategies and fine-tuning to improve LLM-based recommendations.

Evaluating LLMs in Recommender Systems: A Multidimensional Framework

The paper "Beyond Utility: Evaluating LLM as Recommender" addresses the evolving role of LLMs within Recommender Systems (RSs). As LLMs like GPT, Claude, and Llama demonstrate significant prowess across diverse NLP tasks, their applicability as recommenders is increasingly being explored. However, conventional evaluations of recommender systems primarily focus on accuracy, leaving other crucial dimensions underexplored when it comes to LLMs. This paper introduces a multidimensional evaluation framework tailored to identify specific LLM-related characteristics in RS applications, going beyond traditional evaluation dimensions.

Multidimensional Evaluation Framework

The paper proposes a comprehensive evaluation framework that includes both traditional dimensions such as utility and novelty, as well as four novel dimensions specific to LLMs: history length sensitivity, candidate position bias, generation-involved performance, and hallucinations. This framework aims to provide a holistic understanding of the capabilities and limitations of LLMs when deployed in RSs.

1. History Length Sensitivity: This dimension examines how the length of user history input affects the performance of LLM-based recommenders. The paper finds LLMs to perform optimally in cold-start scenarios owing to their capacity to leverage world knowledge, demonstrating that even with minimal user data, they can deliver competitive results.
2. Candidate Position Bias: LLMs exhibit a notable bias towards items placed at the start of a candidate list, a trait not relevant to traditional models. The paper quantitatively captures this bias and discusses its detrimental impact on recommendation accuracy, advocating for further methodological innovations to mitigate such bias.
3. Generation-Involved Performance: By generating rich, textual user profiles, LLMs can provide explainable recommendations. This dimension evaluates the effect of incorporating such generative capabilities. Profiling enhances explainability, though the benefit to recommendation accuracy varies, particularly when longer user histories outperform condensed profiles.
4. Hallucinations: The paper addresses the issue of hallucinations, where LLMs may produce non-existent items in recommendations. While the incidence is generally below 5%, the presence of hallucinations continues to present user experience challenges, necessitating robust mapping techniques to eliminate these inaccuracies.

Implications and Future Directions

The empirical evaluation involving prominent LLMs like GPT-4o and Claude-3 reveals the nuanced strengths and weaknesses of LLMs: they excel in domain-specific knowledge applications and perform robustly in cold-start situations but fall short on longer-tail user histories and exhibit significant candidate position biases. The paper suggests that LLM-powered RSs can surpass traditional models, especially in re-ranking tasks, by leveraging inherent LLM features such as world knowledge and generative capabilities.

While LLMs display promise in enhancing RSs, the paper posits several future research directions. Addressing candidate position bias, refining hallucination mitigation strategies, and optimizing the integration of user profiles can significantly uplift the performance and reliability of LLM-based RSs. Furthermore, exploring the potential of fine-tuning LLMs on recommendation data may bridge existing gaps in collaborative filtering insights.

This comprehensive framework not only facilitates the evaluation of current LLM implementations in RSs but also sets the foundation for future research, encouraging the development of more refined, efficient, and user-centric recommendation solutions powered by LLMs. As the field progresses, such multidimensional evaluation frameworks will become essential in comparing, adapting, and ultimately harnessing the full potential of LLMs in diverse real-world applications.