LibRec: Benchmarking Retrieval-Augmented LLMs for Library Migration Recommendations

Abstract: In this paper, we propose LibRec, a novel framework that integrates the capabilities of LLMs with retrieval-augmented generation(RAG) techniques to automate the recommendation of alternative libraries. The framework further employs in-context learning to extract migration intents from commit messages to enhance the accuracy of its recommendations. To evaluate the effectiveness of LibRec, we introduce LibEval, a benchmark designed to assess the performance in the library migration recommendation task. LibEval comprises 2,888 migration records associated with 2,368 libraries extracted from 2,324 Python repositories. Each migration record captures source-target library pairs, along with their corresponding migration intents and intent types. Based on LibEval, we evaluated the effectiveness of ten popular LLMs within our framework, conducted an ablation study to examine the contributions of key components within our framework, explored the impact of various prompt strategies on the framework's performance, assessed its effectiveness across various intent types, and performed detailed failure case analyses.

• The paper presents a novel framework combining LLMs and retrieval-augmented generation to automate library migration recommendations.
• It extracts migration intents from commit messages and constructs a RAG database to provide context-aware target library recommendations.
• Evaluation on LibEval demonstrates superior performance with models like Claude-3.7-Sonnet, emphasizing the benefits of structured prompting and precise intent extraction.

LibRec: Benchmarking Retrieval-Augmented LLMs for Library Migration Recommendations

Introduction

Library migration involves replacing outdated or unsuitable software libraries with more suitable alternatives, which is a common task in software engineering. Despite the empirical research on the types and frequencies of these migrations, an automated system for recommending alternative libraries has been lacking. This paper presents LibRec, a framework that combines the capabilities of LLMs with retrieval-augmented generation (RAG) techniques to recommend alternative libraries. The framework is evaluated against LibEval, a benchmark comprising migration records from Python repositories.

Methodology

The workflow of LibRec consists of three main components:

1. Migration Intents Generation: Utilizes LLMs to extract the purpose of each library migration from commit messages. These intents are enriched using in-context learning, enhancing the recommendation's accuracy.
2. Construction of RAG Database: Forms a dataset of library migration entities to aid the recommendation task, encoded with migration intents and types to provide contextual background.
3. Target Library Recommendation: Leverages structured prompting, whereby LLMs use migration examples and extracted intents to recommend replacement libraries, improving context comprehension.

An example of a library migration that the framework might manage is moving from pycryptodome/Crypto to cryptography due to compatibility with older distributions—a scenario adequately addressed by the framework's intent detection and recommendation capabilities.

Evaluation

LibEval, the proposed benchmarking dataset, evaluates LibRec's performance using precision metrics across intents derived from a substantial collection of library migration records. The benchmark comprises well-documented migration instances, detailing reasons behind library replacements.

Key Findings

1. Effectiveness: LibRec performs exceptionally well, particularly with the Claude-3.7-Sonnet model, which outperforms others across several metrics.
2. Component Contribution: Both the retrieval-augmented component and migration intents are crucial, though the importance of each varies with implementation specifics, suggesting room for optimization depending on the LLM's strengths.
3. Prompt Strategy: The "One-shot" prompt strategy shows effectiveness across general models, but "Chain-of-Thought" strategies could benefit reasoning models with more complex reasoning algorithms.
4. Intent type Performance: Different intents affect recommendation accuracy. The framework shows outstanding performance for intents concerning source library problems but struggles more when advantages of target libraries are the focus, especially those requiring nuanced feature understanding.

   Figure 1: The effectiveness of our framework (take the best-performing LLM of Claude-3.7-Sonnet as an example) across various intent types. The horizontal axis represents all sub-categories of intent types, and different colors indicate distinct categories. The values on the dotted line denote the average score of our framework in each category.

Case Study

Failures often occur when subtleties in library advantages are significant, particularly in the "Enhanced Features" intent category where nuanced feature improvement recognition remains a challenge. By contrasting the performance of model variations, such as Claude-3.7-Sonnet and DeepSeek-R1, the analysis pinpointed areas for improvement and underscored the importance of aligning model capabilities with the nature of the recommendation task.

Figure 2: Comparison of failure rates for Claude-3.7-Sonnet and DeepSeek-R1 when recommending target libraries across different subcategories (top 5 subcategories with the most significant performance disparities).

Conclusion

LibRec, alongside the LibEval benchmark, offers a sophisticated approach to automating library migration recommendations via retrieval-augmented LLMs. The framework provides meaningful insights into leveraging LLMs for software engineering tasks beyond traditional applications. Future research could explore enhancing feature comparison methods and extending the framework's applicability across different programming languages to further refine recommendations and address existing limitations. Additionally, new hybrid models combining LLM properties could be explored to mitigate existing challenges in nuanced task contexts.