GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models (2411.05830v1)

Abstract: The rapid evolution of software libraries presents a significant challenge for code generation models, which must adapt to frequent version updates while maintaining compatibility with previous versions. Existing code completion benchmarks often overlook this dynamic aspect, and the one that does consider it relies on static code prediction tasks without execution-based evaluation, offering a limited perspective on a model's practical usability. To address this gap, we introduce \textbf{\GitChameleon{}}, a novel, manually curated dataset comprising 116 Python code completion problems, each conditioned on specific library versions and accompanied by executable unit tests. \GitChameleon{} is designed to rigorously assess the ability of modern LLMs to generate version-specific code that is not only syntactically correct but also functionally accurate upon execution. Our comprehensive evaluations reveal that state-of-the-art LLMs struggle with this task; for instance, \textbf{GPT-4o} achieves a pass@10 of only 39.9\% (43.7\% when provided with error feedback), highlighting the complexity of the problem and the limitations of current models. By providing an execution-based benchmark that emphasizes the dynamic nature of code libraries, \GitChameleon{} serves as a critical tool to advance the development of more adaptable and reliable code generation models. For facilitation for further exploration of version-conditioned code generation, we make our code repository publicly accessible at \url{https://github.com/NizarIslah/GitChameleon}.

• The paper introduces GitChameleon, a benchmark of 116 version-conditioned problems using assertion-based tests to evaluate LLMs’ coding accuracy.
• The paper finds that models like GPT-4o and CodeLlama 34B achieve pass@10 rates below 45%, highlighting challenges in handling version-specific code.
• The paper suggests that error feedback, model scaling, and multiple solution attempts can improve LLM performance in adapting to evolving software libraries.

An Exploration of Version-Specific Code Generation with GitChameleon

In the field of code generation, LLMs are increasingly integrated into development environments as virtual assistants, streamlining workflows and enhancing productivity. However, the rapidly evolving nature of software libraries poses a significant challenge for LLMs tasked with generating version-specific code. The introduction of GitChameleon provides a novel benchmark designed to assess the ability of LLMs to generate executable, version-specific code accurately.

Overview and Contributions of GitChameleon

GitChameleon is a Python-based benchmark consisting of 116 version-conditioned problems across popular libraries like NumPy, PyTorch, and Scikit-Learn. Each problem is paired with assertion-based unit tests, enabling a rigorous evaluation of LLM performance in realistically dynamic coding environments. This benchmark stands out by focusing on real API changes, contrasting with other datasets that often rely on synthetic variations. Moreover, in demonstrating the limitations of existing models, GitChameleon provides a structured framework to enhance LLM adaptability to library evolution.

The paper indicates that current LLMs, such as GPT-4o, struggle with version-specific code generation. For example, GPT-4o achieves a pass@10 rate of just 39.9%, rising marginally to 43.7% with error feedback. Such findings underscore the complexity of the task and the need for continued development in this domain. These performance metrics highlight that even state-of-the-art models often fall short when required to produce functionally accurate code across different library versions.

Experimental Analysis and Findings

The empirical evaluation uses various execution-based metrics, with Pass@k indicating success rates for generating syntactically correct and functionally valid code. A range of models, including open-source and closed-source options, are benchmarked, revealing a modest positive correlation between model size and performance. The DeepSeek-Coder 33B stands out with the highest baseline Pass@1 score (35.7%), while CodeLlama 34B excels in the Pass@10 category (42.8%).

Error feedback emerges as a significant factor for improvement, yielding an average increase of 5.4% in Pass@1 across instruction-tuned models. Larger models tend to perform better on argument and function name changes, yet struggle uniformly across the board with semantic changes. The paper discusses potential advancements, suggesting that error feedback, model scaling, and multiple solution attempts can enhance LLM capabilities for version-specific tasks.

Implications and Future Directions

GitChameleon represents a critical step forward in addressing the evolving challenges presented by software library updates. The benchmark could spur advances in continual or lifelong learning approaches, capitalizing on the chronological evolution inherent in software development environments. This direction carries practical value, particularly in scenarios involving enterprise-level code migration, where maintaining compatibility across library versions is non-negotiable.

Looking ahead, the research suggests potential enhancements, such as expanding the GitChameleon to encompass other programming languages or frameworks beyond Python, thereby broadening the scope and applicability of the benchmark. Furthermore, integrating prompt optimization and advanced inference techniques like retrieval-augmented generation could provide an upper bound on performance, paving the way for more nuanced improvements in LLM capabilities.

Conclusion

In conclusion, GitChameleon provides an insightful and practical framework for evaluating LLMs' version-conditioned code generation. The findings point towards the inherent challenges in this task and emphasize the need for further exploration and development in this area, particularly concerning model adaptability and error-handling strategies. Through this benchmark, the paper effectively highlights the current limitations of LLMs and sets a foundation for future research that could significantly enhance the practical utility of AI-assisted coding tools in real-world environments.