DevEval: Evaluating Code Generation in Practical Software Projects (2401.06401v4)
Abstract: How to evaluate LLMs in code generation is an open question. Many benchmarks have been proposed but are inconsistent with practical software projects, e.g., unreal program distributions, insufficient dependencies, and small-scale project contexts. Thus, the capabilities of LLMs in practical projects are still unclear. In this paper, we propose a new benchmark named DevEval, aligned with Developers' experiences in practical projects. DevEval is collected through a rigorous pipeline, containing 2,690 samples from 119 practical projects and covering 10 domains. Compared to previous benchmarks, DevEval aligns to practical projects in multiple dimensions, e.g., real program distributions, sufficient dependencies, and enough-scale project contexts. We assess five popular LLMs on DevEval (e.g., gpt-4, gpt-3.5-turbo, CodeLLaMa, and StarCoder) and reveal their actual abilities in code generation. For instance, the highest Pass@1 of gpt-3.5-turbo only is 42 in our experiments. We also discuss the challenges and future directions of code generation in practical projects. We open-source DevEval and hope it can facilitate the development of code generation in practical projects.
- Program synthesis with large language models. CoRR, abs/2108.07732.
- Evaluating large language models trained on code. CoRR, abs/2107.03374.
- Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. CoRR, abs/2310.11248.
- Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. CoRR, abs/2308.01861.
- GitHub. 2023. Github copilot. https://github.com/features/copilot.
- Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 1643–1652. Association for Computational Linguistics.
- Structured chain-of-thought prompting for code generation. arXiv preprint arXiv:2305.06599.
- Skcoder: A sketch-based approach for automatic code generation. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pages 2124–2135. IEEE.
- Structured chain-of-thought prompting for code generation. arXiv preprint arXiv:2303.17780.
- Starcoder: may the source be with you! CoRR, abs/2305.06161.
- Lost in the middle: How language models use long contexts. CoRR, abs/2307.03172.
- OpenAI. 2023a. gpt-3.5-turbo. https://platform.openai.com/docs/models/gpt-3-5.
- OpenAI. 2023b. GPT-4 technical report. CoRR, abs/2303.08774.
- Pyan. 2023. Pyan. https://github.com/davidfraser/pyan.
- PyPI. Pypi. https://pypi.org/.
- Code llama: Open foundation models for code. CoRR, abs/2308.12950.
- In-context pretraining: Language modeling beyond document boundaries. CoRR, abs/2310.10638.
- Repository-level prompt generation for large language models of code. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 31693–31715. PMLR.
- Learning to mine aligned code and natural language pairs from stack overflow. In Proceedings of the 15th International Conference on Mining Software Repositories, MSR 2018, Gothenburg, Sweden, May 28-29, 2018, pages 476–486. ACM.
- Codereval: A benchmark of pragmatic code generation with generative pre-trained models. CoRR, abs/2302.00288.
- CERT: continual pre-training on sketches for library-oriented code generation. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 2369–2375. ijcai.org.
- RepoCoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2471–2484, Singapore. Association for Computational Linguistics.
- Self-edit: Fault-aware code editor for code generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 769–787. Association for Computational Linguistics.
- Jia Li (380 papers)
- Ge Li (213 papers)
- Yunfei Zhao (13 papers)
- Yongmin Li (32 papers)
- Zhi Jin (160 papers)
- Hao Zhu (212 papers)
- Huanyu Liu (15 papers)
- Kaibo Liu (17 papers)
- Lecheng Wang (8 papers)
- Zheng Fang (103 papers)
- Lanshen Wang (2 papers)
- Jiazheng Ding (5 papers)
- Xuanming Zhang (20 papers)
- Yihong Dong (35 papers)
- Yuqi Zhu (25 papers)
- Bin Gu (86 papers)
- Mengfei Yang (6 papers)