CoCoST: Automatic Complex Code Generation with Online Searching and Correctness Testing
Abstract: LLMs have revolutionized code generation ability by converting natural language descriptions into executable code. However, generating complex code within real-world scenarios remains challenging due to intricate structures, subtle bugs, understanding of advanced data types, and lack of supplementary contents. To address these challenges, we introduce the CoCoST framework, which enhances complex code generation by online searching for more information with planned queries and correctness testing for code refinement. Moreover, CoCoST serializes the complex inputs and outputs to improve comprehension and generates test cases to ensure the adaptability for real-world applications. CoCoST is validated through rigorous experiments on the DS-1000 and ClassEval datasets. Experimental results show that CoCoST substantially improves the quality of complex code generation, highlighting its potential to enhance the practicality of LLMs in generating complex code.
- Santacoder: don’t reach for the stars! arXiv preprint arXiv:2301.03988.
- Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
- Evaluating large language models trained on code.
- Teaching large language models to self-debug.
- Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation.
- Text2analysis: A benchmark of table question answering with advanced data analysis and unclear queries.
- Measuring coding challenge competence with apps. NeurIPS.
- Selfevolve: A code evolution framework via large language models.
- Ds-1000: A natural and reliable benchmark for data science code generation. ArXiv, abs/2211.11501.
- Starcoder: may the source be with you!
- Wizardcoder: Empowering code large language models with evol-instruct.
- Zohar Manna and Richard J. Waldinger. 1971. Toward automatic program synthesis. Commun. ACM, 14(3):151–165.
- OpenAI. 2023. Gpt-4 technical report.
- Code llama: Open foundation models for code.
- Self-edit: Fault-aware code editor for code generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 769–787, Toronto, Canada. Association for Computational Linguistics.
- Docprompting: Generating code by retrieving the docs. In International Conference on Learning Representations (ICLR), Kigali, Rwanda.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
