Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective (2404.07549v1)
Abstract: Code generation aims to understand the problem description and generate corresponding code snippets, where existing works generally decompose such complex tasks into intermediate steps by prompting strategies, such as Chain-of-Thought and its variants. While these studies have achieved some success, their effectiveness is highly dependent on the capabilities of advanced LLMs such as GPT-4, particularly in terms of API calls, which significantly limits their practical applicability. Consequently, how to enhance the code generation capabilities of small and medium-scale code LLMs without significantly increasing training costs is an appealing challenge. In this paper, we suggest that code comments are the natural logic pivot between natural language and code language and propose using comments to boost the code generation ability of code LLMs. Concretely, we propose MANGO (comMents As Natural loGic pivOts), including a comment contrastive training strategy and a corresponding logical comment decoding strategy. Experiments are performed on HumanEval and MBPP, utilizing StarCoder and WizardCoder as backbone models, and encompassing model parameter sizes between 3B and 7B. The results indicate that MANGO significantly improves the code pass rate based on the strong baselines. Meanwhile, the robustness of the logical comment decoding strategy is notably higher than the Chain-of-thoughts prompting. The code is publicly available at \url{https://github.com/pppa2019/Mango}.
- Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
- Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
- Improving code generation by training with natural language feedback. arXiv preprint arXiv:2303.16749.
- Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations.
- Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
- How does gpt obtain its ability? tracing emergent abilities of language models to their sources. Yao Fu’s Notion.
- Self-planning code generation with large language model. arXiv preprint arXiv:2303.06689.
- Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.
- Structured chain-of-thought prompting for code generation. arXiv preprint arXiv:2305.06599.
- Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161.
- Think outside the code: Brainstorming boosts large language models in code generation. arXiv preprint arXiv:2305.10679.
- Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568.
- At which training stage does code data help llms reasoning? arXiv preprint arXiv:2309.16298.
- Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07124.
- Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.
- Reflexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems.
- Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
- Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244.
- Can programming languages boost each other via instruction tuning? arXiv preprint arXiv:2308.16824.
- Parsel: Algorithmic reasoning with language models by composing decompositions. In Thirty-seventh Conference on Neural Information Processing Systems.
- A survey of large language models. arXiv preprint arXiv:2303.18223.
- Improving code generation by dynamic temperature sampling. arXiv preprint arXiv:2309.02772.
- Yijie Chen (10 papers)
- Yijin Liu (29 papers)
- Fandong Meng (174 papers)
- Yufeng Chen (58 papers)
- Jinan Xu (64 papers)
- Jie Zhou (687 papers)