Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Improving Natural Language Capability of Code Large Language Model (2401.14242v1)

Published 25 Jan 2024 in cs.CL

Abstract: Code LLMs (Code LLMs) have demonstrated remarkable performance in code generation. Nonetheless, most existing works focus on boosting code LLMs from the perspective of programming capabilities, while their natural language capabilities receive less attention. To fill this gap, we thus propose a novel framework, comprising two modules: AttentionExtractor, which is responsible for extracting key phrases from the user's natural language requirements, and AttentionCoder, which leverages these extracted phrases to generate target code to solve the requirement. This framework pioneers an innovative idea by seamlessly integrating code LLMs with traditional natural language processing tools. To validate the effectiveness of the framework, we craft a new code generation benchmark, called MultiNL-H, covering five natural languages. Extensive experimental results demonstrate the effectiveness of our proposed framework.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. Florian Boudin. 2018. Unsupervised keyphrase extraction with multipartite graphs. arXiv preprint arXiv:1803.08721.
  2. Topicrank: Graph-based topic ranking for keyphrase extraction. In International joint conference on natural language processing (IJCNLP), pages 543–551.
  3. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pages 89–96.
  4. A scalable and extensible approach to benchmarking nl2code for 18 programming languages. ArXiv, abs/2208.08227.
  5. CodeT: Code generation with generated tests. In The Eleventh International Conference on Learning Representations.
  6. Evaluating large language models trained on code. ArXiv, abs/2107.03374.
  7. PanGu-Coder: Program synthesis with function-level language modeling. ArXiv, abs/2207.11280.
  8. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  9. Corina Florescu and Cornelia Caragea. 2017. Positionrank: An unsupervised approach to keyphrase extraction from scholarly documents. In Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1105–1115.
  10. Huggingface. 2021. Training CodeParrot from Scratch. https://huggingface.co/blog/codeparrot.
  11. Yongmin Li Jia Li, Ge Li and Zhi Jin. 2023. Structured chain-of-thought prompting for code generation.
  12. Slava Kalyuga. 2011. Cognitive load theory: How many types of load does it really need? Educational Psychology Review, 23:1–19.
  13. Structured chain-of-thought prompting for code generation. arXiv preprint arXiv:2305.06599.
  14. StarCoder: may the source be with you!
  15. Competition-level code generation with alphacode. Science, 378:1092 – 1097.
  16. Uncovering and quantifying social biases in code generation. arXiv preprint arXiv:2305.15377.
  17. WizardCoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568.
  18. Deep keyphrase generation. arXiv preprint arXiv:1704.06879.
  19. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404–411.
  20. Author’s Name. 2023. Code llama beats gpt-4: A deep dive. Accessed: 2023-10-31.
  21. Gpt-4 technical report.
  22. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155.
  23. Code Llama: Open foundation models for code.
  24. PanGu-Coder2: Boosting large language models for code with ranking feedback.
  25. Reflexion: Language agents with verbal reinforcement learning.
  26. John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive science, 12(2):257–285.
  27. John Sweller. 2011. Cognitive load theory. In Psychology of learning and motivation, volume 55, pages 37–76. Elsevier.
  28. Peter D Turney. 2000. Learning algorithms for keyphrase extraction. Information retrieval, 2:303–336.
  29. Xiaojun Wan and Jianguo Xiao. 2008. Single document keyphrase extraction using neighborhood knowledge. In AAAI, volume 8, pages 855–860.
  30. Chain-of-thought prompting elicits reasoning in large language models.
  31. A systematic evaluation of large language models of code. Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming.
  32. When language model meets private library. In Conference on Empirical Methods in Natural Language Processing.
  33. CERT: Continual pre-training on sketches for library-oriented code generation. In International Joint Conference on Artificial Intelligence.
  34. Large language models meet NL2Code: A survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7443–7464, Toronto, Canada. Association for Computational Linguistics.
  35. RepoCoder: Repository-level code completion through iterative retrieval and generation.
  36. Unifying the perspectives of nlp and software engineering: A survey on language models for code.
  37. CodeGeeX: A pre-trained model for code generation with multilingual evaluations on humaneval-x. ArXiv, abs/2303.17568.
  38. A survey of large language models for code: Evolution, benchmarking, and future trends.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Wei Li (1122 papers)
  2. Daoguang Zan (24 papers)
  3. Bei Guan (11 papers)
  4. Ailun Yu (6 papers)
  5. Xiaolin Chen (27 papers)
  6. Yongji Wang (21 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now