Explaining Competitive-Level Programming Solutions using LLMs (2307.05337v1)
Abstract: In this paper, we approach competitive-level programming problem-solving as a composite task of reasoning and code generation. We propose a novel method to automatically annotate natural language explanations to \textit{<problem, solution>} pairs. We show that despite poor performance in solving competitive-level programming problems, state-of-the-art LLMs exhibit a strong capacity in describing and explaining solutions. Our explanation generation methodology can generate a structured solution explanation for the problem containing descriptions and analysis. To evaluate the quality of the annotated explanations, we examine their effectiveness in two aspects: 1) satisfying the human programming expert who authored the oracle solution, and 2) aiding LLMs in solving problems more effectively. The experimental results on the CodeContests dataset demonstrate that while LLM GPT3.5's and GPT-4's abilities in describing the solution are comparable, GPT-4 shows a better understanding of the key idea behind the solution.
- Deepcoder: Learning to write programs. In International Conference on Learning Representations.
- Codet: Code generation with generated tests.
- Evaluating large language models trained on code. CoRR, abs/2107.03374.
- Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.
- Teaching large language models to self-debug.
- Wenyue Hua and Yongfeng Zhang. 2022. System 1 + system 2 = better world: Neural-symbolic chain of logic reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 601–612, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems.
- Antti Laaksonen. 2020. Guide to Competitive Programming - Learning and Improving Algorithms Through Contests, Second Edition. Undergraduate Topics in Computer Science. Springer.
- Comparing code explanations created by students and large language models. CoRR, abs/2304.03938.
- Solving quantitative reasoning problems with language models. CoRR, abs/2206.14858.
- Competition-level code generation with AlphaCode. Science, 378(6624):1092–1097.
- Faithful chain-of-thought reasoning.
- Experiences from using code explanations generated by large language models in a web software development e-book. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education, Volume 1, SIGCSE 2023, Toronto, ON, Canada, March 15-18, 2023, pages 931–937. ACM.
- Codeforces as an educational platform for learning programming in digitalization.
- Lever: Learning to verify language-to-code generation with execution.
- OpenAI. 2023a. ChatGPT: Optimizing Language Models for Dialogue. https://openai.com/blog/chatgpt.
- OpenAI. 2023b. Gpt-4 technical report.
- Training language models to follow instructions with human feedback. CoRR, abs/2203.02155.
- Illia Polosukhin and Alexander Skidanov. 2018. Neural program search: Solving programming tasks from description and examples. CoRR, abs/1802.04335.
- Adaptive test generation using a large language model.
- Execution-based code generation using deep reinforcement learning.
- Steven Skiena and Miguel A. Revilla. 2003. Programming challenges: the programming contest training manual. SIGACT News, 34:68–74.
- Iteratively prompt pre-trained language models for chain of thought. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2714–2730, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Self-consistency improves chain of thought reasoning in language models. CoRR, abs/2203.11171.
- Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903.
- Parsel: A (de-)compositional framework for algorithmic reasoning with language models.
- Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488.
- Least-to-most prompting enables complex reasoning in large language models.
- Jierui Li (6 papers)
- Szymon Tworkowski (7 papers)
- Yingying Wu (26 papers)
- Raymond Mooney (21 papers)