ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifiers (2305.14591v3)
Abstract: LLMs excel at implementing code from functionality descriptions but struggle with algorithmic problems that require not only implementation but also identification of the suitable algorithm. Moreover, LLM-generated programs lack guaranteed correctness and require human verification. To address these challenges, we propose ALGO, a framework that synthesizes Algorithmic programs with LLM-Generated Oracles to guide the generation and verify their correctness. ALGO first generates a reference oracle by prompting an LLM to exhaustively enumerate all the combinations of relevant variables. This oracle is then utilized to guide an arbitrary search strategy in exploring the algorithm space and to verify the synthesized algorithms. Our study shows that the LLM-generated oracles are correct for 88% of the cases. With the oracles as verifiers, ALGO can be integrated with any existing code generation model in a model-agnostic manner to enhance its performance. Experiments show that when equipped with ALGO, we achieve an 8x better one-submission pass rate over the Codex model and a 2.6x better one-submission pass rate over CodeT, the current state-of-the-art model on CodeContests. We can also get 1.3x better pass rate over the ChatGPT Code Interpreter on unseen problems. The problem set we used for testing, the prompts we used, the verifier and solution programs, and the test cases generated by ALGO are available at https://github.com/zkx06111/ALGO.
- Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
- GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. URL https://doi.org/10.5281/zenodo.5297715. If you use this software, please cite it using these metadata.
- Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
- Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations, 2023a. URL https://openreview.net/forum?id=ktrw68Cmu9c.
- Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
- Type-directed synthesis of visualizations from natural language queries. Proceedings of the ACM on Programming Languages, 6(OOPSLA2):532–559, 2022.
- Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023b.
- Introduction to algorithms. MIT press, 2022.
- Toga: a neural method for test oracle generation. In Proceedings of the 44th International Conference on Software Engineering, pp. 2130–2141, 2022.
- Obsynth: An interactive synthesis system for generating object models from natural language specifications. arXiv preprint arXiv:2210.11468, 2022.
- Measuring coding challenge competence with apps. NeurIPS, 2021.
- Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022.
- Fault-aware neural code rankers. Advances in Neural Information Processing Systems, 35:13419–13432, 2022.
- A theory of formal synthesis via inductive learning. Acta Informatica, 54:693–726, 2017.
- I speak, you verify: Toward trustworthy neural program synthesis. arXiv preprint arXiv:2210.00848, 2022.
- Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems, 32, 2019.
- Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022.
- Automatic test case and test oracle generation based on functional scenarios in formal specifications for conformance testing. IEEE Transactions on Software Engineering, 48(2):691–712, 2020.
- Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023.
- A deductive approach to program synthesis. ACM Transactions on Programming Languages and Systems (TOPLAS), 2(1):90–121, 1980.
- Reading between the lines: Modeling user behavior and costs in ai assisted programming. arXiv preprint arXiv:2210.14306, 2022.
- Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning, pp. 26106–26128. PMLR, 2023.
- Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=iaYcJKpY2B_.
- Model-based test oracle generation for automated unit testing of agent systems. IEEE Transactions on Software Engineering, 39(9):1230–1244, 2013.
- Flashmeta: A framework for inductive program synthesis. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, pp. 107–126, 2015.
- Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- Natural language to code translation with execution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3533–3546, 2022.
- From program verification to program synthesis. In Proceedings of the 37th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL ’10, pp. 313–326, New York, NY, USA, 2010. Association for Computing Machinery. ISBN 9781605584799. doi: 10.1145/1706299.1706337. URL https://doi.org/10.1145/1706299.1706337.
- Template-based program verification and program synthesis. International Journal on Software Tools for Technology Transfer, 15, 04 2013. doi: 10.1007/s10009-012-0223-4.
- Learn from mistakes through cooperative interaction with study assistant. The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) 2022, 2023.
- Planning with large language models for code generation. In The Eleventh International Conference on Learning Representations, 2023a. URL https://openreview.net/forum?id=Lr8cOOtYbfL.
- Coder reviewer reranking for code generation. In International Conference on Machine Learning, pp. 41832–41846. PMLR, 2023b.
- Kexun Zhang (21 papers)
- Danqing Wang (37 papers)
- Jingtao Xia (3 papers)
- William Yang Wang (254 papers)
- Lei Li (1293 papers)