Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Program Repair with Minimal Edits Using CodeT5 (2309.14760v1)

Published 26 Sep 2023 in cs.CL, cs.AI, and cs.SE

Abstract: Programmers often struggle to identify and fix bugs in their programs. In recent years, many LLMs (LMs) have been proposed to fix erroneous programs and support error recovery. However, the LMs tend to generate solutions that differ from the original input programs. This leads to potential comprehension difficulties for users. In this paper, we propose an approach to suggest a correct program with minimal repair edits using CodeT5. We fine-tune a pre-trained CodeT5 on code pairs of wrong and correct programs and evaluate its performance with several baseline models. The experimental results show that the fine-tuned CodeT5 achieves a pass@100 of 91.95% and an average edit distance of the most similar correct program of 6.84, which indicates that at least one correct program can be suggested by generating 100 candidate programs. We demonstrate the effectiveness of LMs in suggesting program repair with minimal edits for solving introductory programming problems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. H. A. Nguyen, A. T. Nguyen, T. T. Nguyen, T. N. Nguyen, and H. Rajan, “A study of repetitiveness of code changes in software evolution,” in Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 180–190, 2013.
  2. S. Chakraborty, Y. Ding, M. Allamanis, and B. Ray, “CODIT: Code editing with tree-based neural models,” IEEE Transactions on Software Engineering, vol. 48, no. 4, pp. 1385–1399, 2022.
  3. B. Ray, M. Kim, S. Person, and N. Rungta, “Detecting and characterizing semantic inconsistencies in ported code,” in Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 367–377, 2013.
  4. A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan, “IntelliCode Compose: Code generation using transformer,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), p. 1433–1443, 2020.
  5. K. Terada and Y. Watanobe, “Code completion for programming education based on deep learning,” International Journal of Computational Intelligence Studies, vol. 10, no. 2-3, pp. 78–98, 2021.
  6. B. Berabi, J. He, V. Raychev, and M. Vechev, “TFix: Learning to fix coding errors with a text-to-text transformer,” in Proceedings of the 38th International Conference on Machine Learning (ICML), vol. 139, pp. 780–791, 2021.
  7. M. M. Rahman, Y. Watanobe, and K. Nakamura, “A bidirectional lstm language model for code evaluation and repair,” Symmetry, vol. 13, no. 2, 2021.
  8. T. Matsumoto, Y. Watanobe, and K. Nakamura, “A model with iterative trials for correcting logic errors in source code,” Applied Sciences, vol. 11, no. 11, 2021.
  9. H. Joshi, J. Cambronero Sanchez, S. Gulwani, V. Le, G. Verbruggen, and I. Radiček, “Repair is nearly generation: Multilingual program repair with llms,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 4, pp. 5131–5140, 2023.
  10. A. Madaan, A. Shypula, U. Alon, M. Hashemi, P. Ranganathan, et al., “Learning performance-improving code edits,” arXiv preprint, 2023.
  11. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, et al., “Self-Refine: Iterative refinement with self-feedback,” arXiv preprint, 2023.
  12. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, et al., “Attention is all you need,” in Advances in Neural Information Processing Systems (NIPS), vol. 30, 2017.
  13. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, 2019.
  14. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 1877–1901, 2020.
  15. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, et al., “Evaluating large language models trained on code,” arXiv preprint, 2021.
  16. Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, et al., “Competition-level code generation with AlphaCode,” Science, vol. 378, no. 6624, pp. 1092–1097, 2022.
  17. Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8696–8708, 2021.
  18. D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, et al., “InCoder: A generative model for code infilling and synthesis,” in Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023.
  19. E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, et al., “CodeGen: An open large language model for code with multi-turn program synthesis,” in Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023.
  20. Q. Zheng, X. Xia, X. Zou, Y. Dong, S. Wang, et al., “CodeGeeX: A pre-trained model for code generation with multilingual benchmarking on humaneval-x,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), p. 5673–5684, 2023.
  21. R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, et al., “StarCoder: may the source be with you!,” arXiv preprint, 2023.
  22. Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, et al., “WizardCoder: Empowering code large language models with evol-instruct,” arXiv preprint, 2023.
  23. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in Proceedings of the 9th International Conference on Learning Representations (ICLR), 2021.
  24. A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and C. Schmid, “ViViT: A video vision transformer,” in Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6816–6826, 2021.
  25. Y. Watanobe, “Aizu Online Judge,” 2004. https://onlinejudge.u-aizu.ac.jp.
  26. S. Bhatia and R. Singh, “Automated correction for syntax errors in programming assignments using recurrent neural networks,” 2016.
  27. Y. Pu, K. Narasimhan, A. Solar-Lezama, and R. Barzilay, “Sk_p: A neural program corrector for moocs,” in Companion Proceedings of the 2016 ACM SIGPLAN International Conference on Systems, Programming, Languages and Applications: Software for Humanity (SPLASH), p. 39–40, 2016.
  28. M. Tufano, C. Watson, G. Bavota, M. D. Penta, M. White, and D. Poshyvanyk, “An empirical study on learning bug-fixing patches in the wild via neural machine translation,” ACM Transactions on Software Engineering and Methodology, vol. 28, no. 4, 2019.
  29. Z. Chen, S. Kommrusch, M. Tufano, L.-N. Pouchet, D. Poshyvanyk, and M. Monperrus, “SequenceR: Sequence-to-sequence learning for end-to-end program repair,” IEEE Transactions on Software Engineering, vol. 47, no. 9, pp. 1943–1959, 2021.
  30. M. M. Rahman, Y. Watanobe, P. Szmeja, P. Sowiński, M. Paprzycki, and M. Ganzha, “Code semantics learning with deep neural networks: An ai-based approach for programming education,” in Proceedings of the 23rd International Conference on Computational Science (ICCS), pp. 737–750, 2023.
  31. H. Hata, E. Shihab, and G. Neubig, “Learning to generate corrective patches using neural machine translation,” arXiv preprint, 2019.
  32. E. Dinella, H. Dai, Z. Li, M. Naik, L. Song, and K. Wang, “Hoppity: Learning graph transformations to detect and fix bugs in programs,” in Proceedings of the 16th International Conference on Learning Representations (ICLR), 2020.
  33. T. Lutellier, H. V. Pham, L. Pang, Y. Li, M. Wei, and L. Tan, “CoCoNuT: Combining context-aware neural translation models using ensemble for program repair,” in Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), p. 101–114, 2020.
  34. S. Gulwani, I. Radiček, and F. Zuleger, “Automated clustering and program repair for introductory programming assignments,” in Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), p. 465–480, 2018.
  35. Y. Lu, N. Meng, and W. Li, “FAPR: Fast and accurate program repair for introductory programming courses,” arXiv preprint, 2021.
  36. S. Parihar, Z. Dadachanji, P. K. Singh, R. Das, A. Karkare, and A. Bhattacharya, “Automatic grading and feedback using program repair for introductory programming courses,” in Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE), p. 92–97, 2017.
  37. D. Drain, C. Wu, A. Svyatkovskiy, and N. Sundaresan, “Generating bug-fixes using pretrained transformers,” in Proceedings of the 5th ACM SIGPLAN International Symposium on Machine Programming (MAPS), p. 1–8, 2021.
  38. S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, et al., “CodeXGLUE: A machine learning benchmark dataset for code understanding and generation,” in Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, vol. 1, 2021.
  39. Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, et al., “CodeBERT: A pre-trained model for programming and natural languages,” in Findings of the Association for Computational Linguistics: EMNLP, pp. 1536–1547, 2020.
  40. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020.
  41. J. A. Prenner, H. Babii, and R. Robbes, “Can openai’s codex fix bugs? an evaluation on quixbugs,” in Proceedings of the 3rd International Workshop on Automated Program Repair (APR), p. 69–75, 2022.
  42. J. Zhang, J. Cambronero, S. Gulwani, V. Le, R. Piskac, et al., “Repairing bugs in python assignments using large language models,” arXiv preprint, 2022.
  43. M. Jin, S. Shahriar, M. Tufano, X. Shi, S. Lu, et al., “InferFix: End-to-end program repair with llms,” arXiv preprint, 2023.
  44. H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt, “Examining zero-shot vulnerability repair with large language models,” in Proceedings of the 2023 IEEE Symposium on Security and Privacy (SP), pp. 2339–2356, 2023.
  45. Y. Watanobe, M. M. Rahman, T. Matsumoto, U. K. Rage, and P. Ravikumar, “Online judge system: Requirements, architecture, and experiences,” International Journal of Software Engineering and Knowledge Engineering, vol. 32, no. 4, pp. 1–30, 2022.
  46. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: A method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL), pp. 311–318, 2002.
  47. C.-Y. Lin and F. J. Och, “ORANGE: a method for evaluating automatic evaluation metrics for machine translation,” in COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pp. 501–507, 2004.
  48. S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, et al., “CodeBLEU: a method for automatic evaluation of code synthesis,” arXiv preprint, 2020.
  49. B. Roziere, M.-A. Lachaux, L. Chanussot, and G. Lample, “Unsupervised translation of programming languages,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 20601–20611, 2020.
  50. V. I. Levenshtein, “Binary Codes Capable of Correcting Deletions, Insertions and Reversals,” Soviet Physics Doklady, vol. 10, p. 707, 1966.
  51. M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, et al., “fairseq: A fast, extensible toolkit for sequence modeling,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pp. 48–53, 2019.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Atsushi Shirafuji (6 papers)
  2. Md. Mostafizer Rahman (3 papers)
  3. Md Faizul Ibne Amin (2 papers)
  4. Yutaka Watanobe (10 papers)
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now