Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
143 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

AI Coders Are Among Us: Rethinking Programming Language Grammar Towards Efficient Code Generation (2404.16333v2)

Published 25 Apr 2024 in cs.SE, cs.AI, and cs.PL

Abstract: AI models have emerged as another important audience for programming languages alongside humans and machines, as we enter the era of LLMs. LLMs can now perform well in coding competitions and even write programs like developers to solve various tasks, including mathematical problems. However, the grammar and layout of current programs are designed to cater the needs of human developers -- with many grammar tokens and formatting tokens being used to make the code easier for humans to read. While this is helpful, such a design adds unnecessary computational work for LLMs, as each token they either use or produce consumes computational resources. To improve inference efficiency and reduce computational costs, we propose the concept of AI-oriented grammar. This aims to represent code in a way that better suits the working mechanism of AI models. Code written with AI-oriented grammar discards formats and uses a minimum number of tokens to convey code semantics effectively. To demonstrate the feasibility of this concept, we explore and implement the first AI-oriented grammar for Python, named SimPy. SimPy is crafted by revising the original Python grammar through a series of heuristic rules. Programs written in SimPy maintain identical AST structures to those in standard Python. This allows for not only execution via a modified AST parser, but also seamless transformation between programs written in Python and SimPy, enabling human developers and LLMs to use Python and SimPy, respectively, when they need to collaborate. In the experiments, compared with Python, SimPy enables a reduction in token usage by 13.5% and 10.4% for CodeLlama and GPT-4, respectively, when completing the same set of code-related tasks. Additionally, these models can maintain or even improve their performance when using SimPy instead of Python for these tasks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. DeepSeek AI. 2023. DeepSeek Coder: Let the Code Write Itself. https://github.com/deepseek-ai/DeepSeek-Coder.
  2. SantaCoder: don’t reach for the stars! arXiv preprint arXiv:2301.03988 (2023).
  3. Google DeepMind AlphaCode Team. 2023. AlphaCode 2 Technical Report. https://storage.googleapis.com/deepmind-media/AlphaCode2/AlphaCode2_Tech_Report.pdf.
  4. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning. PMLR, 2397–2430.
  5. AutoFocus: Interpreting Attention-Based Neural Networks by Code Perturbation. 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) (2019), 38–41. https://api.semanticscholar.org/CorpusID:208877064
  6. A Theory of Dual Channel Constraints. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results (Seoul, South Korea) (ICSE-NIER ’20). Association for Computing Machinery, New York, NY, USA, 25–28. https://doi.org/10.1145/3377816.3381720
  7. Harrison Chase. 2022. LangChain. https://github.com/langchain-ai/langchain
  8. Evaluating Large Language Models Trained on Code. ArXiv abs/2107.03374 (2021). https://api.semanticscholar.org/CorpusID:235755472
  9. CAT-probing: A Metric-based Approach to Interpret How Pre-trained Models for Programming Language Attend Code Structure. In Conference on Empirical Methods in Natural Language Processing. https://api.semanticscholar.org/CorpusID:252780909
  10. Nadezhda Chirkova and Sergey Troshin. 2023. CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code. ArXiv abs/2308.00683 (2023). https://api.semanticscholar.org/CorpusID:252600018
  11. Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning. Sustainable Computing: Informatics and Systems 38 (2023), 100857.
  12. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. ArXiv abs/2002.08155 (2020). https://api.semanticscholar.org/CorpusID:211171605
  13. Seymour Ginsburg and Joseph S. Ullian. 1966. Ambiguity in context free languages. J. ACM 13 (1966), 62–89. https://api.semanticscholar.org/CorpusID:5851601
  14. Google. 2023. ChatGPT. https://bard.google.com/.
  15. Dick Grune and Ceriel J. H. Jacobs. 2007. Parsing Techniques - A Practical Guide. In Monographs in Computer Science. https://api.semanticscholar.org/CorpusID:33077869
  16. Large Language Models for Software Engineering: A Systematic Literature Review. arXiv:2308.10620 [cs.SE]
  17. https://www.wsj.com/tech/ai/ais-costly-buildup-could-make-early-products-a-hard-sell-bdd29b9f [n. d.].
  18. Program Translation via Code Distillation. ArXiv abs/2310.11476 (2023). https://api.semanticscholar.org/CorpusID:264289043
  19. Scaling Laws for Neural Language Models. ArXiv abs/2001.08361 (2020). https://api.semanticscholar.org/CorpusID:210861095
  20. Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code. 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE) (2020), 1073–1085. https://api.semanticscholar.org/CorpusID:211161525
  21. The Stack: 3 TB of permissively licensed source code. Preprint (2022).
  22. Bernard Lang. 1974. Deterministic Techniques for Efficient Non-Deterministic Parsers. In International Colloquium on Automata, Languages and Programming. https://api.semanticscholar.org/CorpusID:27069587
  23. StarCoder: may the source be with you! ArXiv abs/2305.06161 (2023). https://api.semanticscholar.org/CorpusID:258588247
  24. WizardCoder: Empowering Code Large Language Models with Evol-Instruct. ArXiv abs/2306.08568 (2023). https://api.semanticscholar.org/CorpusID:259164815
  25. Is Self-Attention Powerful to Learn Code Syntax and Semantics? ArXiv abs/2212.10017 (2022). https://api.semanticscholar.org/CorpusID:254877330
  26. Anthony Moi and Nicolas Patry. 2023. HuggingFace’s Tokenizers. https://github.com/huggingface/tokenizers
  27. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In International Conference on Learning Representations. https://api.semanticscholar.org/CorpusID:252668917
  28. OpenAI. 2023a. ChatGPT. https://chat.openai.com/.
  29. OpenAI. 2023b. GPT-3.5. https://platform.openai.com/docs/models/gpt-3-5.
  30. OpenAI. 2023c. GPT-4 Technical Report. ArXiv abs/2303.08774 (2023). https://api.semanticscholar.org/CorpusID:257532815
  31. Tim Peters. 2023. PEP 20 – The Zen of Python. https://peps.python.org/pep-0020/.
  32. Python. 2023. Full Grammar specification. https://docs.python.org/3/reference/grammar.html.
  33. Understanding neural code intelligence through program simplification. Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (2021). https://api.semanticscholar.org/CorpusID:235359051
  34. Syntax-guided program reduction for understanding neural code intelligence models. Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming (2022). https://api.semanticscholar.org/CorpusID:249191662
  35. Language Models are Unsupervised Multitask Learners. https://api.semanticscholar.org/CorpusID:160025533
  36. Repilt. 2023. ReplitLM. https://github.com/replit/replitLM.
  37. Code Llama: Open Foundation Models for Code. ArXiv abs/2308.12950 (2023). https://api.semanticscholar.org/CorpusID:261100919
  38. Neural Machine Translation of Rare Words with Subword Units. ArXiv abs/1508.07909 (2015). https://api.semanticscholar.org/CorpusID:1114678
  39. Structural-semantics Guided Program Simplification for Understanding Neural Code Intelligence Models. Proceedings of the 14th Asia-Pacific Symposium on Internetware (2023). https://api.semanticscholar.org/CorpusID:263672536
  40. Significant Gravitas. [n. d.]. AutoGPT. https://github.com/Significant-Gravitas/AutoGPT
  41. Llama 2: Open Foundation and Fine-Tuned Chat Models. ArXiv abs/2307.09288 (2023). https://api.semanticscholar.org/CorpusID:259950998
  42. tree sitter. 2023. tree-sitter/tree-sitter: An incremental parsing system for programming tools. https://github.com/tree-sitter/tree-sitter.
  43. Sergey Troshin and Nadezhda Chirkova. 2022. Probing Pretrained Models of Source Codes. ArXiv abs/2202.08975 (2022). https://api.semanticscholar.org/CorpusID:246996634
  44. PEP 8 – Style Guide for Python Code. https://peps.python.org/pep-0008/.
  45. What Do They Capture? - A Structural Analysis of Pre-Trained Language Models for Source Code. 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE) (2022), 2377–2388. https://api.semanticscholar.org/CorpusID:246823289
  46. CodeT5+: Open Code Large Language Models for Code Understanding and Generation. ArXiv abs/2305.07922 (2023). https://api.semanticscholar.org/CorpusID:258685677
  47. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. ArXiv abs/2109.00859 (2021). https://api.semanticscholar.org/CorpusID:237386541
  48. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, Online, 38–45. https://www.aclweb.org/anthology/2020.emnlp-demos.6
  49. Robustness, Security, Privacy, Explainability, Efficiency, and Usability of Large Language Models for Code. ArXiv abs/2403.07506 (2024). https://api.semanticscholar.org/CorpusID:268364103
  50. An Extensive Study on Pre-Trained Models for Program Understanding and Generation. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual, South Korea) (ISSTA 2022). Association for Computing Machinery, New York, NY, USA, 39–51. https://doi.org/10.1145/3533767.3534390
  51. TinyLlama: An Open-Source Small Language Model. arXiv:2401.02385 [cs.CL]
  52. Diet code is healthy: simplifying programs for pre-trained models of code. Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (2022). https://api.semanticscholar.org/CorpusID:250113729
  53. Probing model signal-awareness via prediction-preserving input minimization. Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (2020). https://api.semanticscholar.org/CorpusID:227227733
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube