Model Cascading for Code: Reducing Inference Costs with Model Cascading for LLM Based Code Generation (2405.15842v1)
Abstract: The rapid development of LLMs has led to significant advancements in code completion tasks. While larger models have higher accuracy, they also cost much more to run. Meanwhile, model cascading has been proven effective to conserve computational resources while enhancing accuracy in LLMs on natural language generation tasks. It generates output with the smallest model in a set, and only queries the larger models when it fails to meet predefined quality criteria. However, this strategy has not been used in code completion tasks, primarily because assessing the quality of code completions differs substantially from assessing natural language, where the former relies heavily on the functional correctness. To address this, we propose letting each model generate and execute a set of test cases for their solutions, and use the test results as the cascading threshold. We show that our model cascading strategy reduces computational costs while increases accuracy compared to generating the output with a single model. We also introduce a heuristics to determine the optimal combination of the number of solutions, test cases, and test lines each model should generate, based on the budget. Compared to speculative decoding, our method works on black-box models, having the same level of cost-accuracy trade-off, yet providing much more choices based on the server's budget. Ours is the first work to optimize cost-accuracy trade-off for LLM code generation with model cascading.
- Mark Chen et al. Evaluating large language models trained on code. 2021.
- Yujia Li et al. Competition-level code generation with alphacode. Science, 378:1092–1097, 2022.
- Erik Nijkamp et al. Codegen: An open large language model for code with multi-turn program synthesis. 2022.
- Github copilot · your ai pair programmer, 2021. URL https://copilot.github.com/.
- Xunyu Zhu et al. A survey on model compression for large language models. 2023.
- Zechun Liu et al. Llm-qat: Data-free quantization aware training for large language models. 2023.
- Xinyin Ma et al. Llm-pruner: On the structural pruning of large language models. 2023.
- Cheng-Yu Hsieh et al. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. 2023.
- Tri Dao et al. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022.
- Erik Nijkamp et al. Codegen2: Lessons for training llms on programming and natural languages. 2023.
- Raymond Li et al. Starcoder: may the source be with you! 2023.
- Daya Guo et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. 2024.
- Bei Chen et al. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations, 2023a.
- Baptiste Rozière et al. Code llama: Open foundation models for code, 2023.
- Ziyang Luo et al. Wizardcoder: Empowering code large language models with evol-instruct. 2023.
- Lingjiao Chen et al. Frugalgpt: How to use large language models while reducing cost and improving performance. 2023b.
- Victor Sanh et al. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. 2020.
- Yaniv Leviathan et al. Fast inference from transformers via speculative decoding. 2023.
- Weimin Xiong et al. The program testing ability of large language models for code. 2023.
- RunPod. Gpu cloud service, 2023. URL http://runpod.io.
- Daniel Fried et al. Incoder: A generative model for code infilling and synthesis. 2022.
- Qinkai Zheng et al. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. 2023.
- Hugo Touvron et al. Llama 2: Open foundation and fine-tuned chat models. 2023.
- Sahil Chaudhary. Code alpaca: An instruction-following llama model for code generation. Github repository. https://github.com/sahil280114/codealpaca, 2023.
- Jacob Austin et al. Program synthesis with large language models. 2021.
- Dan Hendrycks et al. Measuring coding challenge competence with apps. 2021.
- Thomas Wolf et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, 2020.
- Sylvain Gugger et al. Accelerate: Training and inference at scale made simple, efficient and adaptable. Github repository. https://github.com/huggingface/accelerate, 2022.
- Woosuk Kwon et al. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, page 611–626, 2023.
- Qidong Su et al. The synergy of speculative decoding and batching in serving large language models. 2023.
- Charlie Chen et al. Accelerating large language model decoding with speculative sampling. 2023c.
- Boyuan Chen (75 papers)
- Mingzhi Zhu (6 papers)
- Brendan Dolan-Gavitt (24 papers)
- Muhammad Shafique (204 papers)
- Siddharth Garg (99 papers)