Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Models (2309.02772v3)
Abstract: Recently, LLMs have shown impressive abilities in code generation. However, existing LLMs' decoding strategies are designed for Natural Language (NL) generation, overlooking the differences between NL and programming languages (PL). Due to this oversight, a better decoding strategy for code generation remains an open question. In this paper, we conduct the first systematic study to explore a decoding strategy specialized in code generation. With an analysis of loss distributions of code tokens, we find that code tokens can be divided into two categories: challenging tokens that are difficult to predict and confident tokens that can be easily inferred. Among them, the challenging tokens mainly appear at the beginning of a code block. Inspired by the above findings, we propose a simple yet effective method: Adaptive Temperature (AdapT) sampling, which dynamically adjusts the temperature coefficient when decoding different tokens. We apply a larger temperature when sampling for challenging tokens, allowing LLMs to explore diverse choices. We employ a smaller temperature for confident tokens avoiding the influence of tail randomness noises. We apply AdapT sampling to LLMs with different sizes and conduct evaluations on two popular datasets. Results show that AdapT sampling significantly outperforms state-of-the-art decoding strategy.
- A learning algorithm for Boltzmann machines. Cognitive science, 9(1): 147–169.
- Program Synthesis with Large Language Models.
- greedy algorithm, Dictionary of Algorithms and Data Structures. US Nat. Inst. Std. & Tech Report, 88: 95.
- Measurement error. BMJ: British medical journal, 312(7047): 1654.
- An estimate of an upper bound for the entropy of English. Computational Linguistics, 18(1): 31–40.
- Brown, S. 2011. Measures of shape: Skewness and kurtosis.
- Language Models are Few-Shot Learners. CoRR, abs/2005.14165.
- Brownlee, J. 2019. Probability for machine learning: Discover how to harness uncertainty with Python. Machine Learning Mastery.
- Evaluating Large Language Models Trained on Code. CoRR, abs/2107.03374.
- Hierarchical Neural Story Generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 889–898. Melbourne, Australia: Association for Computational Linguistics.
- Beam Search Strategies for Neural Machine Translation. In Luong, T.; Birch, A.; Neubig, G.; and Finch, A. M., eds., Proceedings of the First Workshop on Neural Machine Translation, NMT@ACL 2017, Vancouver, Canada, August 4, 2017, 56–60. Association for Computational Linguistics.
- InCoder: A Generative Model for Code Infilling and Synthesis. In The Eleventh International Conference on Learning Representations.
- InCoder: A Generative Model for Code Infilling and Synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
- Measuring Coding Challenge Competence With APPS. In Vanschoren, J.; and Yeung, S., eds., Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
- The Curious Case of Neural Text Degeneration. In International Conference on Learning Representations.
- CodeEditor: Learning to Edit Source Code with Pre-Trained Models. ACM Trans. Softw. Eng. Methodol. Just Accepted.
- Large Language Model-Aware In-Context Learning for Code Generation. arXiv preprint arXiv:2310.09748.
- Structured Chain-of-Thought Prompting for Code Generation. arXiv preprint arXiv:2305.06599.
- SkCoder: A Sketch-based Approach for Automatic Code Generation. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, 2124–2135. IEEE.
- AceCoder: Utilizing Existing Code to Enhance Code Generation. arXiv preprint arXiv:2303.17780.
- Competition-level code generation with alphacode. Science, 378(6624): 1092–1097.
- On End-to-End Program Generation from User Intention by Deep Neural Networks. CoRR, abs/1510.07211.
- CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In The Eleventh International Conference on Learning Representations.
- A Conversational Paradigm for Program Synthesis. CoRR, abs/2203.13474.
- Runnenburg, J. T. 1978. Mean, median, mode. Statistica Neerlandica, 32(2): 73–79.
- Do Massively Pretrained Language Models Make Better Storytellers? In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), 843–861. Hong Kong, China: Association for Computational Linguistics.
- Improving Diversity of Neural Text Generation via Inverse Probability Weighting. CoRR, abs/2103.07649.
- CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X. CoRR, abs/2303.17568.