EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (2410.10209v4)

Abstract: As LLMs play an increasingly important role in code generation, enhancing both correctness and efficiency has become crucial. Current methods primarily focus on correctness, often overlooking efficiency. To address this gap, we introduce EffiCoder to improve both aspects by fine-tuning LLMs on a high-quality dataset comprising correct and efficient code samples. Our methodology involves leveraging multiple LLMs to generate diverse candidate code solutions for various tasks across different programming languages. We then evaluate these solutions by measuring their execution time and memory usage through local execution. The code solution with the lowest execution time and memory consumption is selected as the final output for each task. Experimental results demonstrate significant improvements when fine-tuning with Effi-Instruct. For instance, Qwen2.5-Coder-7B-Instruct's pass@1 score increases from 44.8\% to 57.7\%, while the average execution time for correct tasks decreases by 48.4\%. EffiCoder offers a scalable and effective solution for advancing AI-driven code generation, benefiting software development and computational problem-solving. The source code of Effi-Code was released at https://github.com/huangd1999/EffiCoder.

• The paper presents FFI-CODE, a method to fine-tune LLMs that enhances both code correctness and efficiency using iterative optimization based on overhead profiling.
• It constructs a high-quality dataset by aggregating, cleaning, and validating open-source code, ensuring robust performance on algorithmic tasks.
• Experimental results show notable gains, with pass@1 improving from 43.3% to 76.8% and execution time decreasing by 30.5% on key benchmarks.

FFI-CODE: Enhancing Code Efficiency in LLMs

The rapid advancement of LLMs has revolutionized software development with automated code generation. Despite their increased correctness, LLMs often produce code that is inefficient compared to solutions crafted by human developers. The paper presents FFI-CODE, an innovative approach to fine-tuning LLMs for both correctness and efficiency, thereby addressing a critical shortcoming in current methods that overly focus on correctness.

Central to the methodology is the development of the FFI-CODE dataset that amalgamates multiple open-source code datasets, followed by rigorous preprocessing to ensure high-quality input. This paper introduces a Self-Optimization process based on Overhead Profiling designed to iteratively refine code samples for efficiency, as measured by execution time and resource usage, while maintaining correctness.

Key Contributions

1. Dataset Construction and Preprocessing: The research aggregates source code from existing open-source datasets and undergoes comprehensive cleaning and validation processes. Here, tasks are filtered and non-algorithmic tasks are excluded to maintain a focus on tasks with optimization potential. Moreover, the use of test cases ensures the measuring efficiency and correctness.
2. Optimization Pipeline: FFI-CODE enhances code efficiency through an iterative optimization framework. This framework leverages overhead profiling to identify and mitigate inefficiencies, effectively tuning the model with optimized and metadata-rich code.
3. Experimental Validation: The paper reports substantial improvements in both correctness and efficiency on benchmarks such as HumanEval and EffiBench. For instance, the pass@1 of DeepSeek-Coder-6.7B-Instruct improved from 43.3% to 76.8%, and execution time decreased by 30.5%.
4. Scalability and Generalization: The scalable nature of FFI-CODE is validated by fine-tuning several models, demonstrating efficacy across varying LLM sizes and contexts. This includes efficiency gains across different models and configurations with both supervised fine-tuning (SFT) and optimization-based approaches (DPO, ORPO).

Implications and Future Directions

The presented work has profound implications in the field of AI-driven software development, offering a pathway to reducing computational overhead and enhancing sustainability. With efficient code consuming less computational resources, the environmental impact and operational costs can be significantly curtailed, an aspect particularly vital for deployment in resource-constrained environments.

Future work can explore integrating FFI-CODE with other synthetic instructional tuning methods to potentially amplify coding capabilities further, thus enhancing model robustness and adaptability. With the open sourcing of FFI-CODE and associated model weights, the authors aim to spur continued research and development efforts in optimizing LLM-generated code.

This paper sets a solid foundation for enriching LLMs with not only the power to generate syntactically correct code but also a capability to match or surpass human developers in terms of efficiency. The implications extend to optimizing neural architectures, fine-tuning strategies, and further iterations of benchmark metrics, thereby continually driving advancements in automated code generation.