On the Compression of Language Models for Code: An Empirical Study on CodeBERT (2412.13737v1)

Abstract: LLMs have proven successful across a wide range of software engineering tasks, but their significant computational costs often hinder their practical adoption. To address this challenge, researchers have begun applying various compression strategies to improve the efficiency of LLMs for code. These strategies aim to optimize inference latency and memory usage, though often at the cost of reduced model effectiveness. However, there is still a significant gap in understanding how these strategies influence the efficiency and effectiveness of LLMs for code. Here, we empirically investigate the impact of three well-known compression strategies -- knowledge distillation, quantization, and pruning -- across three different classes of software engineering tasks: vulnerability detection, code summarization, and code search. Our findings reveal that the impact of these strategies varies greatly depending on the task and the specific compression method employed. Practitioners and researchers can use these insights to make informed decisions when selecting the most appropriate compression strategy, balancing both efficiency and effectiveness based on their specific needs.

• The paper shows that knowledge distillation, quantization, and pruning reduce CodeBERT’s size and latency with distinct impacts on performance.
• Quantization maintained high task effectiveness for vulnerability detection and code summarization despite some GPU inference slowdowns.
• The study highlights the need for hardware- and task-specific strategies, paving the way for optimized deployments in software engineering.

An Empirical Study on LLM Compression for Software Engineering Tasks

The paper "On the Compression of LLMs for Code: An Empirical Study on CodeBERT" focuses on examining the effects of three prominent model compression strategies—knowledge distillation, quantization, and pruning—on a specific LLM, CodeBERT, when deployed across diverse software engineering tasks. These tasks include vulnerability detection, code summarization, and code search, which represent different paradigms such as classification, code-to-text generation, and text-to-code recommendation.

Research Motivations and Goals

The rapid advancement and deployment of transformer-based LLMs in software engineering are often hampered by the high computational costs associated with their use. This paper attempts to bridge the understanding gap regarding how compression strategies optimize inference latency and memory usage while assessing any trade-offs in effectiveness across code-related tasks. The overarching aim is to provide empirical insights that can guide practitioners and researchers to balance efficiency with effectiveness when selecting compression strategies.

Methodological Overview

Using CodeBERT as a baseline, the authors fine-tuned models for vulnerability detection, code summarization, and code search, subsequently compressing these models with the aforementioned strategies. Each compression method was assessed based on its effect on the model's memory size, inference speed (in both CPU and GPU settings), and task-specific effectiveness metrics, such as Accuracy, F1 Score, and MCC for vulnerability detection; Bleu, BERTScore, and SIDE for code summarization; and MRR and its variants for code search.

Key Findings

Inference Time and Model Size:

• Knowledge Distillation consistently improved inference times and reduced model sizes across all tasks and environments. However, it incurred a noticeable negative impact on the model's effectiveness, especially in non-classification tasks.
• Quantization significantly reduced model sizes with minimal effectiveness degradation. However, it often resulted in increased inference times, particularly in GPU settings, suggesting an efficiency-effectiveness trade-off that requires careful management depending on the user's hardware configuration.
• Pruning exhibited mixed results with improvements in some specific configurations and tasks (notably CPU inference time reduction for code summarization), while generally failing to provide consistent benefits across the board.

Effectiveness:

• Quantization strategies maintained relatively strong effectiveness across tasks with the least compromise, whereas knowledge distillation and pruning led to more noticeable decreases in performance.
• The dependency of task complexity on compression outcomes was evident, with more straightforward tasks being resilient to compression impact, while complex tasks such as code summarization and search faced more substantial effectiveness degradation.

Implications and Future Directions

The paper underscores the importance of considering hardware environments and task-specific requirements when selecting a compression strategy for code LLMs. The nuanced results advocate for further research into automated selection frameworks that adaptively recommend optimal compression configurations based on the target task and execution environment.

Considering the persistence of efficiency-effectiveness trade-offs, future work could explore extending these analyses to other LLMs like CodeT5 or Codex, and to additional software engineering tasks. Furthermore, assessing energy metrics and investigating the performance of compressed models on edge devices with constrained resources might offer additional insights. The paper serves as a valuable stepping stone towards achieving more sustainable AI deployments in software engineering applications.