Are Decoder-Only Large Language Models the Silver Bullet for Code Search? (2410.22240v1)

Abstract: Code search is crucial for code reuse, enabling developers to efficiently locate relevant snippets. Current methods rely on encoder-based models, which suffer from limitations such as poor generalization and restricted input lengths. Decoder-only LLMs, with their extensive pre-training, larger size, and longer input capabilities, offer potential solutions to these issues, yet their effectiveness in code search remains underexplored. To fill this gap, our study presents the first systematic exploration of decoder-only LLMs for code search. We evaluate nine state-of-the-art decoder-only models using two fine-tuning methods, two datasets (CSN and CoSQA$^+$), and three model sizes. Our findings reveal that fine-tuned CodeGemma significantly outperforms encoder-only models like UniXcoder, achieving a 5.57% improvement in MRR on CSN and a 49.6% increase in MAP on CoSQA$^+$ compared to zero-shot UniXcoder. These results highlight the superior performance and adaptability of decoder-only models. Additionally, we provide valuable insights into optimizing these models for code search, covering aspects such as model selection, fine-tuning methods, training data, and model size, and discussing their strengths and limitations.

• The paper finds that decoder-only LLMs require task-specific fine-tuning to outperform encoder-based models in code search.
• Fine-tuning yields notable gains, with models like CodeGemma improving Mean Reciprocal Rank on datasets such as CSN and CoSQA+.
• The study highlights that decoder-only architectures harness longer input sequences, underscoring the importance of specialized training techniques.

Overview of the Paper "Are Decoder-Only LLMs the Silver Bullet for Code Search?"

The paper focuses on evaluating the effectiveness of decoder-only LLMs for code search tasks, particularly in comparison to the traditionally used encoder-based models. The researchers investigate whether the pre-training and architectural advantages of decoder-only models, which allow for longer input processing and potentially better generalization, can lead to significant improvements in code search, a critical task for developers aiming to reuse code efficiently.

Contributions and Methodology

The authors conduct a systematic exploration involving nine prominent decoder-only LLMs. Their investigation centers on three main questions: the performance of these models in zero-shot settings, improvements from fine-tuning, and underlying reasons for performance variations.

1. Zero-shot Performance: The paper begins by comparing the zero-shot capabilities of decoder-only models with encoder-only models such as UniXcoder. Despite their large size and sophisticated pre-training, decoder-only models initially underperform their encoder-based counterparts in zero-shot code search tasks. This finding underscores that task-specific pre-training is vital for achieving adequate zero-shot performance.
2. Fine-tuning Improvements: Through fine-tuning, decoder-only models derive substantial gains in performance across benchmark datasets like CSN and CoSQA$^+$. Fine-tuned decoder-only models, particularly CodeGemma, demonstrate superior adaptability, achieving a 5.57% improvement in Mean Reciprocal Rank (MRR) on the CSN dataset over fine-tuned UniXcoder. This improvement is even more pronounced on the CoSQA$^+ dataset, highlighting the stronger generalization ability of decoder-only frameworks when fine-tuned properly.
3. Analysis of Improvements: The authors explore why fine-tuning enhances performance. The paper contrasts supervised contrastive learning with unsupervised methods, illustrating the superiority of the former in terms of clarity and embedding optimization. Additionally, they assess the role of dataset specificity and model size, identifying that both architecture and comprehensive, task-specific datasets significantly influence efficacy.

Results and Implications

The findings have several implications. Firstly, decoder-only models, given sufficient task-specific fine-tuning, can outperform traditional encoder-based models, thus validating their potential in handling code search tasks. However, this potential is not inherent and requires tailored training methods and data.

The research also presents evidence that the architecture of decoder-only LLMs provides advantages in handling longer queries and diverse datasets. However, challenges persist with ultra-short queries due to the curse of dimensionality and lack of contextual clarity. These limitations suggest that ongoing efforts to optimize model architectures for specific input types remain necessary.

Future Directions

This paper invites numerous future research avenues. One area of interest lies in improving zero-shot performance, potentially through hybrid architectures that blend decoder and encoder properties. Additionally, the findings encourage further exploration of specialized fine-tuning datasets and training strategies to alleviate the limitations associated with short inputs.

Overall, the paper showcases the versatility and potential decoder-only LLMs hold for code search, while also highlighting the need for continued innovation in model training and fine-tuning to leverage these strengths fully. As the capabilities and applications of LLMs expand, decoder-only architectures are poised to play a critical role in advancing software engineering workflows, facilitating more efficient code reuse and improving developer productivity.