LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

Abstract: LLMs applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry. However, as new and improved LLMs are developed, existing evaluation benchmarks (e.g., HumanEval, MBPP) are no longer sufficient for assessing their capabilities. In this work, we propose LiveCodeBench, a comprehensive and contamination-free evaluation of LLMs for code, which continuously collects new problems over time from contests across three competition platforms, namely LeetCode, AtCoder, and CodeForces. Notably, our benchmark also focuses on a broader range of code related capabilities, such as self-repair, code execution, and test output prediction, beyond just code generation. Currently, LiveCodeBench hosts four hundred high-quality coding problems that were published between May 2023 and May 2024. We have evaluated 18 base LLMs and 34 instruction-tuned LLMs on LiveCodeBench. We present empirical findings on contamination, holistic performance comparisons, potential overfitting in existing benchmarks as well as individual model comparisons. We will release all prompts and model completions for further community analysis, along with a general toolkit for adding new scenarios and model

• The paper introduces LiveCodeBench, a comprehensive benchmark that uses new coding challenges to prevent evaluation contamination.
• The methodology involves temporal segmentation and holistic testing across tasks like code generation, self-repair, code execution, and test output prediction.
• Empirical findings reveal that while open-source models improve with instruction tuning, they still lag behind models like GPT-4 due to contamination effects.

LiveCodeBench: Holistic and Contamination Free Evaluation of LLMs for Code

Introduction

The utilization of LLMs in code generation and related areas has seen a significant rise. Existing benchmarks, such as HumanEval and MBPP, fall short in evaluating the diverse capabilities of modern LLMs due to their limited scope and susceptibility to contamination. "LiveCodeBench: Holistic and Contamination Free Evaluation of LLMs for Code" introduces a comprehensive benchmark, LiveCodeBench, to address these shortcomings by continuously incorporating new coding challenges from platforms like LeetCode, AtCoder, and CodeForces, ensuring a contamination-free evaluation.

Benchmark Design and Implementation

Contamination-Free Evaluation

LiveCodeBench systematically avoids data contamination by evaluating models solely on problems released after their respective training data cutoff dates. This temporal segmentation is pivotal in ensuring that model evaluations are not biased by prior exposure to benchmark problems, which is evident in the performance discrepancies of the DeepSeek-Instruct model when tested on pre- and post-September 2023 problems (Figure 1).

Figure 1: LiveCodeBench effectively detects performance drops in models, like DeepSeek-Instruct, post their release dates, indicating possible contamination in earlier tasks.

Holistic Evaluation Framework

LiveCodeBench extends beyond traditional code generation benchmarks by including additional scenarios that reflect the multifaceted nature of programming:

• Code Generation: Standard task of generating code from natural language descriptions.
• Self-Repair: Evaluates a model's ability to debug and fix incorrect code based on execution errors.
• Code Execution: Assesses comprehension through executing code snippets.
• Test Output Prediction: Challenges the model's reasoning ability by generating expected outputs without providing the function's implementation.

These scenarios, visualized in a radial plot, highlight the variable performance of models across different tasks (Figure 2).

Figure 2: Evaluation of LLMs across diverse coding-related scenarios, underscoring performance variability across tasks.

Quality and Diversity of Problems

LiveCodeBench hosts nearly 400 problems sourced over the span of several months, ensuring a balanced difficulty distribution and robust evaluation. The benchmark capitalizes on high-quality problem filters derived from competitive programming platforms, thereby delivering reliable assessment metrics for LLMs.

Empirical Findings

Detecting Contamination

Empirical analysis reveals significant contamination effects in models, such as DeepSeek, which exhibit stark performance reductions on problems post their release dates, affirming LiveCodeBench's efficacy in identifying contaminated evaluations.

Holistic Performance Insights

Overall, LiveCodeBench results demonstrate that open-source models, despite recent optimizations, lag behind closed-access counterparts like GPT-4 and Claude-3-Opus in diversified settings. Notably, instruction-tuned variants show marked improvements over base models, stressing the importance of fine-tuning datasets in enhancing LLM performance.

Comparison with HumanEval+ reveals potential overfitting, particularly in instruction-tuned models that perform well on isolated problems but falter on broader, varied tasks like those in LiveCodeBench (Figure 3).

Figure 3: Scatter plot contrasts model performance on HumanEval+ and LCB-Easy, indicating possible overfitting in open-access models.

Conclusion

LiveCodeBench sets a new standard for evaluating code-focused LLMs through its dynamic and contamination-free framework. By leveraging real-world coding scenarios across multiple platforms, the benchmark provides invaluable insights into both the strengths and limitations of contemporary LLMs, fostering advancements in the development and fine-tuning of these models. The holistic evaluation not only facilitates fair model comparisons but also guides future research directions in refining code generation capabilities.