CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings (2501.01257v1)

Abstract: With the increasing code reasoning capabilities of existing LLMs and breakthroughs in reasoning models like OpenAI o1 and o3, there is a growing need to develop more challenging and comprehensive benchmarks that effectively test their sophisticated competition-level coding abilities. Existing benchmarks, like LiveCodeBench and USACO, fall short due to the unavailability of private test cases, lack of support for special judges, and misaligned execution environments. To bridge this gap, we introduce CodeElo, a standardized competition-level code generation benchmark that effectively addresses all these challenges for the first time. CodeElo benchmark is mainly based on the official CodeForces platform and tries to align with the platform as much as possible. We compile the recent six months of contest problems on CodeForces with detailed information such as contest divisions, problem difficulty ratings, and problem algorithm tags. We introduce a unique judging method in which problems are submitted directly to the platform and develop a reliable Elo rating calculation system that aligns with the platform and is comparable with human participants but has lower variance. By testing on our CodeElo, we provide the Elo ratings of 30 existing popular open-source and 3 proprietary LLMs for the first time. The results show that o1-mini and QwQ-32B-Preview stand out significantly, achieving Elo ratings of 1578 and 1261, respectively, while other models struggle even with the easiest problems, placing in the lowest 20 percent among all human participants. Detailed analysis experiments are also conducted to provide insights into performance across algorithms and comparisons between using C++ and Python, which can suggest directions for future studies.

Evaluation of Competition-Level Code Generation with CodeForces Benchmark

The paper introduces CodeForces, a specialized benchmark for evaluating the reasoning capabilities of LLMs in competition-level code generation. Developed using the CodeForces platform, it offers an advanced evaluation framework that challenges LLMs with coding problems under parameters akin to human assessments, incorporating standardized Elo ratings. This paper critiques existing benchmarks and positions CodeForces as a solution addressing their limitations through a zero false-positive evaluation methodology, support for special judges, and a precisely aligned execution environment.

CodeForces extends the landscape of code generation assessment by leveraging the competitive environment of CodeForces. Unlike previous benchmarks, it establishes a direct submission model where solutions from LLMs are submitted via a bot to the platform, ensuring an entirely genuine judging process. The evaluation utilizes the comprehensive range of problems categorized by contest divisions, difficulty ratings, and algorithm tags, providing a detailed analysis of models’ performance across various dimensions.

Key Insights and Results

1. Unique Benchmarking Methodology: CodeForces' zero false-positive claim is attributed to its innovative evaluation approach of submitting model-generated solutions directly to the CodeForces platform. This process guarantees the authenticity and objectivity of test results. Additionally, supporting special judges enables the benchmark to handle problems without unique solutions, thus offering an assessment closer to human competitions.
2. Elo Rating System: The benchmark provides a standardized Elo rating system aligned with CodeForces' system. This comparative metric allows for a significant assessment of LLMs against human competitors, facilitating a unique perspective on models’ abilities relative to human participants.
3. Model Assessment and Trends: The evaluation of 34 diverse LLMs, including models like OpenAI's o1-mini and QwQ-32B-Preview, revealed a promising performance in reasoning capabilities. The benchmark notes that most models gauge poorly on lower-div complex problems, with the o1-mini model achieving a notable Elo rating of 1578 - surpassing a safety margin reflective of 90% of human participants. The results underscore the importance of models with enhanced chain-of-thought reasoning.
4. Implication of Programming Languages: The outcome of the paper also shed light on language preferences, contrasting typical model inclinations towards Python with superior performance noted when using C++. This insight highlights how execution efficiency fundamentally impacts competitive coding and reveals LLMs’ potential bias in language selection, advocating for training nuances favoring languages like C++ in time-critical contexts.

Implications for Future Research

The introduction of the CodeForces benchmark has profound implications for both practical and theoretical advancements in natural language processing and AI-driven coding capabilities. By focusing on direct submissions and platform-based evaluations, it opens avenues for LLM developers to exploit real-world competitive constraints in their training paradigms.

Moreover, the implications drawn from varying algorithmic performance introduce potential strategic enhancements within LLM architectures, particularly emphasizing a focus on algorithms like dynamic programming and tree traversal, where current models underperform.

Conclusion

CodeForces marks an important contribution to the NLP community, offering a structured, accurately aligned platform to test and refine the code generation abilities of LLMs. Through its comprehensive evaluation features and forward-looking insights, it sets a foundation for future explorations into capability augmentations and fair comparisons between machine and human competencies in competitive coding.