Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages

Abstract: LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating LLMs on code-generation tasks. By curating competitive programming problems, constantly adding fresh problems to the set, and filtering them by release dates, LCB provides contamination-aware evaluation and offers a holistic view of coding capability. However, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering. We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python. Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages while preserving LCB's contamination controls and evaluation protocol. Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment of cross-language code generation competence and requiring models to sustain performance well beyond Python. We evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence of Python overfitting, language-specific contamination, and substantial disparities in multilingual performance. Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities.

• The paper extends LiveCodeBench to 12 languages, ensuring contamination-controlled and unified evaluation of LLMs in code generation.
• It employs a language-agnostic STDIN/STDOUT format and automated test conversion to maintain task fidelity across competitive programming challenges.
• Experimental results reveal significant cross-language disparities, highlighting challenges of non-Python code generation and overfitting in LLM pretraining.

Multi-LCB: Contamination-Controlled Multilingual Code Generation Benchmark for LLMs

Motivation and Benchmark Extension

LiveCodeBench (LCB) quickly established itself as a rigorous benchmark for code generation, emphasizing contamination-aware evaluation by collecting competitive programming problems with release-date filtering. However, its scope was strictly Pythonic, severely limiting conclusions about the cross-lingual coding generalization of LLMs. This is a significant deficiency in real-world deployment, where multi-language competency is required. Multi-LCB fills this gap by extending LCB to 12 programming languages—C++, C#, Python, Java, Rust, Go, TypeScript, JavaScript, Ruby, PHP, Kotlin, Scala—without loss of task fidelity or contamination-control. Each task is semantically preserved across all languages, leveraging unified STDIN/STDOUT protocols and automated test conversion, with close adherence to the original competitive environment.

Figure 1: Multi-LCB pipeline: Python tasks are converted and wrapped for multi-language code generation, with platform-specific test handling and Pass@1 evaluation.

Technical Construction

Multi-LCB retains LCB’s live-update contamination controls and evaluation protocol. All tasks are converted to a language-agnostic STDIN/STDOUT format. For platforms like LeetCode that employ functional signatures, an automated pipeline parses examples and test cases and reformats them for direct input/output evaluation. This unification enables fair, language-robust assessment, reduces the demand for brittle language-specific wrappers, and facilitates prompt engineering agnostic of language idiosyncrasies.

Prompting adheres to zero-shot criteria, using language-specific system headers and explicit code-block delimiters to ensure compliance and parsing consistency across language backends. All generated code is executed within a deterministic, resource-constrained container per language runtime, with correctness marked via Pass@1 (all hidden and public tests passed, no runtime exception or timeout).

Experimental Evaluation

Twenty-four recent LLMs (7B–685B parameters) spanning diverse pretraining paradigms were evaluated on Multi-LCB’s latest dataset. Performance is measured using Pass@1 across all languages, with tasks strictly filtered by post-training release date to enforce zero contamination.

Key results:

• There are considerable cross-language disparities. Python dominance is consistent, with top models losing up to 20 absolute Pass@1 points in statically typed languages (notably Scala, Rust, and Go).
• Python Pass@1 is not a reliable proxy for multi-language proficiency. For example, GPT-OSS-120B*, with 69.9% Pass@1 in Go, outperforms Qwen3-235B-A22B-Thinking-2507*, which dominates Python but not Go, JS, or Rust.
• Strong evidence of both Python overfitting and language-specific training contamination remains. Pre-cutoff tasks yield inflated scores; strict evaluation windows reveal significantly lower and more generalizable results.
• Platform and difficulty stratified evaluations indicate that model robustness degrades consistently with problem complexity, and compiled/static languages introduce substantial failure modes (type errors, compilation errors, and parsing exceptions).

  Figure 2: Pass@1 performance heatmap across Python, C++, Java, C#, stratified by difficulty and model.

  

  Figure 3: Pass@1 performance for Ruby, PHP, Kotlin, JavaScript, demonstrating mid-tier model performance and language-specific difficulty.

  

  Figure 4: Pass@1 performance for TypeScript, Go, Rust, Scala; lowest model scores are routinely observed for Rust and Scala.

Monthly trend analysis shows a systematic decline in performance on new tasks, further supporting the necessity of contamination-aware live benchmarks.

Error Mode and Language Analysis

Detailed error breakdowns reveal that:

• Wrong answers are the dominant failure mode in all languages, but compilation and type failures increase dramatically in C++, Java, Rust, Go, and Scala.
• Models pretrained predominantly on Python code struggle with parsing and type inference in other languages, leading to runtime failures and inefficient solutions (notably timeouts in Rust and Java).
• Format compliance issues contribute to loss in strict STDIN/STDOUT regimes, especially among lower-parameter models.

Figure 5: Monthly Pass@1 trends for Python across models, illustrating the performance decline on newer, post-cutoff tasks.

Implications, Limitations, and Theoretical Consequences

Multi-LCB exposes critical robustness failures in current LLMs regarding code generalization. Large-scale code models exhibit a persistent overfitting to Python, and neither scale nor chain-of-thought augmentation alone closes the generalization gap in strongly typed or less-represented language niches. This has significant implications:

• Industrial deployment of LLMs for code is limited in safety-critical or legacy ecosystems (C++, Rust, Java).
• Current evaluation pipelines overstate practical utility if they rely solely on Python or do not control for contamination.
• Model improvement requires explicit non-Python language coverage during pretraining and sophisticated prompt/architecture adaptation for parsing, type-safety, and compilation compliance.

Multi-LCB also strongly motivates further benchmark expansion (e.g., to Swift, Haskell, R, Julia), as its modular framework can integrate additional languages with minimal engineering.

Directions for Future Work

Future research should focus on:

• Enhanced multilingual pretraining objectives targeting low-resource languages and paradigmatic diversity.
• Generation-time format validation and pre-compilation verification steps, to reduce losses from non-algorithmic errors.
• Broader evaluation on real-world, multi-language, and multi-module software engineering tasks beyond the competitive programming regime.

Additionally, comparison with proprietary models (GPT-4, Gemini, Claude) would contextualize the current open model leaderboard.

Conclusion

Multi-LCB directly addresses the core limitation of LCB’s Python exclusivity through rigorous, contamination-minimized multilingual benchmarking. Pass@1 evaluation across 12 languages and 24 models shows that cross-language code generation remains an open problem, with large, persistent gaps driven by syntax complexity, typing discipline, and training corpus bias. Multi-LCB provides an essential foundation for tracking and stimulating research in robust, language-agnostic code generation, and its public infrastructure will accelerate progress toward universal program synthesis.

Reference:

"Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages" (2606.20517)