HumanEval-X Multilingual Code Generation Benchmark

• HumanEval-X is a multilingual and multi-paradigm benchmark that standardizes LLM code synthesis evaluation across various programming languages and domains.
• It employs execution-based tests and hand-authored harnesses to accurately measure strict accuracy and pass@k metrics.
• The benchmark has driven innovations in debugging frameworks and dataset design, influencing LLM evaluations and repair research.

HumanEval-X is a multilingual and multi-paradigm code generation benchmark designed to provide rigorous, execution-based evaluation of LLMs on program synthesis tasks across a spectrum of programming languages, natural languages, and domains. Originating as an extension of the Python-centric HumanEval, HumanEval-X addresses the growing need for standardized, functionally-grounded, and linguistically diverse benchmarks in LLM-based code generation research. HumanEval-X tasks have influenced and been incorporated into various recent benchmarking efforts, with significant impact on model assessment, dataset design, and cross-lingual evaluation.

1. Benchmark Motivation and Design Principles

HumanEval-X was created to overcome critical limitations of previous code generation benchmarks: heavy monolingual bias (chiefly English-to-Python), limited test coverage, and lack of cross-linguistic or cross-paradigm comparability. By generalizing the HumanEval task set to encompass other programming languages (C++, Java, JavaScript, Go) and, in further expansions, multiple natural languages, HumanEval-X enables:

• Direct cross-language comparison of functional correctness using hand-written or reliably translated test suites.
• Evaluation of LLMs’ ability to generate code across diverse syntactic and semantic paradigms in real-world languages.
• Mitigation of data contamination by templating and combinatorial problem instantiations to limit overlap with seen data (Bradbury et al., 2 Dec 2024).
• More nuanced measurement of code generation failures including logical, syntax, and runtime errors.

The construction of HumanEval-X emphasizes parallelism (one-to-one mapping between tasks across languages), semantic fidelity (manual adaptation accounting for language idioms and type systems), and execution-based evaluation rather than string similarity metrics such as BLEU or CodeBLEU (Zheng et al., 2023).

2. Core Dataset Structure and Multilingual Extensions

The canonical HumanEval-X dataset comprises 164 distinct algorithmic/functional problems, each defined by:

• A docstring-style natural language specification translated/adapted as needed.
• Language-specific function signatures.
• Canonical reference solutions and hand-authored test harnesses.

Programming languages included in early releases are Python, C++, Java, JavaScript, and Go, representing both dynamically and statically typed paradigms (Zheng et al., 2023). Subsequent benchmarks (e.g., mHumanEval, HumanEval-XL) extend the coverage to dozens of programming and over 200 natural languages through large-scale manual and high-quality machine translation pipelines (Raihan et al., 19 Oct 2024, Peng et al., 26 Feb 2024).

Prompts in each language are adapted to adhere to domain conventions: e.g., snake_case vs. camelCase, specific import styles, and idiomatic type usage. Semantic differences—such as binary formatting prefixes or language-specific rounding modes—are resolved manually to maintain task equivalence (Zheng et al., 2023).

3. Evaluation Protocols and Metrics

All HumanEval-X derivatives employ execution-based evaluation, where a candidate solution must pass all test cases to be considered correct. The principal metrics are:

• Strict Accuracy: Fraction of problems for which a single generated solution passes all associated unit tests. Used when only limited samples are available, as in proprietary model evaluations (Heisler et al., 29 Sep 2025).
• pass@k: For independent samples, the expected probability that at least one of the first $k$ solutions passes all tests:

$$\mathrm{pass}@k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

where $n$ is the number of samples and $c$ the number of correct ones (Zheng et al., 2023, Raihan et al., 19 Oct 2024, Peng et al., 26 Feb 2024).

• Multilingual Budget-Allocation pass@k: For multilingual models, $k$ solutions are split across languages; aggregate pass rates across varying budget distributions offer insight into cross-language generalization (Zheng et al., 2023).

Test harness design consistently aims for rigorous coverage, with 5–20 assertions per problem (JavaScript), 7+ per Python/C++ task, and frequent use of complex edge cases (Heisler et al., 29 Sep 2025, Zheng et al., 2023). Model solutions are evaluated in language-appropriate environments (e.g., Node.js v18+ for JavaScript).

4. HumanEval-X Variants and Adaptations

4.1 Combinatorial Instantiation

To address leakage and memorization from LLM training data, recent work proposes generating HumanEval-X-like benchmarks using parameterized templates and combinatorial test designs:

• Each problem becomes a template $T(P_1,...,P_n)$, with $P_i$ sampled from designed domains $D_i$.
• Covering arrays are constructed to ensure all $t$-way parameter interactions are represented with minimal overlap to original tasks (Bradbury et al., 2 Dec 2024).
• Strict guidelines enforce interchangeable difficulty, measured by cyclomatic complexity, time/space cost, and prompt length.
• This approach substantially reduces the risk of contamination and increases task diversity at scale.

4.2 Language and Domain Extensions

Benchmarks such as mHumanEval and HumanEval-XL scale HumanEval-X to hundreds of thousands of NL/PL combinations, supporting low-resource languages (e.g., Rundi, Zulu), legacy PLs (Fortran, COBOL), and parallel evaluation grids (Raihan et al., 19 Oct 2024, Peng et al., 26 Feb 2024). Qiskit HumanEval extends the methodology to quantum programming, introducing domain-specific test harness design and a new spectrum of functional difficulty (Vishwakarma et al., 20 Jun 2024).

5. Comparative Model Evaluation and Insights

HumanEval-X is widely adopted for evaluating proprietary and open-source LLMs, enabling head-to-head comparisons:

Model (JavaScript)	Strict Accuracy (%) (Heisler et al., 29 Sep 2025)
Claude-3.5-Sonnet-20241022	85.98
GPT-4o-2024-08-06	85.98
GPT-4-Turbo-2024-04-09	84.15
Qwen2.5-32B	81.71
SAP Joule (2024-10)	80.49

SAP Joule, though not JavaScript-specialized, ranks fifth (80.49%) among 30 models, trailing the top-performing closed models by less than six points (Heisler et al., 29 Sep 2025). Open-source models (e.g., Qwen2.5-32B) are competitive with proprietary leaders.

Key empirical findings include:

• Model scale shows only a weak correlation with strict accuracy among open-source, code-specialized LLMs ($r\approx0.23$), implying that fine-tuning and architecture choices are more decisive than parameter count (Heisler et al., 29 Sep 2025).
• "Wrong answer" logical bugs are the dominant mode of failure, except in languages with strict compilation (e.g., Go, where syntax errors are common) (Zheng et al., 2023).
• HumanEval-X reveals that cross-lingual code generalization remains significantly weaker in low-resource NL prompts and less common programming languages (Raihan et al., 19 Oct 2024, Peng et al., 26 Feb 2024).

6. Applications in LLM Debugging and Repair Research

HumanEval-X establishes itself as the de facto functional benchmark in program synthesis and LLM-driven automated debugging:

• Multi-agent repair frameworks such as SEIDR combine synthesis, testing, and iterative instruction-driven repair to close the "near-miss" syndrome gap (code nearly correct, but failing a few edge cases).
• On HumanEval-C++, SEIDR with Llama 3-8B achieves pass@100 = 84.2%; GPT-3.5 solves 163/164 problems at least once, despite being much smaller than the largest competing models (Grishina et al., 10 Mar 2025).
• Single-round, targeted bug fixes allow LLMs to leap directly from failing all tests to passing all in one step for over 90% of problems, highlighting the synergy between code generation and iterative feedback (Grishina et al., 10 Mar 2025).

7. Impact, Limitations, and Future Directions

The adoption of HumanEval-X and its derivatives has shaped the evaluation landscape for code-generating LLMs:

• HumanEval-X established the execution-based, multilingual gold standard for LLM benchmarking in code synthesis, stimulating parallel efforts in NL diversity and domain-specific extensions (e.g., quantum computing).
• Limitations include combinatorial explosion for high-dimensional templates, the labor intensity of high-fidelity manual adaptation, and persistent semantic drift as parameters diversify (Bradbury et al., 2 Dec 2024).
• For comprehensive evaluation, richer test suites—measuring coverage and employing mutation testing—are recommended to avoid overestimating functional correctness (Raihan et al., 19 Oct 2024).
• Democratizing code synthesis reliability across the world's linguistic diversity, narrowing cross-lingual generalization gaps, and robustly defending against training-contaminated evaluation remain key open research challenges (Raihan et al., 19 Oct 2024, Peng et al., 26 Feb 2024).

HumanEval-X and its broader family of benchmarks continue to inform both the empirical paper and methodological innovation in LLM-powered program synthesis.