HumanEval-X JavaScript Dataset

• The HumanEval-X JavaScript dataset is a curated benchmark suite of 164 programming tasks with idiomatic prompts and comprehensive test harnesses.
• It rigorously evaluates code generation models using metrics like strict accuracy and pass@k with executable Node.js test suites.
• It is open-sourced under the Apache-2.0 license, fostering cross-model, multilingual research and reproducible code generation benchmarks.

The HumanEval-X JavaScript dataset is a hand-curated benchmark suite for evaluating code generation models on JavaScript programming tasks. Derived from the original Python-based HumanEval, HumanEval-X enables the rigorous comparison of LLMs across a unified suite of real-world tasks using language-idiomatic prompts, canonical solutions, and executable test harnesses. The dataset plays a central role in both monolingual and multilingual code generation research, supporting direct cross-model and cross-language evaluation under standard experimental protocols.

1. Dataset Structure and Content

The HumanEval-X JavaScript subset consists of 164 independent programming problems, each with a unique problem identifier. Every problem comprises four key elements: a natural-language task description, a JavaScript function skeleton specifying argument and return conventions (implied via the signature and JSDoc-style comments), an array of 4–10 unit tests, and an independently validated reference implementation ("canonical_solution"). Neither the original HumanEval nor subsequent HumanEval-X work provides a difficulty stratification; all 164 problems are treated as a single undivided held-out test set (Heisler et al., 29 Sep 2025, Zheng et al., 2023).

Each JavaScript item encodes inputs and outputs in both natural language (via docstring examples) and concrete test harnesses. The test harnesses utilize Node.js, importing the candidate solution and asserting correctness over the complete set of test cases. No interactive or custom parsing is employed: every problem is formulated as a pure function invocation, deterministically validated against provided examples.

Dataset Aspect	Details	Source Papers
Number of problems	164	(Zheng et al., 2023, Heisler et al., 29 Sep 2025)
Test cases/problem	4–10 (mirrors HumanEval Python's style)	(Heisler et al., 29 Sep 2025, Zheng et al., 2023)
Prompt format	NL description, function stub, docstring, test suite	(Zheng et al., 2023, Heisler et al., 29 Sep 2025)
Canonical solution	Hand-written, tested, reviewed	(Zheng et al., 2023)

2. Dataset Construction and Methodology

The HumanEval-X JavaScript suite was constructed by hand-translating all 164 Python problems from HumanEval into JavaScript. Two independent developers authored JavaScript drafts, followed by a third-stage review focusing on semantic fidelity, idiomatic JavaScript usage, and robust test coverage. Problems follow ES6 conventions, including use of let and const, as well as modular exports for test integration. Prompts and docstrings employ CamelCase and JSDoc commenting styles. Documentary examples accompany each function, encoding at least one illustrative I/O mapping per task (Zheng et al., 2023).

Canonical solutions are reviewed for language-specific corner cases, such as integer-binary conversion, floating-point precision, and other JavaScript idiosyncrasies. All test suites are executable and export test arrays using standard Node.js assertion libraries, supporting reliable, automated batch evaluation.

3. Evaluation Protocol and Metrics

Strict and execution-based evaluation protocols are core to HumanEval-X-JS usage. The principal metric, "strict accuracy," is defined as the fraction of problems for which a single generated solution passes all associated unit tests:

$$\mathrm{StrictAccuracy} = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\left(\text{solution}_i \text{ passes all tests}\right)$$

where $N = 164$ is the total number of problems and $\mathbf{1}(\cdot)$ is the indicator function (Heisler et al., 29 Sep 2025). Strict accuracy requires solutions to be entirely correct with respect to all test assertions for a given task.

Benchmarking protocols often generate multiple candidate solutions per prompt; alternative metrics such as pass@$k$ (i.e., the empirical probability that at least one of $k$ generations passes all tests) are rigorously defined as:

$$\mathrm{pass}@k = \mathbb{E}\left[1 - \frac{{n-c \choose k}}{{n \choose k}}\right]\,, \quad k \in \{1,10,100\},\: n=200$$

where $c$ is the number of passing completions and the expectation is taken over all problems (Zheng et al., 2023). However, in some studies, including manual evaluation settings (e.g., SAP Joule), only single-sample strict accuracy is reported due to practical constraints in generation throughput (Heisler et al., 29 Sep 2025).

Other discussed but unused metrics include BLEU, CodeBLEU, ROUGE, and METEOR for text similarity, and test-case average or $n@k$ for partial correctness.

4. Prompt, Test Suite, and Implementation Format

Each JavaScript problem provides a model-ready prompt—the concatenation of a JSDoc comment block and a function stub—alongside a canonical hand-written solution and a standalone Node.js-compatible test script. Test scripts import the candidate function/module, instantiate the declared test cases, and validate success via strict equality (assert.strictEqual or assert.deepStrictEqual). All prompts, solutions, and tests are distributed as UTF-8 plain text files; the complete suite resides within the open-source HumanEval-X repository (data/HumanEval-X/javascript/) (Zheng et al., 2023).

For benchmarking studies, a standardized prompt modification ("Use JavaScript.") may be prepended to the natural-language description to avoid inadvertent language switching in multilingual models, but no changes are made to the unit test logic itself (Heisler et al., 29 Sep 2025).

Sample structure (abridged from (Zheng et al., 2023)):

/** * add two numbers and return the sum. * Example: * add(1, 2) → 3 * add(-5, 5) → 0 */ function add(a, b) { // implementation }

Test harness excerpt: $N = 164$0

5. Access, Distribution, and Licensing

The entire HumanEval-X JavaScript dataset, with canonical solutions and test harnesses, is open-sourced and publicly accessible. The main repository (https://github.com/THUDM/CodeGeeX) provides JSON-indexed prompts, solutions, and test scripts for all 164 JavaScript problems. All files are distributed under an Apache-2.0 license, granting unrestricted research use, modification, and redistribution (Zheng et al., 2023).

Integration and usage are straightforward: scripts ingest the prompt and candidate solution, run test harnesses under Node.js, and compute strict accuracy or pass@$k$ against the standard test set. Official Docker images support out-of-the-box benchmarking.

6. Benchmarking Results and Comparative Analysis

The HumanEval-X JavaScript dataset is widely adopted for empirical comparison of code-generation models. Aggregate results across 30 LLMs in recent evaluations reveal a mean strict accuracy of 60.6% (std: 19%). Top-performing proprietary models, including Claude 3.5 Sonnet and GPT-4o, exceed 85% strict accuracy. SAP Joule achieves a strict accuracy of 80.49%, ranking fifth overall; the leading open-source model (Qwen 2.5-32B) attains 81.71%. No results are reported as a function of problem difficulty, as neither the original nor derivative benchmarks annotate tasks by tier (Heisler et al., 29 Sep 2025).

A plausible implication is that the dataset's spectrum—from basic arithmetic to more algorithmically involved routines—offers a standardized testbed for both general-purpose and specialized code LLMs, facilitating meaningful head-to-head assessment without confounding from language translation artifacts or non-executable test formats.

7. Relation to Multilingual and Extended HumanEval Variants

HumanEval-X-JS is a central reference point for multilingual and natural language generalization studies. "HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization" (Peng et al., 2024) adapts a variant with 80 JS problems, each translated into 23 natural languages for prompt diversity, but strictly parallel in prompt/test schema. While smaller in scale, this multilingual JS set follows the same single-function, docstring, and test case conventions as HumanEval-X, and adopts identical accuracy metrics (pass@1 per prompt). Evaluation reveals that LLMs generally achieve lower pass@1 performance on JavaScript compared to Python but similar to Java and C#, affirming the alignment of HumanEval-X-JS with broader program synthesis evaluation standards.

• (Zheng et al., 2023) "CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X"
• (Heisler et al., 29 Sep 2025) "Evaluating SAP Joule for Code Generation"
• (Peng et al., 2024) "HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization"