---
title: HumanEval-X JavaScript Dataset
url: https://www.emergentmind.com/topics/humaneval-x-javascript-dataset
type: topic
---

# HumanEval-X JavaScript Dataset

The HumanEval-X JavaScript dataset is a hand-curated benchmark suite for evaluating code generation models on JavaScript programming tasks. Derived from the original Python-based HumanEval, HumanEval-X enables the rigorous comparison of large language models (LLMs) across a unified suite of real-world tasks using language-idiomatic prompts, canonical solutions, and executable test harnesses. The dataset plays a central role in both monolingual and multilingual code generation research, supporting direct cross-model and cross-language evaluation under standard experimental protocols.

## 1. Dataset Structure and Content

The HumanEval-X JavaScript subset consists of 164 independent programming problems, each with a unique problem identifier. Every problem comprises four key elements: a natural-language task description, a JavaScript function skeleton specifying argument and return conventions (implied via the signature and JSDoc-style comments), an array of 4–10 unit tests, and an independently validated reference implementation ("canonical_solution"). Neither the original HumanEval nor subsequent HumanEval-X work provides a difficulty stratification; all 164 problems are treated as a single undivided held-out test set [2509.24828][2303.17568].

Each JavaScript item encodes inputs and outputs in both natural language (via docstring examples) and concrete test harnesses. The test harnesses utilize Node.js, importing the candidate solution and asserting correctness over the complete set of test cases. No interactive or custom parsing is employed: every problem is formulated as a pure function invocation, deterministically validated against provided examples.

| Dataset Aspect        | Details                                              | Source Papers                 |
|----------------------|------------------------------------------------------|-------------------------------|
| Number of problems   | 164                                                  | [2303.17568][2509.24828]      |
| Test cases/problem   | 4–10 (mirrors HumanEval Python's style)              | [2509.24828][2303.17568]      |
| Prompt format        | NL description, function stub, docstring, test suite | [2303.17568][2509.24828]      |
| Canonical solution   | Hand-written, tested, reviewed                       | [2303.17568]                  |

## 2. Dataset Construction and Methodology

The HumanEval-X JavaScript suite was constructed by hand-translating all 164 Python problems from HumanEval into JavaScript. Two independent developers authored JavaScript drafts, followed by a third-stage review focusing on semantic fidelity, idiomatic JavaScript usage, and robust test coverage. Problems follow ES6 conventions, including use of `let` and `const`, as well as modular exports for test integration. Prompts and docstrings employ CamelCase and JSDoc commenting styles. Documentary examples accompany each function, encoding at least one illustrative I/O mapping per task [2303.17568].

Canonical solutions are reviewed for language-specific corner cases, such as integer-binary conversion, floating-point precision, and other JavaScript idiosyncrasies. All test suites are executable and export test arrays using standard Node.js assertion libraries, supporting reliable, automated batch evaluation.

## 3. Evaluation Protocol and Metrics

Strict and execution-based evaluation protocols are core to HumanEval-X-JS usage. The principal metric, "strict accuracy," is defined as the fraction of problems for which a single generated solution passes all associated unit tests:

\[
\mathrm{StrictAccuracy} = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\left(\text{solution}_i \text{ passes all tests}\right)
\]

where $N = 164$ is the total number of problems and $\mathbf{1}(\cdot)$ is the indicator function [2509.24828]. Strict accuracy requires solutions to be entirely correct with respect to all test assertions for a given task.

Benchmarking protocols often generate multiple candidate solutions per prompt; alternative metrics such as pass@$k$ (i.e., the empirical probability that at least one of $k$ generations passes all tests) are rigorously defined as:

\[
\mathrm{pass}@k = \mathbb{E}\left[1 - \frac{{n-c \choose k}}{{n \choose k}}\right]\,, \quad k \in \{1,10,100\},\: n=200
\]

where $c$ is the number of passing completions and the expectation is taken over all problems [2303.17568]. However, in some studies, including manual evaluation settings (e.g., SAP Joule), only single-sample strict accuracy is reported due to practical constraints in generation throughput [2509.24828].

Other discussed but unused metrics include BLEU, CodeBLEU, ROUGE, and METEOR for text similarity, and test-case average or $n@k$ for partial correctness.

## 4. Prompt, Test Suite, and Implementation Format

Each JavaScript problem provides a model-ready prompt—the concatenation of a JSDoc comment block and a function stub—alongside a canonical hand-written solution and a standalone Node.js-compatible test script. Test scripts import the candidate function/module, instantiate the declared test cases, and validate success via strict equality (`assert.strictEqual` or `assert.deepStrictEqual`). All prompts, solutions, and tests are distributed as UTF-8 plain text files; the complete suite resides within the open-source HumanEval-X repository (`data/HumanEval-X/javascript/`) [2303.17568].

For benchmarking studies, a standardized prompt modification ("Use JavaScript.") may be prepended to the natural-language description to avoid inadvertent language switching in multilingual models, but no changes are made to the unit test logic itself [2509.24828].

Sample structure (abridged from [2303.17568]):

```javascript
/** 
 * add two numbers and return the sum.
 * Example:
 *   add(1, 2) → 3
 *   add(-5, 5) → 0
 */
function add(a, b) {
  // implementation
}
```
Test harness excerpt:
```javascript
const assert = require("assert");
const { add } = require("./solutions_JavaScript");
const testCases = [
  { in: [1, 2], out: 3 },
  { in: [-5, 5], out: 0 },
  { in: [0, 0], out: 0 }
];
testCases.forEach(({in: args, out: expected}) => {
  assert.strictEqual(add(...args), expected);
});
```

## 5. Access, Distribution, and Licensing

The entire HumanEval-X JavaScript dataset, with canonical solutions and test harnesses, is open-sourced and publicly accessible. The main repository (https://github.com/THUDM/CodeGeeX) provides JSON-indexed prompts, solutions, and test scripts for all 164 JavaScript problems. All files are distributed under an Apache-2.0 license, granting unrestricted research use, modification, and redistribution [2303.17568].

Integration and usage are straightforward: scripts ingest the prompt and candidate solution, run test harnesses under Node.js, and compute strict accuracy or pass@$k$ against the standard test set. Official Docker images support out-of-the-box benchmarking.

## 6. Benchmarking Results and Comparative Analysis

The HumanEval-X JavaScript dataset is widely adopted for empirical comparison of code-generation models. Aggregate results across 30 LLMs in recent evaluations reveal a mean strict accuracy of 60.6% (std: 19%). Top-performing proprietary models, including Claude 3.5 Sonnet and GPT-4o, exceed 85% strict accuracy. SAP Joule achieves a strict accuracy of 80.49%, ranking fifth overall; the leading open-source model (Qwen 2.5-32B) attains 81.71%. No results are reported as a function of problem difficulty, as neither the original nor derivative benchmarks annotate tasks by tier [2509.24828].

A plausible implication is that the dataset's spectrum—from basic arithmetic to more algorithmically involved routines—offers a standardized testbed for both general-purpose and specialized code LLMs, facilitating meaningful head-to-head assessment without confounding from language translation artifacts or non-executable test formats.

## 7. Relation to Multilingual and Extended HumanEval Variants

HumanEval-X-JS is a central reference point for multilingual and natural language generalization studies. "HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization" [2402.16694] adapts a variant with 80 JS problems, each translated into 23 natural languages for prompt diversity, but strictly parallel in prompt/test schema. While smaller in scale, this multilingual JS set follows the same single-function, docstring, and test case conventions as HumanEval-X, and adopts identical accuracy metrics (pass@1 per prompt). Evaluation reveals that LLMs generally achieve lower pass@1 performance on JavaScript compared to Python but similar to Java and C#, affirming the alignment of HumanEval-X-JS with broader program synthesis evaluation standards.

## References

- [2303.17568] "CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X"
- [2509.24828] "Evaluating SAP Joule for Code Generation"
- [2402.16694] "HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization"

Source: https://www.emergentmind.com/topics/humaneval-x-javascript-dataset