mHumanEval: Multilingual Code Evaluation

• mHumanEval is a multilingual benchmark that translates English docstrings into 204 languages while preserving HumanEval’s original code tasks and execution tests.
• It extends evaluation to 25 programming languages by keeping canonical solutions intact and enabling cross-lingual analysis of code generation performance.
• The benchmark supports low-resource NL2Code use cases, exemplified by Bangla-to-Python tasks, and employs execution metrics like Pass@1 for robust assessment.

mHumanEval is a massively multilingual extension of OpenAI’s HumanEval benchmark for execution-based evaluation of code generation from natural-language prompts. It preserves the original HumanEval code tasks, canonical solutions, and unit tests while replacing the English docstrings or prompts with multilingual versions, thereby probing cross-lingual generalization under controlled task semantics (Raihan et al., 2024). In subsequent work, the benchmark has also been used in a narrower low-resource setting—most notably Bangla-to-Python NL2Code for BLP-2025 Task 2—where Bengali instructions paired with Python assertions serve as the central evaluation substrate for iterative agentic code generation (Islam et al., 27 Nov 2025).

1. Definition and lineage

Within the HumanEval family, “mHumanEval” refers to multilingual HumanEval-style benchmarking that moves beyond the original English-to-Python setting by varying the language of the prompt and, in some benchmark families, the target programming language as well (Peng et al., 2024). The benchmark specifically titled mHumanEval was introduced to address three limitations of established code-generation benchmarks: limited task and test diversity, narrow programming-language coverage, and overwhelmingly English-only natural-language prompts (Raihan et al., 2024).

HumanEval and MBPP had been foundational for code LLM evaluation, but both were English-centric and primarily Python-only. HumanEval comprises 164 Python tasks, each with only three unit tests, and MBPP has 974 tasks with similarly sparse tests. Subsequent analyses such as HumanEval+ and MBPP+ showed that performance drops when test suites are expanded to improve node and edge coverage, indicating that small original test suites can overestimate correctness (Raihan et al., 2024). mHumanEval does not solve the coverage problem by adding tests; rather, its primary innovation is breadth of natural-language coverage while keeping the HumanEval execution semantics intact.

This design is significant because most LLMs are trained predominantly on English-centric corpora and underperform on lower-resourced languages. mHumanEval therefore targets actual accessibility and fairness of AI coding assistants by evaluating code generation from prompts written in the world’s diverse languages (Raihan et al., 2024).

2. Dataset architecture and construction

mHumanEval adopts the 164 HumanEval problems and preserves function signatures, canonical solutions, and unit tests; only the natural-language docstrings or prompts are translated (Raihan et al., 2024). Each original docstring, enclosed in triple quotes in the source benchmark, is manually extracted and translated, while function signatures and examples remain otherwise unchanged. The benchmark provides 164 prompts per language for 204 languages, yielding 33,456 prompt instances for Python alone. Across programming languages, it covers 25 programming languages and produces a maximal set of 836,400 prompt–language instances when all NL–PL combinations are considered (Raihan et al., 2024).

The benchmark also extends programming-language coverage beyond prior multi-PL HumanEval derivatives by including MATLAB, Visual Basic, Fortran, and COBOL, with handwritten canonical solutions that pass all tests (Raihan et al., 2024). Its principal reported evaluations focus on Python, although representative subsets are also tested for Java, C++, JavaScript, and Ruby.

To balance machine and human translation quality, the benchmark includes an expert-translated subset, mHumanEval-Expert, for 15 natural languages spanning all six resource classes defined following Joshi et al. (2020). The classes represented are: Class 5 with English, Spanish, French, Japanese, Arabic, Chinese; Class 4 with Portuguese, Italian, Korean, Hindi; Class 3 with Bangla; Class 2 with Swahili and Zulu; Class 1 with Telugu; and Class 0 with Sinhala (Raihan et al., 2024).

Construction uses three translation systems: GPT-4o, Meta’s NLLB, and Google Translate for the languages it supports. For each prompt–language pair, the pipeline generates multiple candidate translations—13 per prompt overall—and selects the candidate maximizing the mean of round-trip BERTScore and CometKiwi when both are available, or BERTScore alone otherwise (Raihan et al., 2024). Human and machine translations are reported as comparable across the 15 expert languages, with BERTScore differences within $\pm 0.02$ and CometKiwi differences within $\pm 0.03$. The benchmark also provides subset views such as mHumanEval-{NL}, mHumanEval-mini, mHumanEval-T500, mHumanEval-R500, mHumanEval-B500, and mHumanEval-{PL}, together with metadata including Flores-200 identifiers, resource-class labels, and translation-quality scores (Raihan et al., 2024).

3. Evaluation semantics and protocol

mHumanEval is an execution-based benchmark. Models are prompted with multilingual docstrings and asked to generate code blocks, and correctness is determined by running the candidate solution against the original HumanEval unit tests in a sandboxed environment (Raihan et al., 2024). In the main mHumanEval study, the primary metric is Pass@1, computed by executing generated code via Python’s subprocess in a local sandbox.

The standard HumanEval-style estimator reported in the literature is

$$\text{pass@}k = 1 - \frac{\binom{N-g}{k}}{\binom{N}{k}},$$

where $N$ is the number of samples per task and $g$ is the number of correct generations that pass all unit tests (Raihan et al., 2024). Although the benchmark supports this broader framing, the main reported experiments focus on Pass@1 for cost reasons.

The benchmark keeps HumanEval’s original tests unchanged, which preserves comparability to prior work but also means that mHumanEval inherits the original benchmark’s limited per-task test coverage of three tests (Raihan et al., 2024). This is central to interpreting results: multilingual performance differences are exposed cleanly because task semantics and test oracles stay fixed, but full functional robustness is still bounded by the underlying test suite.

The reference evaluation setup for code generation uses standard prompting to request a code block, recommended API hyperparameters of max_tokens=1000 and temperature=0.7, and simple regex extraction of the returned code block (Raihan et al., 2024). This minimal harness is intended to isolate multilingual prompt understanding rather than benchmark-specific prompt engineering.

4. Cross-lingual performance patterns

The benchmark study evaluates GPT-4o, Claude-3.5-Opus, GPT-3.5, DeepSeek-Coder-V2, WizardCoder-33B, and Aya-33B (Raihan et al., 2024). For Python across all 204 languages, mean Pass@1 is highest and most stable for Claude-3.5 and GPT-4o, which maintain strong performance even on lower-resource languages. GPT-3.5 and DeepSeek show marked degradation as language resource level decreases. WizardCoder performs well in English relative to its multilingual behavior but collapses on non-English prompts, while Aya is relatively consistent across resource classes but lags in English Pass@1 relative to code-specialized models (Raihan et al., 2024).

Averaged over all 204 languages, the reported mean Pass@1 values are:

• GPT-4o: Python 0.738, Java 0.650, C++ 0.652, JavaScript 0.477, Ruby 0.480
• Claude-3.5: Python 0.739, Java 0.651, C++ 0.649, JavaScript 0.483, Ruby 0.477
• GPT-3.5: Python 0.360, Java 0.270, C++ 0.270, JavaScript 0.099, Ruby 0.103
• DeepSeek-Coder: Python 0.229, Java 0.139, C++ 0.136, JavaScript 0.000, Ruby 0.000
• WizardCoder: Python 0.098, Java 0.009, C++ 0.007, JavaScript 0.000, Ruby 0.000
• Aya: Python 0.445, Java 0.355, C++ 0.356, JavaScript 0.186, Ruby 0.183 (Raihan et al., 2024)

Across resource classes, performance generally declines monotonically from Class 5 to Class 0, with the steepest declines for code-specialized but monolingual-tuned models. JavaScript and Ruby are consistently more challenging across classes, often near zero for some code-tuned models in lower-resource settings. A notable asymmetry is that DeepSeek-Coder’s strongest pairing is Chinese–Python rather than English–Python, reflecting its training data (Raihan et al., 2024).

Error analysis identifies several multilingual-specific failure modes. These include misinterpretation of translated requirements, identifier or keyword leakage into the target language, off-language code generation by code-tuned models when prompted in non-English, and difficulties with non-Latin scripts such as Arabic, Devanagari, Khmer, and Myanmar that compound tokenization problems (Raihan et al., 2024). The benchmark therefore exposes not only code-synthesis limitations but also failures in multilingual semantic grounding.

5. Bangla-centered usage in BLP-2025 Task 2

In the context of Bangla-to-Python code generation, mHumanEval is used as a multilingual adaptation of HumanEval tailored to low-resource NL2Code, specifically Bangla-to-Python evaluation for BLP-2025 Task 2 (Islam et al., 27 Nov 2025). In this setting, the benchmark preserves the core HumanEval philosophy of functional correctness via unit tests while shifting the prompt language to Bengali prose. Prompts may include an “Example” line indicating the target function name and parameters, and the decisive supervision signal is the associated Python assertion set rather than explicit docstrings.

The Bangla slice emphasizes algorithmic coding tasks, control structures, string and list processing, data manipulation, and function design. Representative tasks cited in the study include is_palindrome(s), reverse_words(string), and opposite_Signs(n1, n2), with unit tests such as assert is_palindrome("TENET") == True, assert reverse_words("hello world") == "world hello", and assert opposite_Signs(1,-2) == True (Islam et al., 27 Nov 2025). The study highlights Bangla’s morphological richness and occasional orthographic or encoding irregularities as factors that increase ambiguity and complicate precise logical interpretation and edge-case handling.

The benchmark’s pass/fail structure is exploited directly by BanglaCodeAct, a ReAct-style CodeAct system built around a Thought–Code–Observation loop. The agent plans in Bangla within a <thought> section, generates Python inside <code>, executes the code in a sandboxed Python REPL with a fixed timeout of 5 seconds, observes errors or failing assertions, and iteratively self-corrects until all tests pass or the maximum iteration or timeout is reached (Islam et al., 27 Nov 2025). The implementation uses vLLM with tensor parallelism and prefix caching. It also introduces a safe_run retry handler that restarts reasoning when the agent returns invalid or empty outputs, with Retries = 25.

The reported decoding and sampling configuration includes Max tokens: 8192, Temperature: 0.7, Top-p: 0.9, Best-of: 1, Repetition penalty: 1.05 (with chain-of-thought), Decoding: Self-consistency (n=5), Num paths: 16 / 5 (SC), Seed: 42, Timeout: 5 seconds, and Retries: 25 (Islam et al., 27 Nov 2025). No Bangla-specific pre- or post-processing pipeline is reported; the method relies on multilingual prompting and iterative code refinement rather than normalization or translation-quality filtering.

On mHumanEval, BanglaCodeAct with Qwen3-8B achieves pass@1 = 94.0% on the development set and pass@1 = 71.6% on the blind test set, ranking 8th on dev and 17th on test in the shared task (Islam et al., 27 Nov 2025). The same study reports development-set baselines on mHumanEval for Qwen3-8B—Self-Consistency 88.0, Majority Voting 66.0, Few-Shot 46.0, and Zero-Shot 36.0—as well as Qwen2.5-Coder-7B, Llama-3.1-8B, DeepSeek-Coder-V2-Lite, and TigerLLM-1B-it. Common failure modes include ambiguity in Bangla instructions, type and edge-case errors, and non-convergence within iteration caps for semantically complex tasks. Qualitative examples such as remove_Occ and sort_matrix show that failing assertions can drive the agent from incomplete logic toward more fully correct implementations (Islam et al., 27 Nov 2025).

6. Limitations, related benchmarks, and research significance

mHumanEval inherits HumanEval’s limited test coverage and does not add new tests, so benchmark scores may still overestimate correctness relative to richer suites such as HumanEval+ or MBPP+ (Raihan et al., 2024). For most languages, prompts rely on machine translation, and translation artifacts or scoring limitations can affect prompt fidelity, especially because CometKiwi supports only 100 languages. The reported experiments focus on Pass@1, and broader pass@k evaluations remain more expensive. Although the dataset supports 25 programming languages, the main reported evaluations are concentrated on Python (Raihan et al., 2024).

The Bangla-centered usage surfaces additional limitations. The BLP-2025 study does not experiment with models at or above 32B parameters because of GPU limits, depends on high-quality unit tests as a feedback oracle, and continues to face challenges in Bangla semantic disambiguation (Islam et al., 27 Nov 2025). Future directions proposed there include larger multilingual models, improved test-case design, and strategies for clarifying ambiguous Bangla instructions, potentially through retrieval augmentation or specialized Bangla reasoning prompts.

In the broader multilingual code-generation landscape, HumanEval-XL provides an adjacent benchmark family: 23 natural languages, 12 programming languages, 22,080 prompts, and an average of 8.33 test cases per problem, with parallel prompts across NLs and PLs and back-translation filtering using BERTScore $\ge 0.95$ (Peng et al., 2024). HumanEval-XL complements HumanEval, MBPP, MultiPL-E, HumanEval-X, and ODEX by adding a rigorously parallel natural-language axis. mHumanEval differs primarily in scale of prompt-language coverage—204 natural languages, with 15 expert human translations—and in its explicit emphasis on mid- and low-resource languages (Raihan et al., 2024).

The benchmark’s research significance lies in making cross-lingual NL-to-code evaluation comparable, execution-based, and large-scale. By holding tasks, canonical solutions, and tests fixed while varying prompt language, it exposes multilingual weaknesses that are obscured by English-only evaluation. In the Bangla setting, the same design also supports iterative agentic correction, because Python assertions serve as an immediate oracle for refinement (Islam et al., 27 Nov 2025). This suggests that mHumanEval functions both as a benchmark for multilingual robustness and as an enabling substrate for methods that exploit execution feedback in low-resource code generation.