HumanEval Pro: Advanced Code Benchmarking

• HumanEval Pro is a suite of advanced benchmarks for LLM code synthesis featuring self-invocation, project-level simulation, and multilingual evaluation.
• The evaluation protocol rigorously tests LLMs on realistic multi-file projects and interactive user scenarios, highlighting practical deployment challenges.
• Benchmark methodologies emphasize leakage-resistant design and combinatorial test variants to uncover genuine model capabilities and limitations.

HumanEval Pro is a suite of advanced benchmarking methodologies and datasets designed to rigorously evaluate LLMs on code generation tasks that substantially exceed the difficulty and scope of the original HumanEval benchmark. Developed through multiple independent research initiatives, HumanEval Pro and its extensions stress test LLMs on multi-stage reasoning, self-invocation, project-level synthesis, multilingual code generation, data leakage resilience, and domain transfer—aiming to uncover true capabilities and limitations in deployable software automation.

1. Motivation and Conceptual Advances

Classic code evaluation benchmarks such as HumanEval focus on single-function completion within Python, with unit-test-based metrics. While highly influential, these paradigms lack coverage along several critical axes:

• Compositional reasoning: Isolated functions do not test a model's ability to reuse its own code or coordinate helper logic.
• Project-scale generation: Real software requires orchestrating files, templates, routes, dependencies, and user interaction flows.
• Cross-lingual robustness: English-centric, Python-only paradigms obscure performance differentials in other languages.
• Data leakage resistance: Static hand-written benchmarks are increasingly compromised by accidental inclusion in LLM training corpora.
• Quantum and specialized domains: Emerging applications demand code generation in non-classical settings, such as quantum SDKs.

“HumanEval Pro” thus refers to a set of new benchmarks and protocols developed to address these limitations by increasing task complexity, evaluation rigor, and linguistic/domain reach (Yu et al., 2024, Liu et al., 10 Mar 2025, Bradbury et al., 2024, Raihan et al., 2024, Vishwakarma et al., 2024).

2. Methodologies for “Pro-Level” Benchmark Construction

2.1 Self-Invoking Code Generation

HumanEval Pro (as formalized in (Yu et al., 2024)) operationalizes self-invoking complexity. Each original HumanEval problem is paired with a more complex, related problem requiring explicit invocation of the base solution inside a meta-function. The three-stage pipeline is:

• For each base task, generate a new “meta” task that strictly reuses the original logic, but increases algorithmic and semantic complexity.
• Synthesize canonical solutions for both tasks; iteratively debug until all provided tests are passed.
• Formalize the test suite jointly over base and meta tasks, guaranteeing evaluation contiguity.

This design ensures that models must not only implement correct helpers, but also compose them robustly within higher-level structures, penalizing shallow pattern matching.

2.2 Project-Level and User-Centric Evaluation

ProjectEval (Liu et al., 10 Mar 2025) expands evaluation to realistic, multi-file software artifacts. Key steps include:

• Task prompts escalate from high-level NL descriptions (Level 1), to bulletized feature checklists (Level 2), to masked code skeletons (Level 3).
• Solution validation automates user interaction via Selenium (for web) or subprocess-driven CLI scripts, requiring LLMs to generate code that survives full end-to-end user simulation.
• Automated metrics (checklist, skeleton, and code similarity; parameter mapping) quantify not just pass rates but also alignment with intended structure and logic.

This “project-level” approach advances beyond toy functions to stress systematic engineering, module linking, and UI flow orchestration.

2.3 Leakage-Resistant Template Design

HumanEval_T (Bradbury et al., 2024) combats data leakage risks through template-based combinatorial test design. Each abstract template parameterizes the original task using variable placeholders (e.g., <input_type>, <threshold_descriptor>), systematically instantiated via covering arrays for t-wise interaction coverage. Benchmark variants, lexically and semantically distinct, are then sampled to ensure mutual exclusivity and comparable difficulty, resisting memorization and supporting fair comparison across LLMs.

2.4 Multilingual and Domain-Transfer Benchmarking

mHumanEval (Raihan et al., 2024) provides a massively multilingual extension—over 200 natural languages and 25 programming languages—by pairing the core HumanEval task set with automated and human-verified high-fidelity translations and code templates. Qiskit HumanEval (Vishwakarma et al., 2024) extends the testbed to quantum SDKs, with prompt/solution/test triplets validated via circuit simulation and hardware.

3. Evaluation Protocols and Metrics

All “pro” benchmarks inherit the pass@k metric from HumanEval, defined for n samples and c correct cases as:

$$\mathrm{pass}@k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

where $k$ is typically 1 (greedy decoding) or higher for sampled completions.

Distinguishing features for “pro” benchmarks:

• Complexity scaling: Problems systematically increase in length and compositional depth—HumanEval Pro tasks are empirically 30–50% longer than their HumanEval counterparts (Yu et al., 2024).
• Multi-level input modes: ProjectEval supports three input abstractions, from NL prompt to code skeleton, testing progressive reasoning depth.
• User-centric harnesses: ProjectEval requires code to pass interactive, end-to-end user simulation for full credit, rather than isolated test calls.
• Subtask similarity metrics: ProjectEval measures checklist, skeleton, and code similarity using evaluation frameworks such as CodeBLEU and sentence transformers.
• Cross-linguistic coverage: mHumanEval measures pass@1 across hundreds of source and programming languages, scoring both code correctness and translation fidelity (BERTScore, COMETKiwi).
• Variance and interchangeability: Leakage-resistant benchmarks compute the mean and standard deviation of pass@1 across template variants, ensuring task fairness (Bradbury et al., 2024).

4. Empirical Findings and Comparative Outcomes

4.1 Self-invoking and Project-Level Tasks

• On HumanEval Pro, frontier models drop 10–20 percentage points versus standard HumanEval: GPT-4o’s pass@1 drops from 90.2% to 75.0%; Deepseek-V2.5 from 90.2% to 73.8% (Yu et al., 2024).
• For ProjectEval (project-level, user-centric), even GPT-4o achieves only 13.9% pass@5, with most open-source LLMs below 2% (Liu et al., 10 Mar 2025).
• Subtask scores (GPT-4o, ProjectEval): checklist similarity ≈16.1%, skeleton ≈10.1%, code similarity (CodeBLEU) ≈36.4%, PV similarity ≈15.4% (Liu et al., 10 Mar 2025).

4.2 Multilingual and Data Leakage-Resistant Evaluation

• mHumanEval documents a robust decline in pass@1 as NL resource class decreases: GPT-4o and Claude-3.5 sustain high-resource accuracy (≈0.88) but drop to ≈0.50 on rare/low-resource NLs; GPT-3.5 and Deepseek fall further (Raihan et al., 2024).
• HumanEval_T (template-based) experiments show all major models drop 4.8%–13.8% from HumanEval to HumanEval_T in pass@1, directly attributing lift in the original to memorization. Across-template variance is low, supporting interchangeability (Bradbury et al., 2024).

4.3 Quantum Code Generation

• Qiskit-tuned LLMs (granite-8b-code-qk) improve from 28.71% to 46.53% pass@1 on Qiskit HumanEval, outperforming untuned models by +17.8 percentage points. No evaluated LLM solves the most difficult QHE tasks (e.g., BB84 key generation) (Vishwakarma et al., 2024).

4.4 Failure Modes

• HumanEval Pro: AssertionError remains the leading error (≈60%), followed by function naming errors (~15%) and type/index mismatches (~20%) (Yu et al., 2024).
• ProjectEval: Systematic engineering and project orchestration are frequent points of failure. Cascade (multi-stage) generation adds only marginal improvements.

5. Implications for Model Development and Benchmark Design

• Benchmarks emphasizing self-invoking logic and project-level simulation (as in HumanEval Pro and ProjectEval) reveal failure points masked by single-shot task leaders, redirecting focus toward robust function reuse and large-scale code planning.
• Data leakage—proven to inflate scores on static testbeds—necessitates the adoption of combinatorial-templated benchmarks like HumanEval_T for fair, longitudinal model assessment.
• Multilingual and domain-transfer capacities (mHumanEval, Qiskit HumanEval) expose marked deficits in LLMs lacking extensive cross-lingual or specialized pretraining, underscoring the need for broader, more diverse training corpora.
• Measurement of code and project similarity alongside pass@k offers a more nuanced, explainable assessment, facilitating the diagnosis of not just correctness but fidelity to structured specifications.
• Future advances may depend on integrating user-simulation primitives, explanation modules, and enhanced scaffold-generation strategies into generative agents (Liu et al., 10 Mar 2025).

6. Research Benchmarks and Dataset Overview

Benchmark	Modality	Novelty	Key Metric(s)
HumanEval Pro (Yu et al., 2024)	Python function (self-invoking)	Two-stage code reuse, composition	Pass@1
ProjectEval (Liu et al., 10 Mar 2025)	Multi-file projects	User-centric evaluation, subtask similarity	Pass@K, CodeBLEU, PV etc.
HumanEval_T (Bradbury et al., 2024)	Python function (templates)	Leakage-resistant, combinatorial variants	Pass@1, across-variant std.
mHumanEval (Raihan et al., 2024)	Polyglot (204 NL × 25 PL)	Multilingual, multi-PL, robust evaluation	Pass@1, BERTScore, COMETKiwi
Qiskit HumanEval (Vishwakarma et al., 2024)	Python (Qiskit SDK)	Quantum domain, simulator/hardware test	Pass@1 (QHE)

7. Limitations and Future Directions

Current HumanEval Pro methodologies face open challenges:

• Full coverage: Only a minority of all possible real-world scenarios are addressed; scaling template-based approaches and multilingual pipelines remains crucial.
• Semantic drift: Ensuring combinatorial variants retain equal computational content requires meticulous curation.
• Automation: Human-in-the-loop processes (for solution correction and variant approval) are bottlenecks for broader deployment.
• Difficulty calibration: Developing automated measures for solution complexity, test suite richness, and real-world skill alignment are ongoing research goals.
• Expansion to new domains: Non-Python environments (e.g., Rust, Swift, OpenQASM-3) and other vertical applications are active areas for benchmark extension.

As LLMs approach “trivial” performance on classic benchmarks, HumanEval Pro-style datasets and evaluation standards are central to distinguishing genuine reasoning, composition, and deployment-level reliability in next-generation code synthesis models.

---

References

• HumanEval Pro and MBPP Pro: Evaluating LLMs on Self-invoking Code Generation (Yu et al., 2024)
• ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generation (Liu et al., 10 Mar 2025)
• Addressing Data Leakage in HumanEval Using Combinatorial Test Design (Bradbury et al., 2024)
• mHumanEval -- A Multilingual Benchmark to Evaluate LLMs for Code Generation (Raihan et al., 2024)
• Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models (Vishwakarma et al., 2024)