HumanEval-Java: LLM Java Code Benchmark

• HumanEval-Java is a benchmark suite series designed to evaluate LLMs' ability to generate correct and idiomatic Java code at both function and project levels.
• The benchmarks employ rigorous test-driven validation using multi-granularity metrics like pass@k to assess code completion, compilation, and logical correctness.
• They highlight Java’s unique challenges, including static typing and advanced OOP features, driving advancements in prompt engineering and model synthesis.

HumanEval-Java is a collective term referring to benchmark suites designed for the systematic evaluation of code generation capabilities of LLMs on canonical programming problems in Java. It specifically denotes the Java translation and augmentation of the HumanEval benchmarking paradigm, encompassing both function-level (as in CodeGeeX’s HumanEval-Java and the multilingual HumanEval-XL) and project-level, object-oriented code (as epitomized by JavaBench). These benchmarks are crucial for quantifying the extent to which LLMs can synthesize correct and idiomatic Java code under rigorous test-based evaluation, and for exposing weaknesses in model performance relative to Python, particularly in the presence of Java’s static typing and rich OOP idioms.

1. Origin and Motivation

Historically, code generation benchmarking has been dominated by Python-centric datasets. A survey of 24 leading benchmarks revealed that 95.8% target Python; only five include Java, and none previously exercised large-scale Java projects or advanced OOP features. Existing function-level benchmarks, such as the original HumanEval, typically assess basic control flows and data structures, largely neglecting encapsulation, inheritance, and polymorphism—core tenets of the Java programming paradigm. This imbalance impedes progress in multilingual and statically-typed code synthesis research and overlooks challenges posed by Java’s verbose, strongly-typed, and object-oriented syntax. The “HumanEval-Java” initiative, as realized in various recent benchmarks, addresses these gaps by providing Java-specific, test-driven tasks spanning both elementary function-level problems and complex project-level assignments (Cao et al., 2024, Peng et al., 2024, Zheng et al., 2023).

2. Benchmark Design and Dataset Structure

Multiple instantiations of HumanEval-Java exist, each with distinct design principles:

• HumanEval-Java (CodeGeeX/HumanEval-X):
  • Comprises 164 Java problems directly handtranslated from Python HumanEval tasks. Each includes a Main.java file with a public static method (matching the Python signature) and a Javadoc docstring, plus a TestMain.java harness leveraging JUnit for assertion-based testing.
  • Ensures semantic equivalence to Python originals, incorporating Java-specific adjustments (e.g., camelCase naming, static typing, import conventions).
• HumanEval-XL (multilingual):
  • Extends HumanEval to 80 canonical tasks × 23 natural languages × 12 programming languages, yielding 1,840 Java prompts. Each Java prompt is validated via round-trip translation and BERTScore filtering, and paired with an average of 8.33 assertion-based tests.
• JavaBench (project-level “HumanEval-Java”):
  • Represents an expansion to project-level evaluation, built from four real student assignments: 106 classes and 389 methods, covered by 396 JUnit tests (92% class and 87% method coverage).
  • Prioritizes advanced OOP constructs, embedding encapsulation, inheritance, and polymorphism as essential evaluation criteria.
  • Benchmark integrity is attested by 282 undergraduates under exam conditions, with peer-review on both canonical solutions and test suites (Cao et al., 2024).

Benchmark Suite	Problem Count (Java)	Level	OOP Features	Avg. #Tests/Task
HumanEval-Java	164	Function	Limited	5–12
HumanEval-XL (Java)	80 × 23 NLs = 1840	Function	Limited	~8.33
JavaBench	4 projects, 389 methods	Project/Class	Advanced	99/project (~396 total)

3. Evaluation Methodology

The evaluation pipeline for HumanEval-Java benchmarks is defined by test-driven validation and the standard pass@$k$ statistical metric (Zheng et al., 2023):

• Test Harnesses: Each candidate completion is inserted into the problem template and compiled via javac. Only code passing all tests in the harness is considered correct.
• Metrics:
  • Completion@$k$: All TODOs in the stub are nonempty.
  • Compilation@$k$: Compilation succeeds for at least one of $k$ samples.
  • Pass@$k$: All relevant tests pass for at least one of $k$ sampled generations.
• Formula: For $n$ generated samples, $c$ of which are correct,

$$\mathrm{Pass}@k = \mathbb{E}\Biggl[1 - \frac{\binom{n-c}{k}}{\binom{n}{k}} \Biggr].$$

The same estimator is used for all three metrics by appropriately defining what counts as “correct” per metric.

• Sampling Regimes: Experiments employ top-$p$ sampling ($k$0), typical $k$1 samples, and temperatures $k$2 (pass@1) or $k$3 (higher $k$4) (Zheng et al., 2023).
• Granularities:
  • Function, class, or project-level depending on the benchmark (class/testwise granularity is preferred for finer diagnostic power in JavaBench (Cao et al., 2024)).

4. Context Selection and Synthesis Strategies

Prompt context profoundly impacts LLM performance. Three distinct context settings are used in project-level HumanEval-Java (JavaBench):

• Maximum Context: Whole-project skeleton.
• Minimum Context: Only the relevant class’s skeleton.
• Selected (Goldilocks) Context: Method signatures plus signatures of dependencies, most closely mirroring the function-level HumanEval protocol.

Synthesis strategies include holistic (full class at once), independent (one method at a time), and incremental (sequential, reversed, or random order). Holistic generation yields state-of-the-art results, outperforming incremental variants; random order can surpass strictly sequential approaches but does not exceed holistic (Cao et al., 2024). For function-level tasks (HumanEval-XL, CodeGeeX), the prompt is typically a Java stub plus natural-language docstring.

5. Performance Analysis and Error Modes

Recent results illustrate clear trends in LLM performance on HumanEval-Java. Table A shows leading open-access LLMs’ pass@$k$5 rates on the CodeGeeX/HumanEval-X Java set (Zheng et al., 2023):

Model	Pass@1	Pass@10	Pass@100
GPT-J-6B	7.86%	14.37%	32.96%
GPT-NeoX-20B	8.87%	19.55%	42.23%
InCoder-6.7B	9.05%	18.64%	40.70%
CodeGen-Multi-16B	14.95%	36.73%	60.62%
CodeGeeX-13B	20.04%	36.70%	58.42%

HumanEval-XL demonstrates that GPT-4 achieves 62–68% pass@1 on Java, but outperforms substantially on Python (82.5% pass@1); GPT-3.5 scores ∼1–2% on Java, ∼66% on Python. CodeGen2 16B achieves ∼20–25% on Java (Peng et al., 2024).

Error breakdowns indicate that for CodeGeeX on Java, the dominant failure mode is “wrong answer” (53%), with smaller fractions attributable to runtime errors (14%), syntax errors (8%), and unfinished completions (5%) (Zheng et al., 2023). Java’s static typing, stricter method signatures, and import requirements exacerbate LLM failure rates—signature mismatches, missing imports, object identity issues, and unhandled edge cases are recurrent.

Project-level JavaBench evaluations reveal a more severe gap: human undergraduate pass rates (90.93%) far exceed the best LLMs (best Pass@5 at test level: 41.17%; zero projectwise, i.e., no model generated a fully correct project) (Cao et al., 2024). Selected context (method + dependencies) provides optimal balance, and holistic generation strategies outperform others.

6. Implications and Design Guidance

HumanEval-Java benchmarks expose persistent challenges for LLM-based Java synthesis:

• Multilingual and statically-typed coverage: The introduction of function-level (CodeGeeX/HumanEval-XL) and project-level (JavaBench) HumanEval-Java fills a vital empirical gap for evaluating Java code generation.
• Prompt engineering: Prompts containing only method signatures and dependency signatures maximize Pass@$k$6 without overwhelming models with excessive context.
• Evaluation completeness: Hierarchical, multi-granularity metrics (Completion→Compilation→Pass@k) distinguish compilation failures from logical bugs, while class/testwise evaluation exposes partial successes in large projects otherwise missed by all-or-nothing scoring.
• Benchmark best practices: Employing real assignments, high-coverage and peer-reviewed tests, and OOP feature coverage is indispensable for robust LLM benchmarking in Java.

A plausible implication is that further improvements in Java code LLMs will require advances in handling static types, imports, context summarization, and complex OOP semantics—areas underrepresented in canonical Python-centric benchmarks.

7. Availability and Future Directions

Key HumanEval-Java resources are open-source and actively maintained:

• JavaBench: https://github.com/java-bench/JavaBench (Cao et al., 2024)
• HumanEval-XL: https://github.com/FloatAI/humaneval-xl (Peng et al., 2024)
• HumanEval-X/CodeGeeX: https://github.com/THUDM/CodeGeeX (Zheng et al., 2023)

These serve as reference platforms for future LLM research in code generation, OOP reasoning, and cross-lingual synthesis evaluation. Ongoing benchmarking is likely to further differentiate model capability along the axes of static type handling, prompt engineering, project-scale synthesis, and multilingual robustness in code generation.