BigCodeBench (Hard) Dataset

• BigCodeBench (Hard) is a benchmark of 1,140 high-difficulty Python tasks that require multi-tool integration and precise compositional reasoning.
• It employs detailed PEP-257 compliant docstrings and high-coverage unit tests, achieving an average branch coverage of 99% per task.
• The dataset presents complex challenges by combining diverse library calls and task domains, advancing LLM evaluation in realistic coding scenarios.

BigCodeBench (Hard) refers to the full "complete" evaluation set of the BigCodeBench benchmark, a rigorously constructed suite of 1,140 high-difficulty Python code-generation tasks designed to evaluate LLMs on realistic, multi-tool, multi-step problems that demand precise compositional reasoning and robust tool use. Each task mandates the orchestration of multiple standard and external Python libraries, is specified with detailed PEP-257-style docstrings, and is paired with a suite of high-coverage, deterministic unit tests. The benchmark aims to assess and advance the ability of LLMs to solve challenging tasks that reflect complex practical requirements in software engineering, data analysis, and related domains, exceeding the scope of prior benchmarks focused on algorithmic or single-call code generation (Zhuo et al., 22 Jun 2024).

1. Construction and Definitional Criteria

All tasks in BigCodeBench (Hard) share stringent construction criteria constituting the intended high-difficulty benchmark:

• Multi-tool requirement: Every task utilizes at least two external or standard Python libraries, selected from a pool of 139 libraries (77 from the standard library, 62 from external PyPI packages).
• Compositional reasoning: Task specifications are encoded as multi-step, PEP-257-compliant docstrings, detailing functional goals, parameters, return values, error conditions, and multiple usage examples. These docstrings average 1,112 characters and 33.5 lines.
• Rigorous evaluation: Each task is accompanied by a minimum of five unit tests (mean 5.6), implemented using Python's unittest framework with pytest compatibility, achieving a measured average branch coverage of 99%.
• Task complexity: No explicit "difficulty score" is assigned; instead, the set achieves difficulty through the number of distinct library calls per task (average 4.7), cyclomatic complexity of the reference implementation (average 3.1 as per McCabe’s metric), and the structural depth of the task description.

There are no easy/hard splits within BigCodeBench; the entire "complete" set is intended as a hard, compositional challenge (Zhuo et al., 22 Jun 2024).

2. Dataset Composition and Coverage

BigCodeBench (Hard) comprises a diverse and expansive range of tasks engineered to stress-test current LLM capabilities:

• Total number of tasks: 1,140
• Library/tool utilization:
  • 77 distinct standard library modules
  • 62 unique external PyPI libraries
  • 281 unique standard library calls and 442 unique external library calls featured in reference solutions
  • Average per task: 2.8 libraries, 4.7 function calls
• Unique combinations:
  • 577 distinct sets of libraries
  • 1,045 unique function call sets, indicating high tool-combinatorial variety

Domain coverage (fraction of tasks involving at least one library per domain):

Domain	Fraction of tasks using ≥1 library
Computation	63%
System	60%
General utils	55%
Network	43%
Time & Date	39%
Visualization	33%
Cryptography	20%

Test-case statistics per task:

• Average number of test cases: 5.6
• Average branch coverage: 99% (via standard coverage tooling)

The breadth and depth of tooling and domain representation distinguish BigCodeBench (Hard) from previously established benchmarks.

3. Benchmark Organization, Splits, and Protocol

The BigCodeBench (Hard) set is distributed as a single, undivided evaluation set of 1,140 tasks. There are no train, validation, or test splits, nor is there a separate protocol for "Hard"-only benchmarking—the "complete" collection represents the intended evaluation corpus.

• Evaluation protocol: All tasks are suitable for zero-shot or few-shot code generation benchmarking.
• Absence of training partitions: The benchmark is strictly for evaluation, with no training or tuning split provided.
• Open-ended challenge: The protocol is designed to reflect real-world assessment conditions, where models are expected to generalize to unfamiliar, high-complexity tasks.

A plausible implication is that this benchmark is particularly well-suited for forward-looking studies of LLM generalization and robustness under strict evaluation regimes.

4. Methodology and Task Engineering Pipeline

The BigCodeBench (Hard) suite was developed through a systematic, hybrid pipeline combining LLM-driven data synthesis, semi-automatic test generation, and iterative human curation:

• Data synthesis: Initial task seeds and API usage examples were generated using GPT-4, serving as a foundation for complex multi-tool scenarios.
• Refinement and test generation: Tasks were semi-automatically refactored and supplemented with new unit tests via GPT-4 Code Interpreter, with intermediated human feedback to guide correctness and coverage.
• Human-in-the-loop curation:
  • Manual editing of task docstrings for clarity, completeness, and adherence to PEP-257.
  • Reconciliation of imports, extension and correction of unit tests, and pre-evaluation using GPT-3.5 to detect ambiguities.
  • Multi-pass checks to ensure each task’s consistency, determinism, and alignment with specification.

Task design principles:

1. Diverse function-call sequences: Every task mandates the chaining of at least two distinct libraries, often composing calls such as pytz.timezone(...).astimezone(...) or layered usages involving network, computation, and visualization libraries.
2. Complex, multi-step instructions: Specifications demand the integration of branching logic, exception handling, file/network/database I/O, and construction of custom data structures.
3. Rigorous, open-ended evaluation: Each task is instrumented with at least five deterministic, high-coverage unit tests, frequently leveraging mocking for system-dependent calls.

This hybrid methodology ensures both breadth and depth of task coverage, challenging code-generation models to integrate multiple toolchains and reason over compositional, real-world instructions.

5. Representative Task Types and Testing Paradigms

BigCodeBench (Hard) includes a diverse array of task archetypes, spanning computation, system, network, and data visualization contexts. Representative examples illustrate multi-tool and compositional demands:

• Network/Parsing: Extracting domains from URLs and mapping them to IPv4 addresses using re, urllib.parse, and socket, with unit tests spanning correct resolution, malformed input handling, and negative scenarios.
• Computation/Flattening: Constructing a random integer matrix via numpy, flattening it with itertools, and validating deterministic output and shape constraints through tests.
• Visualization/Word Cloud: Fetching Wikipedia page content, rendering a word cloud using wikipedia, wordcloud.WordCloud, and matplotlib.pyplot; tests check for successful rendering and fallbacks on missing pages.

These examples demonstrate the benchmark’s emphasis on toolchain integration, determinism, and fine-grained specification.

6. Comparative Summary and Metrics

Compared to prior benchmarks such as HumanEval, DS-1000, and ODEX, BigCodeBench (Hard) delivers substantially greater breadth in task tooling, specification complexity, and evaluation rigor. Key comparative metrics are summarized in the tables below:

	# Tasks	Test # (avg.)	Branch Cov.	Prompt Char.	Prompt Lines	Solution Char.	Solution Lines	Cyclomatic Comp.
BigCodeBench	1,140	5.6	99%	1,112.5	33.5	426.0	10.0	3.1
HumanEval	164	7.8	98%	450.6	13.7	180.9	6.8	3.6
DS-1000	452	1.5	98%	831.4	26.2	115.5	4.2	1.4
ODEX	945	1.8	96%	87.5	1.0	50.4	1.9	1.4

	# Domains	# Std Libs / Ext Libs	# Std Calls / Ext Calls	Avg. Libs/Task	Avg. Calls/Task
BigCodeBench (Hard)	7	77 / 62	281 / 442	2.8	4.7
HumanEval	3	4 / 0	7 / 0	0.1	0.1
DS-1000	4	5 / 9	7 / 321	0.8	1.1
ODEX	7	40 / 26	128 / 102	0.6	0.5

BigCodeBench (Hard) constitutes a comprehensive and ambitious evaluation framework for LLM-based code generation, advancing beyond previous benchmarks in both multi-tool orchestration and instruction-following complexity (Zhuo et al., 22 Jun 2024).