EvalPerf: Code Efficiency Benchmark

• EvalPerf is a benchmark dataset comprising 121 efficiency-demanding Python tasks, emphasizing measurable hardware instruction counts to differentiate solution performance.
• The evaluation protocol employs a Differential Performance Score and automated input scaling via a Synthesizer to objectively rank algorithmic efficiency.
• The dataset’s structure and curation criteria ensure reproducible, platform-agnostic assessments, enabling precise comparisons of LLM-generated code efficiency.

EvalPerf is a benchmark dataset and evaluation suite comprising 121 efficiency-demanding Python programming tasks, specifically curated to expose the computational performance spectrum of code outputs generated by LLMs. EvalPerf is constructed atop correctness-focused coding benchmarks such as HumanEval+ and MBPP+, but rigorously filters tasks to capture only those for which algorithmic optimizations yield orders-of-magnitude differences in runtime, as measured by hardware instruction counts under computational stress. The benchmark is integral to the Differential Performance Evaluation (DPE) framework, which provides a precise, reproducible, platform-agnostic protocol for scoring LLM-generated code according to global code efficiency metrics, rather than traditional correctness or wall-clock based assessments (Liu et al., 2024).

1. Dataset Rationale and Design Principles

EvalPerf was developed to correct deficiencies in prior benchmarks, which focused on functional correctness but failed to rigorously differentiate models along the axis of code efficiency. Standard practice with simplistic or undersized test cases leads to an inability to discriminate between naïve and efficient algorithms, as all correct implementations may appear equivalent when problem sizes are small. Consequently, EvalPerf isolates tasks wherein input scaling can elicit substantial efficiency gaps—making it possible to objectively profile, compare, and analyze the efficiency “intelligence” of code-generating LLMs (Liu et al., 2024).

Candidate tasks were drawn from an initial pool of 563 Python programming problems (164 from HumanEval+, 399 from MBPP+), spanning major algorithmic domains: prime testing, sorting/searching, graph traversal, combinatorial enumeration, sequence generation, and matrix operations. Only tasks passing three curation criteria were kept:

• Minimum workload: Even the fastest correct solution must, on largest test input, execute at least $t_{\mathrm{thresh}} = 10,000$ hardware instructions.
• Measurement stability: Reference solutions’ instruction counts must have low coefficient of variation under repeated measurement (below $CV_{\mathrm{thresh}}$), to rule out environmental noise.
• Performance diversity: Each task must have at least $K=4$ disjoint efficiency clusters among correct solutions, as determined by clustering instruction counts; tasks lacking such diversity were excluded.

2. Stress-Test Input Generation: Synthesizing a Synthesizer

A key innovation is EvalPerf’s automated scaling of input difficulty, facilitated by the “Synthesizing a Synthesizer” (SaS) protocol. For each task, a large code model is prompted (using Chain-of-Thought few-shot examples) to generate a function $10^7$9 This function generates batches of valid test cases of monotonically increasing computational hardness as scale increases. The scaling follows $2^1, 2^2, ..., 2^k$, with the process halting once a correct solution exceeds either 20 seconds in wall time or 16 GiB memory usage—guaranteeing practical inputs that induce at least tens of millions of machine instructions.

Illustrative examples of generated input stress patterns:

• Prime testing (is_prime(n)): Prime candidates near $10^7$ to $10^8$, requiring full $\sqrt{n}$ trial division for naïve algorithms.
• Matrix multiplication (matmul(A, B)): Square matrices with $N=512$–$1024$, leading to workloads sensitive to $O(N^3)$ complexity differences.
• Sorting (sort_list(arr)): Arrays with $CV_{\mathrm{thresh}}$0–$CV_{\mathrm{thresh}}$1 random integers, sufficient to distinguish $CV_{\mathrm{thresh}}$2 from $CV_{\mathrm{thresh}}$3 implementations.

3. Dataset Structure and Reference Solutions

Each EvalPerf task is organized in a directory containing:

• prompt.py: Human-readable problem statement and signature.
• input_gen.py: The SaS-generated Python input synthesizer.
• test_inputs/scale_{2^i}.json: Serialized input batches at each scale.
• references/: One representative correct solution for each performance tier (“cluster_1.py” through “cluster_m.py”, slowest to fastest).
• metadata.json: Task-level annotations, including:
  • Task name and algorithmic domain
  • $CV_{\mathrm{thresh}}$4 (instruction-count threshold)
  • Number of clusters $CV_{\mathrm{thresh}}$5
  • Cluster statistics $CV_{\mathrm{thresh}}$6: where $CV_{\mathrm{thresh}}$7 is the mean instruction count (hardware-measured), and $CV_{\mathrm{thresh}}$8 is the cumulative percentage of all validated reference solutions at or below cluster $CV_{\mathrm{thresh}}$9.

This structure enables consistent evaluation, comparison, and reporting across heterogeneous systems. All reference implementations are validated for correctness, ensuring that only efficiency is differentiated during profiling (Liu et al., 2024).

4. Efficiency Scoring: Differential Performance Score (DPS)

EvalPerf leverages the Differential Performance Score (DPS), a compound, global metric that places any new solution $K=4$0 on a reproducible 0–100 scale by comparing its hardware instruction profile to the established performance clusters for that task. Formally, for $K=4$1 clusters with strictly decreasing mean instruction counts $K=4$2 and cumulative ratios $K=4$3, upon observing a new solution’s average instruction count $K=4$4,

$K=4$5

A normalized variant treats each efficiency tier equally:

$K=4$6

DPS provides interpretable percentiles: a solution that sits at DPS = 80% is strictly faster than 80% of all prior validated correct solutions.

Evaluation protocol: For each new (correct) solution, run it on the established stress inputs, record instruction counts via hardware counters (e.g., Linux perf_event), and apply the DPS calculation by comparison to reference clusters.

5. Usage Guidelines and Practical Considerations

EvalPerf is publicly available via the “evalperf/” subfolder at github.com/evalplus/evalplus. Researchers typically follow these steps:

1. Clone the repository.
2. For each task:
   • Execute the candidate solution using python run_code.py --task T --solution my_solution.py --scale S, which profiles instruction count over designated test inputs.
   • Continue scaling until failure or resource limits.
3. Score the solution using python score_evalperf.py --task T, which provides both DPS and $K=4$7 for that task; aggregate scripts compile results across all tasks.

By profiling with hardware instruction counters instead of wall-clock, EvalPerf ensures reproducibility and stability across diverse Linux systems; only perf_event access is required, with no need for simulators or privileged execution (Liu et al., 2024).

6. Representative Examples and Interpretability

Two canonical examples from EvalPerf illustrate its discrimination power:

• Prime Checking (is_prime(n)):
  • Stress test at $K=4$8.
  • Clusters: trial division up to $K=4$9 ($2^1, 2^2, ..., 2^k$0, $2^1, 2^2, ..., 2^k$1), trial to $2^1, 2^2, ..., 2^k$2 ($2^1, 2^2, ..., 2^k$3), trial to $2^1, 2^2, ..., 2^k$4 ($2^1, 2^2, ..., 2^k$5, $2^1, 2^2, ..., 2^k$6), Miller–Rabin ($2^1, 2^2, ..., 2^k$7, $2^1, 2^2, ..., 2^k$8), with corresponding cumulative ratios $2^1, 2^2, ..., 2^k$9.
  • A solution with $10^7$0 instructions is scored at $10^7$1, $10^7$2.
• Matrix Multiplication (matmul(A, B)):
  • Stress test at $10^7$3 matrices.
  • Clusters: triple-nested Python loops ($10^7$4), list comprehensions, NumPy dot (BLAS), blocked Strassen ($10^7$5).
  • A solution with $10^7$6 matches vectorized BLAS, scoring $10^7$7, $10^7$8.

These cases demonstrate EvalPerf’s sensitivity to both algorithmic choices and low-level implementation differences. DPS percentiles directly extrapolate to the competitive landscape of prior LLM-generated solutions, enabling nuanced, interpretable benchmarking of efficiency (Liu et al., 2024).

7. Impact and Research Implications

EvalPerf enables systematic, reproducible assessment of LLMs in efficiency-sensitive code generation, making it possible to separate functional competence from algorithmic or implementation efficiency. Its rigorous input scaling, noise filtering, and efficiency clustering protocols ensure that only those tasks yielding substantial performance differentials are retained. Empirical results show that model scaling laws for code generation do not account for efficiency, while general instruction tuning improves both correctness and efficiency. EvalPerf's platform-independence and robust hardware profiling provide researchers and practitioners with a practical tool for evaluating the “efficiency intelligence” of code outputs on real-world, stress-tested computational benchmarks (Liu et al., 2024).