CodegenBench: Multi-Architecture HPC Benchmark

• CodegenBench is a multi-architecture benchmark that evaluates LLMs' ability to generate numerically correct, efficient CPU-oriented HPC code across heterogeneous platforms.
• It employs both a broad BLAS baseline and specialized architecture-tailored kernels to measure compile success, numerical accuracy, and performance metrics like Pass@1 and Fast_1@1.
• The automated pipeline handles code generation, building, execution, and validation, revealing optimization patterns and common failure modes in hardware-specific code synthesis.

CodegenBench is a multi-architecture benchmark and automated evaluation pipeline for assessing whether LLMs can generate not only correct, but efficient CPU-oriented HPC code across heterogeneous architectures. It targets three CPU platforms—x86_64, Sunway, and Kunpeng—and couples execution-based validation with performance profiling over both a broad BLAS baseline and architecture-tailored kernels. In contrast to code-generation evaluations centered on Python snippets or GPU stacks, CodegenBench treats cross-platform generalization itself as the central problem: the model must compile, satisfy numerical checks, and deliver speed relative to established CPU baselines under architecture-specific constraints (Li et al., 1 Jun 2026).

1. Scope and evaluation target

CodegenBench is designed to answer a focused question: can contemporary LLMs generate not only correct, but efficient CPU-oriented HPC code across heterogeneous architectures? The benchmark addresses a gap left by evaluations that emphasize NVIDIA/CUDA ecosystems, PyTorch-like environments, or high-level programming tasks. Its domain is general-purpose scientific computing and low-level, performance-critical kernels on three CPU platforms with markedly different ISAs and memory organizations—x86_64, Sunway, and Kunpeng—where hardware-aware optimization, portability, and sparse public training data are central difficulties (Li et al., 1 Jun 2026).

The benchmark’s goals are threefold. First, it assesses whether generated C/C++ kernels compile and are numerically correct. Second, it measures speed relative to established CPU baselines. Third, it studies the impact of problem complexity and code length on success and performance. This yields a benchmark that is narrower than general code-generation suites in domain coverage, but much stricter in its architectural demands. A plausible implication is that CodegenBench functions less as a generic programming test than as a stress test for architecture-conditioned synthesis.

2. Benchmark composition and target platforms

CodegenBench comprises two benchmark families. The first is a BLAS baseline of 106 routines spanning Levels 1–3, with varied parameters such as dimensions, strides, data types, and both real and complex arithmetic. Representative routines include AXPY, $y \leftarrow \alpha x + y$; DOT, $s = x^\top y$; SCAL, $x \leftarrow \alpha x$; NRM2, $\|x\|_2 = \sqrt{\sum_i x_i^2}$; GEMV, $y \leftarrow \alpha A x + \beta y$; and GEMM, $C \leftarrow \alpha A B + \beta C$. The second consists of 20 specialized computational kernels for each of the unique supercomputing architectures, released as LeetSunway and LeetKunpeng, which stress hardware-aware parallelization, vector ISA usage, synchronization, and memory-hierarchy control (Li et al., 1 Jun 2026).

The three target systems differ substantially in programming model and optimization burden.

Platform	Toolchain and baseline	Distinctive constraint
x86_64	gcc; OpenBLAS	AVX/AVX2/AVX-512, cache blocking, loop tiling
Sunway	swgcc	MPE controlling 64 CPEs per core group
Kunpeng	bisheng/clang; kblas	SVE/SME, on-die HBM, data placement

On x86_64, the benchmark uses an Intel Xeon Platinum 8488C server platform, where high performance ordinarily depends on vectorization via AVX/AVX2/AVX-512 intrinsics and on library-style blocking optimizations. On Sunway, the target is the SW26010 many-core processor with four Core Groups per chip; each Core Group contains one Management Processing Element and 64 Computing Processing Elements, making explicit master–slave orchestration and local-memory staging essential. On Kunpeng, the setup uses latest ARM-based processors with SVE/SME vector or matrix extensions and on-die HBM, evaluated on a multi-node cluster with dual Kunpeng sockets per node (Li et al., 1 Jun 2026).

3. Evaluation pipeline and metrics

The evaluation framework automates four stages across platforms: pre-processing, prompt and code generation, build, and run plus validation. Pre-processing cleans environments, sets environment variables, and parses metadata. Prompt construction uses problem statements, metadata, and references, then queries the LLM via API. The build stage compiles generated code with platform compilers and harnesses, recording failures such as syntax errors, hallucinated APIs, and missing headers. The final stage executes the code with task-specific parameters, collects stdout, stderr, and logs, and validates outputs against references using tolerance-based numerical checks (Li et al., 1 Jun 2026).

The relative numerical error is defined as

$$\mathrm{err}_{\mathrm{rel}} = \frac{\|y_{\mathrm{pred}} - y_{\mathrm{ref}}\|_2}{\|y_{\mathrm{ref}}\|_2}.$$

Performance is then measured after warm-up across multiple sizes per routine. The benchmark records throughput, such as GEMM GFLOPS $= 2mnk/t$; effective bandwidth, $BW = \mathrm{bytes\_transferred}/t$; speedup, $S = T_{\mathrm{baseline}} / T_{\mathrm{model}}$; and efficiency, $s = x^\top y$0. Normalization is to well-known CPU baselines—OpenBLAS on x86_64 and kblas on Kunpeng—while the Leet tasks compare optimized model code to reference CPU implementations supplied with the benchmark. Success criteria are compile success, passing all numerical checks, and, for Fast_1@1, demonstrating speedup over baseline among first-attempt passes (Li et al., 1 Jun 2026).

The primary benchmark metrics are Pass@1, Pass@5, and Fast_1@1:

• Pass@1: $s = x^\top y$1
• Pass@5: $s = x^\top y$2
• Fast_1@1: $s = x^\top y$3

Nine models were evaluated zero-shot via APIs with a uniform, brief system instruction requesting optimized code; no fine-tuning, tool-use, or architecture-specific plugins were used. The closed-source models were Claude Sonnet 4.6, Claude Opus 4.6, Claude Opus 4.7, and Qwen 3.6 Plus. The open-source models, accessed through APIs, were DeepSeek V3.2, DeepSeek V4 Flash, DeepSeek V4 Pro, Qwen 3.6 Flash, and Qwen 3.5 Plus. The evaluation attempts up to 5 generations per task for Pass@5, while temperature, top-p, and context details are not specified (Li et al., 1 Jun 2026).

4. Empirical results and performance structure

On the x86_64 BLAS suite, closed-source models dominate correctness. Claude Opus 4.7 attains Pass@1 = 0.74 and Pass@5 = 0.85, while Claude Sonnet 4.6 reaches Pass@1 = 0.70 and Pass@5 = 0.79. The best Fast_1@1 on this suite is Sonnet 4.6 at 0.62. Strong open-source contenders, specifically DeepSeek V4 Pro and DeepSeek V4 Flash, reach Pass@1 of approximately 0.48–0.53 and Pass@5 of approximately 0.71–0.74. On Kunpeng BLAS, correctness drops for nearly all models relative to x86_64, although Claude Opus 4.7 still leads with Pass@1 = 0.76 and Pass@5 = 0.85, followed by Claude Opus 4.6 at 0.67 and 0.81. Fast_1@1 on Kunpeng BLAS lies around 0.32–0.39 for closed-source models and around 0.20–0.36 for open-source models (Li et al., 1 Jun 2026).

The architecture-tailored suites are harder. On LeetKunpeng, the best Pass@1 is Claude Opus 4.7 at 0.64 and Pass@5 = 0.70, followed by Opus 4.6 at 0.58 and 0.65. Among tasks that do pass, optimized code is often faster than the provided baseline: Qwen 3.6 Plus achieves Fast_1@1 = 0.93, Opus 4.6 reaches 0.90, Sonnet 4.6 is 0.84, and Opus 4.7 is 0.83. On LeetSunway, Sunway is the hardest target; Claude Opus 4.7 attains Pass@1 = 0.48 and Pass@5 = 0.50, Opus 4.6 attains 0.33 and 0.50, and Sonnet 4.6 is 0.06 and 0.15. For tasks that do pass, Fast_1@1 is often very high, including 1.00 for Sonnet 4.6 and Opus 4.6, reflecting that unlocking any architecture feature tends to beat the simple baseline (Li et al., 1 Jun 2026).

Several structural trends recur across the benchmark. Correctness declines as BLAS level increases from Level 1 to Level 3, consistent with more complex data layouts and boundary logic. Fast_1@1, however, tends to improve from Level 1 to Levels 2–3, because Level 1 routines are largely memory-bound with limited optimization headroom, whereas higher levels are more compute-bound and reward tiling and vectorization. On x86_64, real-number tasks show higher Pass@1 than complex tasks, but Fast_1@1 is higher for complex routines; on Kunpeng, real arithmetic dominates both correctness and acceleration for most models. Pass@1 also drops sharply as implementation length grows, and concise kernels of under approximately 50 lines are more reliable across models. Current LLMs are therefore most effective for moderately difficult problems requiring concise code snippets (Li et al., 1 Jun 2026).

5. Optimization strategies, successes, and failure modes

CodegenBench’s error analysis isolates several recurrent causes of degradation on Sunway and Kunpeng. The first is data scarcity: there are few high-quality, open-source examples and guides for Sunway and, to a lesser extent, Kunpeng, leaving models without robust priors for MPE/CPE programming, DMA-like data movement, and SVE/SME idioms. The second is hallucinated intrinsics and non-portable assumptions, which produce immediate compilation failures. Concrete examples include the nonexistent ARM SVE intrinsic svcmul_f64_z, the invalid form svst1_scatter_index_f64, and x86 intrinsics such as _mm256_reduce_add_ps and _mm256_reduce_add_pd used as though they were available. The third is poor use of memory hierarchy and threading, visible in naive loop nests, absent cache blocking and tiling, and incorrect data routing in Sunway’s master–slave memory architecture. The fourth is unreliable complex-number vectorization, where absence of native complex ISAs forces manual real or imaginary decomposition and makes performance lag optimized libraries by 6–10× when register blocking, cache tiling, and vector packing are missing (Li et al., 1 Jun 2026).

The benchmark also identifies successful optimization patterns. On x86_64, models sometimes exploit AVX2 FMAs correctly. In ssyr2 and dgemv, Claude Opus 4.7 and DeepSeek V3.2 used AVX2 FMA intrinsics and surpassed OpenBLAS by pairing vectorized FMAs with correct stride handling and loop ordering. On Kunpeng, Claude Opus 4.7 applied SVE intrinsics to an attention kernel and achieved at least 1.6× FLOPS speedup once correctness was satisfied. These cases show that the benchmark is not limited to exposing failures; it also records instances in which the model synthesizes competitive low-level code (Li et al., 1 Jun 2026).

Sunway remains the clearest boundary case. Efficient code must orchestrate the MPE controlling many CPEs with explicit data distribution, synchronization, and local-memory staging. This is rarely achieved by LLMs without hints, although when a correct parallelization is synthesized, speedups over simple baselines are common. A common misconception is that high Fast_1@1 necessarily implies broad competence. In CodegenBench, high Fast_1@1 on LeetSunway is conditional on a comparatively small set of successful generations; it measures acceleration among first-attempt passes rather than global task coverage (Li et al., 1 Jun 2026).

6. Position within the benchmark landscape

Within the broader code-generation benchmark landscape, CodegenBench occupies a distinct CPU-HPC and cross-architecture niche. EffiBench-X measures execution time, memory peak, and memory integral for competitive-programming solutions across six languages, reporting that even the best LLM-generated solutions achieve only about 60–62% of human runtime efficiency on average (Qing et al., 19 May 2025). Web-Bench instead evaluates sequential web-project construction with 50 projects of 20 tasks each and Playwright-based end-to-end testing (Xu et al., 12 May 2025). DeepBench targets 520 function-level deep learning code problems spanning pipeline phases, tasks, and input modalities (Daghighfarsoodeh et al., 26 Feb 2025). CodeIF-Bench focuses on instruction-following in interactive code generation with 122 Python tasks and 879 verifiable instructions (Wang et al., 5 Mar 2025). JavaBench addresses project-level Java with 106 classes and 389 methods under OOP-heavy dependencies (Cao et al., 2024). DynaCode, by contrast, dynamically composes nested Python tasks and reports up to 189,263,141 unique problems through call-graph control and complexity-aware scoring (Hu et al., 13 Mar 2025).

This comparison indicates that CodegenBench is not simply another efficiency benchmark. COFFE, for example, also centers on efficiency, but does so through CPU instruction count and efficient@k on Python problems at function and file level (Peng et al., 5 Feb 2025). CodegenBench differs in binding efficiency to architecture-specific CPU-HPC portability: the generated artifact must survive translation across mainstream x86_64, ARM-based Kunpeng, and the Sunway master–slave model, where public exemplars and intrinsics documentation are unevenly available. It therefore complements, rather than replaces, correctness-oriented or workflow-oriented suites by testing a narrower but technically deeper dimension of code synthesis (Li et al., 1 Jun 2026).

7. Reproducibility, limitations, and research implications

CodegenBench releases its dataset and automated evaluation infrastructure, including benchmark tasks, metadata-driven prompts, build scripts, runners, and reference baselines such as OpenBLAS for x86_64, kblas for Kunpeng, and CPU baselines for the Leet suites. Reproduction requires installing the platform compilers—gcc for x86_64, swgcc for Sunway, and bisheng or clang for Kunpeng—configuring the environment per the provided metadata, and running the pipeline, which handles prompt assembly, code generation via API, build via CMake or Make, execution, numerical correctness checks, and performance profiling. Compute requirements depend on the number of LLM queries and on-platform benchmarking runs (Li et al., 1 Jun 2026).

The benchmark’s limitations are explicit. It evaluates zero-shot API-based generation without fine-tuning, tool-use, or architecture-specific plugins, so reported results isolate intrinsic model capabilities rather than agentic workflows. Temperature, top-p, and context details are not specified. Cross-platform comparison is also intrinsically difficult: x86_64, Sunway, and Kunpeng differ not only in ISA but in memory organization, compiler behavior, and programming model. A plausible implication is that CodegenBench should be read as a controlled relative benchmark for heterogeneous CPU-HPC code generation, not as a universal measure of programming ability.

The benchmark nonetheless yields direct methodological implications. It recommends explicit architecture-aware prompting, such as instructing the model to use AVX2 FMAs on x86_64, ACLE SVE intrinsics and predication on Kunpeng, or MPE-led CPE parallelization on Sunway. It also recommends compile-test loops with strict toolchain flags, validation through the relative error criterion, guardrails for intrinsics usage through allowlists, and portability strategies that generate a portable baseline first and then conditionally specialize with #ifdef paths. Future research directions proposed alongside the benchmark include augmenting training corpora with high-quality Sunway and Kunpeng kernels, integrating retrieval or tool-use for intrinsics catalogs and optimization recipes, adding feedback-driven autotuning loops, and studying curriculum prompts that progressively teach memory hierarchies and vector idioms (Li et al., 1 Jun 2026).