LeetKunpeng: ARM HPC Code Benchmark
- LeetKunpeng is a specialized kernel suite for Kunpeng, an ARM-based HPC platform with features like SVE, SME, and on-die HBM.
- It uses 20 computational kernels identical to LeetSunway to assess LLMs’ portability and architecture-specific optimization capabilities.
- The benchmark’s four-stage pipeline and metrics (Pass@1, Pass@5, Fast_1@1) reveal high efficiency in successful code despite challenges of limited training data.
LeetKunpeng is the Kunpeng-specific specialized-kernel suite within "CodegenBench: Can LLMs Write Efficient Code Across Architectures?" and is intended to test whether LLMs can generate not only correct code but efficient, architecture-aware parallel code for a non-x86, CPU-oriented HPC platform. Within CodegenBench, it complements a broad BLAS baseline and the Sunway-oriented LeetSunway subset, and targets the Kunpeng ARM-based platform, whose salient features in the benchmark context are Scalable Vector Extension, Scalable Matrix Extension, and on-die High Bandwidth Memory (Li et al., 1 Jun 2026).
1. Position within CodegenBench
CodegenBench is organized into three primary parts: BLAS, LeetSunway, and LeetKunpeng. The benchmark comprises 106 standard BLAS routines as a general baseline and 20 specialized computational kernels adapted for each of the two supercomputing architectures, namely LeetSunway and LeetKunpeng. LeetKunpeng is therefore not a standalone benchmark family detached from CodegenBench; it is one of the two specialized supercomputer-oriented subsets designed to probe architecture-specific code generation beyond standardized BLAS workloads (Li et al., 1 Jun 2026).
The rationale for introducing LeetKunpeng is explicit. The benchmark was constructed to address whether LLMs can generate efficient code for non-x86, CPU-oriented HPC architectures with architecture-specific optimization requirements. The paper identifies three pressures that distinguish this setting from mainstream code generation evaluation: portability, because models must port baseline code while preserving or improving performance; architecture-awareness, because models must exploit nonstandard hardware features; and data scarcity, because such platforms have less public code and fewer examples.
Kunpeng occupies a deliberate middle position in this design. In the authors’ framing, x86_64 is comparatively easy because it is common and well represented in training data, while Sunway is the hardest target because it is highly specialized and sparse in public examples. Kunpeng is presented as a middle-ground test of cross-architecture generalization: harder than x86 because it requires ARM/Kunpeng-specific optimization, but less exotic than Sunway.
2. Kunpeng as the target architecture
In the benchmark, Kunpeng is described as an ARM-based processor family with HPC-oriented features including on-die High Bandwidth Memory (HBM), Scalable Vector Extension (SVE), and Scalable Matrix Extension (SME). In the appendix’s evaluation environment, the Kunpeng platform is further described as a cluster of multiple interconnected nodes, each node equipped with dual Kunpeng processors exposing SVE/SVE2 and SME and providing on-die HBM (Li et al., 1 Jun 2026).
This architectural framing is central to what LeetKunpeng measures. The paper explicitly contrasts Kunpeng with Sunway by noting that Kunpeng has more public optimization tutorials and more code examples because of the standardized ARM ISA, even if it is less prevalent than x86 in conventional HPC centers. This suggests that failure on LeetKunpeng cannot be reduced to total absence of public documentation. Rather, Kunpeng is used to test whether standardized ARM semantics and somewhat better public information are sufficient for LLMs to internalize platform-specific optimization patterns.
The benchmark also emphasizes concrete Kunpeng-specific optimization opportunities. These include vectorization via SVE, SVE2, and SME, and the use of on-die HBM for programs addressing memory-bound bottleneck behavior. The attention case study is used as an example of this regime: the model selected proper SVE intrinsics, and the generated implementation achieved at least speedup in FLOPS.
3. Benchmark content and task design
LeetKunpeng contains 20 computation kernels, and its most distinctive structural property is that it mirrors LeetSunway exactly in terms of the algorithmic tasks required. The paper states that the 20 computing kernels in LeetKunpeng are identical to those formulated for LeetSunway. LeetKunpeng is therefore not a separate algorithmic problem set; it is the same problem set evaluated under Kunpeng-specific implementation and optimization expectations (Li et al., 1 Jun 2026).
The paper does not enumerate all 20 kernels in the excerpted text, but it does characterize their scope. The specialized suites cover a wide range of kernels, from linear algebra computation to machine learning optimizer. One explicit LeetKunpeng example is an attention kernel, which demonstrates that the suite is not confined to classical BLAS-like routines.
This design has two consequences. First, it makes LeetKunpeng a portability test: algorithmic content is held fixed while implementation strategy must change with the target architecture. Second, it isolates architecture-awareness more cleanly than a benchmark that changes both the problem set and the hardware target at once. A common misconception is that LeetKunpeng introduces Kunpeng-specific algorithms; the benchmark instead keeps the algorithmic tasks aligned with LeetSunway and shifts the burden to hardware-specific realization.
The authors also note a limitation: 20 kernels may be insufficient to fully represent Kunpeng and Sunway workloads. That caveat matters when interpreting LeetKunpeng as a probe of architecture-aware code generation rather than as an exhaustive workload model of Kunpeng-based HPC.
4. Evaluation pipeline, toolchain, and metrics
CodegenBench operationalizes LeetKunpeng through a four-stage pipeline: pre-processing, prompt generation and code generation, build, and run and validation. Pre-processing cleans temporary files, initializes a deterministic environment, and sets environment variables. Prompt generation gathers metadata, instructions, problem descriptions, source snippets, and documentation, constructs a comprehensive prompt, and queries the target LLM. The build stage compiles generated code with the benchmark harness and detects syntax errors, hallucinated APIs, and missing dependencies. The final stage executes programs with metadata-defined runtime parameters, collects outputs and logs, and validates numerically using tolerance-based comparisons (Li et al., 1 Jun 2026).
The Kunpeng toolchain differs between the BLAS and Leet portions. For BLAS/Kunpeng, the benchmark uses bisheng, a customized clang compiler, and links with kblas. For LeetKunpeng, evaluation uses the clang compiler, and the appendix states that evaluations explicitly utilize the specialized on-die HBM feature in Kunpeng chip. This distinction is methodologically important because BLAS/Kunpeng is benchmarked against a tuned library stack, whereas LeetKunpeng evaluates architecture-targeted kernel generation more directly.
The benchmark reports three metrics: , , and . In the paper’s interpretation, is the fraction of tasks solved correctly on the first try, is the fraction solved within five tries, and measures, among tasks that are correct on the first try, the fraction whose generated code is actually faster than the baseline. A generated kernel counts as “fast” when its speedup is greater than 1.
LeetKunpeng should not be conflated with the BLAS portion. BLAS covers 106 distinct BLAS subroutines across Levels 1–3 and is evaluated against optimized baselines such as OpenBLAS on x86 and kblas on Kunpeng. By contrast, LeetKunpeng contains 20 specialized kernels, is architecture-targeted, expects use of Kunpeng-specific features, and is compared against a basic baseline implementation rather than an industrial-strength hand-tuned library. This explains why LeetKunpeng can exhibit relatively poor correctness yet high efficiency among successful samples.
5. Empirical results and characteristic failure modes
Table 3 reports the following LeetKunpeng results for the evaluated models (Li et al., 1 Jun 2026).
| Model | Category | Pass@1 / Pass@5 / Fast_1@1 |
|---|---|---|
| Claude Sonnet 4.6 | Closed-source | 0.49 / 0.65 / 0.84 |
| Claude Opus 4.6 | Closed-source | 0.58 / 0.65 / 0.90 |
| Claude Opus 4.7 | Closed-source | 0.64 / 0.70 / 0.83 |
| Qwen 3.6 Plus | Closed-source | 0.14 / 0.30 / 0.93 |
| DeepSeek V3.2 | Open-source | 0.00 / 0.00 / N/A |
| DeepSeek V4 Flash | Open-source | 0.11 / 0.35 / 0.64 |
| DeepSeek V4 Pro | Open-source | 0.21 / 0.50 / 0.81 |
| Qwen 3.6 Flash | Open-source | 0.00 / 0.00 / N/A |
| Qwen 3.5 Plus | Open-source | 0.05 / 0.15 / 0.80 |
The best LeetKunpeng and are both achieved by Claude Opus 4.7, with 0.64 and 0.70, respectively. The best is Qwen 3.6 Plus at 0.93, but this coexists with much lower correctness. That pattern is one of the benchmark’s central findings: LeetKunpeng can yield high efficiency among the successful cases even when overall solve rates remain limited.
Cross-platform comparison clarifies the benchmark’s difficulty profile. On BLAS-x86, Claude Opus 4.7 reaches 0, 1, and 2. On LeetKunpeng, the same model obtains 0.64, 0.70, and 0.83. The paper interprets this mixed picture as a consequence of benchmark construction: x86 BLAS is easier to solve correctly because of richer training data, but harder to beat in speed because the baseline is highly tuned; LeetKunpeng is harder to solve correctly because the target platform is less represented, but easier to beat in speed once the output is correct because the baseline is simpler.
The paper also positions LeetKunpeng relative to Sunway. For Claude Opus 4.7, LeetSunway 3, whereas LeetKunpeng 4. This supports the claim that standardized ARM-based architectures are easier for LLMs than highly proprietary architectures, while still remaining substantially harder than x86.
Several failure modes recur in Kunpeng-targeted generation. The clearest is hallucination of non-existent intrinsics. The paper gives svcmul_f64_z in zgemm and svst1_scatter_index_f64 in dsymv as examples of plausible-looking but invalid SVE intrinsic names, which the authors interpret as evidence that models have learned naming patterns without grounding in the valid ACLE intrinsic set. A second recurrent problem is complex arithmetic, especially in BLAS/Kunpeng. The authors attribute degradation to lack of native single-instruction support for complex operations, manual decomposition into real and imaginary components, and difficulty leveraging aarch64 SVE intrinsics for complex arithmetic. Reported slowdowns include at least 5 for zsyr2k, more than 6 for zsymm, and at least 7 for csymm. A third pattern is that longer manual architecture-specific code correlates with lower 8, which particularly affects the Leet subsets.
6. Interpretation, misconceptions, and related Kunpeng-specific work
The paper’s overarching interpretation is that cross-platform code generation degrades as the target moves away from x86, and that the degradation is mediated by the availability of public optimization knowledge and examples rather than by ISA novelty alone. The authors state that the primary obstacle is the scarcity of training data, and more specifically the critical lack of real-world open-source code examples needed to learn how to utilize unique hardware features. In this reading, LeetKunpeng is evidence that standardized ARM semantics and public documentation are not by themselves sufficient for robust architecture-specific generalization (Li et al., 1 Jun 2026).
Two misconceptions are explicitly addressed by the benchmark design. First, a high 9 on LeetKunpeng does not imply parity with production-grade optimized libraries. The Leet suites compare generated kernels against a basic baseline implementation, whereas BLAS tasks are measured against highly tuned libraries such as OpenBLAS and kblas. Second, poor Kunpeng performance should not be interpreted simply as a documentation problem. The paper explicitly notes that Kunpeng has more public optimization tutorials and code examples than Sunway because of the standardized ARM ISA; the claimed bottleneck is the shortage of real-world optimized open-source exemplars.
LeetKunpeng is also best understood in relation to broader Kunpeng-oriented systems work. "KBest: Efficient Vector Search on Kunpeng CPU" does not mention LeetKunpeng, but it provides a concrete example of the kind of Kunpeng-first engineering that LeetKunpeng attempts to elicit from LLMs: NEON and SVE usage, 1-to-0 vectorized operations, pipelined segmented accumulation, software prefetch via prfm PLD1KEEP, huge pages through madvise(MADV_HUGEPAGE), 64-byte aligned memory allocation, graph reordering, and a cumulative throughput gain of over 2x in ablation on Kunpeng 920 (Ma et al., 5 Aug 2025). This suggests that successful LeetKunpeng performance requires exactly the sort of grounded microarchitectural and memory-system knowledge demonstrated by hand-engineered Kunpeng-native software.
A further terminological clarification is useful. The similarly named KunServe is an LLM serving system for GPU-memory throttling, and the paper explicitly states that it provides no textual support for any connection to Huawei Kunpeng CPUs, a Kunpeng ecosystem, or a LeetKunpeng project (Cheng et al., 2024). LeetKunpeng is therefore specifically a benchmark subset for Kunpeng-oriented code generation, not a general Kunpeng software stack, serving runtime, or vendor platform label.