---
title: 'LeetKunpeng: ARM HPC Code Benchmark'
url: https://www.emergentmind.com/topics/leetkunpeng
type: topic
---

# LeetKunpeng: ARM HPC Code Benchmark

LeetKunpeng is the Kunpeng-specific specialized-kernel suite within "CodegenBench: Can LLMs Write Efficient Code Across Architectures?" and is intended to test whether large language models can generate not only correct code but efficient, architecture-aware parallel code for a non-x86, CPU-oriented HPC platform. Within CodegenBench, it complements a broad BLAS baseline and the Sunway-oriented LeetSunway subset, and targets the Kunpeng ARM-based platform, whose salient features in the benchmark context are Scalable Vector Extension, Scalable Matrix Extension, and on-die High Bandwidth Memory [2606.04023].

## 1. Position within CodegenBench

CodegenBench is organized into three primary parts: BLAS, LeetSunway, and LeetKunpeng. The benchmark comprises **106 standard BLAS routines** as a general baseline and **20 specialized computational kernels adapted for each** of the two supercomputing architectures, namely LeetSunway and LeetKunpeng. LeetKunpeng is therefore not a standalone benchmark family detached from CodegenBench; it is one of the two specialized supercomputer-oriented subsets designed to probe architecture-specific code generation beyond standardized BLAS workloads [2606.04023].

The rationale for introducing LeetKunpeng is explicit. The benchmark was constructed to address whether LLMs can generate efficient code for **non-x86, CPU-oriented HPC architectures with architecture-specific optimization requirements**. The paper identifies three pressures that distinguish this setting from mainstream code generation evaluation: **portability**, because models must port baseline code while preserving or improving performance; **architecture-awareness**, because models must exploit nonstandard hardware features; and **data scarcity**, because such platforms have less public code and fewer examples.

Kunpeng occupies a deliberate middle position in this design. In the authors’ framing, **x86_64** is comparatively easy because it is common and well represented in training data, while **Sunway** is the hardest target because it is highly specialized and sparse in public examples. **Kunpeng** is presented as a middle-ground test of cross-architecture generalization: harder than x86 because it requires ARM/Kunpeng-specific optimization, but less exotic than Sunway.

## 2. Kunpeng as the target architecture

In the benchmark, Kunpeng is described as an **ARM-based processor family** with HPC-oriented features including **on-die High Bandwidth Memory (HBM)**, **Scalable Vector Extension (SVE)**, and **Scalable Matrix Extension (SME)**. In the appendix’s evaluation environment, the Kunpeng platform is further described as a **cluster of multiple interconnected nodes**, each node equipped with **dual Kunpeng processors** exposing **SVE/SVE2 and SME** and providing **on-die HBM** [2606.04023].

This architectural framing is central to what LeetKunpeng measures. The paper explicitly contrasts Kunpeng with Sunway by noting that Kunpeng has **more public optimization tutorials and more code examples** because of the standardized ARM ISA, even if it is less prevalent than x86 in conventional HPC centers. This suggests that failure on LeetKunpeng cannot be reduced to total absence of public documentation. Rather, Kunpeng is used to test whether standardized ARM semantics and somewhat better public information are sufficient for LLMs to internalize platform-specific optimization patterns.

The benchmark also emphasizes concrete Kunpeng-specific optimization opportunities. These include vectorization via **SVE**, **SVE2**, and **SME**, and the use of **on-die HBM** for programs addressing **memory-bound bottleneck** behavior. The attention case study is used as an example of this regime: the model selected proper **SVE intrinsics**, and the generated implementation achieved **at least \(1.6\times\) speedup in FLOPS**.

## 3. Benchmark content and task design

LeetKunpeng contains **20 computation kernels**, and its most distinctive structural property is that it **mirrors LeetSunway exactly in terms of the algorithmic tasks required**. The paper states that the 20 computing kernels in LeetKunpeng are identical to those formulated for LeetSunway. LeetKunpeng is therefore not a separate algorithmic problem set; it is the same problem set evaluated under **Kunpeng-specific implementation and optimization expectations** [2606.04023].

The paper does not enumerate all 20 kernels in the excerpted text, but it does characterize their scope. The specialized suites cover a wide range of kernels, **from linear algebra computation to machine learning optimizer**. One explicit LeetKunpeng example is an **attention kernel**, which demonstrates that the suite is not confined to classical BLAS-like routines.

This design has two consequences. First, it makes LeetKunpeng a portability test: algorithmic content is held fixed while implementation strategy must change with the target architecture. Second, it isolates architecture-awareness more cleanly than a benchmark that changes both the problem set and the hardware target at once. A common misconception is that LeetKunpeng introduces Kunpeng-specific algorithms; the benchmark instead keeps the algorithmic tasks aligned with LeetSunway and shifts the burden to hardware-specific realization.

The authors also note a limitation: **20 kernels may be insufficient** to fully represent Kunpeng and Sunway workloads. That caveat matters when interpreting LeetKunpeng as a probe of architecture-aware code generation rather than as an exhaustive workload model of Kunpeng-based HPC.

## 4. Evaluation pipeline, toolchain, and metrics

CodegenBench operationalizes LeetKunpeng through a four-stage pipeline: **pre-processing**, **prompt generation and code generation**, **build**, and **run and validation**. Pre-processing cleans temporary files, initializes a deterministic environment, and sets environment variables. Prompt generation gathers metadata, instructions, problem descriptions, source snippets, and documentation, constructs a comprehensive prompt, and queries the target LLM. The build stage compiles generated code with the benchmark harness and detects syntax errors, hallucinated APIs, and missing dependencies. The final stage executes programs with metadata-defined runtime parameters, collects outputs and logs, and validates numerically using tolerance-based comparisons [2606.04023].

The Kunpeng toolchain differs between the BLAS and Leet portions. For **BLAS/Kunpeng**, the benchmark uses **bisheng**, a customized clang compiler, and links with **kblas**. For **LeetKunpeng**, evaluation uses the **clang compiler**, and the appendix states that evaluations explicitly utilize the **specialized on-die HBM feature in Kunpeng chip**. This distinction is methodologically important because BLAS/Kunpeng is benchmarked against a tuned library stack, whereas LeetKunpeng evaluates architecture-targeted kernel generation more directly.

The benchmark reports three metrics: **\(Pass@1\)**, **\(Pass@5\)**, and **\(Fast\_1@1\)**. In the paper’s interpretation, \(Pass@1\) is the fraction of tasks solved correctly on the first try, \(Pass@5\) is the fraction solved within five tries, and \(Fast\_1@1\) measures, among tasks that are correct on the first try, the fraction whose generated code is actually faster than the baseline. A generated kernel counts as “fast” when its **speedup is greater than 1**.

LeetKunpeng should not be conflated with the BLAS portion. BLAS covers **106 distinct BLAS subroutines** across Levels 1–3 and is evaluated against optimized baselines such as **OpenBLAS** on x86 and **kblas** on Kunpeng. By contrast, LeetKunpeng contains **20 specialized kernels**, is **architecture-targeted**, expects use of **Kunpeng-specific features**, and is compared against a **basic baseline implementation** rather than an industrial-strength hand-tuned library. This explains why LeetKunpeng can exhibit relatively poor correctness yet high efficiency among successful samples.

## 5. Empirical results and characteristic failure modes

Table 3 reports the following LeetKunpeng results for the evaluated models [2606.04023].

| Model | Category | Pass@1 / Pass@5 / Fast_1@1 |
|---|---|---|
| Claude Sonnet 4.6 | Closed-source | 0.49 / 0.65 / 0.84 |
| Claude Opus 4.6 | Closed-source | 0.58 / 0.65 / 0.90 |
| Claude Opus 4.7 | Closed-source | 0.64 / 0.70 / 0.83 |
| Qwen 3.6 Plus | Closed-source | 0.14 / 0.30 / 0.93 |
| DeepSeek V3.2 | Open-source | 0.00 / 0.00 / N/A |
| DeepSeek V4 Flash | Open-source | 0.11 / 0.35 / 0.64 |
| DeepSeek V4 Pro | Open-source | 0.21 / 0.50 / 0.81 |
| Qwen 3.6 Flash | Open-source | 0.00 / 0.00 / N/A |
| Qwen 3.5 Plus | Open-source | 0.05 / 0.15 / 0.80 |

The best **LeetKunpeng \(Pass@1\)** and **\(Pass@5\)** are both achieved by **Claude Opus 4.7**, with **0.64** and **0.70**, respectively. The best **\(Fast\_1@1\)** is **Qwen 3.6 Plus** at **0.93**, but this coexists with much lower correctness. That pattern is one of the benchmark’s central findings: LeetKunpeng can yield high efficiency among the successful cases even when overall solve rates remain limited.

Cross-platform comparison clarifies the benchmark’s difficulty profile. On **BLAS-x86**, Claude Opus 4.7 reaches **\(Pass@1 = 0.74\)**, **\(Pass@5 = 0.85\)**, and **\(Fast\_1@1 = 0.52\)**. On **LeetKunpeng**, the same model obtains **0.64**, **0.70**, and **0.83**. The paper interprets this mixed picture as a consequence of benchmark construction: x86 BLAS is easier to solve correctly because of richer training data, but harder to beat in speed because the baseline is highly tuned; LeetKunpeng is harder to solve correctly because the target platform is less represented, but easier to beat in speed once the output is correct because the baseline is simpler.

The paper also positions LeetKunpeng relative to Sunway. For Claude Opus 4.7, **LeetSunway \(Pass@1 = 0.48\)**, whereas **LeetKunpeng \(Pass@1 = 0.64\)**. This supports the claim that standardized ARM-based architectures are easier for LLMs than highly proprietary architectures, while still remaining substantially harder than x86.

Several failure modes recur in Kunpeng-targeted generation. The clearest is **hallucination of non-existent intrinsics**. The paper gives `svcmul_f64_z` in **zgemm** and `svst1_scatter_index_f64` in **dsymv** as examples of plausible-looking but invalid SVE intrinsic names, which the authors interpret as evidence that models have learned naming patterns without grounding in the valid ACLE intrinsic set. A second recurrent problem is **complex arithmetic**, especially in BLAS/Kunpeng. The authors attribute degradation to lack of native single-instruction support for complex operations, manual decomposition into real and imaginary components, and difficulty leveraging **aarch64 SVE intrinsics for complex arithmetic**. Reported slowdowns include at least **\(10\times\)** for `zsyr2k`, more than **\(10\times\)** for `zsymm`, and at least **\(6\times\)** for `csymm`. A third pattern is that longer manual architecture-specific code correlates with lower \(Pass@1\), which particularly affects the Leet subsets.

## 6. Interpretation, misconceptions, and related Kunpeng-specific work

The paper’s overarching interpretation is that cross-platform code generation degrades as the target moves away from x86, and that the degradation is mediated by the availability of public optimization knowledge and examples rather than by ISA novelty alone. The authors state that the primary obstacle is **the scarcity of training data**, and more specifically the **critical lack of real-world open-source code examples** needed to learn how to utilize unique hardware features. In this reading, LeetKunpeng is evidence that standardized ARM semantics and public documentation are not by themselves sufficient for robust architecture-specific generalization [2606.04023].

Two misconceptions are explicitly addressed by the benchmark design. First, a high **\(Fast\_1@1\)** on LeetKunpeng does **not** imply parity with production-grade optimized libraries. The Leet suites compare generated kernels against a **basic baseline implementation**, whereas BLAS tasks are measured against highly tuned libraries such as **OpenBLAS** and **kblas**. Second, poor Kunpeng performance should not be interpreted simply as a documentation problem. The paper explicitly notes that Kunpeng has more public optimization tutorials and code examples than Sunway because of the standardized ARM ISA; the claimed bottleneck is the shortage of real-world optimized open-source exemplars.

LeetKunpeng is also best understood in relation to broader Kunpeng-oriented systems work. "KBest: Efficient Vector Search on Kunpeng CPU" does **not mention LeetKunpeng**, but it provides a concrete example of the kind of Kunpeng-first engineering that LeetKunpeng attempts to elicit from LLMs: **NEON** and **SVE** usage, **1-to-\(B\) vectorized operations**, **pipelined segmented accumulation**, software prefetch via **`prfm PLD1KEEP`**, huge pages through **`madvise(MADV_HUGEPAGE)`**, **64-byte aligned memory allocation**, graph reordering, and a cumulative throughput gain of **over 2x** in ablation on **Kunpeng 920** [2508.03016]. This suggests that successful LeetKunpeng performance requires exactly the sort of grounded microarchitectural and memory-system knowledge demonstrated by hand-engineered Kunpeng-native software.

A further terminological clarification is useful. The similarly named **KunServe** is an LLM serving system for GPU-memory throttling, and the paper explicitly states that it provides **no textual support** for any connection to Huawei Kunpeng CPUs, a Kunpeng ecosystem, or a LeetKunpeng project [2412.18169]. LeetKunpeng is therefore specifically a **benchmark subset for Kunpeng-oriented code generation**, not a general Kunpeng software stack, serving runtime, or vendor platform label.

Source: https://www.emergentmind.com/topics/leetkunpeng