LeetSunway: Sunway Code Generation Benchmark

• LeetSunway is a benchmark tailored for Sunway architectures, featuring 20 curated computing kernels to test LLMs on generating optimized, parallel code.
• It evaluates models on their ability to exploit Sunway’s master-slave core paradigm, thread synchronization, and memory strategies, measuring performance against baseline implementations.
• The benchmark exposes recurrent LLM failure modes such as low-level API hallucinations and challenges in mapping algorithm semantics to the unique Sunway processor design.

Searching arXiv for the benchmark and related paper so the article can be grounded in the cited preprint. LeetSunway is the Sunway-specific component of CodegenBench, a benchmark suite for evaluating whether LLMs can generate efficient parallel code across heterogeneous CPU-oriented high-performance computing platforms. In the cited literature, it is defined as “a meticulously curated problem set consisting of exactly 20 typical and frequently utilized computation kernels tailored specifically for the Sunway architecture.” Its purpose is not limited to functional code synthesis: each task is intended to test whether generated code can exploit the Sunway platform’s master-slave core paradigm, thread synchronization requirements, and memory access strategies in a way that yields measurable performance on the target hardware (Li et al., 1 Jun 2026).

1. Position within CodegenBench

CodegenBench is divided into three primary components: a BLAS suite, LeetSunway, and LeetKunpeng. The BLAS portion provides a broad baseline over 106 standard Basic Linear Algebra Subprograms routines, while LeetSunway and LeetKunpeng are architecture-specific subsets targeting the Sunway and Kunpeng architectures, respectively (Li et al., 1 Jun 2026).

Within that structure, LeetSunway serves a distinct role. The BLAS tasks evaluate code generation against standardized numerical kernels and strong optimized libraries, whereas LeetSunway evaluates the generation of tailored implementations for a non-x86, non-CUDA, CPU-focused supercomputing architecture. The benchmark therefore separates two capabilities that are often conflated: generic numerical code synthesis and architecture-aware HPC code generation.

This distinction is central to the interpretation of the benchmark. In the BLAS setting, outputs are compared against highly hand-tuned standard libraries such as OpenBLAS. In LeetSunway, by contrast, the generated code is measured relative to baseline CPU reference implementations. The paper explicitly argues that this makes LeetSunway harder in terms of correctness and architecture porting, even though speedup over the baseline can become relatively easy once a semantically correct architecture-aware implementation exists (Li et al., 1 Jun 2026).

2. Sunway architectural substrate

LeetSunway is anchored in the Sunway execution environment, specifically the SW26010 many-core processor. The paper describes each processor chip as containing four Core Groups, and each Core Group as containing one Management Processing Element, or MPE, together with 64 Computing Processing Elements, or CPEs. The MPE is responsible for control and task scheduling, while the CPE cluster is dedicated to intensive parallel data processing (Li et al., 1 Jun 2026).

The evaluation environment for Sunway tasks is the Sunway TaihuLight supercomputer. The reported infrastructure consists of 40,960 SW26010 many-core processors with an aggregate memory capacity of 1,310,720 GB. For LeetSunway, all test cases are compiled using the swgcc compiler in order to utilize the master-slave paradigm (Li et al., 1 Jun 2026).

The benchmark’s difficulty follows directly from this hardware model. The paper characterizes Sunway as requiring explicit handling of distributed memory, master-slave orchestration, thread synchronization, and low-level tuning. It also contrasts Sunway with x86_64 and Kunpeng. Relative to x86_64, Sunway is described as less standardized and much less represented in public code. Relative to Kunpeng, which is ARM-based, Sunway is presented as more proprietary and more dependent on architecture-specific implementation knowledge (Li et al., 1 Jun 2026).

A notable nuance is that the paper contains a mild internal tension on documentation availability. One passage states that extensive architecture documentation exists in the public domain for platforms like Kunpeng and Sunway, whereas other passages stress scarcity of documentation and proprietary instruction sets. The most consistent interpretation is that architectural descriptions may exist publicly, but optimized, real-world open-source exemplars remain scarce; the paper repeatedly identifies that scarcity as a major source of model failure.

3. Task composition and curation

LeetSunway contains exactly 20 kernels. These are described as “typical and frequently utilized computation kernels” and are tailored specifically for the Sunway architecture. The benchmark does not present a full task list in the cited text, but it does state that the covered kernels span “from linear algebra computation to machine learning optimizer” (Li et al., 1 Jun 2026).

The paper also states that the 20 computing kernels in LeetKunpeng are identical to those formulated for LeetSunway. This means that the two architecture-specific subsets share the same algorithmic task set while differing in target architecture and implementation constraints. A plausible implication is that CodegenBench isolates architecture adaptation from task semantics more cleanly than a benchmark in which each platform would use unrelated kernels.

LeetSunway is not described as a collection of free-form prompts alone. CodegenBench is said to include reference implementations, baseline CPU reference implementations, and platform-specific tasks. The prompting system aggregates metadata together with source code snippets, documentation, or problem descriptions from related files, and uses these materials to construct a comprehensive prompt for the model. The framework is further described as allowing prompt templates and metadata schemas to be dynamically adjusted or enhanced (Li et al., 1 Jun 2026).

This curation strategy is significant because LeetSunway is intended to test more than the recovery of familiar API idioms. The model is expected to translate baseline counterparts into Sunway-targeted implementations that exploit native hardware features. The benchmark is therefore positioned at the intersection of program synthesis, architecture-specific optimization, and code porting.

4. Evaluation workflow and measurement

The LeetSunway evaluation pipeline follows the general CodegenBench workflow: pre-processing and workspace reset, prompt generation and LLM code generation, build, and run plus validation. During build, the system parses and executes compilation directives specified in the setup metadata. During execution, runtime parameters are injected according to the metadata. Numerical correctness is validated using tolerance-based comparisons against expected outputs (Li et al., 1 Jun 2026).

Three metrics are used: $Pass@1$, $Pass@5$, and $Fast\_1@1$. In the paper’s description, $Pass@1$ measures whether a task passes all test cases in one generation, $Pass@5$ measures whether it passes within five generation attempts, and $Fast\_1@1$ measures the fraction of first-attempt valid tasks that also achieve speedup greater than 1 over the baseline. The appendix states that the sample size is 5, matching the use of $Pass@5$ (Li et al., 1 Jun 2026).

LeetSunway’s efficiency criterion is important. Unlike BLAS tasks, whose performance is judged against strong tuned libraries, LeetSunway evaluates speedup relative to the baseline CPU reference implementation. This design emphasizes whether the model can activate architecture-specific performance mechanisms at all, rather than whether it can match expert library engineering.

All evaluated models were used in a zero-shot setting with a short system instruction asking the model to answer in code with proper optimizing mechanism applied. No task-specific fine-tuning was reported. The nine evaluated models were Claude Sonnet 4.6, Claude Opus 4.6, Claude Opus 4.7, Qwen 3.6 Plus, DeepSeek V3.2, DeepSeek V4 Flash, DeepSeek V4 Pro, Qwen 3.6 Flash, and Qwen 3.5 Plus (Li et al., 1 Jun 2026).

5. Empirical results on LeetSunway

The LeetSunway results show a pronounced separation between correctness and conditional speedup. The best reported $Pass@1$ is 0.48 for Claude Opus 4.7. The best reported $Pass@5$ is 0.50, achieved by both Claude Opus 4.6 and Claude Opus 4.7. Several open-source models score 0.00 on both $Pass@1$ and $Pass@5$0, including DeepSeek V3.2, DeepSeek V4 Pro, Qwen 3.6 Flash, and Qwen 3.5 Plus (Li et al., 1 Jun 2026).

Model	$Pass@5$1	$Pass@5$2
Claude Sonnet 4.6	0.06	0.15
Claude Opus 4.6	0.33	0.50
Claude Opus 4.7	0.48	0.50
Qwen 3.6 Plus	0.22	0.35
DeepSeek V3.2	0.00	0.00
DeepSeek V4 Flash	0.05	0.20
DeepSeek V4 Pro	0.00	0.00
Qwen 3.6 Flash	0.00	0.00
Qwen 3.5 Plus	0.00	0.00

For the subset of models with nonzero valid tasks, the reported $Pass@5$3 values are strikingly high: 1.00 for Claude Sonnet 4.6, 1.00 for Claude Opus 4.6, 0.90 for Claude Opus 4.7, 0.91 for Qwen 3.6 Plus, and 1.00 for DeepSeek V4 Flash. The paper highlights this as a “compelling anomaly”: absolute generation correctness remains poor, but the few successful generations often exceed the baseline speed threshold (Li et al., 1 Jun 2026).

The authors’ interpretation is explicit. LeetSunway is difficult because most generations are not usable at all, yet once a model produces semantically correct code that activates native hardware features, outperforming the unoptimized baseline CPU code becomes relatively easy. This suggests that the dominant bottleneck is architecture-grounded correctness rather than marginal micro-optimization.

6. Failure modes, interpretive cautions, and broader significance

The benchmark is used to expose several recurrent failure modes in current LLM code generation. One is hallucination of low-level architecture-specific APIs or intrinsics. The paper gives examples on other architectures, such as svcmul_f64_z and _mm256_reduce_add_pd, to illustrate that models often produce plausible but nonexistent primitives. Although no Sunway-specific hallucinated intrinsic is listed in the cited text, the paper’s broader conclusion is that this problem becomes most severe on Sunway because public code examples and documentation are scarcest there (Li et al., 1 Jun 2026).

A second failure mode is inability to map algorithmic semantics into the Sunway parallel structure. For LeetSunway, that mapping requires MPE/CPE orchestration, synchronization, memory routing, and explicit use of the master-slave model. A third is the burden of prolonged architecture-specific code length; the discussion states that $Pass@5$4 scores in the Leet series remained uniformly low, likely exacerbated by the longer implementations needed for manual architecture-specific coding together with the lack of open-access reference material (Li et al., 1 Jun 2026).

Several misconceptions are preempted by the reported results. High $Pass@5$5 should not be read as evidence that LeetSunway is an easy benchmark. The paper argues the opposite: correctness is the principal obstacle, and most generated programs remain unusable. Nor should LeetSunway be understood as a conventional library-binding benchmark. Its point is to test specialized code optimization, parallel computing logic, and architecture-specific porting under real hardware constraints.

In the broader argument of CodegenBench, LeetSunway functions as the strongest evidence for a cross-platform generalization gap. The paper concludes that performance degradation is most severe on the Sunway architecture, which has the least public documentation and code examples. In that sense, LeetSunway is not merely a Sunway benchmark; it is a stress test for whether LLM-based code generation can operate beyond mainstream, heavily documented software-hardware ecosystems (Li et al., 1 Jun 2026).