NPUEval Benchmark

• NPUEval is a benchmark that evaluates LLM-generated, optimized NPU kernels by assessing both functional correctness and vectorization efficiency on real hardware.
• It integrates a compiler-in-the-loop approach with HumanEval-style prompts, leveraging LLVM-AIE and cycle-level measurement to guide domain-specific code synthesis.
• The dataset covers 102 machine-learning operators with explicit vector intrinsics and tile-buffer details, highlighting the challenges of fragmentary NPU programming.

Searching arXiv for the NPUEval benchmark and closely related work. {} NPUEval is a benchmark and evaluation harness for measuring the ability of LLMs to generate highly optimized, vectorized kernels for Neural Processing Units. It was introduced in the context of growing deployment of NPUs in power-sensitive client devices, where efficient execution of AI workloads depends on libraries of optimized kernels written in domain-specific C++ with vector intrinsics and data-movement APIs. The benchmark packages prompts, behavioral models, an LLVM-based compiler stack, on-device execution, and cycle-level measurement around an AMD Ryzen 9 7940HS NPU, thereby turning NPU kernel synthesis into an end-to-end, hardware-grounded code-generation problem rather than a purely textual programming task (Kalade et al., 18 Jul 2025).

1. Definition, scope, and motivation

NPUEval is described as the first end-to-end, open-source benchmark and evaluation harness specifically designed to measure the ability of LLMs to generate highly optimized, vectorized kernels for NPUs (Kalade et al., 18 Jul 2025). Its motivation rests on two observations. First, unlike GPUs, NPUs are a relatively new class of device whose programming communities and optimized-kernel examples are highly fragmented across vendors. Second, while LLMs often produce functionally correct high-level code, generating domain-specialized, vectorized C++ kernels remains difficult because such kernels typically rely on vendor-specific intrinsics and explicit data movement (Kalade et al., 18 Jul 2025).

The benchmark targets a setting in which kernel quality cannot be reduced to compilation success or semantic correctness alone. NPUEval therefore evaluates generated code on actual hardware according to both functional correctness and vectorization efficiency. This dual criterion is central to the benchmark’s design: a kernel that merely passes tests through scalar looping is treated differently from one that meaningfully utilizes the vector processing unit (VPU) (Kalade et al., 18 Jul 2025).

A plausible implication is that NPUEval situates NPU programming closer to systems-compiler evaluation than to conventional code-generation benchmarks. The benchmark couples prompt engineering, compilation, graph construction, hardware execution, and performance instrumentation into a single experimental protocol, which is why it functions as both a dataset and an executable measurement environment.

2. Dataset composition and operator coverage

NPUEval consists of a dataset of 102 “common” machine-learning operators expressed as HumanEval-style C++ kernel prompts, together with reference behavioral models in Python and test vectors (Kalade et al., 18 Jul 2025). Each prompt provides a C++ function signature and docstring describing kernel semantics, example inputs and outputs, explicit data-movement information such as tile buffer sizes, and a NumPy or JAX behavioral model for correctness checking; an optional canonical C++ reference may also be included (Kalade et al., 18 Jul 2025).

The dataset spans eight operator categories.

Category	Approximate count and examples
Convolutions	≈25; conv2dk1_i8, depthwise 3×3 conv, dilated conv
Matrix-multiply/GEMM	≈10; fp16 GEMM, int8-by-int8 matrix multiply with accumulation
Activations	≈15; ReLU, LeakyReLU, Sigmoid, Tanh, Softmax, Swish
Pooling	≈8; MaxPool2D, AvgPool2D, global pooling variants
Element-wise arithmetic and reductions	≈20; Add/Sub/Mul/Div, Abs, Sum, Mean, Argmax
Normalizations	≈8; BatchNorm, LayerNorm
Data-movement/transforms	≈10; Identity, Transpose, Pack/Unpack
Miscellaneous	≈6; FFT-style shuffles, Gather/Scatter

This coverage matters because the operators are not limited to a single kernel family such as GEMM or convolution. Instead, the benchmark includes arithmetic, reductions, transforms, and data-movement kernels that stress different aspects of the NPU toolchain and different programming idioms in the AIE API. The inclusion of explicit tile-buffer information also means that generated kernels are embedded in a concrete dataflow setting rather than evaluated as isolated source fragments (Kalade et al., 18 Jul 2025).

The benchmark’s use of HumanEval-style prompts is also significant. It aligns the task with established LLM code-generation workflows while preserving the low-level constraints of NPU programming. This suggests that NPUEval is intended not only as a hardware benchmark but also as a bridge between mainstream LLM evaluation paradigms and vendor-specific accelerator programming.

3. Evaluation harness, hardware execution, and metrics

NPUEval’s evaluation methodology has two components: functional correctness on real hardware and a vectorization efficiency metric (Kalade et al., 18 Jul 2025).

For correctness, the generated C++ kernel is compiled with LLVM-AIE, described as a fork of LLVM customized for AMD AIE intrinsics. MLIR-AIE and IRON are then used to assemble an on-chip dataflow graph, configure each AIE tile’s ports and memories, and load the kernel. A Python driver pushes test vectors into the NPU, runs the kernel, and compares the outputs against the Python behavioral model using absolute or relative error tolerances. The summary specifies typical tolerances of $1 \times 10^{-2}$ for integers, $2 \times 10^{-2}$ for low-precision floating point, and $3 \times 10^{-2}$ for complex activations. Compilation failures or incorrect outputs are counted as non-passing (Kalade et al., 18 Jul 2025).

For performance, NPUEval instruments cycle counters and measures total cycles and vector-instruction cycles: $$C_{\text{total}} = \text{total cycles to execute the kernel on the test input}$$

$$C_{\text{vector}} = \text{cycles spent issuing vector (VPU) instructions}$$

The vectorization score is then defined as

$$V = (C_{\text{vector}} / C_{\text{total}}) \times 100\%.$$

Kernels that fail correctness are assigned $V = 0\%$ (Kalade et al., 18 Jul 2025).

In practice, state-of-the-art hand-tuned open-source kernels achieve $V$ in the $10\text{–}30\%$ range on this hardware (Kalade et al., 18 Jul 2025). That calibration is important because it anchors the benchmark’s performance metric in a realistic ceiling derived from available open-source kernels rather than from a synthetic notion of perfect utilization.

A recurrent misconception in accelerator-code evaluation is that passing semantic tests is sufficient evidence of useful code generation. NPUEval explicitly rejects that assumption. The benchmark records that almost every LLM can be coaxed into passing functional tests by falling back to simple scalar loops, but those solutions yield $V \approx 0\text{–}5\%$ (Kalade et al., 18 Jul 2025). Functional correctness is therefore necessary but not sufficient.

4. Compiler stack, prompting protocol, and retrieval-augmented generation

NPUEval relies on an open-source compiler toolchain comprising LLVM-AIE as compiler and MLIR-AIE plus IRON as graph builder (Kalade et al., 18 Jul 2025). The benchmark emphasizes that there are no magic auto-vectorize flags: vectorization must be expressed through C++ AIE APIs and intrinsics such as aie::load_v, aie::store_v, aie::vector<>, and aie::mac (Kalade et al., 18 Jul 2025). This design choice makes the task materially different from conventional C++ optimization, since the model must emit architecture-specific vector code rather than generic scalar code that a backend later optimizes automatically.

The generation pipeline is compiler-in-the-loop. During generation, the same LLVM-AIE compiler is invoked on each LLM draft, and compilation errors are sent back verbatim as feedback. Each model is allowed up to ten compile-correct-compile iterations, with diminishing returns observed after approximately five iterations (Kalade et al., 18 Jul 2025). The evaluation overview in the summary is: LLM $2 \times 10^{-2}$0 post-processing $2 \times 10^{-2}$1 LLVM-AIE compile $2 \times 10^{-2}$2 MLIR-AIE graph build $2 \times 10^{-2}$3 on-device execution $2 \times 10^{-2}$4 correctness plus cycle counters $2 \times 10^{-2}$5 feedback (Kalade et al., 18 Jul 2025).

The prompting protocol uses a system prompt that insists on “only produce a single, self-contained C++ kernel function with correct signature, imports, and intrinsics—no main(), no markdown fences, no English explanations.” Generation uses temperature $2 \times 10^{-2}$6 for greedy decoding, except that reasoning-capable variants are set to $2 \times 10^{-2}$7 by default; random seed is locked to $2 \times 10^{-2}$8 where supported (Kalade et al., 18 Jul 2025).

Retrieval-Augmented Generation is implemented by indexing approximately 200 open-source vectorized AIE kernels, manually filtered to remove scalar examples, using text-embedding-ada-002 and LlamaIndex. Two nearest-neighbor vectorized kernels are appended to each user prompt (Kalade et al., 18 Jul 2025). The paper states that RAG improves both compilation success and VPU utilization for most models (Kalade et al., 18 Jul 2025). At the same time, the model-by-model pass-rate table is heterogeneous, so a cautious reading is that the effect is pipeline- and model-dependent rather than uniformly monotone.

The benchmark also evaluates reasoning-oriented models. For models such as DeepSeek R1 that support multi-turn reasoning, better planning of loop bounds and broadcast patterns is reported, but API hallucinations remain if compiler feedback is omitted (Kalade et al., 18 Jul 2025). This suggests that reasoning alone is insufficient in a domain where the error surface is constrained by exact compiler legality and hardware APIs.

5. Model evaluations and quantitative findings

NPUEval evaluates a mix of proprietary and open-weight models, including GPT-4.1, GPT-4o and Mini variants, Claude 3.5 Haiku, Claude 3.7 Sonnet, Meta Llama 3.1-405B and 70B, DeepSeek R1 and V3, and Qwen2.5-Coder (Kalade et al., 18 Jul 2025). The benchmark reports both correctness pass-rates and vectorization outcomes.

Selected functional-correctness pass-rates are reported for settings without RAG and with RAG under 0 to 5 recompilations (Kalade et al., 18 Jul 2025).

Model	Pass-rate progression
Claude 3.7 Sonnet	21.6%→73.5% (no RAG), 23.5%→70.6% (RAG)
GPT-4.1	29.4%→71.6% (no RAG), 22.5%→58.8% (RAG)
DeepSeek R1	20.6%→38.2% (no RAG), 19.6%→29.4% (RAG)
Qwen2.5-Coder	50.0%→68.6% (no RAG), 6.9%→13.7% (RAG)
GPT-4o Mini	58.8%→66.7% (no RAG), 26.5%→35.3% (RAG)

The paper also reports out-of-the-box vectorization behavior on select kernels. DeepSeek R1 achieved $2 \times 10^{-2}$9 vectorization on certain activations such as ReLU_bf16 in zero-shot mode, but averaged only approximately $3 \times 10^{-2}$0 across the full 102-kernel suite. GPT-4.1 increased its mean vectorization score from approximately $3 \times 10^{-2}$1 in zero-shot mode to approximately $3 \times 10^{-2}$2 with RAG plus compiler feedback. Across all models and kernels, average $3 \times 10^{-2}$3 hovers near $3 \times 10^{-2}$4, and the dataset-wide average remains approximately $3 \times 10^{-2}$5 even for frontier models (Kalade et al., 18 Jul 2025).

These results establish a characteristic pattern. Stronger models such as GPT-4o, Claude 3.7, and DeepSeek R1 often attempt vector intrinsics out of the box and sometimes reach $3 \times 10^{-2}$6 on easier kernels, but they are also more prone to hallucinated APIs and compilation failures. Smaller or code-tuned models such as GPT-4o Mini and Claude 3.5 are described as delivering higher pass-rates with “boring” scalar code, indicating a trade-off between reliability and performance ambition (Kalade et al., 18 Jul 2025).

The benchmark includes a concrete qualitative comparison between a scalar passthrough kernel and a vectorized 64-byte load/store kernel (Kalade et al., 18 Jul 2025). That contrast captures the central empirical distinction in the benchmark: not whether a model can write code that runs, but whether it can synthesize architecture-appropriate vectorized code that exploits the NPU execution model.

6. Interpretation, limitations, and projected extensions

NPUEval’s main conclusion is that NPU kernel generation remains difficult even for state-of-the-art models. The abstract states that latest reasoning models such as DeepSeek R1 show promising results, achieving out-of-the-box $3 \times 10^{-2}$7 vectorization on select kernels, but that the average score across the entire dataset remains roughly $3 \times 10^{-2}$8 even with compiler feedback and vectorized kernel examples (Kalade et al., 18 Jul 2025). The benchmark therefore functions less as a leaderboard for solved performance and more as a stress test for the current limits of LLM-driven low-level code generation.

One important interpretive point is that the benchmark separates two failure modes that are often conflated. The first is inability to satisfy syntax, type, or API constraints, which surfaces as compilation failure. The second is inability to produce code with substantive VPU utilization, which surfaces even when the kernel is functionally correct. This separation makes NPUEval useful for diagnosing whether improvements come from better compiler compliance, better semantic reasoning about tensor operations, or better architectural understanding of vector intrinsics.

The benchmark also exposes a tension between open-source compiler tooling and fragmented vendor ecosystems. LLVM-AIE, MLIR-AIE, and IRON make the AMD NPU target experimentally accessible, yet the paper notes that optimized-kernel examples are underrepresented in LLM pre-training data and fragmented across hardware platforms (Kalade et al., 18 Jul 2025). This suggests that the benchmark measures not only raw model capability but also the sparsity of domain-specific training evidence available to present-day LLMs.

Future directions identified in the paper include extending NPUEval to other NPU families such as Google Edge TPU and Apple Neural Engine, building compiler-backend-specific RAG to surface correct pragmas and intrinsics for each toolchain, and integrating more sophisticated chain-of-thought prompting and agentic multi-kernel composition workflows (Kalade et al., 18 Jul 2025). The dataset and evaluation code are to be released with a permissive open-source license (Kalade et al., 18 Jul 2025), and the benchmark is positioned as infrastructure for research on code generation and NPU kernel optimization rather than merely as a static collection of prompts.

In that sense, NPUEval occupies a specific place in the recent benchmarking landscape. It turns NPU kernel synthesis into a reproducible, compiler-backed, hardware-validated evaluation problem with explicit attention to vectorization, making it a benchmark for optimization quality as much as for code correctness (Kalade et al., 18 Jul 2025).