SimdBench: Benchmark for SIMD Intrinsic Code Generation

• SimdBench is a benchmark for evaluating SIMD-intrinsic code generation with 136 tasks spanning SSE, AVX, Neon, SVE, and RVV.
• It pairs differential testing against scalar baselines with performance measurement using the Google Benchmark library to assess speedup.
• The benchmark highlights challenges in intrinsic programming including interface complexity, manual data alignment, and masking-heavy control flow that LLMs struggle to overcome.

Searching arXiv for SimdBench and related SIMD-intrinsic code generation papers. SimdBench is a benchmark for evaluating LLMs on SIMD-intrinsic code generation: the production of correct and performant vectorized C/C++ implementations using explicit SIMD intrinsics rather than ordinary scalar code. It was introduced as the first benchmark specifically designed for this setting, with 136 tasks spanning SSE, AVX, Neon, SVE, and RVV, and it evaluates both functional correctness and execution performance under architecture-specific compilation and runtime conditions (He et al., 21 Jul 2025). Its motivation is that existing code-generation benchmarks largely ignore low-level hardware features, so they do not capture the real difficulty of programming with intrinsics, where interface complexity, alignment and memory-layout constraints, and masking-heavy control flow are central. The name should be distinguished from unrelated benchmarks titled “SimBench” in human-behavior simulation and digital-twin generation (Hu et al., 20 Oct 2025, Wang et al., 2024).

1. Motivation and problem setting

SimdBench was proposed to measure whether LLMs can assist with explicit SIMD programming, a domain in which developers trade coding productivity for high performance by directly invoking architecture-specific intrinsic APIs (He et al., 21 Jul 2025). The benchmark is motivated by three specific challenges of SIMD programming identified in the paper: the complexity and poor readability of intrinsic interfaces, the need for manual data alignment and memory-layout handling, and the need to express complex control and data flow such as masking.

These difficulties make SIMD intrinsic programming materially different from generic code generation. A scalar benchmark can reward syntactic plausibility or high-level functional correctness without probing whether a model understands register width, lane type, masking semantics, or target-ISA naming conventions. SimdBench was therefore designed to test whether an LLM can generate correct, fast, architecture-aware SIMD intrinsic code under conditions closer to explicit vectorization practice than ordinary benchmark suites.

A further motivation is comparative evaluation. The paper argues that code-generation benchmarks had focused on scalar code, leaving unclear how LLMs perform when the target is not merely correct code, but vectorized code that must preserve semantics while exploiting hardware-specific SIMD support. SimdBench fills that gap by pairing correctness testing with performance testing for the same tasks.

2. Coverage across instruction sets, task sources, and data types

SimdBench contains 136 tasks targeting five representative SIMD intrinsic families across mainstream processor ecosystems (He et al., 21 Jul 2025).

Intrinsic family	Architecture	Model characteristics
SSE	x86	128-bit fixed-width SIMD
AVX	x86	256-bit and 512-bit variants
Neon	ARM	128-bit fixed-width
SVE	ARM	Scalable from 128 to 2048 bits in 128-bit increments
RVV	RISC-V	Vector-length-agnostic model whose length depends on hardware

This coverage is significant because the targeted families differ substantially in width, masking, and programming model. Fixed-width ISAs such as SSE and Neon impose one class of coding constraints, while SVE and RVV require reasoning about scalable or vector-length-agnostic execution. A plausible implication is that SimdBench does not merely test lexical familiarity with intrinsics, but also whether a model can adapt to heterogeneous vector abstractions.

The benchmark is built from two sources. The first consists of 62 hand-crafted tasks based on intrinsic documentation. The second consists of 74 modified HumanEval-derived tasks, obtained by filtering and adapting tasks from HumanEval and HumanEval-CPP until they became suitable for SIMD vectorization. These two sources sum to the full 136-task corpus (He et al., 21 Jul 2025).

The hand-crafted tasks are organized around six basic SIMD-relevant operation types identified from intrinsic documentation: data movement, integer operations, floating-point operations, comparison operations, logical operations, and reinterpret-cast operations. The benchmark constructs roughly 10 tasks per operation type. The supported data types include signed and unsigned integers of 8, 16, 32, and 64 bits, floating-point types of 32 and 64 bits, plus bool and string. Types not uniformly supported across all five families are deliberately excluded, including 128-bit integers, low-precision floating point, and architecture-dependent types such as short and long.

The HumanEval-derived portion is filtered to remove tasks that are poor fits for explicit vectorization. Excluded categories include sequential or recursion-heavy tasks, tasks with inherently sequential behavior, tasks whose data structures involve unpredictable memory layouts, tasks requiring function calls inside the target loops, and tasks relying on unsupported third-party libraries. After this filtering, 74 of the 164 HumanEval tasks are retained and rewritten into the SimdBench format.

3. Benchmark construction and prompt design

Each SimdBench task includes a natural-language description, a target function signature, a correctness test case, a performance test case, and annotated metadata such as the scalar entry point and the SIMD entry point (He et al., 21 Jul 2025). This richer task structure distinguishes it from standard code benchmarks that often provide only a short prompt and a few assertions.

The hand-crafted task-generation workflow proceeds in four stages: implementing a vectorizable code snippet, writing a natural-language functional description, building correctness and performance tests, and balancing operation and type coverage. The HumanEval-derived tasks are normalized to the same format. Their descriptions are refined to expose vectorizable structure, parallelism prompts are added, correctness and performance tests are created, and input ranges are expanded where necessary so that large-scale performance testing becomes meaningful.

A notable prompt-level feature is the token [simd], which is replaced during preprocessing by the actual intrinsic family name for the chosen evaluation setting. For scalar baselines, the same functional descriptions and tests are reused, but the SIMD-enabling prompt is removed and the entry point is switched back to the scalar function. This design supports controlled scalar-versus-vector comparisons on functionally matched tasks.

The benchmark also makes the problem statements more explicit than ordinary code-generation suites. Descriptions specify element-wise operations, types, widths, and possible undefined or unsafe behaviors such as integer overflow or invalid memory access. This is consequential for intrinsic programming, where lane width, register width, and memory behavior are not peripheral details but often determine both correctness and code shape.

4. Evaluation protocol, correctness criteria, and performance measurement

SimdBench evaluates correctness by differential testing against a canonical scalar solution (He et al., 21 Jul 2025). For each task, generated SIMD code is compiled and executed alongside the scalar reference on the same inputs, and outputs are compared. Input generation has three modes: special values for corner cases, randomized values within valid ranges, and fully randomized values for safe arrays. Each correctness test repeats the procedure 1,000 iterations, and a sample passes only if all iterations pass.

The benchmark introduces a special “no intrinsic” check. If generated code compiles even without the required SIMD headers and compiler options, it is labeled as having ignored the SIMD requirement and is counted separately. This matters because a code generator could otherwise evade the task by returning scalar code that satisfies the functional specification but fails the vectorization objective.

Performance is measured with the Google Benchmark library. Each performance test invokes both the generated SIMD implementation and the scalar baseline under identical compiler, machine, and option settings. The benchmark distinguishes stable inputs, which affect algorithmic complexity and must remain fixed across repetitions, from arbitrary inputs, which may vary. Performance tests use larger input sizes than correctness tests and repeat measurements many times to obtain stable nanosecond-level timings. Speedup is measured relative to the scalar baseline, and the paper trims outliers by removing the top and bottom 20% of measured speedups before averaging.

To summarize multi-sample behavior, SimdBench reports pass@1, pass@5, efficient@1, and efficient@5. The text states that pass@k is the probability that at least one of $k$ samples passes the correctness test. The “efficient@k” metric mirrors pass@k but counts only samples that are both correct and performance-improving. Performance is evaluated only for correct code; invalid code is not meaningfully benchmarked.

The paper evaluates 18 representative LLMs, comprising 15 closed-source and 3 open-source models from Anthropic, Mistral, DeepSeek, Google, OpenAI, xAI, and Alibaba. For each task, the authors sample 5 generations at temperature $T=0.2$. Correctness testing runs in Docker on an x86_64 machine with two AMD EPYC 7H12 64-core CPUs and 512 GB RAM. Scalar, SSE, and AVX code are compiled with clang 18.1.3; Neon, SVE, and RVV use cross-compilers with clang 19.1.7 and QEMU for correctness execution. Correctness timeout is 15 seconds. Performance testing uses real hardware when possible: SSE and AVX on the x86 system, Neon on AWS EC2 C6g, and SVE on AWS EC2 C8g. RVV is excluded from performance evaluation because too few models generate valid RVV code. Performance timeout is 150 seconds, and the Google Benchmark commit used is b20cea6.

5. Empirical results and observed failure modes

The central empirical result is that all evaluated LLMs suffer a substantial drop in correctness when generating SIMD-intrinsic code compared with scalar code generation (He et al., 21 Jul 2025). In scalar mode, all but GPT-3.5-Turbo and Qwen-Coder-Turbo exceed 75% pass@1. Under SIMD-intrinsic generation, however, correctness degrades sharply across the board. The degradation is especially severe for Codestral, GPT-3.5-Turbo, GPT-4o-mini, Mistral-Large, Qwen-Coder-Plus, Qwen-Coder-Turbo, and Qwen2.5-Turbo, all of which have pass@1 at or below 15% and pass@5 at or below 20% across all intrinsic families.

Among the evaluated models, DeepSeek-R1 is best overall, with the highest correctness results in all six scenarios and an average pass@5 of 75.44% across the five intrinsics. Gemini-2.5-Flash is reported as the second-strongest model overall and remains close behind DeepSeek-R1 on several metrics. The benchmark thus discriminates not just between weak and strong code models, but between models that cope differently with fixed-width and scalable vector ISAs.

Results differ markedly by intrinsic family. AVX is the easiest family overall: 17 of the 18 models achieve their highest pass@5 on AVX. RVV is the hardest: all 18 models have their lowest pass@5 on RVV, and seven models fail to generate any correct RVV compilation at all. SVE is also difficult, and SVE together with RVV accounts for the weakest rankings for 17 of the 18 models. This suggests that scalable and vector-length-agnostic SIMD models are substantially harder for current LLMs than fixed-width SSE, AVX, and Neon.

The performance results are more nuanced. The paper finds that valid SIMD code can still produce performance benefits, and that across many tasks the generated vectorized solutions outperform the scalar baseline, especially when combined with compiler optimizations. It further states that LLM-assisted SIMD code generation can overcome some limitations of compiler auto-vectorization and sometimes achieve further gains at -O3. At the same time, performance is not guaranteed: some valid SIMD implementations do not beat compiler-optimized scalar code, and one cited case shows AVX outputs often beating scalar code at -O0 but much less often at -O3.

The error analysis partitions invalid outputs into five classes: compilation error, runtime error, wrong result, no intrinsic, and timeout. The dominant problems are compilation errors and wrong results. Across all invalid cases, the most frequent issue is “use of undeclared identifier”, especially for SVE and RVV, reflecting incorrect intrinsic names or missing required prefixes. Logical bugs producing wrong results are the second major failure mode, particularly for SSE, AVX, and Neon, where code may compile and execute but fail differential testing. A further common error is “no matching function”, typically caused by argument-count or type mismatches in overloaded intrinsics. For RVV specifically, the most common problem is omission of the __riscv_ prefix, which the authors attribute to stale or missing training data for updated RVV documentation.

6. Limitations, implications, and subsequent use

The benchmark paper acknowledges several limitations (He et al., 21 Jul 2025). Correctness and performance are not measured on the same architecture family for all intrinsic types in the most direct way: correctness is centralized on an x86-based server with cross-compilation and QEMU for non-x86 targets, while performance uses real ARM hardware for Neon and SVE and excludes RVV because valid RVV outputs are too rare. The benchmark also does not explore more advanced prompting or translation settings such as chain-of-thought, retrieval-augmented generation, or scalar-to-vector translation in the main experiments.

The implications drawn by the authors are specific. First, current LLMs are not yet reliable at explicit SIMD intrinsic generation, especially for newer or more complex vector ISAs such as SVE and RVV. Second, future progress likely depends on better training data containing up-to-date intrinsic definitions. Third, retrieval-augmented generation is suggested as a practical way to ground models in current intrinsic documentation. Fourth, a step-by-step approach in which a model first generates scalar code and then transforms it into SIMD code is proposed as a promising direction, potentially aided by vectorization-aware transformation systems such as VecTrans. The paper also points to broader applications in optimizing performance-critical libraries, improving cross-platform portability of intrinsic code, and helping test or fuzz SIMD toolchains.

Subsequent work has used SimdBench as an external benchmark for training-time methods specialized to explicit vectorization. In AutoVecCoder, SimdBench serves as the main evaluation suite, though only the SSE and AVX subsets are used because that system targets x86 intrinsic generation (Li et al., 18 May 2026). That paper evaluates models in a strict zero-shot setting at inference time and reports AutoVecCoder-8B as the best overall model on both subsets for correctness, fast_1, and median speedup (P50). A plausible implication is that SimdBench has already become a reference workload for research on LLM-assisted explicit vectorization, particularly for methods that try to optimize correctness and execution efficiency jointly.

In that sense, SimdBench occupies a specific niche in the benchmark landscape. It is not a general-purpose code benchmark, nor a compiler benchmark in the traditional sense. Rather, it operationalizes explicit SIMD intrinsic generation as a distinct evaluation problem: one requiring architectural specificity, semantic preservation under vectorization, and empirical validation of speedup in addition to textual or functional plausibility.