---
title: 'SimdBench: Benchmark for SIMD Intrinsic Code Generation'
url: https://www.emergentmind.com/topics/simdbench
type: topic
---

# SimdBench: Benchmark for SIMD Intrinsic Code Generation

Searching arXiv for SimdBench and related SIMD-intrinsic code generation papers.
SimdBench is a benchmark for evaluating large language models on **SIMD-intrinsic code generation**: the production of correct and performant vectorized C/C++ implementations using explicit SIMD intrinsics rather than ordinary scalar code. It was introduced as the first benchmark specifically designed for this setting, with **136 tasks** spanning **SSE**, **AVX**, **Neon**, **SVE**, and **RVV**, and it evaluates both functional correctness and execution performance under architecture-specific compilation and runtime conditions [2507.15224]. Its motivation is that existing code-generation benchmarks largely ignore low-level hardware features, so they do not capture the real difficulty of programming with intrinsics, where interface complexity, alignment and memory-layout constraints, and masking-heavy control flow are central. The name should be distinguished from unrelated benchmarks titled “SimBench” in human-behavior simulation and digital-twin generation [2510.17516] [2408.11987].

## 1. Motivation and problem setting

SimdBench was proposed to measure whether LLMs can assist with explicit SIMD programming, a domain in which developers trade coding productivity for high performance by directly invoking architecture-specific intrinsic APIs [2507.15224]. The benchmark is motivated by three specific challenges of SIMD programming identified in the paper: the complexity and poor readability of intrinsic interfaces, the need for manual data alignment and memory-layout handling, and the need to express complex control and data flow such as masking.

These difficulties make SIMD intrinsic programming materially different from generic code generation. A scalar benchmark can reward syntactic plausibility or high-level functional correctness without probing whether a model understands register width, lane type, masking semantics, or target-ISA naming conventions. SimdBench was therefore designed to test whether an LLM can generate **correct, fast, architecture-aware SIMD intrinsic code** under conditions closer to explicit vectorization practice than ordinary benchmark suites.

A further motivation is comparative evaluation. The paper argues that code-generation benchmarks had focused on scalar code, leaving unclear how LLMs perform when the target is not merely correct code, but vectorized code that must preserve semantics while exploiting hardware-specific SIMD support. SimdBench fills that gap by pairing correctness testing with performance testing for the same tasks.

## 2. Coverage across instruction sets, task sources, and data types

SimdBench contains **136 tasks** targeting five representative SIMD intrinsic families across mainstream processor ecosystems [2507.15224].

| Intrinsic family | Architecture | Model characteristics |
|---|---|---|
| SSE | x86 | 128-bit fixed-width SIMD |
| AVX | x86 | 256-bit and 512-bit variants |
| Neon | ARM | 128-bit fixed-width |
| SVE | ARM | Scalable from 128 to 2048 bits in 128-bit increments |
| RVV | RISC-V | Vector-length-agnostic model whose length depends on hardware |

This coverage is significant because the targeted families differ substantially in width, masking, and programming model. Fixed-width ISAs such as SSE and Neon impose one class of coding constraints, while SVE and RVV require reasoning about scalable or vector-length-agnostic execution. A plausible implication is that SimdBench does not merely test lexical familiarity with intrinsics, but also whether a model can adapt to heterogeneous vector abstractions.

The benchmark is built from two sources. The first consists of **62 hand-crafted tasks** based on intrinsic documentation. The second consists of **74 modified HumanEval-derived tasks**, obtained by filtering and adapting tasks from HumanEval and HumanEval-CPP until they became suitable for SIMD vectorization. These two sources sum to the full **136-task** corpus [2507.15224].

The hand-crafted tasks are organized around **six basic SIMD-relevant operation types** identified from intrinsic documentation: data movement, integer operations, floating-point operations, comparison operations, logical operations, and reinterpret-cast operations. The benchmark constructs roughly 10 tasks per operation type. The supported data types include signed and unsigned integers of **8, 16, 32, and 64 bits**, floating-point types of **32 and 64 bits**, plus **bool** and **string**. Types not uniformly supported across all five families are deliberately excluded, including 128-bit integers, low-precision floating point, and architecture-dependent types such as `short` and `long`.

The HumanEval-derived portion is filtered to remove tasks that are poor fits for explicit vectorization. Excluded categories include sequential or recursion-heavy tasks, tasks with inherently sequential behavior, tasks whose data structures involve unpredictable memory layouts, tasks requiring function calls inside the target loops, and tasks relying on unsupported third-party libraries. After this filtering, **74 of the 164 HumanEval tasks** are retained and rewritten into the SimdBench format.

## 3. Benchmark construction and prompt design

Each SimdBench task includes a natural-language description, a target function signature, a correctness test case, a performance test case, and annotated metadata such as the scalar entry point and the SIMD entry point [2507.15224]. This richer task structure distinguishes it from standard code benchmarks that often provide only a short prompt and a few assertions.

The hand-crafted task-generation workflow proceeds in four stages: implementing a vectorizable code snippet, writing a natural-language functional description, building correctness and performance tests, and balancing operation and type coverage. The HumanEval-derived tasks are normalized to the same format. Their descriptions are refined to expose vectorizable structure, parallelism prompts are added, correctness and performance tests are created, and input ranges are expanded where necessary so that large-scale performance testing becomes meaningful.

A notable prompt-level feature is the token `[simd]`, which is replaced during preprocessing by the actual intrinsic family name for the chosen evaluation setting. For scalar baselines, the same functional descriptions and tests are reused, but the SIMD-enabling prompt is removed and the entry point is switched back to the scalar function. This design supports controlled scalar-versus-vector comparisons on functionally matched tasks.

The benchmark also makes the problem statements more explicit than ordinary code-generation suites. Descriptions specify element-wise operations, types, widths, and possible undefined or unsafe behaviors such as integer overflow or invalid memory access. This is consequential for intrinsic programming, where lane width, register width, and memory behavior are not peripheral details but often determine both correctness and code shape.

## 4. Evaluation protocol, correctness criteria, and performance measurement

SimdBench evaluates correctness by **differential testing** against a canonical scalar solution [2507.15224]. For each task, generated SIMD code is compiled and executed alongside the scalar reference on the same inputs, and outputs are compared. Input generation has three modes: special values for corner cases, randomized values within valid ranges, and fully randomized values for safe arrays. Each correctness test repeats the procedure **1,000 iterations**, and a sample passes only if all iterations pass.

The benchmark introduces a special “no intrinsic” check. If generated code compiles even without the required SIMD headers and compiler options, it is labeled as having ignored the SIMD requirement and is counted separately. This matters because a code generator could otherwise evade the task by returning scalar code that satisfies the functional specification but fails the vectorization objective.

Performance is measured with the **Google Benchmark** library. Each performance test invokes both the generated SIMD implementation and the scalar baseline under identical compiler, machine, and option settings. The benchmark distinguishes stable inputs, which affect algorithmic complexity and must remain fixed across repetitions, from arbitrary inputs, which may vary. Performance tests use larger input sizes than correctness tests and repeat measurements many times to obtain stable nanosecond-level timings. Speedup is measured relative to the scalar baseline, and the paper trims outliers by removing the top and bottom **20%** of measured speedups before averaging.

To summarize multi-sample behavior, SimdBench reports **pass@1**, **pass@5**, **efficient@1**, and **efficient@5**. The text states that pass@k is the probability that at least one of \(k\) samples passes the correctness test. The “efficient@k” metric mirrors pass@k but counts only samples that are both correct and performance-improving. Performance is evaluated only for correct code; invalid code is not meaningfully benchmarked.

The paper evaluates **18 representative LLMs**, comprising **15 closed-source** and **3 open-source** models from Anthropic, Mistral, DeepSeek, Google, OpenAI, xAI, and Alibaba. For each task, the authors sample **5 generations** at temperature \(T=0.2\). Correctness testing runs in Docker on an x86_64 machine with two AMD EPYC 7H12 64-core CPUs and 512 GB RAM. Scalar, SSE, and AVX code are compiled with clang 18.1.3; Neon, SVE, and RVV use cross-compilers with clang 19.1.7 and QEMU for correctness execution. Correctness timeout is **15 seconds**. Performance testing uses real hardware when possible: SSE and AVX on the x86 system, Neon on AWS EC2 C6g, and SVE on AWS EC2 C8g. RVV is excluded from performance evaluation because too few models generate valid RVV code. Performance timeout is **150 seconds**, and the Google Benchmark commit used is `b20cea6`.

## 5. Empirical results and observed failure modes

The central empirical result is that **all evaluated LLMs suffer a substantial drop in correctness when generating SIMD-intrinsic code compared with scalar code generation** [2507.15224]. In scalar mode, all but GPT-3.5-Turbo and Qwen-Coder-Turbo exceed **75% pass@1**. Under SIMD-intrinsic generation, however, correctness degrades sharply across the board. The degradation is especially severe for Codestral, GPT-3.5-Turbo, GPT-4o-mini, Mistral-Large, Qwen-Coder-Plus, Qwen-Coder-Turbo, and Qwen2.5-Turbo, all of which have **pass@1 at or below 15%** and **pass@5 at or below 20%** across all intrinsic families.

Among the evaluated models, **DeepSeek-R1** is best overall, with the highest correctness results in all six scenarios and an average **pass@5 of 75.44% across the five intrinsics**. **Gemini-2.5-Flash** is reported as the second-strongest model overall and remains close behind DeepSeek-R1 on several metrics. The benchmark thus discriminates not just between weak and strong code models, but between models that cope differently with fixed-width and scalable vector ISAs.

Results differ markedly by intrinsic family. **AVX** is the easiest family overall: **17 of the 18 models** achieve their highest pass@5 on AVX. **RVV** is the hardest: **all 18 models** have their lowest pass@5 on RVV, and **seven models fail to generate any correct RVV compilation at all**. **SVE** is also difficult, and SVE together with RVV accounts for the weakest rankings for **17 of the 18 models**. This suggests that scalable and vector-length-agnostic SIMD models are substantially harder for current LLMs than fixed-width SSE, AVX, and Neon.

The performance results are more nuanced. The paper finds that valid SIMD code can still produce performance benefits, and that across many tasks the generated vectorized solutions outperform the scalar baseline, especially when combined with compiler optimizations. It further states that LLM-assisted SIMD code generation can overcome some limitations of compiler auto-vectorization and sometimes achieve further gains at `-O3`. At the same time, performance is not guaranteed: some valid SIMD implementations do not beat compiler-optimized scalar code, and one cited case shows AVX outputs often beating scalar code at `-O0` but much less often at `-O3`.

The error analysis partitions invalid outputs into five classes: compilation error, runtime error, wrong result, no intrinsic, and timeout. The dominant problems are compilation errors and wrong results. Across all invalid cases, the most frequent issue is **“use of undeclared identifier”**, especially for SVE and RVV, reflecting incorrect intrinsic names or missing required prefixes. Logical bugs producing wrong results are the second major failure mode, particularly for SSE, AVX, and Neon, where code may compile and execute but fail differential testing. A further common error is **“no matching function”**, typically caused by argument-count or type mismatches in overloaded intrinsics. For RVV specifically, the most common problem is omission of the `__riscv_` prefix, which the authors attribute to stale or missing training data for updated RVV documentation.

## 6. Limitations, implications, and subsequent use

The benchmark paper acknowledges several limitations [2507.15224]. Correctness and performance are not measured on the same architecture family for all intrinsic types in the most direct way: correctness is centralized on an x86-based server with cross-compilation and QEMU for non-x86 targets, while performance uses real ARM hardware for Neon and SVE and excludes RVV because valid RVV outputs are too rare. The benchmark also does not explore more advanced prompting or translation settings such as chain-of-thought, retrieval-augmented generation, or scalar-to-vector translation in the main experiments.

The implications drawn by the authors are specific. First, current LLMs are not yet reliable at explicit SIMD intrinsic generation, especially for newer or more complex vector ISAs such as SVE and RVV. Second, future progress likely depends on better training data containing up-to-date intrinsic definitions. Third, retrieval-augmented generation is suggested as a practical way to ground models in current intrinsic documentation. Fourth, a step-by-step approach in which a model first generates scalar code and then transforms it into SIMD code is proposed as a promising direction, potentially aided by vectorization-aware transformation systems such as VecTrans. The paper also points to broader applications in optimizing performance-critical libraries, improving cross-platform portability of intrinsic code, and helping test or fuzz SIMD toolchains.

Subsequent work has used SimdBench as an external benchmark for training-time methods specialized to explicit vectorization. In **AutoVecCoder**, SimdBench serves as the main evaluation suite, though only the **SSE** and **AVX** subsets are used because that system targets x86 intrinsic generation [2605.17978]. That paper evaluates models in a strict zero-shot setting at inference time and reports **AutoVecCoder-8B** as the best overall model on both subsets for **correctness**, **fast\_1**, and **median speedup (P50)**. A plausible implication is that SimdBench has already become a reference workload for research on LLM-assisted explicit vectorization, particularly for methods that try to optimize correctness and execution efficiency jointly.

In that sense, SimdBench occupies a specific niche in the benchmark landscape. It is not a general-purpose code benchmark, nor a compiler benchmark in the traditional sense. Rather, it operationalizes explicit SIMD intrinsic generation as a distinct evaluation problem: one requiring architectural specificity, semantic preservation under vectorization, and empirical validation of speedup in addition to textual or functional plausibility.

Source: https://www.emergentmind.com/topics/simdbench