MSKernelBench: CUDA Kernel Optimization Benchmark

• MSKernelBench is a comprehensive benchmark that evaluates CUDA kernel optimization across heterogeneous workloads including dense, sparse, and scientific routines.
• It implements 50 tasks with FP32 and BF16 versions, using complexity-weighted scoring and detailed profiling to assess performance and guide optimizations.
• The benchmark underpins CUDAMaster, a multi-agent system that iteratively refines kernel code, demonstrating significant speedups versus vendor libraries and compiler approaches.

MSKernelBench is a benchmark for general-purpose, multi-scenario CUDA kernel optimization introduced to evaluate whether LLM-based systems can optimize kernels beyond the ML/PyTorch operators emphasized in earlier work. It spans multiple scenarios—including fundamental algebraic operations, common LLM kernels, sparse matrix operators, and scientific computing routines—and each task is implemented in both FP32 and BF16, yielding 100 optimization tasks from 50 underlying tasks. In the same work, the benchmark serves as the evaluation foundation for CUDAMaster, a multi-agent, hardware-aware system that uses profiling information and automatically constructs the full compilation and execution toolchain (Han et al., 7 Mar 2026).

1. Origin and problem setting

MSKernelBench was introduced to address what its authors describe as the absence of systematic evaluation for multi-scenario CUDA optimization. The motivating critique is that prior LLM-based kernel benchmarks focused too narrowly on machine-learning operators and fused LLM kernels, which leaves out sparse linear algebra, scientific computing routines, irregular memory access patterns, numerical methods, and stencil codes. The paper argues that this narrow scope biases evaluation toward regular, dense, compute-heavy kernels whose optimization patterns are often publicly known, making it easier for models to recall standard solutions rather than demonstrate broader optimization ability. It therefore frames MSKernelBench as a low-level testbed for kernels with heterogeneous bottlenecks rather than as another framework-centric benchmark (Han et al., 7 Mar 2026).

The same paper situates the benchmark against two established classes of optimization systems. One class consists of hand-tuned vendor libraries such as cuBLAS and cuSPARSE, which are fast but expensive to develop and not flexible. The other consists of compiler-based approaches such as TVM and Triton, which are more automated but often fail to match expert performance across heterogeneous workloads. In this setting, multi-scenario CUDA optimization is presented as difficult precisely because different operators are constrained by different bottlenecks: compute-bound, memory-latency-bound, and memory-bandwidth-bound. This organization is central to MSKernelBench’s design rather than a post hoc analysis (Han et al., 7 Mar 2026).

2. Benchmark scope and operator families

MSKernelBench contains 50 tasks, each with separate FP32 and BF16 implementations. Its stated scenario coverage comprises four broad groups: fundamental algebraic operations, common LLM kernels or operator sequences, sparse matrix operators, and scientific computing routines. The benchmark is also described through a more detailed operator-family organization. Dense linear algebra includes dot product, vector add, square matrix multiplication, scalar-matrix multiplication, matrix multiplication, matrix power, and matrix-vector multiplication. Sparse matrix operators cover COO, CSR, CSC, and ELL formats through SpMV and SpMM variants. Normalization tasks include BatchNorm, RMSNorm, LayerNorm, and Softmax, while activation functions include Sigmoid, SELU, ELU, GELU, Tanh, ReLU, and Leaky ReLU (Han et al., 7 Mar 2026).

The detailed task list extends beyond those families. It includes loss functions such as MSE Loss, Hinge Loss, and Cross-Entropy Loss; LLM operator sequences such as Softmax Attention, Triangle Attention, Merge Attention States, SwiGLU, SiLU and Mul, and RoPE; tensor-processing operators such as Reverse Array, Matrix Copy, Matrix Transpose, and High-Order Contraction; stencil and spatial routines such as 1D Convolution, 2D Max Pooling, 2D Convolution, 2D Stencil, and 3D Stencil; and math or algorithm kernels such as Sorting, Top-K, Simpson Integration, Monte Carlo Integration, FFT, and Prefix Sum. The paper also reports a bottleneck distribution across the 50 tasks: 11 compute-bound tasks, 24 memory-latency-bound tasks, and 15 memory-bandwidth-bound tasks. That distribution is used to show that the corpus was assembled to span hardware bottlenecks, not merely application labels (Han et al., 7 Mar 2026).

3. Evaluation protocol and scoring

The evaluation protocol is explicitly multi-stage. Submitted CUDA code must compile under a fixed interface, including the required function name, parameter list, and wrapper conventions. Execution correctness is then checked across a predefined set of data sizes rather than a single fixed input. For each size, random test cases are generated and the produced outputs are compared against a ground-truth kernel. The coder and debugger prompts use an absolute error tolerance of $10^{-5}$. Performance is measured as speedup relative to the provided ground-truth baseline kernel, using 3 warm-up runs and 50 measured runs, with the average runtime taken as the reported value (Han et al., 7 Mar 2026).

A distinguishing element is the complexity-weighted score. For data size $N_i$, the benchmark assigns a weight proportional to the theoretical computational complexity of the baseline implementation,

$w_i \propto T(N_i),$

and aggregates speedups through

$P=\frac{\sum_i w_i S_i}{\sum_i w_i},$

where $S_i$ is the measured end-to-end speedup at size $N_i$. This makes the final score a complexity-weighted average speedup rather than a flat mean. The reported rationale is that larger and more computationally expensive cases should matter more, and that algorithmic improvements with better scaling behavior should be rewarded. The benchmark is implemented in pure C/CUDA rather than PyTorch, specifically to strip away framework abstractions and evaluate low-level kernel quality directly (Han et al., 7 Mar 2026).

4. CUDAMaster and profiling-guided optimization

MSKernelBench is used in conjunction with CUDAMaster, a multi-agent optimization pipeline composed of four specialized agents. The Planner Agent reads the operator description, code, hardware profile, and test results, then proposes one incremental optimization strategy at a time. The Coder Agent implements that strategy in CUDA code while preserving the original interface. The Compiler Agent generates the appropriate nvcc compilation command, including build, link, and architecture settings. The Debug Agent attempts to fix compilation and runtime failures, subject to a fixed limit on debug rounds. The pipeline iterates for a fixed number of optimization rounds $R$, with up to $D$ debug attempts per candidate (Han et al., 7 Mar 2026).

A core design feature is filtered profiling guidance. The system collects NVIDIA Nsight Compute metrics, then filters them according to bottleneck class. The three bottleneck categories are compute-bound, memory-latency-bound, and memory-bandwidth-bound, with thresholds determined using Otsu’s method on throughput distributions and reported as near 30% for the major throughput metrics. For compute-bound kernels, the retained metrics are Compute(SM) Throughput, Issue Slots Busy, Executed IPC Active, and SM Busy. For memory-latency-bound kernels, they are L2 Hit Rate, L1/TEX Hit Rate, Executed IPC Elapsed, and Mem Busy. For memory-bandwidth-bound kernels, they are DRAM Throughput, Memory Throughput, Max Bandwidth, and Mem Pipes Busy. The paper presents this filtering as a way to emulate expert practice by focusing attention on the counters most relevant to the current bottleneck class (Han et al., 7 Mar 2026).

5. Reported results and ablations

The reported experimental setup uses 50 tasks in FP32 and BF16, for 100 total optimization tasks, on an NVIDIA RTX 4090. The evaluated LLMs are OpenAI o4-mini and DeepSeek-V3.2, with optimization parameters $R=3$ iterations and $D=3$ debug trials. Cumulative success is reported as the percentage of tasks whose weighted speedup exceeds threshold $N_i$0. For o4-mini versus DeepSeek-V3.2, the benchmark reports $N_i$1 versus $N_i$2 at $N_i$3, $N_i$4 versus $N_i$5 at $N_i$6, and $N_i$7 versus $N_i$8 at $N_i$9. The same discussion notes that BF16 gives a slight edge for o4-mini at moderate thresholds, while FP32 is more stable at stringent thresholds (Han et al., 7 Mar 2026).

The paper also reports comparisons against authoritative baselines on six representative kernels. Against vendor libraries, SpMV CSR is compared to cuSPARSE, 2D Convolution to cuDNN, and Dot Product to cuBLAS. Against Astra, the paper evaluates RMSNorm, SiLU and Mul, and Merge Attention States. It reports that both tested models beat cuSPARSE on SpMV CSR, and that the generated kernels achieve up to $w_i \propto T(N_i),$0 speedup over cuDNN on 2D convolution and cuBLAS on dot product. Relative to Astra, o4-mini leads by about 35% on RMSNorm and matches or exceeds Astra on the other two fused LLM operators. The broader interpretation in the paper is that sparse kernels and dense kernels can both be improved substantially, while irregular memory behavior remains the harder regime and benefits most from profiling guidance (Han et al., 7 Mar 2026).

Ablation studies are used to isolate the contribution of iteration, debugging, and profiling. The configurations compared are Full $w_i \propto T(N_i),$1, No Debug $w_i \propto T(N_i),$2, Single Iteration $w_i \propto T(N_i),$3, and Single Run $w_i \propto T(N_i),$4. The Full setting consistently performs best, and removing either debug or iteration lowers success rates, especially at high thresholds. For profiling, the comparison is between Filtered, No Profile, and Full Profile. Filtered profiling is reported to match or nearly match Full Profile on success while outperforming No Profile, and it reduces cost and token usage substantially: up to 32% cost reduction and roughly 30–40% fewer tokens. The paper further analyzes hardware-state changes after optimization, reporting that 6 memory-bound tasks are transformed into compute-bound tasks, 14 tasks shift from memory latency to bandwidth bottlenecks, memory-latency-bound tasks drop by 67% from 24 to 8, compute-bound tasks increase by 45%, and memory-bandwidth-bound tasks increase by 73%. Metric-level changes are likewise reported: Issue Slots Busy $w_i \propto T(N_i),$5 and Executed IPC Active $w_i \propto T(N_i),$6 for compute-bound tasks; Mem Busy $w_i \propto T(N_i),$7 and Executed IPC Elapsed $w_i \propto T(N_i),$8 for memory-latency-bound tasks; and DRAM Throughput $w_i \propto T(N_i),$9 and Mem Pipes Busy $P=\frac{\sum_i w_i S_i}{\sum_i w_i},$0 for memory-bandwidth-bound tasks (Han et al., 7 Mar 2026).

6. Relation to adjacent benchmark lines and naming usage

MSKernelBench is explicitly positioned against KernelBench. In the comparison reported by the MSKernelBench paper, both benchmarks include dense matrix operators and LLM operator sequences, but only MSKernelBench includes sparse matrix operators, scientific kernels, and numerical methods. The same comparison states that KernelBench uses Python, whereas MSKernelBench uses C/C++; that scalable data sizes are absent in KernelBench and present in MSKernelBench; that complexity-weighted evaluation is absent in KernelBench and present in MSKernelBench; and that hardware profiling is limited in KernelBench but integrated in MSKernelBench. KernelBench itself is an open-source benchmark with 250 tasks drawn from real PyTorch ML workloads, organized into three difficulty levels, and centered on the fast_p metric for joint correctness and speedup evaluation (Han et al., 7 Mar 2026, Ouyang et al., 14 Feb 2025).

Subsequent benchmark work broadens the landscape further. MultiKernelBench extends kernel-generation evaluation to 285 tasks across 14 categories on Nvidia GPUs, Huawei NPUs, and Google TPUs (Wen et al., 20 Jul 2025). MobileKernelBench moves the problem to mobile deployment, using 190 tasks derived from 95 primitive operators organized into 12 categories and evaluated through an automated on-device MNN pipeline (Zou et al., 12 Mar 2026). KernelBench-X shifts emphasis toward category-aware failure analysis and hardware efficiency, using 176 tasks across 15 categories to argue that task structure explains more variance in semantic correctness than method identity (Wang et al., 6 May 2026). This suggests that MSKernelBench occupies a specific niche within the benchmark literature: general-purpose CUDA optimization across heterogeneous workload scenarios, rather than multi-platform portability, mobile deployment, or category-level failure diagnosis.

The name itself is not fully uniform across the supplied literature. A 2023 autotuning paper is described in the data block as “MSKernelBench/BAT 2.0,” but the work it introduces is BAT 2.0, a benchmark suite of tunable GPU kernels for evaluating optimization algorithms used by autotuners through a shared problem interface (Tørring et al., 2023). By contrast, the explicit benchmark name MSKernelBench is used by the 2026 paper on multi-scenario CUDA kernel optimization. A common misconception is therefore to treat all similarly named kernel-benchmark papers as the same resource; the literature instead separates at least three distinct lines: autotuning suites such as BAT 2.0, ML-centric kernel-generation benchmarks such as KernelBench, and the broader multi-scenario CUDA benchmark explicitly named MSKernelBench (Han et al., 7 Mar 2026).