CUDABench: Text-to-CUDA Benchmark
- CUDABench is a benchmark suite for text-to-CUDA generation that evaluates kernels by recovering algorithms and mapping GPU tasks from natural-language descriptions.
- It employs a three-dimensional evaluation of breadth, depth, and difficulty across 1,500 prompts spanning applications like deep learning, image processing, and scientific simulations.
- Its generative verification pipeline checks for compilation correctness, execution-based functional validation, and roofline-based performance to diagnose LLM capabilities.
CUDABench, introduced in "CUDABench: Benchmarking LLMs for Text-to-CUDA Generation" (Zhu et al., 13 Feb 2026), is a benchmark suite for text-to-CUDA generation: the task is to generate a CUDA kernel directly from a natural-language specification rather than translating an existing high-level program such as PyTorch or C/C++ into CUDA. The benchmark is designed to evaluate whether a model can recover the algorithm, data movement pattern, parallelization strategy, memory hierarchy usage, and CUDA-specific implementation details from text alone, and to assess the result through compilation correctness, execution-based functional verification, and a roofline-based performance metric. Its structure combines CUDABench-Set, a dataset of 500 CUDA kernel tasks expanded into 1,500 prompts, with CUDABench-Score, a unified score that gives credit only to kernels that are both compilable and functionally correct (Zhu et al., 13 Feb 2026).
1. Problem formulation and scope
CUDABench is built around a claim that text-to-CUDA is a fundamentally different and harder problem than code-to-code translation. In text-to-CUDA, the model is not given explicit control flow or an already-written source program whose intent is visible; instead, it must infer the algorithm and its GPU mapping from text. The benchmark therefore targets a broader capability than syntax completion or API recall: it asks whether an LLM can map domain knowledge to an efficient GPU implementation across diverse application areas and input scales (Zhu et al., 13 Feb 2026).
A common misconception is to treat CUDA generation benchmarks as variants of translation benchmarks. CUDABench explicitly rejects that equivalence. It is framed as more realistic because many GPU kernels are specified verbally or at the level of a problem statement, not already written in a source language whose control flow and intent are explicit. This design choice makes the benchmark sensitive not only to CUDA fluency, but also to algorithm retrieval, parallel decomposition, and hardware-aware implementation quality (Zhu et al., 13 Feb 2026).
The benchmark’s motivating concern is also methodological. Because CUDA kernel generation is hardware-specific and performance-critical, a kernel that compiles and returns correct outputs may still be a poor GPU implementation that wastes bandwidth or compute capacity. CUDABench is therefore not limited to compilation or wall-clock timing; it couples correctness with a roofline-based notion of efficiency that is intended to be hardware-independent (Zhu et al., 13 Feb 2026).
2. CUDABench-Set: Breadth, Depth, and Difficulty
CUDABench-Set is organized along a three-dimensional Breadth–Depth–Difficulty evaluation space and contains 500 tasks × 3 difficulty levels = 1,500 prompts (Zhu et al., 13 Feb 2026). The benchmark spans six representative GPU application domains:
- Fundamental Linear Algebra: GEMM, transpose, elementwise ops, least squares.
- Deep Learning Operators: activations, losses, normalization, optimizers.
- Computer Vision and Image Processing: image filters, edge detection, point-cloud ops like Ball Query and Furthest Point Sampling.
- Data Analytics: sorting, top-k, histogram, prefix sum, reduction.
- Signal Processing: FIR filtering, wavelet transforms, related transforms.
- Scientific Simulation / Finance: Monte Carlo, PDE solvers like FDTD, Black-Scholes and other numerical finance kernels.
The breadth dimension is meant to counter a benchmark bias toward machine-learning-only workloads. The paper emphasizes that prior benchmarks were heavily skewed toward ML workloads, while real-world CUDA programming covers much more than deep learning. This suggests that CUDABench treats domain diversity as a first-class variable rather than a peripheral add-on (Zhu et al., 13 Feb 2026).
The depth dimension is implemented through five input scales: Tiny, Small, Medium, Large, and Huge. The “Huge” category can exceed 1 GB of data. Each scale is paired with its own reference kernel, data generator, and validator because both the correct algorithmic behavior and the validation criteria can vary with scale. In effect, scale is not merely a runtime multiplier; it is part of the task specification (Zhu et al., 13 Feb 2026).
The difficulty dimension progressively removes information from the prompt:
- Level 1: Guided Implementation includes task name, detailed algorithm description, and CUDA implementation guidance such as memory hierarchy and thread mapping.
- Level 2: Algorithmic Specification includes task name and algorithm description, but removes hardware-specific implementation guidance.
- Level 3: Concept Retrieval only gives the task name, forcing the model to recover the algorithm and implementation strategy from internal knowledge.
Tasks are drawn from open-source CUDA codebases, NVIDIA CUDA Samples, PyTorch-related code, and production GPU repositories, and each task includes a verified reference CUDA kernel. This construction makes the dataset simultaneously a capability probe and a controlled evaluation corpus (Zhu et al., 13 Feb 2026).
3. Verification pipeline and scoring formalism
CUDABench evaluates generated kernels with a Generative Verification Pipeline. The pipeline begins with NVCC compilation. A generated kernel must compile successfully before any further evaluation. Compilation is treated as a syntactic and API-level gate rather than as evidence of semantic validity (Zhu et al., 13 Feb 2026).
Functional consistency is then checked through execution-based verification. A kernel that compiles is executed on test inputs generated by the benchmark’s data generator, and the output is compared against the reference output from the validator. The paper describes this as a strict two-stage filter: compile successfully, then pass execution-based functional verification. Only kernels passing both stages proceed to performance profiling (Zhu et al., 13 Feb 2026).
For functionally correct kernels, the benchmark measures execution time, FLOPs, and data movement volume using NVIDIA Nsight Compute, and defines the roofline upper bound as
The resulting Performance-Score is
In the memory-bound case, this reduces to achieved memory bandwidth divided by peak memory bandwidth; in the compute-bound case, it reduces to achieved GFLOPs/sec divided by peak GFLOPs/sec. The paper’s conceptual point is that a single metric can adapt to whether a kernel is bandwidth-limited or compute-limited (Zhu et al., 13 Feb 2026).
The unified CUDABench-Score is defined as
$\text{CUDABench-Score} = \frac{1}{N}\sum_{i=0}^{N} \left[ \mathbbm{1}(\text{Compilation}_i \land \text{Functional}_i) \times \text{Performance-Score}_i \right].$
A task contributes to the benchmark score only if it is both compilable and functionally correct; otherwise its contribution is zero. This prevents fast-but-wrong code from being rewarded. Experiments are also reported with Pass@1 and Pass@3, where , for compilation, functional correctness, and score-based success (Zhu et al., 13 Feb 2026).
4. Empirical findings
CUDABench is reported to be challenging even for top LLMs. The benchmarked models include GPT-5.2, Claude 4.5 Sonnet, Gemini 3 Flash, DeepSeek-V3.2, Qwen3-Max, Kimi K2 Thinking, and MiniMax-M2.1. Performance drops substantially as prompt difficulty increases: GPT-5.2’s Pass@1 functional accuracy falls from 79.8% at Level 1 to 60.8% at Level 3 (Zhu et al., 13 Feb 2026).
One of the benchmark’s central empirical results is the large gap between compilation success and functional correctness. Some models, such as Claude 4.5 Sonnet, achieve nearly perfect or perfect compilation rates, but functional correctness is much lower. The paper attributes many failures not to syntax, but to logic errors involving synchronization, boundary conditions, parallel indexing, and memory handling. The stated interpretation is that modern LLMs have largely learned CUDA syntax and common APIs, but not reliable GPU algorithm semantics (Zhu et al., 13 Feb 2026).
The benchmark also isolates domain-specific knowledge as a major bottleneck. When only the task name is given at Level 3, model performance drops sharply, especially in specialized areas such as scientific simulation, signal processing, and finance kernels. By contrast, models do much better on more familiar foundations like linear algebra. This suggests that internal retrieval of specialized algorithmic patterns remains unreliable even when general coding competence is strong (Zhu et al., 13 Feb 2026).
Hardware utilization is similarly weak. Even the best models leave a large fraction of GPU capability unused: top models have similar performance scores around 40% at Level 1, meaning roughly 60% of theoretical GPU capability remains untapped. CUDABench therefore separates two failure modes that can otherwise be conflated: semantic failure, in which the kernel is wrong, and efficiency failure, in which the kernel is correct but far from hardware limits (Zhu et al., 13 Feb 2026).
The paper additionally reports evaluation on an RTX 4090 and states that the normalized score remains relatively stable across GPUs, despite very different peak bandwidth and compute capabilities. This is offered as evidence that the roofline-based score is more portable than raw execution time (Zhu et al., 13 Feb 2026).
5. Position within CUDA benchmark literature
CUDABench occupies a specific place in the recent CUDA benchmark landscape: it is a generation benchmark, not an optimization-from-reference benchmark, a debugging benchmark, or a translation benchmark. Adjacent systems emphasize different task formulations and scoring regimes (Li et al., 18 Jul 2025, Han et al., 7 Mar 2026, Li et al., 8 May 2026, Heakl et al., 22 May 2025, Lin et al., 19 Mar 2026).
| Benchmark | Primary task | Distinguishing evaluation axis |
|---|---|---|
| CUDABench (Zhu et al., 13 Feb 2026) | Text-to-CUDA generation from a natural-language specification | Compilation correctness, execution-based functional verification, roofline-based Performance-Score |
| KernelBench context in CUDA-L1 (Li et al., 18 Jul 2025) | Generate or optimize CUDA against a reference PyTorch implementation | Executability, correctness over 1000 random test inputs, wall-clock speedup |
| MSKernelBench (Han et al., 7 Mar 2026) | Multi-scenario CUDA kernel optimization | Multi-scale correctness and complexity-weighted performance score |
| CUDABeaver (Li et al., 8 May 2026) | Broken-start CUDA debugging | Performance-preserving repair, failure-category analysis, protocol-conditional |
| CASS-Bench (Heakl et al., 22 May 2025) | Cross-architecture GPU code translation | Execution-verified source-to-source and assembly-to-assembly translation accuracy |
| SOL-ExecBench (Lin et al., 19 Mar 2026) | Kernel optimization against hardware Speed-of-Light bounds | SOL score measuring how much of the gap to analytically derived hardware bounds is closed |
The contrast with KernelBench-style evaluation is especially sharp. In the CUDA-L1 study, each task provides a reference PyTorch implementation plus fixed inputs/outputs; the system generates alternative CUDA code, checks whether it compiles and runs, validates correctness against the reference, and then scores performance by wall-clock speedup. CUDABench instead starts from text and asks the model to generate the CUDA program itself, which shifts difficulty from local optimization toward algorithm recovery and implementation synthesis (Li et al., 18 Jul 2025).
MSKernelBench extends evaluation in another direction. It is a multi-scenario benchmark covering dense, sparse, LLM, and scientific/HPC kernels, supports both FP32 and BF16, tests multiple input sizes, and uses a complexity-weighted performance score. CUDABench shares the concern for multi-scale evaluation, but its defining variable is prompt incompleteness rather than profiling-guided iterative optimization (Han et al., 7 Mar 2026).
CUDABeaver, CASS-Bench, and SOL-ExecBench further delimit CUDABench’s scope. CUDABeaver studies whether models can repair real failing CUDA workspaces without degeneration into slower fallbacks; CASS-Bench evaluates CUDA↔HIP and SASS↔RDNA3 translation with execution-verified ground truth; SOL-ExecBench reframes benchmarking around proximity to analytically derived hardware limits on Blackwell GPUs. These benchmarks do not supersede CUDABench; they specify adjacent capability regimes that CUDABench does not attempt to measure directly (Li et al., 8 May 2026, Heakl et al., 22 May 2025, Lin et al., 19 Mar 2026).
6. Significance, misconceptions, and implications
CUDABench’s main significance lies in what it demonstrates about evaluation itself. The paper argues that assessing the performance of LLM-generated GPU programs is nontrivial, and its results show why: compilation success can hide major semantic failures, and functionally correct kernels can still underutilize the GPU by a wide margin. The benchmark therefore formalizes three distinct questions—can the kernel compile, does it behave correctly, and does it approach hardware-efficient execution—and refuses to collapse them into a single unqualified “success” notion (Zhu et al., 13 Feb 2026).
This directly addresses a recurring misconception in LLM-for-CUDA evaluation: that syntactic competence is a sufficiently informative proxy for practical ability. CUDABench shows that it is not. The benchmark’s large compilation-to-correctness gap indicates that models often know CUDA syntax and common APIs without reliably implementing correct GPU semantics. Its prompt-level difficulty progression further shows that missing algorithmic knowledge, not merely missing implementation hints, is a major failure mode (Zhu et al., 13 Feb 2026).
A second misconception is that raw execution time on one GPU is an adequate cross-model performance metric. CUDABench instead argues for normalization to hardware limits via a roofline-based score. This design is consistent with a broader trend in later benchmarking work: CUDA-L1 emphasizes hardened speedup measurement and anti-hacking evaluation, while SOL-ExecBench replaces software-relative speedup with hardware-grounded Speed-of-Light targets (Li et al., 18 Jul 2025, Lin et al., 19 Mar 2026). A plausible implication is that CUDABench identified an evaluation problem that subsequent work has pursued with increasingly specialized measurement protocols.
More broadly, the benchmark positions text-to-CUDA generation as a research problem that sits at the intersection of program synthesis, domain-specific algorithm retrieval, and architecture-aware optimization. Its findings imply that progress requires more than better token-level code generation: models need stronger access to specialized algorithm families, more reliable reasoning about synchronization and indexing, and more consistent exploitation of memory bandwidth and compute throughput. In that sense, CUDABench is not merely a leaderboard; it is a diagnostic framework for separating syntax, semantics, and hardware efficiency in LLM-generated CUDA (Zhu et al., 13 Feb 2026).