CUDABench-Set: Text-to-CUDA Benchmark
- CUDABench-Set is a text-to-CUDA benchmark featuring a 3D evaluation space—Breadth, Depth, and Difficulty—to analyze kernel generation from varied natural language prompts.
- It includes 500 tasks across six GPU computing domains with three levels of prompt detail, providing reference implementations along with data generators and validators.
- The dataset exposes challenges in CUDA synthesis such as functional errors and low GPU utilization, guiding improvements in LLM performance and algorithm retrieval.
CUDABench-Set is the text-to-CUDA dataset at the core of the CUDABench benchmark. Each item corresponds to one CUDA kernel task together with natural-language or minimally specified prompts, a canonical reference CUDA implementation, and a data generator and validator for execution-based checking. Its purpose is to evaluate LLM capabilities for direct text-to-CUDA generation rather than code-to-code translation, and it integrates a roofline-based performance evaluation rather than only correctness or wall-clock time (Zhu et al., 13 Feb 2026).
1. Definition and organizing principles
CUDABench-Set is structured around a 3D evaluation space called Breadth–Depth–Difficulty. This design is intended to separate domain coverage, problem scale, and prompt informativeness, so that failures in CUDA generation can be attributed to missing algorithmic knowledge, weak sequential-to-parallel mapping, or poor hardware utilization rather than being conflated into a single aggregate score (Zhu et al., 13 Feb 2026).
| Dimension | Meaning | Instantiation |
|---|---|---|
| Breadth | Coverage of application domains and kernel types | Six GPU computing domains |
| Depth | Coverage of input size and problem scale | Five input scale levels: Tiny, Small, Medium, Large, Huge |
| Difficulty | Amount of task and implementation information in the prompt | Three prompt levels |
Breadth measures domain and workload diversity. Depth measures input size and performance stress, with each kernel defined at five scale levels. Difficulty measures how much information is given in the prompt, ranging from guided CUDA implementation hints to only the task name and minimal I/O description. This organization makes CUDABench-Set a benchmark not only of syntax generation, but of algorithm retrieval, CUDA mapping, and performance-oriented kernel design (Zhu et al., 13 Feb 2026).
The construction methodology is also explicit. For each kernel task, the authors select a real-world kernel from open-source CUDA codebases, derive a textual specification at three difficulty levels, implement a reference CUDA kernel as ground truth, and provide a data generator and validator. This yields a benchmark in which textual intent, executable reference behavior, and performance evaluation are coupled at the task level rather than supplied as separate resources (Zhu et al., 13 Feb 2026).
2. Corpus composition and workload coverage
CUDABench-Set contains 500 distinct tasks across six application domains, and each task has three difficulty levels, yielding 1,500 prompts in total (Zhu et al., 13 Feb 2026). The six domains are fundamental linear algebra, deep learning operators, computer vision and image processing, data analytics, signal processing, and scientific simulation and finance.
The domain mix is deliberately broad. Fundamental linear algebra includes kernels such as matrix multiplication, matrix transpose, element-wise operations, and least-squares solvers. Deep learning operators include activation functions such as GELU and Swish, loss functions such as Cross-Entropy and KL Divergence, normalization such as LayerNorm, and optimizers such as AdamW. Computer vision and image processing includes Gaussian blur, edge detection, Ball Query, and Furthest Point Sampling. Data analytics includes sorting, top-, histogram building, prefix sums, and reductions. Signal processing includes FIR filters and discrete wavelet transforms. Scientific simulation and finance includes Monte Carlo integration, PDE solvers such as FDTD, and Black–Scholes pricing (Zhu et al., 13 Feb 2026).
The dataset is derived from production-level, open-source CUDA codebases, including PyTorch and related deep-learning operator libraries, NVIDIA CUDA Samples, and other production-level GPU repositories. This provenance matters because the benchmark is intended to mimic real-world GPU workloads rather than toy kernels. A plausible implication is that CUDABench-Set stresses both canonical CUDA idioms and domain-specific implementation patterns that arise in practice (Zhu et al., 13 Feb 2026).
Task complexity also spans multiple performance regimes. The paper explicitly leverages the roofline model to characterize kernels through arithmetic intensity, defined as the ratio of the total number of FLOPs executed by the application to the total number of bytes transferred. Low-arithmetic-intensity kernels are memory-bound; high-arithmetic-intensity kernels are compute-bound. The benchmark therefore includes not only embarrassingly parallel elementwise kernels but also reductions, scans, recursive transforms, and numerically intensive scientific codes (Zhu et al., 13 Feb 2026).
3. Prompt hierarchy and task specification
The three prompt levels define the difficulty axis. Level 1, “Guided Implementation,” includes the task name, a detailed algorithm description, and explicit CUDA implementation guidance such as memory hierarchy usage, thread and block mapping strategies, and often explicit formulas. Level 2, “Algorithmic Specification,” includes the task name and algorithm description but omits CUDA-specific guidance, forcing the model to infer the grid, block, and memory mapping. Level 3, “Concept Retrieval,” provides only the task name and minimal I/O description, so the model must recover both the algorithmic definition and the CUDA implementation strategy from prior knowledge (Zhu et al., 13 Feb 2026).
The Haar wavelet transform example illustrates the progression. At Level 1, the prompt specifies an 8192-element signal, number of blocks, threads per block, per-block input size, the formulas for detail and approximation coefficients, and where results must be stored. At Level 2, the prompt describes the algorithm and output organization but does not prescribe CUDA mapping. At Level 3, only the task name and minimal functional description are given. This makes the difficulty axis an explicit probe of whether a model can translate from text to CUDA under varying levels of guidance (Zhu et al., 13 Feb 2026).
Each task includes a prompt, a reference I/O specification, and a reference implementation. Depending on the level, the model sees the task name, algorithmic description, tensor or array shapes, data types, and, for Level 1, CUDA constraints such as block and grid sizes and memory usage. The expected output is one or more CUDA kernels and any required helper functions that compile with NVCC and follow the validator’s input-output contract. No external CUDA libraries such as cuBLAS or cuDNN are relied on for correctness; kernels are expected to implement the operations explicitly (Zhu et al., 13 Feb 2026).
This prompt design distinguishes CUDABench-Set from PyTorch-to-CUDA or C-to-CUDA translation suites. Unlike KernelBench, TritonBench, or BabelTower, CUDABench-Set starts from pure textual specifications, from detailed natural language down to just a task name, and therefore evaluates a more general text-to-CUDA capability (Zhu et al., 13 Feb 2026).
4. Verification pipeline and roofline-based scoring
CUDABench-Set feeds into a Generative Verification Pipeline with three layers of evaluation: compilation correctness, functional consistency, and roofline-based performance. A generated kernel is valid only if it compiles successfully with NVCC and passes execution-based verification against reference outputs within domain-appropriate tolerances. Only kernels that pass both stages proceed to performance evaluation (Zhu et al., 13 Feb 2026).
Performance is measured using a roofline-based metric, Performance-Score. The benchmark collects execution time, total FLOPs, and total data movement. Arithmetic intensity is then used with the roofline model to compute attainable performance:
For a given kernel,
This yields a normalized score in , where 1 means the kernel is operating at the theoretical roofline. For memory-bound kernels, Performance-Score reduces to effective memory-bandwidth utilization; for compute-bound kernels, it becomes compute utilization (Zhu et al., 13 Feb 2026).
CUDABench-Score aggregates over all tasks by assigning zero to tasks that fail compilation or functional checks and otherwise averaging the per-task Performance-Score. This design deliberately prevents high performance on a small correct subset from masking broad failures in compilation or semantics (Zhu et al., 13 Feb 2026).
The paper argues that this hardware-normalized formulation is more informative than raw runtime alone. It demonstrates hardware-independence with an MSE kernel on A40 versus RTX 4090: despite absolute performance differences of 295.2 versus 554.9 GFLOPs/s, the optimized kernel’s Performance-Score remains relatively stable at 78.6% versus 74.8% (Zhu et al., 13 Feb 2026).
5. Empirical findings revealed by the benchmark
CUDABench-Set exposes a notable mismatch between high compilation success rates and much lower functional correctness. At Level 1, Compilation Pass@1 reaches 99.8% for Claude 4.5 Sonnet, 97.6% for Gemini 3 Flash, and 93.4% for GPT-5.2, but Functional Pass@1 is substantially lower, with 79.8% for GPT-5.2, 85.8% for Claude 4.5 Sonnet, and 76.8% for Qwen3-Max. The paper further notes that many failures are semantic rather than syntactic, involving boundary conditions, synchronization, memory hazards, and incorrect parallelization of nontrivial algorithms (Zhu et al., 13 Feb 2026).
Performance also degrades as prompt information is removed. GPT-5.2’s Functional Pass@1 drops from 79.8% at Level 1 to 75.2% at Level 2 and 60.8% at Level 3, and similar drops occur across models. Level 3, which supplies only the task name and minimal I/O description, therefore acts as a test of algorithm retrieval rather than only implementation fidelity (Zhu et al., 13 Feb 2026).
Domain-level failures are uneven. The paper reports lower failure in linear algebra and very high failure in scientific simulation and finance; for example, Claude 4.5 Sonnet has an 85% failure rate in that domain at Level 3. Examples include incorrect Black–Scholes formulas, omission of recursive decomposition in DWT, and incorrect discretization schemes in PDE solvers. This suggests that current models encode basic math and deep-learning operators more robustly than specialized numerical methods (Zhu et al., 13 Feb 2026).
Even when prompts are easy and guided, GPU utilization remains low. For Level 1 Pass@1 Score, GPT-5.2 reaches 40.9%, Claude 4.5 Sonnet 40.2%, and Gemini 3 Flash 40.1%. The paper interprets this as roughly 60% of potential GPU capacity remaining unused. The observed causes include poor memory coalescing, lack of shared-memory tiling and cache reuse, low occupancy from naive block and grid configuration, and warp divergence or excessive global-memory traffic (Zhu et al., 13 Feb 2026).
6. Position in the CUDA benchmark ecosystem
CUDABench-Set is positioned against several neighboring benchmarks. Compared with KernelBench, TritonBench, and BabelTower, it targets direct text-to-CUDA rather than language-to-language translation; compared with ComputeEval, it provides reference implementations, realistic large input sizes, and performance evaluation. Its novelty lies in combining direct text-to-CUDA tasks, multi-domain coverage, fully specified reference kernels with generators and validators, realistic input sizes including “Huge” instances above 1 GB, and a roofline-based performance metric (Zhu et al., 13 Feb 2026).
Within the broader ecosystem, SOL-ExecBench can be understood as analogous in spirit to a “CUDABench-Set” but focused on deep-learning kernels on NVIDIA Blackwell. Its authors explicitly contrast CUDABench as a 500-task, general-purpose text-to-CUDA benchmark with roofline-based scoring against hardware limits, whereas SOL-ExecBench is a Blackwell-focused, AI-kernel subset with analytically derived Speed-of-Light bounds and a more elaborate harness for agentic optimizers (Lin et al., 19 Mar 2026). This comparison highlights that CUDABench-Set is broad in domain scope, while later benchmarks have specialized toward model-extracted kernels and architecture-specific optimization targets.
CUDABeaver defines a complementary axis. It argues that benchmarks such as KernelBench, ComputeEval, CUDABench, and TritonBench evaluate generation from a specification, allowing models to pass by emitting slower but safer code. CUDABeaver instead starts from broken optimized CUDA workspaces and evaluates performance-preserving debugging. This suggests that CUDABench-Set measures synthesis capability from textual intent, whereas debugging-oriented suites measure whether optimized CUDA structure can be repaired without degeneration (Li et al., 8 May 2026).
The limitations identified for CUDABench-Set are correspondingly clear. It is CUDA-specific; its roofline parameters are defined per GPU even though Performance-Score is hardware-normalized; its six domains are broad but not exhaustive; and its unit of evaluation is the single kernel rather than multi-kernel fusion or whole-application execution. The paper points to extensions toward other GPU programming models, more complex task compositions, broader domain coverage, and use as a training or finetuning resource for specialized text-to-CUDA systems (Zhu et al., 13 Feb 2026).