---
title: 'CUDABench-Set: Text-to-CUDA Benchmark'
url: https://www.emergentmind.com/topics/cudabench-set
type: topic
---

# CUDABench-Set: Text-to-CUDA Benchmark

CUDABench-Set is the text-to-CUDA dataset at the core of the CUDABench benchmark. Each item corresponds to one CUDA kernel task together with natural-language or minimally specified prompts, a canonical reference CUDA implementation, and a data generator and validator for execution-based checking. Its purpose is to evaluate LLM capabilities for direct text-to-CUDA generation rather than code-to-code translation, and it integrates a roofline-based performance evaluation rather than only correctness or wall-clock time [2603.02236].

## 1. Definition and organizing principles

CUDABench-Set is structured around a 3D evaluation space called Breadth–Depth–Difficulty. This design is intended to separate domain coverage, problem scale, and prompt informativeness, so that failures in CUDA generation can be attributed to missing algorithmic knowledge, weak sequential-to-parallel mapping, or poor hardware utilization rather than being conflated into a single aggregate score [2603.02236].

| Dimension | Meaning | Instantiation |
|---|---|---|
| Breadth | Coverage of application domains and kernel types | Six GPU computing domains |
| Depth | Coverage of input size and problem scale | Five input scale levels: Tiny, Small, Medium, Large, Huge |
| Difficulty | Amount of task and implementation information in the prompt | Three prompt levels |

Breadth measures domain and workload diversity. Depth measures input size and performance stress, with each kernel defined at five scale levels. Difficulty measures how much information is given in the prompt, ranging from guided CUDA implementation hints to only the task name and minimal I/O description. This organization makes CUDABench-Set a benchmark not only of syntax generation, but of algorithm retrieval, CUDA mapping, and performance-oriented kernel design [2603.02236].

The construction methodology is also explicit. For each kernel task, the authors select a real-world kernel from open-source CUDA codebases, derive a textual specification at three difficulty levels, implement a reference CUDA kernel as ground truth, and provide a data generator and validator. This yields a benchmark in which textual intent, executable reference behavior, and performance evaluation are coupled at the task level rather than supplied as separate resources [2603.02236].

## 2. Corpus composition and workload coverage

CUDABench-Set contains 500 distinct tasks across six application domains, and each task has three difficulty levels, yielding 1,500 prompts in total [2603.02236]. The six domains are fundamental linear algebra, deep learning operators, computer vision and image processing, data analytics, signal processing, and scientific simulation and finance.

The domain mix is deliberately broad. Fundamental linear algebra includes kernels such as matrix multiplication, matrix transpose, element-wise operations, and least-squares solvers. Deep learning operators include activation functions such as GELU and Swish, loss functions such as Cross-Entropy and KL Divergence, normalization such as LayerNorm, and optimizers such as AdamW. Computer vision and image processing includes Gaussian blur, edge detection, Ball Query, and Furthest Point Sampling. Data analytics includes sorting, top-\(k\), histogram building, prefix sums, and reductions. Signal processing includes FIR filters and discrete wavelet transforms. Scientific simulation and finance includes Monte Carlo integration, PDE solvers such as FDTD, and Black–Scholes pricing [2603.02236].

The dataset is derived from production-level, open-source CUDA codebases, including PyTorch and related deep-learning operator libraries, NVIDIA CUDA Samples, and other production-level GPU repositories. This provenance matters because the benchmark is intended to mimic real-world GPU workloads rather than toy kernels. A plausible implication is that CUDABench-Set stresses both canonical CUDA idioms and domain-specific implementation patterns that arise in practice [2603.02236].

Task complexity also spans multiple performance regimes. The paper explicitly leverages the roofline model to characterize kernels through arithmetic intensity, defined as the ratio of the total number of FLOPs executed by the application to the total number of bytes transferred. Low-arithmetic-intensity kernels are memory-bound; high-arithmetic-intensity kernels are compute-bound. The benchmark therefore includes not only embarrassingly parallel elementwise kernels but also reductions, scans, recursive transforms, and numerically intensive scientific codes [2603.02236].

## 3. Prompt hierarchy and task specification

The three prompt levels define the difficulty axis. Level 1, “Guided Implementation,” includes the task name, a detailed algorithm description, and explicit CUDA implementation guidance such as memory hierarchy usage, thread and block mapping strategies, and often explicit formulas. Level 2, “Algorithmic Specification,” includes the task name and algorithm description but omits CUDA-specific guidance, forcing the model to infer the grid, block, and memory mapping. Level 3, “Concept Retrieval,” provides only the task name and minimal I/O description, so the model must recover both the algorithmic definition and the CUDA implementation strategy from prior knowledge [2603.02236].

The Haar wavelet transform example illustrates the progression. At Level 1, the prompt specifies an 8192-element signal, number of blocks, threads per block, per-block input size, the formulas for detail and approximation coefficients, and where results must be stored. At Level 2, the prompt describes the algorithm and output organization but does not prescribe CUDA mapping. At Level 3, only the task name and minimal functional description are given. This makes the difficulty axis an explicit probe of whether a model can translate from text to CUDA under varying levels of guidance [2603.02236].

Each task includes a prompt, a reference I/O specification, and a reference implementation. Depending on the level, the model sees the task name, algorithmic description, tensor or array shapes, data types, and, for Level 1, CUDA constraints such as block and grid sizes and memory usage. The expected output is one or more CUDA kernels and any required helper functions that compile with NVCC and follow the validator’s input-output contract. No external CUDA libraries such as cuBLAS or cuDNN are relied on for correctness; kernels are expected to implement the operations explicitly [2603.02236].

This prompt design distinguishes CUDABench-Set from PyTorch-to-CUDA or C-to-CUDA translation suites. Unlike KernelBench, TritonBench, or BabelTower, CUDABench-Set starts from pure textual specifications, from detailed natural language down to just a task name, and therefore evaluates a more general text-to-CUDA capability [2603.02236].

## 4. Verification pipeline and roofline-based scoring

CUDABench-Set feeds into a Generative Verification Pipeline with three layers of evaluation: compilation correctness, functional consistency, and roofline-based performance. A generated kernel is valid only if it compiles successfully with NVCC and passes execution-based verification against reference outputs within domain-appropriate tolerances. Only kernels that pass both stages proceed to performance evaluation [2603.02236].

Performance is measured using a roofline-based metric, Performance-Score. The benchmark collects execution time, total FLOPs, and total data movement. Arithmetic intensity is then used with the roofline model to compute attainable performance:

$$
\text{Attainable GFLOPs/sec}=
\min\{\text{Peak Floating-Point Performance},\ \text{Peak Memory Bandwidth}\times\text{Arithmetic Intensity}\}.
$$

For a given kernel,

$$
\text{Performance-Score}=
\frac{\text{Achieved GFLOPs/sec}}{\text{Attainable GFLOPs/sec}}.
$$

This yields a normalized score in \([0,1]\), where 1 means the kernel is operating at the theoretical roofline. For memory-bound kernels, Performance-Score reduces to effective memory-bandwidth utilization; for compute-bound kernels, it becomes compute utilization [2603.02236].

CUDABench-Score aggregates over all tasks by assigning zero to tasks that fail compilation or functional checks and otherwise averaging the per-task Performance-Score. This design deliberately prevents high performance on a small correct subset from masking broad failures in compilation or semantics [2603.02236].

The paper argues that this hardware-normalized formulation is more informative than raw runtime alone. It demonstrates hardware-independence with an MSE kernel on A40 versus RTX 4090: despite absolute performance differences of 295.2 versus 554.9 GFLOPs/s, the optimized kernel’s Performance-Score remains relatively stable at 78.6% versus 74.8% [2603.02236].

## 5. Empirical findings revealed by the benchmark

CUDABench-Set exposes a notable mismatch between high compilation success rates and much lower functional correctness. At Level 1, Compilation Pass@1 reaches 99.8% for Claude 4.5 Sonnet, 97.6% for Gemini 3 Flash, and 93.4% for GPT-5.2, but Functional Pass@1 is substantially lower, with 79.8% for GPT-5.2, 85.8% for Claude 4.5 Sonnet, and 76.8% for Qwen3-Max. The paper further notes that many failures are semantic rather than syntactic, involving boundary conditions, synchronization, memory hazards, and incorrect parallelization of nontrivial algorithms [2603.02236].

Performance also degrades as prompt information is removed. GPT-5.2’s Functional Pass@1 drops from 79.8% at Level 1 to 75.2% at Level 2 and 60.8% at Level 3, and similar drops occur across models. Level 3, which supplies only the task name and minimal I/O description, therefore acts as a test of algorithm retrieval rather than only implementation fidelity [2603.02236].

Domain-level failures are uneven. The paper reports lower failure in linear algebra and very high failure in scientific simulation and finance; for example, Claude 4.5 Sonnet has an 85% failure rate in that domain at Level 3. Examples include incorrect Black–Scholes formulas, omission of recursive decomposition in DWT, and incorrect discretization schemes in PDE solvers. This suggests that current models encode basic math and deep-learning operators more robustly than specialized numerical methods [2603.02236].

Even when prompts are easy and guided, GPU utilization remains low. For Level 1 Pass@1 Score, GPT-5.2 reaches 40.9%, Claude 4.5 Sonnet 40.2%, and Gemini 3 Flash 40.1%. The paper interprets this as roughly 60% of potential GPU capacity remaining unused. The observed causes include poor memory coalescing, lack of shared-memory tiling and cache reuse, low occupancy from naive block and grid configuration, and warp divergence or excessive global-memory traffic [2603.02236].

## 6. Position in the CUDA benchmark ecosystem

CUDABench-Set is positioned against several neighboring benchmarks. Compared with KernelBench, TritonBench, and BabelTower, it targets direct text-to-CUDA rather than language-to-language translation; compared with ComputeEval, it provides reference implementations, realistic large input sizes, and performance evaluation. Its novelty lies in combining direct text-to-CUDA tasks, multi-domain coverage, fully specified reference kernels with generators and validators, realistic input sizes including “Huge” instances above 1 GB, and a roofline-based performance metric [2603.02236].

Within the broader ecosystem, SOL-ExecBench can be understood as analogous in spirit to a “CUDABench-Set” but focused on deep-learning kernels on NVIDIA Blackwell. Its authors explicitly contrast CUDABench as a 500-task, general-purpose text-to-CUDA benchmark with roofline-based scoring against hardware limits, whereas SOL-ExecBench is a Blackwell-focused, AI-kernel subset with analytically derived Speed-of-Light bounds and a more elaborate harness for agentic optimizers [2603.19173]. This comparison highlights that CUDABench-Set is broad in domain scope, while later benchmarks have specialized toward model-extracted kernels and architecture-specific optimization targets.

CUDABeaver defines a complementary axis. It argues that benchmarks such as KernelBench, ComputeEval, CUDABench, and TritonBench evaluate generation from a specification, allowing models to pass by emitting slower but safer code. CUDABeaver instead starts from broken optimized CUDA workspaces and evaluates performance-preserving debugging. This suggests that CUDABench-Set measures synthesis capability from textual intent, whereas debugging-oriented suites measure whether optimized CUDA structure can be repaired without degeneration [2605.08455].

The limitations identified for CUDABench-Set are correspondingly clear. It is CUDA-specific; its roofline parameters are defined per GPU even though Performance-Score is hardware-normalized; its six domains are broad but not exhaustive; and its unit of evaluation is the single kernel rather than multi-kernel fusion or whole-application execution. The paper points to extensions toward other GPU programming models, more complex task compositions, broader domain coverage, and use as a training or finetuning resource for specialized text-to-CUDA systems [2603.02236].

Source: https://www.emergentmind.com/topics/cudabench-set