CUDABench-Score: Text-to-CUDA Performance Metric

• CUDABench-Score is a unified metric that measures text-to-CUDA generation by integrating compilation success, functional correctness, and roofline-normalized performance.
• It utilizes a rigorous Generative Verification Pipeline including data preparation, kernel compilation, execution-based validation, and profiling with NVIDIA Nsight Compute.
• By penalizing compilation or correctness failures and normalizing valid kernels’ performance, the score provides a hardware-independent measure to guide CUDA code improvements.

Searching arXiv for the cited benchmark papers to ground the article in current literature. CUDABench-Score is a unified metric for evaluating text-to-CUDA generation that captures not only whether a generated kernel compiles, but also whether it is functionally correct and performance-efficient relative to the hardware’s theoretical limit (Zhu et al., 13 Feb 2026). It was introduced together with CUDABench, a benchmark designed to evaluate the text-to-CUDA capabilities of LLMs, and is computed as the final output of the Generative Verification Pipeline (Zhu et al., 13 Feb 2026). In contrast to benchmark practices that emphasize syntax, compilation success, or raw execution time, CUDABench-Score is intended to provide correctness-aware evaluation, hardware-independent performance evaluation, and a single scalar score that reflects both aspects together (Zhu et al., 13 Feb 2026).

1. Rationale and Benchmarking Context

The introduction of CUDABench-Score is motivated by the claim that existing code-generation benchmarks are insufficient for CUDA and GPU settings because they usually emphasize syntax or compilation success, and/or execution time on a specific machine (Zhu et al., 13 Feb 2026). For CUDA, this is presented as inadequate because a kernel can compile and even run correctly, yet still be a poor implementation if it leaves most of the GPU underutilized (Zhu et al., 13 Feb 2026). The same source states that raw runtime is highly dependent on the exact GPU model, memory bandwidth, and system conditions, so it is not a fair cross-hardware metric (Zhu et al., 13 Feb 2026).

Within CUDABench, this concern is situated in a broader benchmark framework. The benchmark constructs CUDABench-Set, which covers a Breadth-Depth-Difficulty evaluation space in diverse application domains, including artificial intelligence, scientific computing, and data analytics (Zhu et al., 13 Feb 2026). CUDABench-Score is the metric layer that operationalizes evaluation over this task set by combining validity and efficiency into one scalar quantity (Zhu et al., 13 Feb 2026).

This design distinguishes text-to-CUDA evaluation from earlier settings that primarily study translation from high-level languages into CUDA (Zhu et al., 13 Feb 2026). The paper explicitly frames text-to-CUDA generation as more general and more challenging (Zhu et al., 13 Feb 2026). A plausible implication is that the metric is not merely assessing syntactic rendering of known code patterns, but also the model’s ability to generate GPU kernels that satisfy correctness and resource-utilization constraints jointly.

2. Position within the Generative Verification Pipeline

CUDABench-Score is computed as the final output of the Generative Verification Pipeline, which has four stages (Zhu et al., 13 Feb 2026). These stages are Test data preparation, Kernel compilation, Execution-based functional verification, and Performance evaluation (Zhu et al., 13 Feb 2026).

In the first stage, the data generator creates random input data and reference outputs (Zhu et al., 13 Feb 2026). In the second stage, the generated CUDA code is compiled with NVCC (Zhu et al., 13 Feb 2026). In the third stage, the kernel is run on generated inputs, and its output is compared against the reference output (Zhu et al., 13 Feb 2026). In the fourth stage, for kernels that pass correctness checks, the system profiles them with NVIDIA Nsight Compute to measure execution time, FLOPs, and data movement, and then computes the roofline-based Performance-Score (Zhu et al., 13 Feb 2026).

A central property of the pipeline is its strict validity gating. Only kernels that pass both compilation and functional verification are scored for performance (Zhu et al., 13 Feb 2026). Failed kernels contribute zero to the final CUDABench-Score (Zhu et al., 13 Feb 2026). This establishes a two-stage filter: compilation validity and execution-based semantic validity must both hold before any performance credit is assigned (Zhu et al., 13 Feb 2026).

This pipeline structure gives CUDABench-Score a specific semantics. It is not a pure throughput metric and not a pure correctness metric. Rather, it is a correctness-gated efficiency metric in which invalid generations are collapsed to zero contribution, and valid generations are weighted by their roofline-normalized efficiency (Zhu et al., 13 Feb 2026).

3. Formal Definition

The score combines three ingredients: Compilation correctness, Functional correctness, and Performance-Score (Zhu et al., 13 Feb 2026). The paper defines the unified metric as

$\text{CUDABench-Score} = \frac{1}{N}\sum_{i=0}^N \left[\mathbbm{1}(\text{Compilation}_i \land \text{Functional}_i)\times \text{Performance-Score}_i\right]$

where $N$ is the total number of tasks in CUDABench-Set, $i$ is the task index, $\mathbbm{1}(\cdot)$ is an indicator function, $\text{Compilation}_i$ denotes whether task $i$ compiled successfully, $\text{Functional}_i$ denotes whether task $i$ passed execution-based verification, and $\text{Performance-Score}_i$ is the roofline-normalized performance for task $i$ (Zhu et al., 13 Feb 2026).

The operational interpretation is explicit. If a kernel fails compilation or functional checks, its contribution is 0 (Zhu et al., 13 Feb 2026). If it is valid, it contributes its normalized performance score (Zhu et al., 13 Feb 2026). The source further summarizes the metric as effectively the average performance of only the correct kernels, with incorrect kernels counted as zero (Zhu et al., 13 Feb 2026).

This definition gives the metric a sharply asymmetric penalty structure. Syntactically plausible but semantically incorrect CUDA receives no partial credit; likewise, semantically correct but badly underperforming kernels can contribute only in proportion to their normalized efficiency. This suggests that CUDABench-Score is designed to suppress the common ambiguity between “runnable code” and “usable GPU code,” which is particularly consequential in CUDA settings.

4. Roofline-Based Performance-Score

The key performance component of CUDABench-Score is the Performance-Score, which is based on the roofline model (Zhu et al., 13 Feb 2026). The paper defines the attainable performance for a kernel as

$N$0

This is described as the standard roofline upper bound, under which performance is limited either by compute throughput or by memory bandwidth depending on the kernel’s arithmetic intensity (Zhu et al., 13 Feb 2026).

The Performance-Score itself is defined as

$N$1

which normalizes observed performance by the kernel’s theoretically reachable upper bound (Zhu et al., 13 Feb 2026).

The paper further interprets this ratio by regime. For memory-bound kernels, the score simplifies to memory bandwidth utilization:

$N$2

For compute-bound kernels, the score simplifies to compute utilization:

$N$3

Accordingly, Performance-Score is presented as a unified, regime-aware efficiency measure: for memory-bound workloads, it measures how well memory bandwidth is used; for compute-bound workloads, it measures how close the kernel is to peak compute throughput (Zhu et al., 13 Feb 2026). The significance of this construction is that it avoids treating all kernels as if they should be judged by the same raw-time criterion. Instead, evaluation is normalized by the bottleneck class implied by arithmetic intensity and machine limits.

5. Interpretation and Empirical Meaning

A higher CUDABench-Score means that the model generated kernels that are more likely to compile, more likely to be functionally correct, and more efficient relative to hardware limits (Zhu et al., 13 Feb 2026). Conversely, a lower score indicates one or more of the following: compilation failures, semantic or functional errors, or poor GPU utilization even when the code is correct (Zhu et al., 13 Feb 2026). Because incorrect kernels contribute zero, the score strongly penalizes models that produce syntactically valid but incorrect CUDA (Zhu et al., 13 Feb 2026).

The metric is naturally normalized through the Performance-Score ratio. The paper states that 1.0 would mean the kernel reaches the roofline-attainable limit, whereas 0 means the kernel either fails correctness checks or has essentially no meaningful normalized performance contribution (Zhu et al., 13 Feb 2026). In the reported results, top models still only reach around the 20–40% range on CUDABench-Score in many settings, which the paper presents as evidence of substantial headroom (Zhu et al., 13 Feb 2026).

These semantics matter because the score does not conflate two different error modes. Compilation and functional errors are treated as validity failures; inefficient but valid kernels are treated as low-efficiency successes. This separation makes the metric informative about whether a model’s bottleneck lies in code formation, semantic fidelity, or hardware exploitation. The paper’s broader benchmark findings reinforce this interpretation by noting a notable mismatch between high compilation success rates and low functional correctness, as well as suboptimal utilization of GPU hardware resources (Zhu et al., 13 Feb 2026).

6. Limitations and Caveats

The authors identify several practical caveats in the use of CUDABench-Score (Zhu et al., 13 Feb 2026). First, performance is measured only after a kernel passes compilation and functional verification, so the score does not reward fast but incorrect kernels (Zhu et al., 13 Feb 2026). This is a deliberate design choice, but it also means the metric is inseparable from the strictness and coverage of the correctness validator.

Second, roofline-based scoring requires collecting execution time, FLOPs, and bytes moved, which is more involved than using runtime alone (Zhu et al., 13 Feb 2026). Third, the pipeline uses NVIDIA Nsight Compute for profiling, so the evaluation infrastructure is GPU-tooling dependent (Zhu et al., 13 Feb 2026). Fourth, the validator compares outputs against reference outputs, with care taken to avoid underflow by generating sufficiently large input values; this means correctness evaluation is tied to the chosen test generator and validation thresholds (Zhu et al., 13 Feb 2026). Fifth, although normalized, the score is still a benchmark metric derived from roofline modeling: it captures closeness to theoretical hardware limits, but not all possible production considerations (Zhu et al., 13 Feb 2026).

These caveats constrain interpretation. CUDABench-Score is a principled benchmark metric, but it is not equivalent to end-to-end application latency, maintainability, portability, or numerical robustness under all input distributions. A plausible implication is that the metric is best viewed as a controlled research instrument for comparing text-to-CUDA systems under a common evaluation protocol, rather than as a complete substitute for deployment-specific performance assessment.

7. Relation to Prior and Adjacent CUDA Benchmarks

CUDABench-Score is explicitly distinguished from earlier code-generation benchmark metrics that often focus on pass@k, compilation success, functional correctness, raw execution time, or combined heuristics such as $N$4 (Zhu et al., 13 Feb 2026). The same source states that such approaches often target PyTorch-to-CUDA or C-to-CUDA translation rather than true text-to-CUDA, rely heavily on execution time, do not normalize for whether a kernel is memory-bound or compute-bound, and often lack full reference implementations or realistic input sizes (Zhu et al., 13 Feb 2026).

The difference is therefore methodological rather than merely terminological. CUDABench-Score incorporates correctness and performance together, normalizes performance using the roofline model, accounts for different bottlenecks across kernels, is more hardware-independent than raw runtime, and better reflects whether a kernel uses GPU resources effectively (Zhu et al., 13 Feb 2026). In this framing, CUDA performance is fundamentally about balancing compute and memory behavior, not just generating runnable code (Zhu et al., 13 Feb 2026).

Related work on CUDA debugging introduces an adjacent but distinct evaluation concern. CUDABeaver, a benchmark for LLM-based automated CUDA debugging, argues that correctness-only scoring can overestimate debugging ability because a model may repair a failing CUDA program by making it safer but slower, a failure mode termed repair by degeneration (Li et al., 8 May 2026). Its proposed metric, $N$5, makes benchmark scores explicitly conditional on the model, corpus, and evaluation protocol, including a performance-gate threshold (Li et al., 8 May 2026). Although this is not CUDABench-Score, it underscores a common theme: CUDA evaluation is not adequately characterized by correctness alone, and performance preservation or performance efficiency must be made explicit in the metric design (Li et al., 8 May 2026).

An older point of comparison is cuROB, a CUDA-based real-parameter optimization benchmark that serves as GPU-native benchmark infrastructure and emphasizes efficient CUDA implementation and speedup relative to CPU-based suites such as CEC'14 (Ding et al., 2014). However, the source notes that cuROB does not define a new “score” metric in the sense of a single scalar CUDABench-Score formula; its benchmark “score” is effectively represented by runtime or speedup over the CPU baseline and by optimization quality on standard functions (Ding et al., 2014). This comparison helps situate CUDABench-Score within a newer class of CUDA benchmarks in which benchmarking is not only executed on GPUs, but also evaluated through correctness-aware, hardware-normalized scalar metrics tailored to generative models (Zhu et al., 13 Feb 2026).