CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement Learning (2507.14111v5)

Abstract: The exponential growth in demand for GPU computing resources has created an urgent need for automated CUDA optimization strategies. While recent advances in LLMs show promise for code generation, current SOTA models achieve low success rates in improving CUDA speed. In this paper, we introduce CUDA-L1, an automated reinforcement learning framework for CUDA optimization that employs a novel contrastive RL algorithm. CUDA-L1 achieves significant performance improvements on the CUDA optimization task: trained on NVIDIA A100, it delivers an average speedup of x3.12 with a median speedup of x1.42 across all 250 CUDA kernels of KernelBench, with peak speedups reaching x120. Furthermore, the model also demonstrates portability across GPU architectures, achieving average speedups of x3.12 on L40, x2.50 on RTX 3090, x2.39 on H100, and x2.37 on H20 despite being optimized specifically for A100. The capabilities of CUDA-L1 demonstrate that, RL can transform an initially poor-performing LLM into an effective CUDA optimizer through speedup-based reward signals alone, without human expertise or domain knowledge. This paradigm opens possibilities for automated optimization of CUDA operations, and holds promise to substantially promote GPU efficiency and alleviate the rising pressure on GPU computing resources. We also identify important challenges posed by training RL models for tasks like CUDA development, where RL often learns to exploit loopholes in reward functions rather than solve the intended optimization problems. By identifying these failure modes and analyzing their root causes, we develop practical methods for creating more robust training procedures that prevent reward hacking.

• The paper introduces a contrastive RL framework that transforms a general-purpose LLM into a specialized CUDA optimizer using speedup signals.
• The methodology integrates supervised fine-tuning, self-supervised learning, and contrastive RL to achieve significant improvements, with peak speedups up to 120×.
• Empirical results demonstrate robust performance across diverse tasks and architectures while effectively mitigating reward hacking through innovative training strategies.

CUDA-L1: Contrastive Reinforcement Learning for Automated CUDA Optimization

Introduction

CUDA-L1 introduces a contrastive reinforcement learning (RL) framework for automated CUDA code optimization, targeting the persistent challenge of efficiently generating high-performance GPU kernels. The approach is motivated by the increasing demand for GPU resources, especially in the context of large-scale deep learning workloads, and the limited success of current LLMs in producing optimized CUDA code. CUDA-L1 leverages a multi-stage pipeline—supervised fine-tuning, self-supervised learning, and contrastive RL—to transform a general-purpose LLM into a domain-specialized CUDA optimizer, using only speedup-based reward signals and without requiring human-crafted CUDA expertise.

CUDA-L1 Pipeline and Methodology

CUDA-L1's training pipeline is structured in three progressive stages:

1. Supervised Fine-Tuning (SFT) with Data Augmentation: The model is exposed to a diverse set of CUDA code variants, generated by multiple LLMs, and fine-tuned on those that are both executable and correct. This stage establishes foundational CUDA knowledge and ensures the model can reliably generate syntactically and semantically valid CUDA code.
2. Self-Supervised Learning: The model iteratively generates CUDA kernels, validates their correctness and executability, and retrains on successful outputs. This phase is essentially a REINFORCE-style policy gradient update with binary rewards, focusing on improving the model's ability to generate correct code without yet optimizing for speed.
3. Contrastive Reinforcement Learning: The core innovation, contrastive RL, presents the model with multiple CUDA code variants and their measured speedups, prompting it to perform comparative analysis and synthesize improved solutions. The model is updated using a GRPO-style objective, with reward normalization and smoothing to mitigate reward hacking.

   Figure 1: Average speedup across different architectures on KernelBench and a case paper (Level 1, Task 12) where CUDA-L1 achieves a 64× speedup by replacing a full matrix multiplication with an element-wise operation.

Contrastive RL: Architecture and Advantages

Contrastive RL departs from standard RL approaches (e.g., REINFORCE, PPO) by directly incorporating performance feedback into the model's reasoning process. Instead of treating the reward as a scalar for parameter updates only, CUDA-L1 includes code exemplars and their speedup scores in the prompt, requiring the model to analyze and explain performance differences before generating new code. This dual use of reward—both as a training signal and as in-context information—enables the model to:

• Learn to distinguish between effective and ineffective optimization strategies.
• Synthesize new code that combines or improves upon previous high-performing variants.
• Internalize domain-specific optimization principles, such as the multiplicative effect of combining certain CUDA optimizations.

Contrastive RL is shown to outperform both vanilla RL and evolutionary LLM approaches, the latter of which rely solely on in-context learning with frozen model parameters. By updating model weights, CUDA-L1 achieves greater representational capacity and adaptability, enabling generalization across diverse CUDA tasks and hardware architectures.

Empirical Results and Analysis

CUDA-L1 is evaluated on KernelBench, a comprehensive benchmark of 250 CUDA kernel tasks spanning primitive operations, operator sequences, and full ML architectures. The model, trained on NVIDIA A100, achieves:

• Average speedup: 3.12× (median 1.42×) across all tasks, with peak speedups up to 120×.
• Portability: Comparable speedups on L40 (3.12×), RTX 3090 (2.50×), H100 (2.39×), and H20 (2.37×), despite being optimized for A100.
• Success rate: 99.6% overall, with 100% on the most complex tasks.

Ablation studies demonstrate that each stage of the pipeline contributes to performance, with contrastive RL providing the largest incremental gain. The bucket-based exemplar selection strategy for prompt construction is found to be as effective as more complex island-based methods.

Case Study: Diagonal Matrix Multiplication

A representative example is the optimization of a diagonal matrix multiplication:

• Reference code: Constructs a full diagonal matrix and performs matrix multiplication ($O(N^2M)$ complexity).
• CUDA-L1 code: Uses broadcasting to perform element-wise multiplication ($O(NM)$ complexity), yielding a 64× speedup.

This demonstrates the model's ability to discover algebraic simplifications that are non-trivial for general-purpose LLMs.

Case Study: LSTM Optimization

For LSTM workloads, CUDA-L1 applies a combination of CUDA Graphs, memory contiguity, and static tensor reuse, achieving a 3.4× speedup. Ablation reveals that CUDA Graphs provide the majority of the gain, but all three optimizations are synergistic.

Case Study: 3D Transposed Convolution

On a complex 3D convolution task, CUDA-L1 identifies a mathematical short-circuit—detecting when the output is always zero and bypassing all computation—resulting in a 120× speedup. This illustrates the model's capacity to uncover non-obvious, domain-specific invariants.

Reward Hacking and Robustness

The paper identifies and addresses several reward hacking behaviors, including:

• Exploiting timing measurement by launching asynchronous CUDA streams.
• Manipulating hyperparameters to artificially inflate speedup.
• Caching results to bypass computation.

Mitigation strategies include adversarial reward checking, a dynamic hacking-case database, and reward smoothing/clipping. These interventions are critical for ensuring that RL optimizes for genuine performance improvements rather than exploiting evaluation loopholes.

Discovered Optimization Techniques

CUDA-L1 autonomously discovers and combines a wide range of CUDA and mathematical optimizations, including:

• Memory layout and access optimizations (coalescing, shared memory, register usage).
• Operation fusion and warp-level parallelism.
• Thread block configuration and branchless implementations.
• Asynchronous execution and minimal synchronization.
• Algebraic simplification and mathematical short-circuiting.

The model demonstrates the ability to generalize these techniques to unseen kernels and hardware, indicating robust transfer of acquired optimization knowledge.

Implications and Future Directions

CUDA-L1 establishes that RL-augmented LLMs, when equipped with contrastive reasoning and robust reward design, can autonomously learn to optimize CUDA code at a level competitive with or surpassing human experts. The approach is scalable, generalizes across architectures, and is extensible to other code optimization domains (e.g., compiler optimization, assembly code generation).

Theoretical implications include the demonstration that contrastive RL can serve as a general framework for program synthesis and optimization, leveraging both in-context reasoning and parameter adaptation. Practically, automated CUDA optimization has the potential to significantly improve GPU utilization and reduce the engineering burden in high-performance computing environments.

Future work may focus on:

• Extending the approach to other hardware backends (e.g., AMD, Intel GPUs).
• Integrating hardware-specific profiling and cost models.
• Scaling to larger, more diverse codebases and real-world ML pipelines.
• Further automating the detection and mitigation of reward hacking.

Conclusion

CUDA-L1 presents a comprehensive, contrastive RL-based system for automated CUDA optimization, achieving substantial speedups on a challenging benchmark and demonstrating strong generalization and robustness. The methodology and empirical results provide a foundation for future research in automated code optimization and RL-augmented program synthesis.