Generalization of KernelBench-Optimized CUDA Kernels to Real-World Production

Determine how CUDA kernels optimized on benchmarks such as KernelBench—particularly those produced by RL-augmented LLM systems like Sakana AI's AI CUDA Engineer and CUDA-L1, which evaluate each operation on a single fixed configuration—translate to real-world production environments in terms of performance and applicability.

Background

The paper notes a recent surge of work using LLMs and reinforcement learning (RL) to automatically generate CUDA kernels, with many efforts evaluating on benchmarks like KernelBench. KernelBench includes diverse operations but evaluates each on a single, fixed configuration, leaving uncertainty about how such optimizations carry over to production workloads that vary widely in input sizes and conditions.

The authors explicitly state that it remains unclear whether kernels optimized under benchmark settings generalize to real-world deployment. This motivates their focus on matrix multiplication across 1,000 (M, N, K) configurations and comparisons with strong baselines (cuBLAS, cuBLASLt) to address practical applicability beyond benchmark scenarios.

References

Existing work, such as Sakana AI's AI CUDA Engineer and CUDA-L1, primarily optimize kernels from benchmarks like KernelBench, which covers a diverse range of CUDA operations, each evaluated on a single, fixed configuration (e.g., one specific input dimension). However, it remains unclear how these benchmark-optimized kernels translate to real-world production environments.

— CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning  (2512.02551 - Su et al., 2 Dec 2025) in Section 1 (Introduction)

We have not yet applied the three-transform algorithm to a specific physical process. The correlators and kinematic factors depend on the chosen process and determine both the coordinate grids and the GPU calculation. We leave this application in the next-leading order p-A CGC calculations and its performance study to future work.

— Fourier Transforms of Color Glass Condensate Multi-Wilson-Line Correlators via Filon Quadrature  (2608.18589 - Duan et al., 19 Aug 2026) in Section 5.5, “Outlook for physical applications” (also discussed in the Conclusion)