Robust, comprehensive evaluation for AI-driven kernel generation

Develop robust and comprehensive evaluation protocols for AI-driven GPU kernel generation that jointly assess robustness and generalization across input shapes, operator types, and hardware ecosystems, overcoming current limitations of benchmarks confined to fixed shapes and NVIDIA-only forward-pass primitives.

Background

The survey notes that prevailing benchmarks inadequately reflect real-world workloads because they are often restricted to fixed input shapes and forward-pass primitives within the NVIDIA ecosystem. This limitation undermines reliable assessment of generalization and robustness, which are essential for production-grade kernel generation.

To address these gaps, the authors call for evaluation protocols that jointly test across shapes, operators, and heterogeneous hardware ecosystems, providing a stronger foundation for measuring progress in kernel generation research.

References

A key open challenge in AI-driven kernel generation is the lack of robust and comprehensive evaluation.

— Towards Automated Kernel Generation in the Era of LLMs  (2601.15727 - Yu et al., 22 Jan 2026) in Section 7 (Challenges and Opportunities), paragraph "Evaluation Robustness and Generalization"

Existing RTL-agent benchmarks generally do not include a mechanism to detect over-solving or reward hacking . They measure whether the submitted artifact passes the provided harness, but they usually do not separate debugging feedback from final hidden scoring, test robustness under randomized or perturbed stimuli, or audit whether a solution has specialized to artifacts of the evaluator. This is an open problem for the community.

— Agentic Hardware Design as Repository-Level Code Evolution  (2606.28279 - Yu et al., 26 Jun 2026) in Section 6, Discussion and Limitations

Several limitations remain, and each is a future direction this work leaves open. i) Section~\ref{sec:theory} says enough about bit-level semantics for a descriptor to hold across machines and across library versions: none of its fields names a machine, and the split is written down rather than read off the hardware, so it survives the change in SM count at which cuBLAS's own guarantee lapses. Measuring that cross-machine robustness is what the resources behind this work did not reach, due to resource limitations.

— Taming Bitwise Behavior in GPU Kernels with Tensor Core: Black-Box Reconstruction, Compiler Enforcement, and Static Verification  (2609.11356 - Yang et al., 10 Sep 2026) in Section 7, Discussion and future work, item (i)