Effect of a Larger LLM Refinement Budget on the Triton–cuTile Gap

Determine whether increasing the iterative LLM code-generation budget beyond 10 refinement iterations would eliminate the observed performance and coverage gap between Triton-generated and cuTile-generated kernels under the TileBench evaluation protocol.

Background

TileBench evaluates LLM-generated Triton and cuTile kernels using a maximum of 10 refinement iterations per backend. A backend is frozen when it produces a verification-passing implementation reaching at least 80% roofline utilization; otherwise, refinement stops after the ten-iteration budget is exhausted.

The reported results show that Triton achieves broader verify-clean coverage, higher correctness-gated speedup, and greater token efficiency than cuTile for both evaluated LLMs. However, the experiment does not vary the iteration budget, so it leaves unresolved whether additional refinement opportunities would close the observed backend difference or whether the gap would persist.

References

The trajectories show that later refinement can improve both backends, but the final experiment does not vary the iteration budget and therefore does not establish whether a larger budget would eliminate the backend gap.

— TileBench: A Controlled Benchmark for Performance Evaluation and Bottleneck Diagnosis of Tile-Based Programming Models  (2609.29067 - Cui et al., 24 Sep 2026) in Appendix, Section “Additional Evaluation Results,” subsection “RQ4: Iterative Refinement Trajectories”