Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent

Abstract: Existing GPU kernel generation benchmarks draw problems from synthetic or curated sources that diverge from deployed workloads. We present Atrex-Bench, a benchmark whose 30 operators and 440 shapes are sampled directly from full-cluster production inference traces of compute-limited, memory-rich GPUs. Each problem carries an importance weight derived from its share of observed GPU time, weighted by application card-hours and computed separately for the serving phases in which it runs, together with a per-problem roofline ceiling, so the aggregate score emphasizes the kernels that consume the most serving time. Evaluating six frontier coding agents on Atrex-Bench shows that even the best vanilla model reaches only ${\sim}10\%$ of the hardware roofline on production operators; and correctness alone overstates capability, since much of the apparent pass rate comes from PyTorch fallbacks rather than kernels the model wrote. To close this gap, we co-release Atrex-Kernel-Agent (AKA), a profile-driven kernel-optimization agent that combines iterative measure-revise search, optimization dropout for escaping stalled search contexts, and a layered GPU-optimization knowledge base (298 reference-kernel files and 244 optimization-knowledge documents, plus external upstream reference projects for API/ISA lookup). In a controlled case study, the agent converts zero-FlyDSL fallbacks into real kernels that match or exceed hand-tuned production baselines.

• The paper introduces Atrex-Bench, a trace-driven benchmark of 30 operators and 440 production-derived shapes, showing that the best agent achieves only 10.7% of the hardware roofline and no agent matches production kernels.
• Production-weighted scoring reveals that correctness overstates capability because agents often rely on PyTorch or vendor-kernel fallbacks, while compute-bound fused quantization and matrix operations remain especially difficult.
• The paper’s Atrex-Kernel-Agent combines profiling, documented optimization tactics, and controlled iteration to convert fallbacks into target-DSL kernels that reach or exceed production performance in a three-operator case study.

Motivation and positioning

Existing benchmarks for LLM kernel generation—KernelBench, BackendBench, TritonBench, MultiKernelBench, FlashInfer-Bench, CUDABench, and SOL-ExecBench—draw their problem sets from synthetic or curated sources. The authors argue that such suites cannot answer the production-readiness question because they diverge from deployed workloads on three axes: which tensor shapes actually run (production fleets are heavily skewed), which kernels matter (an unweighted average treats a rare elementwise op the same as a fused-attention path), and how good is good enough (a deployable kernel must approach the hardware roofline, not merely beat an unoptimized baseline). Atrex-Bench is designed to close all three gaps simultaneously; no prior benchmark combines production sampling, per-problem roofline bounds, and importance-weighted aggregation.

Benchmark design

Atrex-Bench samples problems directly from online inference traces collected on XPU-A (a desensitized non-NVIDIA accelerator) and H20 clusters with more than 10k deployed accelerators, spanning vLLM, SGLang, AITER, and RTP-LLM and roughly 20 production models including Qwen3 MoE variants, DeepSeek-R1, and dense Qwen deployments. The current release contains 30 operators and 440 hot shapes drawn from 1,303 profiles. Each operator ships as a five-file contract: a visible PyTorch reference, input generator, and shape set, plus hidden metadata.json (provenance) and roofline.json (scoring denominators). This generation–evaluation boundary prevents agents from recovering the upstream kernel by name or tuning to the roofline formula.

Two scoring ingredients distinguish the benchmark. First, each (operator, shape) unit receives a speed-of-light latency derived from calibrated hardware peaks and the reference's semantic FLOPs and memory traffic: $$T_{\text{roofline}} = \max(F/P_{\text{peak}},\ M/\beta)$$. Second, operators receive importance weights $w_i$ equal to their share of observed device time, weighted by application card-hours and computed separately for prefill and decode phases. The headline metric $S_{\text{agg}} = \sum_i w_i S_i$ weights each operator's median per-shape roofline achievement by its production time share, assigning zero to operators with no correct kernel. The weight distribution is heavily skewed: the top five operators (unified_attention at 36.1%, fused_moe, block_scaled_mm, fp8_blockscale_fused_moe, paged_attention_decode) carry approximately 64% of total weight, and the top ten carry roughly 80%. Shape counts are deliberately decoupled from importance—rms_norm ships 56 shapes but only 2.5% of weight—so acing many light operators cannot compensate for failing heavy ones. The suite also spans five precisions and seven orders of magnitude in both FLOPs and bytes per shape, covering memory-bound, compute-bound, and pure data-movement regimes.

Evaluation of frontier agents

Six frontier coding agents—Claude Opus 4.7, GPT-5.5, Qwen3.7-Max, Kimi-K2.6, GLM-5.1, and DeepSeek-V4-Pro—were evaluated on all 440 units targeting FlyDSL, a DSL essentially absent from pre-training corpora, so results probe in-context learning rather than recall. The central finding is stark: the best candidate reaches only 10.7% of the hardware roofline ($S_{\text{agg}} = 0.107$ for GPT-5.5), and no agent matches the deployed production kernel—Opus 4.7 comes closest at 0.99×, while the weakest trails at 0.12×. Correctness spans 46.2% (GLM-5.1) to 92.0% (Opus 4.7), but correctness does not translate into performance.

Model	Compile (%)	Correct (%)	FlyDSL (%)	$S_{\text{agg}}$	vs. torch.compile	vs. prod
Claude Opus 4.7	99.6	92.0	78.5	0.059	2.29×	0.99×
GPT-5.5	100.0	91.1	71.6	0.107	3.06×	0.85×
Qwen3.7-Max	97.1	84.8	43.8	0.047	1.10×	0.19×
Kimi-K2.6	91.5	81.5	40.1	0.043	0.94×	0.33×
GLM-5.1	60.9	46.2	38.6	0.015	0.97×	0.33×
DeepSeek-V4-Pro	81.0	62.3	36.4	0.012	0.63×	0.12×

Importance weighting reorders the leaders: Opus 4.7 edges GPT-5.5 on unweighted medians (0.104 vs. 0.129), but the weighted score puts GPT-5.5 well ahead (0.107 vs. 0.059), because Opus's weakness concentrates precisely in the compute-bound operators that dominate production wall-time—its median achievement there is 0.009 against GPT-5.5's 0.074. Across the panel, every model reaches several times more of the roof on memory-bound shapes than compute-bound ones, indicating that agents have learned bandwidth optimization far more thoroughly than matrix-engine scheduling.

The correctness illusion

A passing kernel is not necessarily a written kernel. Because the compile gate admits PyTorch fallbacks and vendor-kernel calls, models can accumulate correctness while writing little target code: Qwen3.7-Max achieves 84.8% correctness with only 43.8% FlyDSL adoption, answering attention via scaled_dot_product_attention and GEMM via precompiled AITER entry points. For the middle of the field, roughly half of fully correct operators run on non-DSL paths. Opus 4.7 is the exception, with more FlyDSL-dominant operators (26) than fully correct ones (24). This is specification shortcutting rather than answer leakage—the hidden-file contract rules out provenance or roofline exploitation—and the FlyDSL-adoption metric makes it measurable.

Hardness, cost, and failure modes

Operator difficulty is largely model-independent: nine operators are solved by every model, while the hardest cluster—fp8_blockscale_fused_moe (22.2% mean pass rate), fused_rmsnorm_quant (34.8%), per_token_group_quant_fp8 (44.7%), block_scaled_mm (56.9%)—consists uniformly of low-precision quantization fused with a second operation, marking a shared capability frontier. Generation volume does not predict quality: DeepSeek-V4-Pro emits the most tokens (6.56M) for only 18 correct operators, while GPT-5.5 reaches 26 correct operators on 1.19M tokens (46K per correct operator).

Of 683 failing units, 53.4% never compile (concentrated in GLM-5.1 and DeepSeek-V4-Pro), and among compiled kernels the dominant failure is silent numeric mismatch (257 units)—catchable only by multi-seed numerical checking. Notably, Opus 4.7's failures are half correct-but-slow kernels, a signature that its gap is optimization knowledge rather than coding ability.

Atrex-Kernel-Agent

AKA targets the two residual gaps—fallback-dominated passes and low roofline achievement—with three mechanisms: a profile-driven measure–revise loop grounded in official profilers (ncu, rocprofv3), an "optimization dropout" partial restart that masks stale iteration memories while preserving accepted kernels and audit trails, and a layered GPU Wiki knowledge base of 298 reference kernels and 244 optimization documents plus cached upstream projects (CUTLASS, FlyDSL, Triton, AITER, FlashInfer, FlashMLA). Every iteration must convert profiler evidence into a bottleneck hypothesis, retrieve documented tactics, plan explicitly, and change exactly one optimization category before validation; every hardware-spec value must be cited from GPU Wiki.

The evaluation is explicitly scoped as a controlled case study, not a benchmark-wide ranking: two base models (Qwen3.7-Max, Opus 4.7) under vanilla versus optimizer-augmented conditions, on three compute-bound operators at one production shape each, with single timed runs. The results are nonetheless strong:

Model	Operator	FlyDSL	Roofline $S$	vs. prod
Qwen3.7-Max	chunk_gated_delta	0% → ~100%	0.001 → 0.03	0.03× → 1.20×
Qwen3.7-Max	attention_forward	0% → 99%	0.06 → 0.40	0.17× → 1.11×
Qwen3.7-Max	mla_decode	14% → 87%	0.0003 → 0.0035	0.10× → 1.06×
Opus 4.7	chunk_gated_delta	0% → ~100%	0.001 → 0.03	0.04× → 1.31×
Opus 4.7	attention_forward	~100% → 99%	0.28 → 0.42	0.78× → 1.17×
Opus 4.7	mla_decode	92% → 92%	0.0023 → 0.0042	0.71× → 1.27×

For the weaker model the workflow converts 0%-FlyDSL fallbacks into near-100%-FlyDSL kernels—for example, a recurrence 29× slower than production becomes a kernel beating it by 1.2×. For the stronger model it closes the residual roofline gap on already-written kernels. On mla_decode_attention, whose ceiling is a hand-written assembly kernel, the agent adopts a split-KV restructuring with online-softmax reduction and edges past the hand-tuned AITER kernel at 1.06×. The authors note candidly that overtaking AITER on attention_forward is partly enabled by the awkward test shape (head dimension 72, non-power-of-two for MFMA tiling), where the general varlen kernel itself reaches only ~36% of the roof—shape specialization, not general superiority, explains that margin.

Limitations

The paper concedes several constraints plainly. Empirical results are reported only on XPU-A despite traces covering H20 as well; cross-accelerator validation remains open. The trace slice covers compute-limited, memory-rich fleets specifically, so the sampled problems represent that deployment slice rather than all hardware classes. The release is inference-only—backward and optimizer-step kernels are out of scope. The AKA evaluation rests on three operators, one shape each, and single timed runs, so its gains should be read as case-study evidence rather than distribution-level claims. The FlyDSL-only target also means results may not transfer to Triton, Gluon, or CuteDSL, which the prompt framework supports but defers to future releases.

Conclusion

Atrex-Bench establishes that frontier LLM agents, measured against production-derived workloads and importance-weighted roofline ceilings, remain far from deployable—at most ~10% of the hardware roof, with no candidate matching hand-tuned production kernels—and that correctness metrics materially overstate capability due to fallback shortcutting. The co-released AKA demonstrates that profile-driven search over a curated knowledge base can convert fallbacks into target-DSL kernels that match or exceed production baselines, localizing the remaining deficit in retrievable domain knowledge rather than raw coding ability. Both artifacts are open-source, and the benchmark's refreshable weighting scheme is designed to track evolving production traffic under versioned snapshots.