- The paper presents a harness-engineering workflow that combines evaluation, profiling, bounded optimization, promotion gates, and artifact memory to guide LLM agents toward correct, fast GPU kernels.
- Agent-assisted kernels achieved 1.12×–29.68× mean-latency speedups over FlashInfer baselines across five MoE, sparse-attention, and Gated DeltaNet operators, consistently outperforming matched autonomous-agent results.
- The study shows that conservative validation is essential because low-trial testing can miss rare numerical errors, as demonstrated by DSA top-k failures detected through 100-trial replay and resolved with a fallback implementation.
This paper reports a harness-engineering approach to LLM-driven GPU kernel generation, developed for the MLSys 2026 FlashInfer AI Kernel Generation Contest on NVIDIA Blackwell B200 GPUs. Rather than proposing a new model, runtime, or autonomous optimizer, the work studies the surrounding engineering loop—evaluation harness, optimization controller, skills, promotion gates, and artifact memory—that makes LLM coding agents produce correct and fast drop-in kernels. Across five operator definitions spanning three contest tracks (FP8 Fused MoE, DeepSeek Sparse Attention (DSA), and Gated DeltaNet (GDN)), the retained kernels achieve mean-latency speedups over the supplied FlashInfer baselines of 1.62×, 18.05×, 29.68×, 1.12×, and 13.70×, and matched Full-Agent baselines are consistently slower than the Agent-Assisted artifacts (2607.17979).
Motivation and problem setting
The contest definitions reflect mechanisms used in production LLM serving: DeepSeekMoE-style top-8 routed FP8 experts, DeepSeek-V3.2 sparse attention with a lightning indexer and top-k page selection, and the gated delta-rule recurrence from Gated Delta Networks. These operators expose irregular shapes, mixed precision, short decode regimes, and long-context regimes, so a candidate kernel must satisfy a packaging contract, compile in the target container, preserve numerical semantics, and improve mean latency over the full workload distribution—not merely a representative case. The paper's central claim is that failures in LLM-driven kernel generation typically arise from the surrounding loop: stale baselines, incomplete workload coverage, packaging drift, noisy promotion, and loss of profiler or provenance information. Harness engineering is presented as the mechanism that addresses these failure modes, following the position that humans should design environments, constraints, and feedback loops while agents perform bounded implementation search.
Harness and controller architecture
The system separates two layers. The evaluation harness is the measurement layer: it packages candidates, compiles them in an official-aligned environment (CUDA 13.2, PyTorch 2.12, Triton 3.6, cupti-python timing, isolated B200 runners), runs correctness checks, measures latency, and archives benchmark and profiler artifacts. The optimization controller is the decision layer: it converts NCU and Torch Profiler evidence into reusable bottleneck state, selects one bounded optimization hypothesis per round, supervises plateaus and regressions, and records accepted and rejected evidence as trajectory memory.
Expert practice is encoded as reusable skills rather than ad hoc prompts: a generic CUDA optimizer skill standardizes the candidate contract, evaluation, NCU/NSYS profiling, and two-stage search; a FlashInfer B200 contest skill adds reference-first reconnaissance, workload discovery, paired same-round gates, and official-parity evaluation; a unified controller skill codifies a stateful round contract with state extraction, state matching, one-direction optimization, and manifest-based resumption. Promotion is deliberately conservative: a candidate is promoted only if the full workload distribution improves without correctness regressions, and rejected probes are archived as negative evidence to prevent re-exploration of failed launch-bound, fusion, or tile-size hypotheses. Sub-agents are used for parallel exploration and review, but promotion remains centralized—sub-agent outputs are proposals, not authority.
Agent-assisted workflow
The primary workflow uses Codex (GPT-5.3-Codex family) and Claude Code (Claude Opus 4.6) as coding agents under a human-authored harness. Humans design skills and evaluators, curate references (FlashInfer, DeepGEMM, TensorRT-LLM, CUTLASS), decide evaluation budgets, and make final promotion decisions; agents generate CUDA, Triton, and CuTe candidates, interpret profiles, and propose route changes. Each round follows a fixed sequence: identify the bottleneck and workload regime, generate a bounded candidate family around one hypothesis, run paired representative gates, profile promising routes, integrate shape-local wins behind cheap dispatch rules, run a full sweep, and promote only on distribution-wide improvement. Plateau recovery is explicitly human-guided—switching model families, requesting sub-agent debate, supplying references, or changing implementation languages. The paper also ran Full-Agent experiments with LoongFlow plan-execute-summarize search as same-protocol baselines.
Results
Retained Agent-Assisted kernels outperform both the supplied FlashInfer baselines and matched Full-Agent artifacts on every definition:
| Definition |
Agent-Assisted (ms) |
Speedup vs. FlashInfer |
Full-Agent speedup vs. FlashInfer |
| DSA attention |
0.011175 |
29.68× |
14.54× |
| DSA top-k indexer |
0.006893 |
18.05× |
3.81× |
| GDN prefill |
0.051992 |
13.70× |
1.03× |
| MoE FP8 |
0.286342 |
1.62× |
0.27× |
| GDN decode |
0.006201 |
1.12× |
0.83× |
The Full-Agent artifacts are 1.35–13.25× slower than the Agent-Assisted retained kernels, and two remain below the supplied FlashInfer baseline (MoE FP8 at 0.27×, GDN decode at 0.83×). This is the paper's strongest comparative claim: autonomous search under the same protocol does not yet match expert-guided harness use. The largest speedups came from structural, hardware-aware changes—sparse-attention route specialization, a CuTe scorer/selector split for top-k, and a Blackwell chunked CuTe-DSL GDN prefill path using tensor memory, TMA, and warp specialization—rather than micro-optimization. GDN decode shows the smallest gain because the supplied baseline was already near-optimal; remaining wins were batch-local, addressed via a dispatcher combining a Triton recurrent route (small batches), a one-warp specialization (batch 8), a CuTe pretranspose route (mid batches), and a larger-value route (batch 64).
The paper is explicit that these are local official-aligned measurements, not final leaderboard scores, and that the reported speedups use a ratio-of-means reporting normalization rather than the official correctness-gated per-workload-ratio contest score.
Trial-sensitive correctness in DSA top-k
A notable negative result concerns the DSA top-k indexer. A post-submission check exposed a trial-sensitive numerical issue that low-trial validation missed: FlashInfer-Bench evaluates randomized per-trial inputs, and the resolved default budget is only n=3 trials, so rare top-k boundary cases (small margin γ(s) relative to numerical perturbation ϵ) can be sampled in one run and missed in another. High-trial replay (100 trials) reproduced sparse selector-boundary failures on one concern workload (3/200 mismatches, max relative error 0.1587), while a conservative fallback tag passed all 1,206 repeated checks. The paper derives the detection probability 1−(1−pw)n under an independent-trial model and concludes that low-trial sweeps are acceptable inside the search loop for iteration speed, but suspicious or high-impact shapes require targeted high-trial replay as a final validation gate. This directly tempers the automation narrative: stronger autonomous search must be paired with stronger final validation, or rare correctness failures will pass low-trial gates.
Profile-guided evidence
Profiler output was compressed into decision records—dominant kernel, bottleneck class, limiting resource, occupancy, waves per SM—and these records governed both dispatch and promotion. The condensed largest-shape evidence is internally consistent with the final designs: in MoE FP8 the two expert GEMMs account for 81.6% of CUDA time with occupancy limited to ~18.7% by register and shared-memory pressure, motivating joint GEMM-plus-packing optimization; in DSA top-k the selector is a one-block launch at 0.01 waves/SM, identifying selector underfill as the next bottleneck; in DSA attention the split/merge pipeline is low-wave and not bandwidth-saturated; in GDN prefill the Blackwell chunk kernel is 98.5% of CUDA time with register/shared-memory limits and ~50% excessive global sectors, leaving coalescing headroom. Ablation records show the gates doing real work: a forced Triton dispatch in MoE was rejected despite correctness (0.496 ms vs. 0.286 ms retained), a scorer tile size of N=32 regressed, and a 40-pair GDN dispatch table was removed in favor of the chunked default.
Limitations
The paper concedes several constraints plainly. The evaluation covers an Agent-Assisted workflow, not an isolated LLM, so the contribution of individual components (skills, controller, gates, memory) is not ablated in isolation; human harness design, budget decisions, and reference curation were central and cannot be separated from the measured gains. Workload distributions are contest-specific, timings are local official-aligned B200 evidence rather than final leaderboard scoring, and the DSA top-k retained artifact preserves a low-trial result with the high-trial fallback evidence reported separately. The Full-Agent comparison uses LoongFlow as a single representative autonomous baseline, and its trace-local best-score checkpoint can differ from the FlashInfer-relative best-latency checkpoint due to run-local PyTorch normalization.
Conclusion
The paper demonstrates that LLM-driven GPU kernel generation can be made reliable and auditable through harness engineering: reusable skills, a separated evaluation harness and profile-backed controller, workload-grounded shape dispatch, conservative promotion gates, and artifact memory. The 1.12×–29.68× FlashInfer-relative speedups and the consistent gap over matched Full-Agent artifacts indicate that expert-provided optimization directions, references, and workload context remain critical. The open question the paper leaves is how much of the controller state, trajectory memory, and model orchestration can be moved into the harness itself without weakening the measured validation discipline that the DSA top-k replay shows is necessary.
