MusaCoder: Native GPU Kernel Generation

• MusaCoder is a framework for native GPU kernel generation, converting high-level PyTorch code into executable CUDA and MUSA kernels with strict performance and correctness criteria.
• It employs progressive kernel-oriented data synthesis, supervised fine-tuning, and reinforcement learning with execution-feedback to ensure compilability, numerical accuracy, and forbidden fallback avoidance.
• The system integrates advanced stabilizers like PrimeEcho, Buffered Dynamic Retry, and MirrorPop alongside MooreEval to achieve superior benchmark performance.

Searching arXiv for the MusaCoder paper and closely related work on kernel generation and sequential-to-parallel translation. MusaCoder is a full-stack training framework for native GPU kernel generation that targets the direct synthesis of executable low-level CUDA or MUSA kernels from high-level PyTorch programs. It is designed around an execution-feedback training regime in which compilability, numerical correctness, legality with respect to forbidden PyTorch/ATen fallbacks, and empirical speedup are all treated as verifiable objectives rather than purely textual ones. The reported system comprises progressive kernel-oriented data synthesis, supervised post-training, diversity-preserving rejection fine-tuning, and reinforcement learning driven by a distributed verifier and reward environment called MooreEval. In the formulation presented by its authors, MusaCoder addresses a task on which generic code LLMs are weak and on which naïve execution-based RL is unstable because of sparse rewards, reward hacking, and off-policy drift (Cheng et al., 3 Jun 2026).

1. Definition and problem setting

MusaCoder concerns native GPU kernel generation, defined as turning high-level tensor programs into executable low-level code on CUDA and MUSA backends (Cheng et al., 3 Jun 2026). The emphasis on native generation is essential: the objective is not merely to emit plausible GPU-flavored source code, but to produce kernels that execute correctly, avoid forbidden high-level computational fallbacks, and achieve practical speedups on hardware.

The paper frames this as a systems-oriented code generation problem with unusually strict constraints. A valid output must compile, preserve output shape and type, remain numerically close to PyTorch reference outputs, avoid illegal memory behavior, and satisfy device and runtime restrictions (Cheng et al., 3 Jun 2026). This makes the task substantially narrower and harsher than ordinary code synthesis benchmarks, because syntactic fluency is insufficient. A kernel may appear structurally plausible while still failing at compilation, indexing, boundary handling, legality checks, or runtime performance.

The framework is implemented in two model sizes, MusaCoder-9B and MusaCoder-27B, both initialized from Qwen base checkpoints (Cheng et al., 3 Jun 2026). The training recipe is explicitly staged: multi-source data synthesis, multi-task supervised fine-tuning, diversity-preserving rejection fine-tuning, and execution-feedback RL through MooreEval, with additional stabilization components named PrimeEcho, Buffered Dynamic Retry (BDR), and MirrorPop (Cheng et al., 3 Jun 2026).

2. Technical challenges in native kernel generation

The paper identifies several reasons that existing LLMs struggle with this task. First, the initial success rate is very low because generating kernels from scratch requires simultaneous control over GPU execution semantics, thread and block indexing, memory layout, strides, reduction formulas, normalization logic, and boundary conditions (Cheng et al., 3 Jun 2026). Errors in any of these areas frequently produce compile failures, illegal memory accesses, shape mismatches, or incorrect numerical outputs.

Second, the correctness criterion is intrinsically stricter than in ordinary code generation. The model must satisfy a conjunction of properties rather than a single textual metric. This is why the paper treats correctness as behavior under execution rather than similarity to a reference string (Cheng et al., 3 Jun 2026). A plausible implication is that standard lexical or structural metrics are especially weak proxies in this domain.

Third, direct RL on execution outcomes is described as unstable for three specific reasons: reward sparsity, reward hacking, and off-policy drift (Cheng et al., 3 Jun 2026). Reward sparsity arises because many rollout groups contain only failed programs. Reward hacking arises when a model attempts to call PyTorch or ATen operators instead of generating true native kernels. Off-policy drift becomes pronounced in repeated multi-turn interaction, where the training distribution may diverge from the rollout policy.

The paper further stresses that these difficulties are amplified on emerging ecosystems such as MUSA, where datasets, tooling, and verification infrastructure are less mature than in CUDA-centric environments (Cheng et al., 3 Jun 2026). This positions MusaCoder not only as a kernel-generation method but also as a backend-enablement framework for Moore Threads GPUs.

3. Training pipeline and synthesized supervision

MusaCoder’s training pipeline begins with progressive kernel-oriented data synthesis. The paper argues that naïve PyTorch-to-CUDA translation pairs are insufficient, so the corpus is constructed from several sources: open-source PyTorch-to-CUDA/MUSA tasks, real GitHub PyTorch modules, and NNSmith-generated computation graphs (Cheng et al., 3 Jun 2026). Difficult operator families are upsampled, including convolutions, transposed convolutions, reductions, normalization, softmax, broadcasting, and complex indexing (Cheng et al., 3 Jun 2026).

A second component is a GPU Kernel Knowledge Q&A dataset intended to teach CUDA programming model concepts, memory hierarchy, tensor semantics, mathematical derivations, performance optimization, numerical stability, and benchmarking or profiling concepts (Cheng et al., 3 Jun 2026). The stated motivation is that many kernel failures arise from conceptual misunderstanding rather than from purely local syntax errors.

The synthesis pipeline also includes automated unit-test generation. For each reference program, tests are synthesized across shapes, dtypes, value ranges, and edge cases; LLM-generated tests are run in native PyTorch to obtain ground-truth outputs and invalid tests are filtered (Cheng et al., 3 Jun 2026). The training data are then enriched with a structured six-step reasoning trace consisting of: holistic analysis and operator deconstruction; scalar mathematical formula derivation; shape and dimensionality calculus; indexing relations and memory mapping; boundary handling and type safety; and error pre-analysis with negative constraints (Cheng et al., 3 Jun 2026). Samples additionally include an explicit [Tensor Shape Information] block with shapes, strides, contiguity flags, and intermediate tensor layouts (Cheng et al., 3 Jun 2026).

The paper extends this synthesis to multi-turn trajectories that incorporate compilation errors, runtime errors, correctness mismatches, profiling feedback, and iterative fixes (Cheng et al., 3 Jun 2026). Two auxiliary data categories are also introduced: CUDA Kernel Reviewer data, for judging candidate kernels against PyTorch references and identifying bugs or performance bottlenecks, and Profiling / NCU data, for learning to interpret profiler outputs and convert bottlenecks into optimizations (Cheng et al., 3 Jun 2026).

After synthesis, the model undergoes multi-task supervised fine-tuning (SFT) on a mixture of kernel generation, reviewer, profiling, GPU knowledge QA, and general coding or instruction data (Cheng et al., 3 Jun 2026). Approximately 30% of the SFT mixture is retained as general instruction-following or coding data to avoid overfitting to kernel-specific formats (Cheng et al., 3 Jun 2026). For multi-turn samples, only the final turn contributes to the loss, while earlier turns are preserved as context (Cheng et al., 3 Jun 2026).

This SFT stage is followed by diversity-preserving rejection fine-tuning (RFT). Rather than keeping a single best candidate per prompt, MusaCoder retains all valid correct implementations, clusters them by structural or micro-architectural features, and randomly samples from a bounded diverse pool, typically 4–8 trajectories per prompt (Cheng et al., 3 Jun 2026). The explicit purpose is to prevent entropy collapse and preserve exploration capacity.

4. MooreEval and the execution-feedback learning regime

The central infrastructure component is MooreEval, which serves both as verifier and reward environment (Cheng et al., 3 Jun 2026). Architecturally, it is described as distributed and asynchronous, with a Host, Compile Workers, Exec Workers, Redis queue + inflight tracking, Shared filesystem / OSS, and a Reward Adapter (Cheng et al., 3 Jun 2026). Its function is to support high-throughput compilation and execution, crash isolation, structured telemetry, anti-hacking enforcement, and RL reward computation.

For each candidate kernel, MooreEval performs a fixed verification pipeline: code extraction, interface validation, isolated compilation, randomized correctness testing, detection of cheating or fallbacks, and runtime benchmarking (Cheng et al., 3 Jun 2026). The anti-hacking logic specifically detects forbidden high-level PyTorch or ATen operators such as matmul, convolution, reduction, and other computational fallbacks; only non-computational utilities are allowed by the whitelist (Cheng et al., 3 Jun 2026). If a prohibited fallback is found, the sample receives zero reward or hard failure.

The verifier output is formalized as

$$V(c, x) = ( \mathrm{compiled}, \mathrm{correct}, \mathrm{legal}, \mathrm{speedup}, \mathrm{category}, \mathrm{detail} ).$$

Here, compiled denotes compilation success, correct correctness against the PyTorch reference, legal absence of cheating or forbidden fallbacks, and speedup runtime speedup over baseline, while category and detail encode structured failure telemetry (Cheng et al., 3 Jun 2026).

The reward is correctness-gated:

$$s(c)= \begin{cases} -1, & \text{if code extraction fails, compilation fails, or runtime fails}, \ -1, & \text{if a disallowed PyTorch/aten::* fallback is detected}, \ -1, & \text{if } q = 0, \ -0.5 + 0.5q, & \text{if } 0 < q < 1, \ 1 + \lambda \cdot \min(\max(\nu - 1, 0), \nu_{\max}), & \text{if } q = 1 \text{ and the implementation is legal}. \end{cases}$$

In this formulation, $q \in [0,1]$ is the correctness rate on randomized tests and $\nu$ is speedup over the PyTorch baseline (Cheng et al., 3 Jun 2026). The structure ensures that performance reward is available only for outputs that are already fully correct and legal.

RL is based on GRPO and proceeds in two phases: single-turn RL warmup followed by multi-turn feedback RL (Cheng et al., 3 Jun 2026). In the first phase, the model generates a single candidate per task and is updated through group-relative rewards. In the second, MooreEval returns structured feedback after failures and the model retries, allowing it to repair syntax errors, shape errors, dtype mismatches, runtime bugs, and performance problems (Cheng et al., 3 Jun 2026).

5. Stabilization mechanisms: PrimeEcho, BDR, and MirrorPop

A distinguishing feature of MusaCoder is the introduction of three RL stabilizers. The first, PrimeEcho, is a first-turn-anchored trajectory reward:

$$R_{\tau_i} = \alpha s_{i,1} + (1-\alpha)\max_{1 \le k \le K} s_{i,k} + b_{\mathrm{early}(\tau_i)},$$

with an early-success bonus

$$b_{\mathrm{early}(\tau_i)} = \beta_1 \mathbf{1}[\mathrm{success}(c_{i,1})] + \beta_2 \mathbf{1}[\neg \mathrm{success}(c_{i,1}) \land \mathrm{success}(c_{i,2})].$$

The paper states that this mechanism prevents the model from sacrificing first-turn quality in favor of later repair turns, and uses $\alpha = 0.75$ by default (Cheng et al., 3 Jun 2026).

The second, Buffered Dynamic Retry (BDR), addresses rollout groups in which every candidate receives reward $-1$. For such hard prompts, a failed candidate $c^{-}$ and its MooreEval feedback $f^{-}$ are combined into a new feedback-conditioned prompt $$s(c)= \begin{cases} -1, & \text{if code extraction fails, compilation fails, or runtime fails}, \ -1, & \text{if a disallowed PyTorch/aten::* fallback is detected}, \ -1, & \text{if } q = 0, \ -0.5 + 0.5q, & \text{if } 0 < q < 1, \ 1 + \lambda \cdot \min(\max(\nu - 1, 0), \nu_{\max}), & \text{if } q = 1 \text{ and the implementation is legal}. \end{cases}$$0, which is inserted into a FIFO buffer $$s(c)= \begin{cases} -1, & \text{if code extraction fails, compilation fails, or runtime fails}, \ -1, & \text{if a disallowed PyTorch/aten::* fallback is detected}, \ -1, & \text{if } q = 0, \ -0.5 + 0.5q, & \text{if } 0 < q < 1, \ 1 + \lambda \cdot \min(\max(\nu - 1, 0), \nu_{\max}), & \text{if } q = 1 \text{ and the implementation is legal}. \end{cases}$$1 and sampled later with small probability $$s(c)= \begin{cases} -1, & \text{if code extraction fails, compilation fails, or runtime fails}, \ -1, & \text{if a disallowed PyTorch/aten::* fallback is detected}, \ -1, & \text{if } q = 0, \ -0.5 + 0.5q, & \text{if } 0 < q < 1, \ 1 + \lambda \cdot \min(\max(\nu - 1, 0), \nu_{\max}), & \text{if } q = 1 \text{ and the implementation is legal}. \end{cases}$$2 (Cheng et al., 3 Jun 2026). The paper explicitly recommends using BDR late in RL rather than early, to avoid overfitting to repair-conditioned behavior.

The third, MirrorPop, is a sequence-level off-policy masking rule designed to avoid cancellation effects in signed log-ratio averages. For each token,

$$s(c)= \begin{cases} -1, & \text{if code extraction fails, compilation fails, or runtime fails}, \ -1, & \text{if a disallowed PyTorch/aten::* fallback is detected}, \ -1, & \text{if } q = 0, \ -0.5 + 0.5q, & \text{if } 0 < q < 1, \ 1 + \lambda \cdot \min(\max(\nu - 1, 0), \nu_{\max}), & \text{if } q = 1 \text{ and the implementation is legal}. \end{cases}$$3

and the sequence statistic is

$$s(c)= \begin{cases} -1, & \text{if code extraction fails, compilation fails, or runtime fails}, \ -1, & \text{if a disallowed PyTorch/aten::* fallback is detected}, \ -1, & \text{if } q = 0, \ -0.5 + 0.5q, & \text{if } 0 < q < 1, \ 1 + \lambda \cdot \min(\max(\nu - 1, 0), \nu_{\max}), & \text{if } q = 1 \text{ and the implementation is legal}. \end{cases}$$4

The resulting mask is

$$s(c)= \begin{cases} -1, & \text{if code extraction fails, compilation fails, or runtime fails}, \ -1, & \text{if a disallowed PyTorch/aten::* fallback is detected}, \ -1, & \text{if } q = 0, \ -0.5 + 0.5q, & \text{if } 0 < q < 1, \ 1 + \lambda \cdot \min(\max(\nu - 1, 0), \nu_{\max}), & \text{if } q = 1 \text{ and the implementation is legal}. \end{cases}$$5

This treats deviation magnitude symmetrically regardless of sign and filters responses that are too far from the rollout policy (Cheng et al., 3 Jun 2026).

These stabilizers are framed not as auxiliary heuristics but as necessary responses to specific pathologies of execution-feedback RL for kernel generation. The ablation results support that interpretation: removing PrimeEcho, BDR, or MirrorPop reduces both correctness and speed-related metrics, with the largest drop occurring when MirrorPop is removed (Cheng et al., 3 Jun 2026).

6. Empirical performance and benchmark results

MusaCoder is evaluated on KernelBench and a MUSA-ported KernelBench under a strict protocol requiring successful parsing and compilation, correctness on randomized inputs, absence of forbidden computational fallbacks, and performance benchmarking only after correctness and legality have passed (Cheng et al., 3 Jun 2026). The primary reported metrics are Pass@8, Avg.@8, and Faster Rate, with the speed threshold set at $$s(c)= \begin{cases} -1, & \text{if code extraction fails, compilation fails, or runtime fails}, \ -1, & \text{if a disallowed PyTorch/aten::* fallback is detected}, \ -1, & \text{if } q = 0, \ -0.5 + 0.5q, & \text{if } 0 < q < 1, \ 1 + \lambda \cdot \min(\max(\nu - 1, 0), \nu_{\max}), & \text{if } q = 1 \text{ and the implementation is legal}. \end{cases}$$6 over baseline (Cheng et al., 3 Jun 2026).

On KernelBench, the base checkpoints are reported as relatively weak under this strict verifier. Qwen3.5-9B achieves 23.6% Pass@8 and 7.05% Avg.@8, while Qwen3.6-27B achieves 67.2% Pass@8 and 35.60% Avg.@8 (Cheng et al., 3 Jun 2026). After post-training, the gains are substantial:

Model/state	Pass@8	Avg.@8
MusaCoder-9B-SFT	69.6%	61.60%
MusaCoder-9B-RL	83.6%	77.20%
MusaCoder-27B-SFT	84.8%	79.40%
MusaCoder-27B-RL	93.2%	88.60%

The paper reports that MusaCoder-27B-RL exceeds Claude Opus 4.7, which records 87.2% Pass@8 and 77.30% Avg.@8, by +6.0 and +11.3 points respectively (Cheng et al., 3 Jun 2026). On the harder Level 3 subset, Claude is reported at 54% Pass@8 and 39.25% Avg.@8, while MusaCoder-27B-RL reaches 72% Pass@8 and 65.75% Avg.@8 (Cheng et al., 3 Jun 2026). The paper interprets this as evidence of improved handling of shape, indexing, and multi-operator workloads.

For speed, the KernelBench Faster Rate also favors the 27B RL model. Against PyTorch eager and torch.compile, the reported overall rates are: 0.9% / 0.5% for Qwen3.5-9B, 3.4% / 1.6% for Qwen3.6-27B, 11.8% / 7.5% for Claude Opus 4.7, 5.4% / 3.2% for MusaCoder-9B-RL, and 15.0% / 9.2% for MusaCoder-27B-RL (Cheng et al., 3 Jun 2026). This indicates that the reported gains are not limited to executable correctness.

On the MUSA-ported KernelBench, the cross-backend results are also strong. MusaCoder-9B-RL records 84.4% Pass@8, 72.3% Avg.@8, and 6.4% Faster, while MusaCoder-27B-RL records 92.4% Pass@8, 81.7% Avg.@8, and 12.5% Faster (Cheng et al., 3 Jun 2026). Among the listed baselines, DeepSeek-V4-Pro reports 92.0% Pass@8, 56.9% Avg.@8, and 5.7% Faster, while GLM-5.1 reports 88.0% Pass@8, 66.4% Avg.@8, and 6.9% Faster (Cheng et al., 3 Jun 2026). The paper’s interpretation is that MusaCoder matches or exceeds competing systems in correctness while improving average correctness and speed on MUSA.

The training setup itself is part of the empirical claim. SFT uses AdamW with learning rate $$s(c)= \begin{cases} -1, & \text{if code extraction fails, compilation fails, or runtime fails}, \ -1, & \text{if a disallowed PyTorch/aten::* fallback is detected}, \ -1, & \text{if } q = 0, \ -0.5 + 0.5q, & \text{if } 0 < q < 1, \ 1 + \lambda \cdot \min(\max(\nu - 1, 0), \nu_{\max}), & \text{if } q = 1 \text{ and the implementation is legal}. \end{cases}$$7, warmup ratio 3%, weight decay 0.01, bf16, maximum sequence length 40K, global batch size 256, and 1 epoch (Cheng et al., 3 Jun 2026). RL uses two-stage GRPO with rollout group size 8, training batch size 64, maximum multi-turn length 3, temperature 0.9 for training and 0.7 for validation, top-$$s(c)= \begin{cases} -1, & \text{if code extraction fails, compilation fails, or runtime fails}, \ -1, & \text{if a disallowed PyTorch/aten::* fallback is detected}, \ -1, & \text{if } q = 0, \ -0.5 + 0.5q, & \text{if } 0 < q < 1, \ 1 + \lambda \cdot \min(\max(\nu - 1, 0), \nu_{\max}), & \text{if } q = 1 \text{ and the implementation is legal}. \end{cases}$$8 of 0.95 for training and 0.7 for validation, RL learning rate $$s(c)= \begin{cases} -1, & \text{if code extraction fails, compilation fails, or runtime fails}, \ -1, & \text{if a disallowed PyTorch/aten::* fallback is detected}, \ -1, & \text{if } q = 0, \ -0.5 + 0.5q, & \text{if } 0 < q < 1, \ 1 + \lambda \cdot \min(\max(\nu - 1, 0), \nu_{\max}), & \text{if } q = 1 \text{ and the implementation is legal}. \end{cases}$$9, warmup ratio 0.1, weight decay 0.1, gradient clipping 0.5, maximum prompt length 8K, and maximum response length 32K (Cheng et al., 3 Jun 2026). The system is trained on 64 Moore Threads MTT S5000 machines, each with 8 × 80GB accelerator cards (Cheng et al., 3 Jun 2026).

7. Interpretation, relation to adjacent research, and limitations

MusaCoder occupies a distinct position within code-generation research because it treats kernel synthesis as a verifiable execution problem rather than as ordinary program text generation. This differentiates it from work centered on stylistic personalization, such as MPCoder, which formulates multi-user code generation as conditional modeling over user history and coding style (Dai et al., 2024), and from work centered on sequential-to-parallel code translation through mutual supervision, such as MuSL, which uses a Translator–Tester loop to enforce functional equivalence between C/C++ and CUDA programs (Ke et al., 11 Jun 2025).

The contrast is methodologically significant. MuSL defines correctness in terms of source–target behavioral equivalence and uses unit tests to filter translated code (Ke et al., 11 Jun 2025). MusaCoder likewise prioritizes behavior, but its task is lower-level and stricter: it must generate native kernels from high-level tensor programs, detect forbidden high-level fallbacks, and optimize for empirical speedup under backend-specific constraints (Cheng et al., 3 Jun 2026). A plausible implication is that MusaCoder extends the broader execution-feedback paradigm from code translation into the domain of backend-native systems synthesis.

The paper also carries an infrastructural claim about Moore Threads GPUs. It argues that the MTT S5000 cluster can support long-context SFT, asynchronous rollouts, distributed verification, and full post-training for both 9B and 27B models (Cheng et al., 3 Jun 2026). In that sense, MusaCoder is presented not only as a model family but also as evidence that emerging accelerator ecosystems can host the complete LLM post-training stack.

Several limitations are explicitly or implicitly noted. Some benefits of shape annotation are supported only by manual inspection or case study, not a large-scale formal ablation (Cheng et al., 3 Jun 2026). BDR is described as mainly a late-stage stabilizer, rather than a universal training method (Cheng et al., 3 Jun 2026). The MUSA benchmark omits torch.compile comparisons because it is not fully benchmarked on the platform (Cheng et al., 3 Jun 2026). More broadly, the method depends on extensive synthetic and verified data generation, a strong execution infrastructure, and backend-specific reward environments (Cheng et al., 3 Jun 2026). These constraints suggest that the framework is particularly suited to domains where executable verification is available and where legality and performance can be operationalized.

Within that scope, the reported contribution is twofold. First, MusaCoder provides a concrete recipe for combining synthesis, verification, rejection fine-tuning, and RL to learn native kernel generation on CUDA and MUSA (Cheng et al., 3 Jun 2026). Second, it advances the view that correctness, legality, and speed can be jointly optimized in post-training through full-stack execution feedback, rather than delegated solely to test-time search or human repair (Cheng et al., 3 Jun 2026).