MegaKernel: Persistent GPU Execution
- MegaKernel is a GPU execution style that fuses multiple operations into one persistent kernel, reducing kernel launch overhead and inter-operator memory traffic.
- It is applied in diverse domains such as path tracing and LLM decoding, using techniques like warp specialization and in-kernel scheduling to manage complex execution flows.
- Compiler-generated megakernel models incorporate static correctness checks and dynamic scheduling, achieving notable speedups and flexible performance across graphics and AI workloads.
MegaKernel denotes a GPU execution style in which work that would ordinarily be expressed as many kernel launches, stage kernels, or operator boundaries is absorbed into a single monolithic or persistent kernel. In graphics, the term usually refers to a single compute kernel in which one thread traces an entire path from camera ray generation through intersection, BSDF evaluation, radiance accumulation, and termination. In contemporary LLM systems, it denotes a persistent kernel that can encompass an entire decode block, an MoE subgraph, or even a whole forward pass, with device-side scheduling, counters, queues, or warp-specialized roles replacing host-orchestrated kernel launches and inter-operator memory traffic (Padilla et al., 26 May 2026, Dong et al., 12 May 2026, Jaber et al., 8 Jun 2026, Cheng et al., 22 Dec 2025).
1. Core definitions and execution regimes
The term has multiple precise meanings, all centered on collapsing a computation graph into one long-lived GPU program. In forward path tracing, a megakernel is a single, monolithic GPU kernel: one compute dispatch, one thread per pixel or sample, and one control-flow tree carrying local path state until miss, Russian roulette, or maximum depth termination. In Ada-MK, a MegaKernel is a single persistent kernel that stays resident on the GPU SMs and contains the entire Transformer decoder block + LM head execution logic for decode, with warp specialization across Loader, Consumer, Storer, Controller, and Launcher roles. In AutoMegaKernel, it is a single persistent cooperative CUDA kernel launched with cudaLaunchCooperativeKernel, with one resident threadblock per SM, that runs the whole forward pass for one decode step in one launch. In Mirage Persistent Kernel, the mega-kernel is the extreme form of fusion in which all computation and communication of a tensor program are executed inside one persistent kernel and synchronized with device-memory events rather than kernel barriers (Padilla et al., 26 May 2026, Dong et al., 12 May 2026, Jaber et al., 8 Jun 2026, Cheng et al., 22 Dec 2025).
| Context | MegaKernel form | Distinguishing property |
|---|---|---|
| GPU path tracing | One monolithic compute kernel | One thread traces a complete path |
| Dense LLM decode | One persistent fused decode kernel | Eliminates per-token launch overhead and HBM round-trips |
| Whole-model or MoE runtime | One persistent cooperative/task kernel | In-kernel scheduling via events, queues, or counters |
This shared vocabulary conceals materially different scheduling models. Some MegaKernels are essentially per-thread monoliths with predominantly local state; others are persistent runtimes that execute task DAGs, route work across SMs, and coordinate communication or tile readiness through counters and device-side queues. The common invariant is the erasure of conventional host-launched kernel boundaries.
2. Graphics origins: monolithic path tracing and its wavefront contrast
In GPU path tracing, the megakernel formulation corresponds directly to forward path tracing. A path is generated sequentially as a chain of vertices , with estimator
and a megakernel realization keeps ray state, throughput, and radiance accumulators in per-thread local state while a single bounce loop performs ray generation, intersection, BSDF evaluation, radiance accumulation, resampling, and termination. The alternative wavefront formulation retains the same mathematical algorithm but decomposes execution into global buffers plus multiple kernels for ray generation, intersection, shading, accumulation, and, in optimized variants, compaction and indirect dispatch (Padilla et al., 26 May 2026).
The empirical trade-off is not purely conceptual. On an NVIDIA RTX 3060 Ti, using Vulkan compute shaders with inline ray queries and the “A Beautiful Game” scene, the reported wavefront implementation reached 73.6 FPS and 13.58 ms frame time, whereas the megakernel forward path tracer reached 64.7 FPS and 15.47 ms, a 1.16× relative speedup for wavefront. Nsight traces attributed this to wavefront’s higher VRAM Throughput and L2 Throughput despite lower SM Throughput and RTCore Throughput; neither implementation saturated any major GPU unit, and the limiting factors were identified as communication and memory latency, together with synchronization overhead (Padilla et al., 26 May 2026).
CrossRT places this result in a broader graphics context. It explicitly adopts the Laine–Karras–Aila terminology and distinguishes a megakernel model, a plain wavefront model, and a queue-based wavefront model. Its translator can generate path tracing implementations as a single megakernel or as multiple kernels from the same hardware-agnostic C++ source, without altering the programming model or input source code. CrossRT also states that the megakernel model is “the most flexible and currently the most performant for code with relatively low register pressure,” while simultaneously describing wavefront mode as crucial for NeRF and neural SDF workloads, where independent-thread execution is inefficient (Frolov et al., 2024).
This suggests that, even in the domain where the term originally stabilized, MegaKernel is not a universal optimum. Register pressure, divergence structure, memory locality, and the granularity at which rays or samples can be regrouped all materially influence whether a monolithic kernel or a staged wavefront organization is preferable.
3. Persistent decode kernels for dense LLM inference
In LLM serving, MegaKernel emerged from a specific systems pressure: decode is dominated by many very small GPU operations executed under strict latency budgets. Ada-MK reports a production online advertising setting with 1–5 ms end-to-end latency constraints, and profiles TensorRT-LLM serving Qwen2.5-1.5B as incurring 1,655,550 kernel launches, consuming about 3.3 seconds, with kernel launch overhead alone = 14.6% of end-to-end inference time. Its MegaKernel eliminates both repeated launches and inter-operator HBM round-trips by fusing decode into one persistent kernel that passes intermediates through shared memory and registers. The system couples this with a three-dimensional shared-memory constraint model and K-dimension splitting that reduce peak shared memory usage by 50%, an MLIR-based fine-grained DAG offline search that “solidifies” the optimal execution trace and completely removes runtime branching, and a TensorRT-LLM plugin design that retains TensorRT-LLM for Prefill while using MegaKernel only for Decode. On an NVIDIA L20, Ada-MK improves single-batch throughput by up to 23.6% over vanilla TensorRT-LLM and 50.2% over vLLM, with positive gains across all tested scenarios (Dong et al., 12 May 2026).
AutoMegaKernel pushes the same idea toward fully automatic synthesis. It compiles a HuggingFace Llama-family model into a single persistent cooperative CUDA kernel that runs the whole forward pass in one launch, with no per-model hand-written CUDA. The same source retargets sm_80, sm_90, and sm_120 from one codebase, auto-generates correct megakernels for 10 of 10 supported models, and reproduces HuggingFace greedy decode token-for-token on a real SmolLM2-135M checkpoint, with a reported perplexity match of absolute difference. Its agent-driven autotuning loop improves the megakernel over its own baseline by 1.25–1.72×, and a search-found W8A16 megakernel outperforms CUDA-graphed cuBLAS bf16 at batch-1 decode on several inference-class GPUs, including L4 up to 1.33×, L40S 1.25–1.27×, A10G up to 1.08×, and RTX 5090 1.19–1.23× (Jaber et al., 8 Jun 2026).
These systems share a technical thesis: for decode, the critical optimization target is not merely operator-local fusion but the removal of launch boundaries and HBM materialization boundaries from the entire decode subgraph. In this regime, warp specialization, persistent residency, and schedule specialization are not ancillary tactics; they are the defining implementation content of the MegaKernel.
4. Compilerization, schedule IRs, and static correctness
A major shift in the literature is the movement from hand-built MegaKernels toward compiler-generated persistent runtimes. Mirage Persistent Kernel introduces an SM-level graph representation in which task nodes execute on individual SMs and event nodes encode fine-grained dependencies. The compiler performs event fusion, graph normalization, and linearization, and the in-kernel runtime partitions SMs into workers and schedulers, combining JIT and AOT task launch modes. The result is a single mega-kernel that fuses computation, communication, and serving logic; the reported evaluation states end-to-end inference latency reductions of up to 1.7× over kernel-per-operator LLM serving systems, and the detailed multi-GPU evaluation reports up to 10× throughput over native PyTorch for Qwen3-1.7B inference (Cheng et al., 22 Dec 2025).
Event Tensor generalizes this compiler perspective to dynamic workloads. It treats synchronization events as first-class tensors with symbolic shapes and wait counters, so dependencies between tiled tasks can be expressed as tensor-index mappings rather than as an explicitly materialized runtime graph. This enables a single ahead-of-time-compiled dynamic megakernel to support both shape dynamism and data-dependent dynamism, including MoE routing through runtime tensors such as topk and exp_indptr. Built on this abstraction, the Event Tensor Compiler supports static and dynamic scheduling transformations. Reported results include up to 1.40× speedup for fused GEMM + Reduce-Scatter and All-Gather + GEMM, up to 1.23× on MoE layers, and up to 1.48× faster time-per-output-token than vLLM at batch size 1 for Qwen3-30B-A3B; for Qwen3-32B, warmup time is 35 s with ETC, versus 123 s for vLLM and 583 s for SGLang (Jin et al., 14 Apr 2026).
AutoMegaKernel adds a different compiler concern: the correctness of agent-generated schedules. Its frozen validator checks schedule IR well-formedness, deadlock-freedom, race-freedom, and output reachability via static graph checks rather than a mechanized proof. Across 7,160 adversarial schedules, of which 6,091 were unsafe, the validator had zero false-accepts, and it accepted all 360 real lowerings (Jaber et al., 8 Jun 2026).
Taken together, these systems recast MegaKernel as a compilation target rather than merely a fused coding style. Task graphs, event tensors, static validators, and in-kernel schedulers form the control plane; the persistent CUDA or Triton kernel is the execution substrate onto which that control plane is lowered.
5. MoE, distributed communication, and runtime-aware polymorphism
In Mixture-of-Experts systems, MegaKernel usually denotes fusion at the granularity of the expert MLP or the entire expert-parallel MoE subgraph. RaMP defines its megakernel as the single fused GPU kernel that performs the MoE MLP layer—up-projection, activation, down-projection, and scatter—for all experts and all routed tokens in one launch. It parameterizes the kernel by tile configuration , and the resulting CTA grid depends explicitly on the runtime routing histogram: RaMP’s four-parameter wave cost model selects the fastest configuration from the runtime expert histogram with 0.93% mean regret versus exhaustive search, and in deployment it achieves 1.22× kernel speedup over static dispatch and 1.30× end-to-end speedup in vLLM over Triton, alongside 1.41× over DeepGEMM and 1.13× over FlashInfer CUTLASS (Sharma et al., 28 Apr 2026).
UniEP moves the MoE MegaKernel into distributed training. It fuses communication and computation into two persistent kernels per phase: Dispatch + GroupGEMM MegaKernel and GroupGEMM + Combine MegaKernel. Within a single CUDA stream, SMs dynamically fetch communication, computation, and relay/reduction tasks from a device-side queue, while a deterministic global token mapping guarantees numerical consistency with sequential execution even under aggressive overlap. The reported result is 1.03×–1.38× speedups over prior expert-parallel systems while maintaining the accuracy requirements of production LLM training, and the end-to-end 128-GPU training experiment preserves a bitwise identical training trajectory (Zheng et al., 21 Apr 2026).
DITRON extends the MegaKernel concept to a distributed, multi-level tiling compiler. At its task level, the entire model or large subgraphs are represented as a DAG of tile-level tasks with software-scoreboard dependencies, and a persistent MEGA_TRITON_KERNEL fetches per-SM work queues and dispatches task implementations. DITRON reports 6%–30% speedups on isolated kernels and 5%–30% on end-to-end inference in vLLM, together with more than 10% MFU improvement in production training tasks and approximately 500,000 GPU hours of monthly training-cost savings (Zheng et al., 2 May 2026).
At multi-node scale, communication semantics become decisive. “Eliminating Hidden Serialization in Multi-Node Megakernel Communication” shows that proxy-based RDMA transports can silently serialize GPU-initiated put+signal communication because each tile transfer expands to PUT → FENCE → SIGNAL, and the fence drains the NIC pipeline. In a Qwen3-30B example, one PE can issue 96 concurrent inter-node PUTs through a single proxy channel. Perseus addresses this with decoupled signaling, which batches fences at per-destination granularity and reduces fence count by 8×, and NIC-side ordering, which uses hardware fence flags so the proxy never blocks. On proxy-based transports, Perseus yields up to 10.3× end-to-end speedup, and on IBRC it matches or exceeds IBGDA GPU-direct by up to 1.2× (Oh et al., 1 May 2026).
The MoE literature therefore reframes MegaKernel as a runtime-polymorphic object. Its optimal form depends not only on tensor shape but on expert histograms, collective topology, proxy semantics, and the exact overlap budget between communication and per-expert compute.
6. Locality, bottlenecks, and alternatives to the monolithic kernel model
Recent work emphasizes that MegaKernel performance is inseparable from hardware hierarchy. Fleet adapts persistent-kernel execution to multi-die GPUs by introducing Chiplet-tasks, a hierarchy level between CU-level and device-level work that binds computation and data to a chiplet’s private L2 cache. Implemented on AMD Instinct MI350, Fleet designates one workgroup per chiplet as a scheduler and the rest as workers, with per-chiplet scheduling and cooperative weight tiling. On Qwen3-8B it achieves 1.3–1.5× lower decode latency than vLLM at batch sizes 1–8; at larger batches, cooperative tiling raises L2 hit rate from 12% to 54% at batch size 32 and from 39% to 61% at batch size 64, reduces HBM traffic by up to 37%, and delivers 1.27–1.30× speedup over a chiplet-unaware megakernel baseline (Chowdhary et al., 15 Apr 2026).
The limits of megakernelization are equally explicit. AutoMegaKernel reports that its W8A16 megakernel beats CUDA-graphed cuBLAS on inference-class GPUs but trails cuBLAS on A100/H100, where the harness localizes the bottleneck to cross-SM synchronization. In graphics, the RTX 3060 Ti study finds wavefront path tracing ~16% faster than the megakernel forward path tracer because improved cache locality outweighs the megakernel’s higher instantaneous SM and RTCore throughput. CrossRT likewise states that wavefront mode is essential for NeRF and neural SDF, even though it characterizes the megakernel model as most flexible and currently most performant for code with relatively low register pressure (Jaber et al., 8 Jun 2026, Padilla et al., 26 May 2026, Frolov et al., 2024).
VDCores goes further and positions itself explicitly as an alternative to megakernel or monolithic-kernel design for asynchronous GPUs. Rather than encoding overlap in one giant kernel, it exposes Virtual Compute Cores, Virtual Memory Cores, and a dependency-connected micro-op graph. Across four LLM inference workloads on GH200, H100, and RTX 6000 Pro, it improves decoding throughput by 24% on average and by up to 77% under dynamic inputs, while reducing kernel programming and specialization effort by 90% (He et al., 4 May 2026).
A plausible implication is that MegaKernel is best understood not as the endpoint of optimization, but as one point in a broader design space of persistent execution. It is strongest when kernel-launch overheads, inter-operator HBM round-trips, and kernel barriers dominate. It becomes less definitive when routing distributions are highly dynamic, when synchronization frequency overwhelms compute, when cache locality is better recovered by wavefront regrouping, or when hardware exposes asynchronous engines and locality domains that a single monolithic control flow cannot exploit directly.