Event Tensor Compiler (ETC)

• Event Tensor Compiler (ETC) is a unified compilation framework that leverages symbolic event tensor abstraction to manage fine-grained synchronization and dynamic scheduling in GPUs.
• It employs both static and dynamic scheduling strategies to fuse tasks within a persistent kernel, significantly reducing kernel launch overhead and improving computational efficiency.
• Performance benchmarks demonstrate that ETC outperforms traditional methods in LLM low-batch inference and MoE layers by optimizing runtime synchronization and memory usage.

The Event Tensor Compiler (ETC) is a unified compilation framework designed to address key limitations in megakernel approaches for modern GPU workloads, particularly those involving LLM inference. ETC introduces the Event Tensor abstraction to systematically represent fine-grained synchronization, inter-operator dependencies, and dynamic execution patterns in persistent GPU kernels. This enables first-class support for both shape dynamism and data-dependent dynamism, leading to improved GPU utilization, reduced kernel launch overhead, and efficient handling of complex computation graphs (Jin et al., 14 Apr 2026).

1. Event Tensor Abstraction

An Event Tensor $E$ is a symbolic-shaped, multi-dimensional array of events, each representing a fine-grained synchronization point at the tile level. The principal role of $E$ is to encode producer-to-consumer dependencies among tiled tasks—typically individual CTA-tile invocations of operators—across the computational graph.

Each dimension of $E$ may remain symbolic until runtime, providing support for shape dynamism. Data-dependent dynamism is also natively supported, where index mappings and counters can be initialized or updated based on runtime data (e.g., as in Mixture-of-Experts (MoE) top-k routing).

Formalism

• Let $T = \{T_1,\ldots,T_M\}$ denote the set of all tasks, with each $T_p$ corresponding to a single tile invocation.
• $E \in \mathbb{N}^{d_1 \times \cdots \times d_r}$ is an $r$-dimensional Event Tensor, with its concrete shape $(d_1,\ldots,d_r)$ potentially symbolic until runtime.
• Two bipartite relation sets define synchronization:
  • $\text{InEdges} \subseteq T \times \text{Index}(E)$
  • $\text{OutEdges} \subseteq T \times \text{Index}(E)$
  • Where $E$0.
• For $E$1, $E$2 must call $E$3 on completion.
• For $E$4, $E$5 waits via $E$6 before starting.

Synchronization semantics rely on global integer counters, with notify implemented as an atomic decrement and wait as spin-wait until zero.

Example: Split-K Reduction

Given $E$7, tasks $E$8 compute partial sums $E$9 and tasks $E$0 compute $E$1. An Event Tensor $E$2 is defined such that:

• OutEdges: $E$3 for all $E$4
• InEdges: $E$5

$E$6 calls $E$7 on completion; $E$8 invokes $E$9 before consuming.

2. Compiler Pipeline and Scheduling

The ETC compiler transforms computational graphs annotated with Event Tensors into a single persistent GPU kernel, supporting both static and dynamic scheduling backends. Operators are assumed to be written as CTA-tile-level device functions.

High-Level Workflow

1. Frontend IR: Operators carry per-tile shapes, and InEdges/OutEdges/Types of Event Tensors.
2. Graph-Level Optimization: Constant folding and memory planning are performed.
3. Tile-Level Optimization: Includes tensor-core usage and pipelining.
4. Scheduling Transformation: Selection of static or dynamic scheduling passes.
5. Prefetch Rewriting: Optional, for weights.
6. Code Generation: Generation of a persistent kernel.
7. Static Queue Materialization: For static schedules, per-SM static queues are generated per shape.

2.1 Static Scheduling

• For each SM $T = \{T_1,\ldots,T_M\}$0, a fixed ordered task list $T = \{T_1,\ldots,T_M\}$1 is precomputed.
• A fused kernel with $T = \{T_1,\ldots,T_M\}$2 CTAs is emitted.
• Each CTA loops over $T = \{T_1,\ldots,T_M\}$3 in order, executing:
  • For each task: wait on InEdges, execute operator, notify on OutEdges.
• At runtime: a single launch, zero scheduling overhead.

2.2 Dynamic Scheduling

• On-device scheduler maintains a lock-free global queue.
• At kernel start, initialize all Event counters.
• Push tasks with zero InEdges into the ready queue.
• Each CTA pops tasks, waits on InEdges, executes the operator, and upon completion, decrements respective Event counters. If an Event counter reaches zero, enqueue all dependent consumer tasks.
• No host involvement after kernel launch.

2.3 Dynamism Support

• Shape Dynamism: Event Tensor shapes and edge mappings are symbolic and instantiated concretely at runtime.
• Data-Dependent Dynamism: Index expressions can access runtime tensors (e.g., MoE’s $T = \{T_1,\ldots,T_M\}$4 or $T = \{T_1,\ldots,T_M\}$5), enabling compilation ahead-of-time while supporting dynamic event graphs.

3. Persistent-Kernel Design

3.1 Kernel Launch Strategy

ETC eliminates repeated kernel launches by fusing subgraphs—often entire LLM decoding workloads—into a single persistent kernel. CTAs either loop over a static work list or pull new work adaptively from the ready queue.

3.2 Synchronization Semantics

All synchronization is implemented through global integer arrays, with atomic decrements for notifications (possibly spin-free) and spin-wait for waits. This design avoids the need for specialized runtimes.

3.3 Cross-Operator Task Parallelism

ETC exposes tile-level dependencies across traditional operator boundaries, enabling parallel execution patterns such as:

• Overlapping attention Q-Norm+RoPE with K/V Norm+RoPE.
• Fusing MLP GEMM+activation+post-norm in pipeline stages.
• Grouping and GroupGEMM fusion for MoE layers via dynamic events.

3.4 Scheduling Constraints

A task is eligible for dispatch when its InEdge event counters are zero. In static scheduling, the task order is precomputed on the host to respect these constraints; in dynamic scheduling, on-device queueing maintains correctness.

4. Performance Evaluation

Benchmarks on NVIDIA B200 GPUs using CUDA 13 and PyTorch 2.8 provide insights into ETC’s efficiency and latency benefits.

4.1 Cross-Operator Fusion

• GEMM+ReduceScatter (dynamic scheduling): up to 1.40× speedup over cuBLAS+NCCL baseline.
• AllGather+GEMM (static scheduling): up to 1.40× speedup.

4.2 Mixture-of-Experts (MoE) Layer

• Qwen3-30B-A3B, TP=1, tokens=64–1024 (dynamic): up to 1.23× speedup versus Triton for 512–1024 tokens.
• ETC shows consistent gains over Triton and FlashInfer across token counts.

4.3 LLM Low-Batch Inference

Batch	vLLM Normalized Latency	SGLang	ETC
1	1.00	0.83	0.68
4	1.00	0.89	0.75
16	1.00	0.95	0.79
64	1.00	0.98	0.82

Qwen3-32B, TP=4: ETC matches or slightly exceeds vLLM and SGLang across batch sizes, reporting ≈1.05× on small batches.

4.4 Warm-up Overhead

Engine	Warm-up Time (s)	# CUDA-Graph Captures
vLLM	123	67
SGLang	583	203
ETC	35	1

ETC achieves significant reductions in system warm-up times and CUDA-Graph capture overhead.

5. Design Trade-Offs and Practical Considerations

Static vs Dynamic Scheduling

• Static scheduling: Minimizes runtime overhead and is optimal for regular DAGs (e.g., standard transformer layers, AllGather+GEMM).
• Dynamic scheduling: Accommodates data-dependent, unpredictable workloads (MoE, communication jitter), but introduces scheduler push/pop costs and potential contention, especially in multi-GPU tensor-parallel (TP) deployments.

Scalability

Event Tensors support scaling to millions of events with minimal compilation-time overhead. However, the memory footprint for counters grows with the number of tiles, and spin-waiting for events can consume cycles. In practice, such overhead is often amortized through overlapping workloads.

Programming Model and Automation

Device functions require explicit tiling and event annotation. Automatic derivation of event graphs from high-level programs represents an open direction. Coverage of hot input shapes for sampled static schedules may require manual tuning.

6. Implications and Future Directions

The Event Tensor abstraction and ETC compiler unify static and dynamic megakernel fusion in a single consistent framework, supporting both runtime shape and data-dependent dynamism without requiring recompilation or CUDA-Graph recapture. This suggests substantial potential for deploying high-performance LLM inference systems with reduced operational overhead and flexible support for next-generation dynamic workloads. A plausible implication is that continued development in automatic event graph generation and shape management may further lower the barrier for adoption in complex deep learning pipelines (Jin et al., 14 Apr 2026).