Fused Kernel Engine (FKE)

• Fused Kernel Engine (FKE) is an execution paradigm that fuses multiple dependent computational stages to eliminate intermediate memory writes and boost arithmetic intensity.
• It leverages techniques like tiling, asynchronous copy, and on-chip intermediate storage to overcome memory-bound bottlenecks and optimize resource usage.
• FKE implementations have demonstrated significant speedups in GPU inference, transformer models, and distributed systems by reducing kernel-launch overhead and global memory accesses.

Fused Kernel Engine (FKE) denotes a class of execution designs in which a full algorithmic pipeline of multiple dependent operators is fused into a single, carefully scheduled code object so that intermediate values remain in registers, shared memory, caches, or other fast local storage rather than being repeatedly materialized in global memory. In the literature, the term is used explicitly for GPU-side inference and serving subsystems such as FLAME and AdaFuse, and more broadly to characterize deeply fused attention, retrieval, simulation, and linear-algebra kernels whose principal goals are to maximize arithmetic intensity, reduce kernel-launch overhead, and minimize global-memory bandwidth utilization (Guo et al., 17 Sep 2025, Li et al., 12 Mar 2026, Bikshandi et al., 2023).

1. Definition and unifying principles

Across the cited literature, an FKE is not a single framework or ISA feature but a recurring systems pattern. Its defining move is to collapse a sequence of dependent stages—such as gather–GEMM–scatter, $QK^\top \rightarrow$ softmax $\rightarrow PV$, routing $\rightarrow$ adapter merge, or dequantization $\rightarrow$ matrix multiplication $\rightarrow$ epilogue—into one executable unit. The immediate effect is the elimination of intermediate round-trips to high-bandwidth memory (HBM) or DRAM, which is particularly consequential when the workload is memory-bound rather than compute-bound (Bikshandi et al., 2023, Sharma, 24 Jun 2026, Zhang et al., 12 Feb 2026).

This logic appears in several mathematically distinct settings. In fused attention, the unfused computation

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{QK^\top+M}{\sqrt{d_k}}\right)V$$

is reorganized so that score formation, masking, normalization, and value aggregation proceed without materializing the intermediate score or probability matrices in global memory (Guo et al., 17 Sep 2025). In AdaFuse, dynamic LoRA application is rewritten as a fused parameter merge,

$$f_*^l = f^l + \sum_{i=1}^k \alpha_i \cdot \left(\mathrm{LoRA\_DOWN}_i^l \times \mathrm{LoRA\_UP}_i^l\right),$$

and the paper’s SGMM kernel performs the merge for all layers in a single batched GEMM call (Li et al., 12 Mar 2026). In each case, fusion is a dataflow transformation: intermediate tensors cease to be first-class memory objects and become transient register- or SRAM-resident states.

A consistent motivation is the “memory wall.” For memory-bound algorithms such as attention, MaxSim scoring, long-context inference, and generative recommendation, the papers repeatedly identify excessive memory traffic or fragmented launch structure—not nominal FLOP count—as the critical bottleneck (Bikshandi et al., 2023, Sharma, 24 Jun 2026, Guo et al., 17 Sep 2025, Vegasena, 18 Apr 2026). This suggests that FKE is best understood as a memory-traffic minimization strategy coupled to execution scheduling, rather than merely as a syntax-level instance of operator fusion.

2. Precursors and historical development

The contemporary FKE vocabulary sits atop a longer lineage of kernel-fusion work. An early GPU study on massive video data analysis formalized the partitioning problem for image-processing kernels, developed an optimization model over fusion candidates, and reported that fused execution was “2 to 3 times higher” in execution time and throughput than sequential execution (Adnan et al., 2015). That work already emphasized the canonical FKE themes: dependency-constrained fusion, shared-memory reuse, and data-traffic reduction.

A second line of development treated fusion as a program-transformation problem over loop nests and dataflow graphs. HFAV introduced automatic transformation of kernel-based computations in disparate, nested loops into fused, vectorized form, with reductions in intermediate storage and improved performance on contemporary hardware (Sewall et al., 2017). Here the essential abstraction is no longer a hand-written fused CUDA kernel but a representation of kernels, data dependencies, and stencil-like access patterns that permits legality analysis, storage contraction, and code generation.

In high-order CFD, hyperbolic diffusion in flux reconstruction provided a domain-specific route to fusion. By hyperbolising diffusion terms, the method eliminated expensive algorithmic steps needed to form viscous stresses and thereby enabled fusion of GPU kernels within tensor-product elements. The resulting fused kernels achieved “3-4 times speedup,” reduced total runtime by approximately $25\%$ in three-dimensional test cases, and yielded a speedup of $2.3$ times compared to the standard ACM formulation (Trojak et al., 2021). The importance of this episode is conceptual: it shows that fusion opportunity can arise not only from compiler or runtime engineering but also from reformulating the numerical method itself.

Taken together, these precursors indicate that FKE should not be reduced to a recent LLM-serving idiom. It is the current name for a broader convergence of ideas in dataflow analysis, numerical reformulation, memory-hierarchy optimization, and code generation.

3. Kernel anatomy and hardware realization

At kernel level, FKE design is governed by a small set of recurring mechanisms: tiling, on-chip residency of intermediates, asynchronous movement of tiles, overlap of copy and compute, epilogue fusion, and resource balancing among registers, shared memory, occupancy, and launch granularity.

The Hopper FlashAttention-2 case study is a canonical example. It implements the forward pass as a custom fused CUDA kernel that combines online-softmax with back-to-back GEMMs, avoids intermediate writes to HBM for the matrices $S$ and $\rightarrow PV$0, uses Hopper-specific Tensor Memory Accelerator (TMA) for asynchronous copy, and uses Warpgroup Matrix-Multiply-Accumulate (WGMMA) instructions for high-throughput matrix multiplication (Bikshandi et al., 2023). The paper stresses that fusion is not merely a matter of concatenating operators: it requires layout transformations in CUTLASS/CuTe, overlap of COPY and GEMM, and tile-size selection for the $\rightarrow PV$1, $\rightarrow PV$2, and $\rightarrow PV$3 matrices under register-pressure and shared-memory constraints. Its best-performing tile shapes were $\rightarrow PV$4 for head dimension $\rightarrow PV$5 and $\rightarrow PV$6 for head dimension $\rightarrow PV$7, rather than $\rightarrow PV$8, because the latter induced register spillage and serialization issues (Bikshandi et al., 2023).

FLAME presents a more systemized serving interpretation. Its FKE module is implemented directly through the NVIDIA TensorRT API rather than ONNX conversion, integrates custom plug-in operators, includes mask-aware Flash-Attention for “single user, multiple items (SUMI),” and fuses FFN and Layer Normalization blocks into single CUDA kernels (Guo et al., 17 Sep 2025). The implementation also uses asynchronous cp_async instructions on Ampere GPUs so that data movement from global to shared to register memory is pipelined with GEMM computation. In this formulation, the FKE is a serving-engine subsystem rather than a single kernel, but its internal mechanics remain the same: eliminate intermediate writes, compress launch structure, and exploit architecture-specific fast paths.

DeepFusionKernel applies the same principle to transformer MLPs rather than attention. It fuses the SwiGLU pipeline into a single CUDA kernel, eliminating intermediate activation tensors and choosing row-major or column-major tiling to maximize either activation reuse or weight reuse depending on workload (Zhang et al., 12 Feb 2026). A runtime profiler selects the highest-throughput variant during warmup, after which CUDA Graphs retain the fast path. The paper’s premise—that autoregressive decoding is increasingly memory-bandwidth-bound and that MLP blocks are under-optimized relative to attention—extends the FKE idea beyond the usual focus on softmax and KV-cache kernels.

4. Representative systems and application domains

The literature uses the FKE idea across a wide range of workloads. The table summarizes representative instances.

System	Fused path	Reported outcome
FlashAttention-2 on Hopper	online-softmax + GEMM-I/GEMM-II with TMA and WGMMA	20-50% higher FLOPs/s; 2.5–3× over default CUTLASS FMHA (Bikshandi et al., 2023)
AdaFuse	token-level pre-gating + SGMM merge of selected adapters across all layers	over 2.4× lower decoding latency; 3.1 ms/token; one SGMM kernel per token (Li et al., 12 Mar 2026)
FLAME	TensorRT-API engine with fused attention, FFN, and LayerNorm plug-ins	4.6x-6.1x speedup ratio; 4.7x-6.3x throughput gain ratio (Guo et al., 17 Sep 2025)
TileMaxSim	matmul + row-wise max + sum, with fused PQ scoring	80.2% of peak HBM bandwidth; 82M documents/second (Sharma, 24 Jun 2026)
FlashSpread	CSR traversal + hazard evaluation + Bernoulli tau-leaping + state update + write-back	8.09 Giga-NUPS; 217x speedup over optimised CPU tau-leaping (Shakeri et al., 23 Apr 2026)
Open-TQ-Metal	int4 KV quantization + compressed-domain attention in Metal shaders	48x attention speedup at 128K; KV cache from 40 GB to 12.5 GB (Vegasena, 18 Apr 2026)

Several additional examples broaden the range of substrates and execution models. FairyFuse implements multiplication-free inference for ternary LLMs on CPUs by fusing the eight real-valued sub-GEMVs of each widely-linear layer into a single AVX-512 loop with masked additions and subtractions and zero floating-point multiplications in the inner loop; roofline analysis reported a 29.6x kernel speedup, and end-to-end throughput reached 32.4 tokens per second on a single Intel Xeon 8558P (Zuo et al., 22 Apr 2026). Matrix-free 3D SIMP topology optimization fused gather, per-element stiffness multiplication, and scatter accumulation into one CUDA kernel and reported 4.6-7.3x end-to-end SIMP wall-time speedup on cantilever cases (Yang et al., 20 Apr 2026). For diffusion transformers on consumer Ampere GPUs, a single fused Triton INT8 GEMM combined int8xint8$\rightarrow PV$9int32 accumulation with per-token $\rightarrow$0 per-channel dequantization and bias in the epilogue, ran 2.8-4.2x faster than bf16 per GEMM, and delivered an approximately 1.1x end-to-end speedup at 768px (Asaria et al., 12 Jun 2026).

These cases illustrate that FKE is not tied to one model family or one hardware vendor. It appears in CUDA, Triton, Metal, TensorRT plug-ins, CuPy runtime compilation, and AVX-512 CPU kernels, with targets ranging from transformer inference and generative recommendation to retrieval, epidemic simulation, and structural optimization.

5. Compiler, library, and distributed forms

A major development is the shift from hand-written fused kernels to frameworks that search, synthesize, or compose them automatically. FlashFuser is the most explicit compiler-oriented example. It is presented as the first compiler framework to use Distributed Shared Memory (DSM) for kernel fusion on modern GPUs, introducing DSM-based communication primitives, a dataflow analyzer, and a search engine with analytical cost modeling and DSM-aware pruning (Huang et al., 15 Dec 2025). Its cost model is expressed as

$\rightarrow$1

with optimization over memory levels subject to capacity constraints. On an NVIDIA H100 GPU, FlashFuser reduced memory access by $\rightarrow$2, delivered kernel speedups of 3.3x against highly-tuned libraries and 4.1x against state-of-the-art compilers, and produced a 1.24x end-to-end speedup (Huang et al., 15 Dec 2025).

Diffuse generalizes fusion to distributed task-based runtimes. It uses a scale-free intermediate representation of distributed computation, performs dynamic task fusion, and then invokes an MLIR-based JIT compiler to fuse the kernels within the fused tasks (Yadav et al., 2024). The system accelerated unmodified applications by 1.86x on average and by between 0.93x and 10.7x on up to 128 GPUs. The significance for FKE is that the fused execution object need not be a single kernel chosen a priori; it may be the output of a runtime analysis over distributed tasks and cross-library dependencies.

The Fused Kernel Library (FKL) occupies a third point in the design space. Using C++17 metaprogramming rather than a custom compiler, it generates a single optimized fused kernel for arbitrary combinations of GPU library functions at compile time and supports both horizontal fusion and vertical fusion (Amoros et al., 9 Aug 2025). The paper reports speedups in the range of “2x to more than 1000x,” with up to 20,931× in synthetic benchmarks. This shifts FKE from an application-specific optimization to an API methodology for library construction.

Distributed communication can itself be fused with computation. The work on fused computation-collective operations created self-contained GPU kernels in which workgroups communicate partial results to remote GPUs immediately upon finishing local computation, while other workgroups continue executing (Punniyamurthy et al., 2023). Prototype fused operators—embedding + All-to-All, GEMV + AllReduce, and GEMM + All-to-All—reduced combined execution time by up to 22%, 20%, and 31% depending on operator and deployment mode. In such cases, the “engine” fuses not only local operators but also dependent collective communication, extending the FKE idea beyond device-local memory optimization.

6. Performance limits, trade-offs, and terminological scope

The empirical literature is consistent on one point: FKEs are most effective when the baseline is dominated by memory traffic, launch overhead, or both. TileMaxSim showed that naive MaxSim materializes an $\rightarrow$3 similarity matrix and reaches only 5–18% of peak HBM bandwidth, whereas the fused, IO-aware design reached 80.2% of peak HBM bandwidth on H100 (Sharma, 24 Jun 2026). Open-TQ-Metal showed that dequantize-then-attend baselines squander the memory savings of int4 KV compression, while compressed-domain fused attention removes the intermediate dequantization matrices entirely (Vegasena, 18 Apr 2026). AdaFuse similarly attributed large decoding slowdowns not to increased FLOPs but to fragmented, sequential CUDA kernel launches during dynamic routing and merging (Li et al., 12 Mar 2026).

The principal design constraint is resource pressure. The FlashAttention-2 Hopper study identifies register spills, shared-memory pressure, and occupancy loss as central barriers to deeper fusion (Bikshandi et al., 2023). The ACM-HD work finds that one fusion strategy is preferable at low order while another dominates at high order because workload shape changes with polynomial degree (Trojak et al., 2021). FlashSpread must further adapt fusion to graph topology through degree-aware CSR dispatch and active-node compaction (Shakeri et al., 23 Apr 2026). These results indicate that “more fusion” is not automatically better; fusion depth must be balanced against tile size, occupancy, bank conflicts, synchronization cost, and code complexity.

Kernel-level gains also do not imply proportional end-to-end gains. In the fused INT8 GEMM for Ideogram 4.0, individual GEMMs ran 2.8-4.2x faster than bf16, yet the end-to-end speedup was only approximately 1.1x because linear GEMMs accounted for about 12% of total inference time (Asaria et al., 12 Jun 2026). A plausible implication is that FKE effectiveness should be analyzed under Amdahl-style workload composition, not only via microbenchmarks.

The term also has scope limits. Some FKEs are hardware-specific: the Ideogram fused INT8 kernel is explicitly advantageous on consumer Ampere GPUs, while on A100 and B200 the same kernel loses to native bf16/FP8 paths (Asaria et al., 12 Jun 2026). FairyFuse argues that ternary compression offers little benefit on GPUs, even though it is highly effective on bandwidth-limited CPUs (Zuo et al., 22 Apr 2026). Open-TQ-Metal is specialized to Apple Silicon and to compressed-domain attention via Metal shaders (Vegasena, 18 Apr 2026). Finally, the acronym is not unique across the broader literature: DeepRFTv2 uses “FKE” to denote “Fourier Kernel Estimator,” a deblurring module unrelated to fused execution engines (Mao et al., 26 Nov 2025). That collision is terminological rather than conceptual.

Taken as a whole, the literature presents FKE not as a single product category but as an execution principle: transform multi-stage computation so that dependence-respecting work is performed in one tightly scheduled object, exploit the fastest available memory hierarchy, and pay global-memory and launch costs only at the boundaries that remain unavoidable.