---
title: Kernel Fusion and Arithmetic Intensity
url: https://www.emergentmind.com/topics/kernel-fusion-and-arithmetic-intensity
type: topic
---

# Kernel Fusion and Arithmetic Intensity

Kernel fusion is a code transformation and code generation strategy that merges multiple GPU or CPU kernels into single, larger computational units, thereby reducing the volume of off-chip memory traffic and increasing the arithmetic intensity of the resulting computations. Arithmetic intensity (AI), defined as the ratio of the total number of floating-point operations (FLOPs) to the total number of bytes moved between off-chip (or upper-level) memory and the compute units, is a principal metric in roofline-based performance modeling. Fusion systematically raises AI by keeping intermediate results on-chip rather than on DRAM, unlocking higher throughput, especially on modern architectures where compute throughput outpaces memory bandwidth.

## 1. Defining Arithmetic Intensity and Its Role in Performance

Arithmetic intensity is expressed as
\[
I = \frac{\text{Total FLOPs}}{\text{Total Bytes Moved}}
\]
In a roofline model, peak attainable performance is bottlenecked by either computational throughput (compute-bound regime) or memory bandwidth (memory-bound regime); thus, kernels with low AI are memory-bound, while high-AI kernels can potentially achieve a higher fraction of peak FLOP/s [2506.22169, 1911.11576, 1305.1183, 2108.13342, 2406.18109]. For example, in a General Matrix-Matrix Multiply (GEMM) tile,
\[
I = \frac{W}{B}
\]
where \( W \) is the number of FLOPs and \( B \) is the number of bytes moved between DRAM/main memory and on-chip storage [2506.22169].

The movement of a kernel up the roofline—from memory-bound to compute-bound—as AI increases is a principal target of fusion strategies. This can yield kernel performance improvements of 2×–5× in AI and corresponding speedups, as shown empirically across diverse workloads [2506.22169, 1911.11576, 1305.1183, 2108.13342].

## 2. Principles and Formal Models of Kernel Fusion

The essential idea in kernel fusion is to eliminate redundant memory transactions across consecutive kernels. In the unfused scenario, a chain of \( n \) kernels each perform their own global memory loads and stores:
\[
\text{Total Bytes}_{\text{unfused}} = \sum_{i=1}^{n} (\text{Bytes Read}_i + \text{Bytes Written}_i)
\]
After fusion, intermediate values are retained on-chip (registers, shared memory, or DSM), so only the initial loads and final stores touch off-chip memory:
\[
\text{Total Bytes}_{\text{fused}} = \text{Bytes Read}_1 + \text{Bytes Written}_n
\]
AI then increases by a factor roughly equal to the number of kernels fused [1710.08774, 1305.1183, 2108.13342, 2508.07071].

Fusion frameworks formalize this as an optimization problem, where either an integer linear program (ILP) [1911.11576], a beam search [2009.10924], or a combinatorial search space with pruning is constructed [2512.12949, 2506.22169]. Objective functions operate on estimated or measured AI increases, saved memory, and occupancy/resource penalties.

## 3. Fusion and Tiling Methodologies

State-of-the-art kernel fusion frameworks (e.g., MCFuser [2506.22169], FlashFuser [2512.12949], FusionStitching [1911.11576, 2009.10924], Fused Kernel Library [2508.07071], and DNNFusion [2108.13342]) follow a multi-layered approach:

- **Tiling Strategy**: Decompose iteration spaces into tiles, assign tile shapes, and select loop orderings. Tile sizes and orderings have direct effect on both compute (W) and memory (B) in AI computation [2506.22169].
- **Loop Nest and DAG Analysis**: Express the computation as a loop-nest DAG or task-based IR to make dependency, sharing, and data reuse explicit [1710.08774, 2406.18109].
- **Fusion Plan Generation**: Solve for maximal sets of fusible subgraphs/patterns, guided by resource constraints (e.g., shared memory, registers, DSM capacity) and legal fusion rules (no data-dep cycles or cross-barrier fusions) [1911.11576, 2512.12949, 2108.13342, 2406.18109].
- **On-Chip Allocation and Scheduling**: For each fusion candidate, optimize on-chip memory allocation across registers, shared/local memory, and DSM, using buffer sharing, storage contraction, and memory-manager tactics. This maximizes data reuse and limits costly off-chip transfers [2506.22169, 2512.12949].
- **Code Generation**: Emit a single, parameterized kernel for the fused computation, directly mapping data movement patterns into code and managing synchronization as required [1911.11576, 2508.07071].

## 4. Quantitative Effects: AI Gain and Performance

Kernel fusion yields AI improvements and associated speedups as documented in multiple empirical studies:
- **MCFuser** achieves AI inflations from \( \varphi \approx 2 \) (unfused small \( K \)) up to \( \varphi \approx 50 \) (fully fused, large tile, no redundant access), shifting throughput from \( 0.1 P \) to the compute roof on A100 [2506.22169].
- **FusionStitching** boosts AI by a mean of 2.4× per fused kernel (range 1.5–4.2×), with end-to-end speedups up to 5.7× on DL workloads [1911.11576].
- **DNNFusion** reports a 2×–5× increase in AI and up to 3.8× speedups on full models, as in fusing Conv-BN-ReLU or Attention blocks [2108.13342].
- **FLASHFuser** (exploiting DSM) lifts AI by ~2.4×, reduces DRAM traffic by 58%, and achieves per-kernel speedups over SOTA by 3.3–6.4× [2512.12949].
- **TurboFNO** captures a 5–10× AI boost by fusing FFT, GEMM, and iFFT operations and coalescing all shared-memory traffic, reducing kernel launches from 5 to 1 [2504.11681].
- **Classic BLAS fusion** doubles AI in BiCGK and GEMVER sequences and achieves up to 2.6× speedups over CUBLAS [1305.1183].
- **Stencil code** fusions can raise AI from 0.04 to 0.12 flops/byte (3×) and produce 4× speedups on multi-GPU runs [2406.18109].

## 5. Resource Constraints and Trade-Offs

Fusion is constrained by on-chip storage (registers, shared memory, DSM), register and shared-memory pressure (impacting occupancy), and the need for synchronizations (e.g., after partial reductions). Over-fusing can reduce SM occupancy causing diminished latency hiding [2506.22169, 2009.10924, 2508.07071]. For deep fusion chains or data-dependent kernels, codegen/resource limits or launch cost can dominate, requiring heuristics or analytic penalties in cost models to avoid over-fusion [2512.12949, 1911.11576]. Fusing memory-bound operators with compute-bound ones generally realizes the most AI gain [2108.13342]. DSM enables fusion at scales previously limited by SMEM, e.g., in multi-GEMM FFN blocks or large convolution chains [2512.12949].

## 6. Variants and Extensions Across Domains

Fusion is implemented in various programming models:
- **Automatic compile-time fusion** via metaprogramming (FKL, C++17) enables arbitrary-depth fusion for point-wise, reduction, and batched kernels [2508.07071].
- **JIT-IR fusion** (MLIR, runtime fusion in distributed/task-based systems) allows cross-library kernel composition, as in Diffuse [2406.18109].
- **Domain-specific generator frameworks** (e.g., for PDEs or spectral operators) optimize fusion and storage contraction in nested loop and tensor-product settings, raising AI in stencil/PDE codes [1710.08774, 2107.14027].
- **Video/data pipelines** demonstrate 2–3× AI improvements and throughput gains for multi-stage fused image-processing [1509.04394].

The convergence of tiling, DAG abstraction, resource-aware planning, and cost modeling is central to all high-performance fusion systems.

## 7. Empirical Benchmarks and Roofline Analysis

Empirical results across diverse domains systematically corroborate the core AI/speedup relationship:
- **Batched GEMM chains (MCFuser)**: up to 5.9× kernel speedup over Ansor, up to 8.1× over PyTorch or 3.0× over FlashAttention [2506.22169].
- **FNO Fourier layers (TurboFNO)**: up to 1.5× boost over cuFFT+cuBLAS baselines, entirely attributable to elimination of 4–6 global memory passes [2504.11681].
- **Video pipelines**: 67% reduction in global memory traffic and up to 3× increase in arithmetic intensity, mapping to doubled or tripled throughput [1509.04394].
- **Diffusion/PDE codes**: fusion raises intensity and moves from bandwidth-bound to compute-limited, with strong correlation between predicted AI and measured GFLOP/s [1710.08774, 2107.14027].
- **Distributed tasks (Diffuse)**: raising AI by 2–5× produces up to 10× end-to-end speedups on GPU clusters [2406.18109].

A consistent finding is that AI uplift, enabled by fusion, correlates strongly with performance in memory-constrained regimes.

---

References:  
- "MCFuser: High-Performance and Rapid Fusion of Memory-Bound Compute-Intensive Operators" [2506.22169]  
- "FusionStitching: Boosting Execution Efficiency of Memory Intensive Computations for DL Workloads" [1911.11576]  
- "Optimizing CUDA Code By Kernel Fusion---Application on BLAS" [1305.1183]  
- "DNNFusion: Accelerating Deep Neural Networks Execution with Advanced Operator Fusion" [2108.13342]  
- "Composing Distributed Computations Through Task and Kernel Fusion" [2406.18109]  
- "The Fused Kernel Library: A C++ API to Develop Highly-Efficient GPU Libraries" [2508.07071]  
- "FusionStitching: Boosting Memory Intensive Computations for Deep Learning Workloads" [2009.10924]  
- "High-Performance Code Generation though Fusion and Vectorization" [1710.08774]  
- "Efficient Kernel Fusion Techniques for Massive Video Data Analysis on GPGPUs" [1509.04394]  
- "FlashFuser: Expanding the Scale of Kernel Fusion for Compute-Intensive Operators via Inter-Core Connection" [2512.12949]  
- "TurboFNO: High-Performance Fourier Neural Operator with Fused FFT-GEMM-iFFT on GPU" [2504.11681]  
- "Hyperbolic Diffusion in Flux Reconstruction: Optimisation through Kernel Fusion within Tensor-Product Elements" [2107.14027]  
- "Kernel Fusion in Atomistic Spin Dynamics Simulations on Nvidia GPUs using Tensor Core" [2308.07487]

Source: https://www.emergentmind.com/topics/kernel-fusion-and-arithmetic-intensity