Merge Kernel Fusion Techniques

• Merge Kernel is a methodology that fuses multiple kernel operations into a single optimized unit, reducing memory traffic and computational overhead.
• It encompasses techniques in GPU kernel fusion, permutation-based kernel construction, and deep network kernel merging for enhanced performance.
• Utilizing merge kernels yields significant efficiency gains and improved model robustness, with reported speedups up to 10⁴× in specific applications.

A merge kernel refers to a class of methodologies that combine (or “fuse”) multiple kernel operations or kernel matrices into a single, more efficient or more expressive entity. In contemporary machine learning, high-performance computing, and statistical methodology, the term “merge kernel” is used in at least three dominant senses: (1) GPU kernel fusion for computational efficiency, (2) algorithmic kernel construction for permutation spaces and structured data, and (3) statistical or learning-theoretic fusion of multiple kernel matrices or kernel statistics. The following article synthesizes the state-of-the-art across these domains.

1. Merge Kernels in GPU Computing: Principles and Methodologies

In GPU computational libraries, a merge kernel (often used interchangeably with kernel fusion or fused kernel) is an optimization in which multiple GPU kernels—which might otherwise be launched in sequence and incur redundant memory traffic—are fused into a single CUDA kernel. This process can be categorized into:

• Vertical Fusion (VF): Chaining dependent operations (e.g., $K_1 \to K_2 \to \cdots \to K_m$) in a pipeline so that all intermediate data are kept in on-chip registers or shared memory. DRAM reads/writes for intermediate results are avoided, reducing total off-chip traffic from $2 \cdot m \cdot k \cdot N$ (unfused) to $2 \cdot k \cdot N$ (fused), achieving a traffic reduction factor of $m$.
• Horizontal Fusion (HF): Aggregating $k$ independent launches of the same kernel (for instance, on different data planes) into a single kernel execution using gridDim.z, with theoretical speedups $$S_\mathrm{HF}(k)=k T_\mathrm{plane}/T_\mathrm{fused\_hf}(k)$$.

Combined, VF and HF can achieve multiplicative speedup and DRAM savings. The Fused Kernel Library (FKL) automates the on-demand generation of such merge kernels using C++17 metaprogramming with nvcc, eliminating the need for custom compilers and manual fused-kernel authoring. It exposes composable abstractions—read, write, unary, and binary operators—composed into fusionable data-parallel patterns such as TransformDPP and ReduceDPP. Template instantiation and static reflection at compile time guarantee all operations are inlined in a single global kernel, with occupancy and block/grid sizing computed at compile time. This architecture achieves end-to-end speedups ranging from $2\times$ to over $10^4\times$ depending on the instruction density and pattern, erasing host-side and memory-bound bottlenecks compared to legacy libraries (Amoros et al., 9 Aug 2025).

Prior works have formalized kernel fusion for linear algebra workloads, such as BLAS, via source-to-source compilers and linear programs that optimize the fusion plan given shared-memory constraints and data dependencies. Filipović et al. demonstrated up to $2.61\times$ measured speedups on BLAS workloads by systematically fusing chains of map/reduce kernels; fusion was gated by thread/data mappings and occupancy requirements (Filipovič et al., 2013). Further, more complex ILP-based partitioning selects optimal merge points in application pipelines (e.g., video processing) to obey data dependencies and maximize global-memory traffic reduction, with empirical 2–3$\times$ speedups across heterogeneous GPGPUs (Adnan et al., 2015).

2. Merge Kernel Constructions: Permutation Spaces and Efficient Kernel Learning

The merge kernel framework has been extended to the design of specialized kernels for structured (non-vectorial) spaces. The Merge Kernel for permutation spaces constructs a feature map via the sequence of element comparisons in merge sort, reducing feature dimensionality from $2 \cdot m \cdot k \cdot N$0 (Mallows kernel) to $2 \cdot m \cdot k \cdot N$1. The binary feature vector encodes the outcome of each deterministic comparison within the merge sort computation tree. The kernel is then

$2 \cdot m \cdot k \cdot N$2

ensuring positive-definiteness by function composition with a Gaussian RBF kernel. Additional lightweight descriptors—such as shift histograms, cross-partition comparisons, and sliding-window motifs—are concatenated to restore invariances and capture global/local patterns, yielding a composite kernel that outperforms classical baselines on permutation optimization tasks at superior computational complexity (Xie et al., 17 Jul 2025).

3. Kernel Merging in Deep Networks and Model Robustness

Model-internal kernel fusion has emerged as a mechanism to regularize and enhance the robustness of deep convolutional networks. The "in-model merging" (InMerge) approach selectively merges similar convolutional kernels—in cosine similarity—within a single deep CNN during a finetuning phase, using a stochastic, threshold-based, convex-combination update:

$2 \cdot m \cdot k \cdot N$3

for pairs $2 \cdot m \cdot k \cdot N$4 above a similarity threshold $2 \cdot m \cdot k \cdot N$5 (default $2 \cdot m \cdot k \cdot N$6) in deep layers only. This reduces redundancy, acts as a spatially-aware regularizer, and consistently increases test-set robustness in medical imaging tasks (average AUROC or accuracy gains up to $2 \cdot m \cdot k \cdot N$7 over standard training) without any inference-time overhead (Wang et al., 27 Feb 2025). The methodology preserves semantic feature diversity while constraining over-correlation among filters.

4. Statistical Merge Kernels: Matrix Completion and Adaptive Testing

In multi-view learning and bioinformatics, merge kernel algorithms address the challenge of combining incomplete or heterogeneous similarity matrices. The Mutual Kernel Matrix Completion (MKMC) algorithm fuses $2 \cdot m \cdot k \cdot N$8 incomplete kernel (covariance) matrices $2 \cdot m \cdot k \cdot N$9 by iterative expectation-maximization, minimizing the sum of KL-divergences to a model kernel $2 \cdot k \cdot N$0:

$2 \cdot k \cdot N$1

with closed-form E-step and M-step updates for efficient mutual completion and averaging, yielding $2 \cdot k \cdot N$2. Empirical results demonstrate superior structure preservation and classification accuracy for the fused kernel relative to traditional (single-source) completions, especially under high missingness (Kato et al., 2017).

Adaptive kernel merging appears in statistical testing via the MMD-FUSE approach, which computes permutation-safe, soft-max adaptive statistics over collections of normalized MMD estimators from multiple candidate kernels:

$2 \cdot k \cdot N$3

The data-driven kernel fusion test achieves state-of-the-art power, finite-sample calibration (no data splitting required), and theoretically quantifiable deviation bounds under both null and alternative hypotheses due to the permutation-invariance at every stage (Biggs et al., 2023).

5. Merge Kernels for Efficient LLM Inference and Model Compression

Merge kernels occupy a central role in state-of-the-art LLM serving. In long-context LLM inference, memory bottlenecks at the attention KV-cache are mitigated by merging adjacent and highly similar cached key/value pairs. KVMerger clusters locally similar key states (cosine similarity above a threshold), then computes a new merged key by a Gaussian-kernel-weighted average centered at the “pivotal” token (highest aggregated attention score):

$2 \cdot k \cdot N$4

This approach achieves near-maximal downstream accuracy (at 50% or 35% cache budgets) on a range of long-context benchmarks, outperforming both eviction-based and uniform-averaging alternatives (Wang et al., 2024). Similar strategies appear in transformer backpropagation for through-memory efficiency.

Mega-kernel (persistent kernel) compilation fuses entire tensor programs into one launch on multi-GPU systems via block-level task-graph partitioning and decentralized in-kernel scheduling (Mirage Persistent Kernel). MPK’s graph normalization, SM-level tiling, and software-pipelined execution yield up to $2 \cdot k \cdot N$5 lower end-to-end latency in LLM inference while maximizing Tensor-Core and SMEM utilization (Cheng et al., 22 Dec 2025). Deep kernel fusion of MLP blocks in transformers (specifically fusing all constituent GEMMs and pointwise ops of SwiGLU MLPs) reduces global memory traffic by $2 \cdot k \cdot N$6, improving token generation throughput by up to $2 \cdot k \cdot N$7 over hand-tuned modular baselines (Zhang et al., 12 Feb 2026).

6. Kernel Fusion in Multiple Kernel Learning and Deep Architectures

Kernel fusion in statistical learning arises via the combination of multiple feature-specific kernels—either by convex combination, learned weighting, or explicit architectural fusion. In deep multiple kernel fusion, separate kernel Gram matrices are embedded, passed through independent deep sub-networks, and then concatenated at a fusion layer with kernel-level dropout “masking”:

$2 \cdot k \cdot N$8

contributing to significant robustness and accuracy increases over both classical MKL solvers and single-kernel models (Song et al., 2016). Composition kernels (all nonempty sums over a base kernel set) are included to emulate the effect of all convex and polynomial kernel mixes, enhancing representational flexibility.

7. Limitations, Extensions, and Practical Guidelines

Merge kernel methods are subject to architectural and mathematical constraints:

• Occupancy and Code Generation: Excessive fusion may drop GPU occupancy given finite register/shared-memory budgets; compile-time recursion may hit template instantiation or code size limits (Amoros et al., 9 Aug 2025).
• Correctness: Fusion is permitted only where data- and thread-mapping are compatible, or where proper barriers/shared-memory handoff can preserve correctness (Filipovič et al., 2013, Adnan et al., 2015).
• Invariance and Robustness: Some merging strategies (e.g., permutation kernels) lack full invariance; such gaps can be mitigated with carefully selected descriptors (Xie et al., 17 Jul 2025).
• Generality: Current public libraries support only certain patterns (map/reduce, single data-parallel pattern per call). Supporting heterogeneous fusion (multiple patterns, pointer returns, dynamic allocation) requires further advances (Amoros et al., 9 Aug 2025).

Practical recommendations include normalizing/centering kernel matrices prior to fusion (Kato et al., 2017), automating fusion plan selection via empirical performance prediction (Filipovič et al., 2013), and modularizing fusion patterns for maintainability and extensibility (Cheng et al., 22 Dec 2025).

---

In summary, merge kernels represent a unifying abstraction that subsumes fine-grained computational fusion, kernel matrix algebra, expressive kernel design for structured spaces, and adaptive statistical testing. Across these domains, they systematically reduce redundancy, memory bandwidth, or statistical inefficiency by merging operations or data representations at the most efficient algebraic or architectural level, as rigorously formalized in the most recent literature (Amoros et al., 9 Aug 2025, Filipovič et al., 2013, Adnan et al., 2015, Xie et al., 17 Jul 2025, Wang et al., 27 Feb 2025, Kato et al., 2017, Cheng et al., 22 Dec 2025, Biggs et al., 2023, Wang et al., 2024, Zhang et al., 12 Feb 2026, Song et al., 2016).