TileFuse: Mixed-Precision Kernel Library

• TileFuse is a close-to-metal mixed-precision kernel library that fuses weight layout, metadata placement, and microkernels to efficiently execute transformer linear layers on AMD XDNA2 NPUs.
• It supports AWQ-style quantization (W4A16 and W8A16), enabling native execution of off-the-shelf quantized LLM models without needing to reshape the model for proprietary schemes.
• Evaluations report up to 121.6% GEMM throughput improvement and 281% GEMV speedup on Ryzen AI laptops, along with significantly reduced energy consumption.

Searching arXiv for the cited TileFuse paper and closely related work to ground the article. TileFuse is a close-to-metal mixed-precision kernel library for AMD XDNA2 NPUs that targets transformer linear layers in quantized LLM inference. It is designed to bring practical low-bit formats such as AWQ-style W4A16 and W8A16 directly onto XDNA2, rather than requiring models to be reshaped around an NPU-specific quantization scheme. Its core design couples weight layout, metadata placement, mixed-precision microkernels, and array-level dataflow, and it fuses unpacking, dequantization, and GEMM/GEMV execution into a single kernel flow. Reported evaluations cover both kernel-level performance and end-to-end LLM inference on Ryzen AI laptops (Pang et al., 9 Jun 2026).

1. Problem setting and system scope

TileFuse is situated in the context of on-device LLM inference on edge SoCs that increasingly integrate NPUs to improve performance and energy efficiency under tight power and thermal budgets. The motivating obstacle is that practical LLM deployment on current client NPUs remains difficult because widely used quantization formats such as AWQ do not map cleanly onto many existing NPU software stacks, which are often proprietary and expose limited low-level control. TileFuse addresses this problem at the kernel-library level rather than as a general-purpose compiler or end-to-end runtime: it specifically targets transformer linear layers, which dominate GEMM and GEMV workloads in LLM execution (Pang et al., 9 Jun 2026).

The library’s stated goals are native and efficient execution of practical quantized LLM formats, high performance and energy efficiency for both GEMM and GEMV, broad shape support up to 32K, and end-to-end usability in practical LLM inference pipelines. This positioning is important because it frames TileFuse as a mechanism for running off-the-shelf quantized models on XDNA2 without recoding the model around a vendor-specific format. A common misconception is to treat all NPU acceleration work as either compiler research or model-level quantization research; TileFuse instead operates at the boundary between quantization semantics and close-to-metal execution.

2. Quantization model and fused execution path

TileFuse supports two quantization formats. For AWQ-style W4A16, it uses 4-bit asymmetric quantized weights, 16-bit activations, group-wise per-column scales and zero-points, AWQ group size 128, and accumulation in FP32. For W8A16, it uses 8-bit symmetric quantized weights, 16-bit activations, and per-channel scales. The paper gives the dequantization rules as

$\hat{w} = s_g q$

for symmetric W8A16 and

$\hat{w} = s_g (q - z_g)$

for asymmetric W4A16, where $q$ is the quantized value, $s_g$ is the scale for group or channel $g$, and $z_g$ is the zero-point for group $g$ (Pang et al., 9 Jun 2026).

The defining execution strategy is fusion. In a non-fused baseline, quantized weights are expanded to FP16 or BF16, written to memory, and then reloaded by GEMM, which introduces unnecessary memory movement and a global-memory roundtrip for the intermediate representation. TileFuse instead unpacks, dequantizes, and multiplies quantized weights on-the-fly within the computation kernel. For INT4 weights, unpacking widens two-per-byte packed values to INT8; dequantization then applies the scale and, for W4A16, the zero-point; the prepared weights are reused in tiled execution and passed directly to AIE matrix-multiply primitives.

This fused path is not merely an implementation convenience. It preserves the data reuse structure of tiled GEMM and GEMV while removing an otherwise separate materialization phase. A plausible implication is that TileFuse treats dequantization as part of the kernel’s steady-state datapath rather than as a preprocessing stage, which is especially consequential on hardware where memory movement is the dominant constraint.

3. Interleaved pre-tiling, weight layout, and metadata placement

A central systems contribution is the interleaved pre-tiling layout. The paper identifies a hardware constraint on AMD XDNA2 memory controller and DMA buffer-descriptor streaming: conventional column-oriented or interleaved layouts can hit stride limits and cap matrix dimensions, for example at 8K columns. TileFuse addresses this with offline pre-tiling at model deployment time. Quantized weights and their associated scales and zero-points are grouped, packed, and reordered so that all tiles processed by a given NPU tile are contiguous in memory (Pang et al., 9 Jun 2026).

For W4A16, each $k \times n$ weight tile is laid out as contiguous INT4 values followed by BF16 scales and INT8 zero-points, with zero-points duplicated if needed so that each DMA payload is 128 bytes. The paper reports that this supports continuous DMA streaming and enables matrices up to 32K columns or rows. It also states that the layout better matches hardware consumption patterns, improving DRAM burst access and sustained bandwidth. Metadata placement is co-designed with the weight layout so that data and dequantization metadata travel together, minimizing the need for separate streams and increasing locality.

The data block further reports that aliased or duplicated zero-points used for DMA padding incur less than 2% memory overhead for large bandwidth savings. This is a narrow but technically significant design choice: it shows that TileFuse accepts a small storage penalty to simplify streaming regularity and reduce runtime bandwidth inefficiency.

4. Mixed-precision microkernels and array-level dataflow

TileFuse relies on hand-optimized mixed-precision microkernels to maximize local AIE utilization. These kernels fuse bit-level unpacking, dequantization, and BF16 or FP16 multiply-accumulate, and they cache dequantized weight tiles for reuse so that the dequantization cost is paid once per tile rather than once per use. In GEMV, the microkernel processes wider $64 \times 8$ weight blocks per iteration and accumulates outputs in larger chunks, reusing activation fragments across many output rows and using AIE vector registers more fully (Pang et al., 9 Jun 2026).

The paper emphasizes that mixed precision changes the GEMV operating point. By reducing memory traffic and increasing arithmetic per moved datum, GEMV becomes more compute-bound, which motivates further compute-side optimization. A related array-level issue is utilization of the 4x8 AIE array. The default mapping, which streams along a single row, activates only 8 of 32 cores during GEMV. TileFuse introduces a two-stage weight distribution scheme: first, each shim core streams a bundle of tile weights into a memory core; second, the memory core redistributes the tiles to multiple compute cores across all four rows in a column. The reported result is that all 32 AIE cores participate in parallel GEMV computation, yielding up to 4x higher utilization for generation-side workloads (Pang et al., 9 Jun 2026).

The software substrate is also part of the design. TileFuse uses the IRON software stack for direct control over microkernel code and DMA data movement, and custom buffer descriptors and kernel programs are loaded for each workload. This exposes a deployment nuance that the paper states explicitly: large GEMM operations in the prefilling phase amortize dataflow reconfiguration cost, but short GEMV operations in generation can be dominated by dispatch overhead. For that reason, the paper recommends hybrid NPU/iGPU deployment in some regimes.

5. Reported performance and energy characteristics

The reported empirical results span kernel-level throughput, end-to-end prompt filling, and energy use on Ryzen AI laptops. For kernel-level GEMM, TileFuse NPU kernels reach up to 12 TOPS on Ryzen AI 7 350, Krackan Point, for W4A16, and 9 TOPS on Ryzen AI 9 HX 370, Strix Point. For large matrices, the paper reports a 121.6% improvement in kernel-level GEMM throughput over full-precision NPU baselines. For GEMV, it reports up to 281% improvement over the full-precision baseline when fused kernels are combined with full-array mapping. Compared with a tuned iGPU baseline, identified as llama.cpp Q4_K, the NPU delivers more than 2x performance and energy efficiency on GEMM (Pang et al., 9 Jun 2026).

For end-to-end LLM inference, the main reported gain is in prompt filling. TileFuse achieves up to 2.0x lower prefilling latency than the iGPU baseline and more than 64.6% lower energy consumption; one quoted configuration is Krackan Point running Llama3-8B with a 4096-token prompt. The paper also states that speedups are consistent for long prompts, whereas short prompts may still favor the iGPU because of NPU launch overhead.

Scope	Reported result	Context
GEMM throughput	up to 12 TOPS	Ryzen AI 7 350, Krackan Point, W4A16
GEMM throughput	up to 9 TOPS	Ryzen AI 9 HX 370, Strix Point
GEMM improvement	121.6%	over full-precision NPU baseline
GEMV improvement	281%	over full-precision baseline
Prefilling latency	up to 2.0x lower	versus iGPU baseline
Energy	more than 64.6% lower	Ryzen AI laptop end-to-end experiment

The ablation results are also structurally informative. Pre-tiling alone yields up to 50% gain in GEMM throughput due to improved streaming. Adding fused mixed-precision kernels yields up to 121.6% further improvement. For GEMV, optimized microkernels and dataflow add 2x–3x speedup, particularly when GEMV becomes compute-bound. At the request level, the paper gives the latency model

$L = l_p + n \cdot l_g$

where $\hat{w} = s_g (q - z_g)$0 is prefilling latency, $\hat{w} = s_g (q - z_g)$1 is per-token generation latency, and $\hat{w} = s_g (q - z_g)$2 is the number of generated tokens (Pang et al., 9 Jun 2026).

6. Significance, limitations, and placement within edge LLM inference

TileFuse’s main significance is that it makes XDNA2 a practical target for AWQ-style edge LLM inference while preserving native support for widely used quantization formats. The paper’s broader claim is that native NPU support for off-the-shelf quantization can make NPUs substantially more usable in real client deployments. In that sense, TileFuse is not primarily a new quantization method; it is a hardware-software co-design for executing established quantized formats efficiently on a specific NPU family (Pang et al., 9 Jun 2026).

Its limitations are equally explicit. The approach is focused on transformer linear layers rather than arbitrary operators. It depends on close-to-metal access through IRON and on workload-specific kernel programs and buffer descriptors. It also does not eliminate all deployment trade-offs: large GEMM operations benefit most because they amortize dataflow reconfiguration, whereas generation-side GEMV can still be constrained by dispatch overhead. The recommendation for hybrid NPU/iGPU deployment follows directly from this asymmetry.

Within the broader edge inference landscape, TileFuse can be understood as an argument for retaining popular model-side quantization conventions and moving the adaptation burden into kernel design, layout transformation, and array-level scheduling. This suggests a shift from platform-specific model reshaping toward platform-native execution of established quantized representations, with weight layout, metadata locality, and fused dequantization becoming first-class optimization targets.