AMD XDNA2 NPU: Deep Learning Accelerator

• AMD XDNA2 NPU is a second-generation accelerator in Ryzen AI systems, featuring a 4×8 tiled array of 32 cores and up to 50 TOPS performance.
• It employs explicit data movement and a three-level memory hierarchy with software-managed scratchpads to optimize DRAM usage and kernel execution.
• The programming model leverages tools like IRON and MLIR-AIR for precise tile placement and efficient mapping of dense linear algebra and transformer workloads.

AMD XDNA 2 NPU, often written XDNA2, is the second current generation of AMD’s Ryzen AI NPU family and functions as the dedicated deep-learning accelerator in Ryzen AI processors alongside CPUs and an integrated GPU. In the cited literature, it is described as a tiled spatial accelerator with explicit data movement, software-managed scratchpads, and a $4 \times 8$ array of 32 compute cores. Its practical significance is defined less by nominal peak compute than by the ability of compilers and runtimes to control tiling, memory placement, DMA orchestration, fusion, and dispatch on a device that shares DRAM with the rest of the system (Taka et al., 15 Dec 2025, Deshmukh et al., 25 Aug 2025).

1. Generational position within AMD’s Ryzen AI NPU family

The most explicit generational comparison presents XDNA2 as a materially more capable version of the original XDNA NPU rather than a minor refresh. XDNA is described as the earlier generation with 20 cores organized as a $4 \times 5$ array, while XDNA2 expands that to 32 cores in a $4 \times 8$ array. Both generations use the same basic modular architecture—compute tiles, memory tiles, and shim tiles—but XDNA2 is cited as offering up to 50 TOPS versus 10 TOPS for XDNA. The same source also states that XDNA2 adds hardware support for bfp16 (block floating-point), where “a block of eight numbers shares one common exponent,” in addition to native support for int8, int16, and bf16 (Taka et al., 15 Dec 2025).

The continuity with earlier client NPUs is also explicit in prior bare-metal work targeting the first-generation Phoenix XDNA platform rather than XDNA2. That work already described a tiled accelerator with 64 KB local memory per compute core, 512 KB per memory core, host-facing shim cores, and command-processor-driven reconfiguration, indicating that XDNA2 inherits a programming model centered on explicit orchestration rather than cache transparency (Rösti et al., 3 Apr 2025).

Feature	XDNA	XDNA2
Compute-core array	$4 \times 5$	$4 \times 8$
Core count	20	32
Advertised peak capability	10 TOPS	up to 50 TOPS
Datatype note	int8, int16, bf16	int8, int16, bf16, bfp16

2. Spatial organization, memory hierarchy, and data movement

Across the architecture-focused papers, XDNA2 is consistently described as an explicit data-movement architecture rather than a dynamically scheduled one. The NPU is a 2D array of compute tiles with local L1 memory, a row of memory tiles providing L2 memory, and shim tiles that connect to DRAM; an on-chip command processor orchestrates transfers and kernel execution. Only shim tiles can initiate transactions to SoC memory, while compute tiles communicate through local memories and the streaming interconnect. Every tile has a DMA engine, and the performance model therefore depends on matching work partitioning to local storage and communication resources rather than relying on hardware caches (Taka et al., 15 Dec 2025, Wang et al., 16 Oct 2025).

The end-to-end deployment work instantiates this model for Ryzen AI 300/400 Series processors, describing the AMD XDNA 2 NPU as a 4 × 8 tile array in which each compute tile has a 64 KB L1 scratchpad, each column has a 512 KB memory tile (L2) shared by its four compute tiles, a shim tile connects to host DDR, and tiles communicate through streaming interconnects and cascade connections (Li et al., 27 May 2026). TileFuse gives an equivalent hardware view in terms of 32 AIE compute cores, 8 memory cores, and 8 shim cores on Ryzen AI laptops (Pang et al., 9 Jun 2026).

The memory hierarchy is central because XDNA-family client NPUs use software-managed memory rather than hardware caches. Zen-Attention describes a three-level hierarchy consisting of $L1$ core-tile memory, $L2$ shared “memory-tile” scratchpad, and $L3$ host system memory or DRAM shared with the CPU and GPU. On an AMD Ryzen AI 9 HX platform, the NPU is allocated about $60\ \mathrm{GB/s}$, versus $130\ \mathrm{GB/s}$ for the GPU, making DRAM traffic a first-order constraint for transformer execution (Deshmukh et al., 25 Aug 2025).

Several XDNA2-specific memory-system features recur in optimization papers. The GEMM study highlights independent core execution, broadcast capability in which A tiles are broadcast across rows and B tiles across columns, 3D addressing in CompTiles and ShimTiles, 4D addressing in MemTiles, neighboring MemTile memory access on XDNA2, and six MM2S and six S2MM channels in MemTiles for staging output tiles (Taka et al., 15 Dec 2025). Zen-Attention adds the explicit architectural note that on AMD-XDNA2 NPUs, $4 \times 5$0 MM2S read DMA channels support padding across dimensions $4 \times 5$1, which directly affects irregular transformer shapes (Deshmukh et al., 25 Aug 2025).

3. Programming model and compiler infrastructure

The programming model exposed by XDNA2 is low level by accelerator standards. The deployment paper emphasizes that spatial NPUs expose explicit tile placement, explicit data movement scheduling, explicit scratchpad / memory tile management, explicit kernel dispatch, and hard constraints on DMA channels, L1/L2 capacity, BufferObject descriptor limits, and runtime, firmware, and driver overheads (Li et al., 27 May 2026). This differs from GPU or CPU execution models in which locality and scheduling are more extensively mediated by hardware.

Three open compiler layers recur in the literature. At the lowest level, IRON exposes compute tiles, memory tiles, shim tiles, ObjectFifos, and DMA tasks; earlier work describes it as a bare-metal toolflow in which a Python script defines NPU layout, routing, DMA behavior, and synchronization, while compute-core code is written in C++ with the AIE API and compiled into a static configuration and runtime instruction stream (Rösti et al., 3 Apr 2025). TileFuse is compiled through AMD’s open close-to-metal IRON / MLIR-AIE flow, using explicit control over compute tiles, memory layout, and data movement (Pang et al., 9 Jun 2026).

A more structured abstraction is provided by MLIR-AIR, whose AIR dialect introduces air.launch, air.segment, and air.herd for spatial scheduling; air.memcpy and air.channel.put/get for locality and communication; and air.token plus explicit dependency lists for synchronization. The compiler lowers high-level loop and tensor programs into explicit spatial schedules, overlaps communication with computation, and maps workloads onto AMD NPUs without relying on ad hoc runtime coordination. In matrix multiplication, MLIR-AIR reports up to 78.7% compute efficiency and performance almost identical to hand-optimized MLIR-AIE code, while a fused LLaMA 2 multi-head attention prototype is expressed in approximately 150 lines of high-level MLIR and reduces latency from 834 μs to 373 μs, a 2.24× speedup (Wang et al., 16 Oct 2025).

The ecosystem study NPUEval shows how difficult efficient kernel creation remains even with open tooling. It evaluates 102 common operators using LLVM-AIE, MLIR-AIE, and IRON on actual AMD NPU hardware, defines vectorization score as the percentage of cycles spent executing vector instructions, and reports that the average score across the dataset remains roughly 10% even with compiler feedback and vectorized kernel examples. The same study stresses that efficient AMD NPU programming requires vector intrinsics, careful data layout, and compiler-aware pragmas rather than scalar C++ loops (Kalade et al., 18 Jul 2025).

4. Dense linear algebra and the XDNA2 performance model

General matrix multiplication is the most systematic performance case study for XDNA2. The cited methodology is unified across XDNA and XDNA2 and is built around multi-level tiling plus an output-stationary mapping. Four tiling levels are defined: the innermost single-core tile $4 \times 5$2, a core-local tile $4 \times 5$3, an array-level tile covering the whole NPU, and the outer matrix dimensions $4 \times 5$4. Parallelization is performed across the $4 \times 5$5 and $4 \times 5$6 dimensions, while the reduction over $4 \times 5$7 is done in time within each core, so all cores compute independently on different data and avoid cross-core partial-sum communication (Taka et al., 15 Dec 2025).

The output-stationary scheme keeps each output $4 \times 5$8 tile resident in a core’s L1 memory while accumulation over $4 \times 5$9 proceeds. Input tiles $4 \times 8$0 and $4 \times 8$1 are double-buffered at L1 and L2 to overlap DMA with computation, while $4 \times 8$2 is kept as a single buffer to free L1 space for larger tile choices. Performance tuning is explicitly formulated around the balanced point where array compute time and DRAM time are comparable, summarized in the paper as $4 \times 8$3 (Taka et al., 15 Dec 2025).

This method yields large gains on XDNA2 because the architecture can exploit its full $4 \times 8$4 array, unlike first-generation mappings that sometimes used only a symmetric $4 \times 8$5 subset. The best reported full-array XDNA2 throughputs are 38.05 TOPS for int8-int8, 31.52 TOPS for int8-int16, 25.31 TOPS for int8-int32, and 14.71 TOPS for bf16-bf16. Representative balanced kernels include $4 \times 8$6 for int8-int8, reaching 36.13 TOPS, and $4 \times 8$7 for bf16-bf16, reaching 14.52 TOPS in one reported case and 14.71 TOPS in the roofline sweep (Taka et al., 15 Dec 2025).

A recurrent lesson is that XDNA2 performance is not determined by peak core compute alone. The same paper states that XDNA2 is more sensitive to DRAM access patterns than XDNA and that the difference between column-major and row-major $4 \times 8$8 is larger on XDNA2, because column-major $4 \times 8$9 enables larger contiguous accesses via $4 \times 5$0. This makes data layout and DMA address generation part of the kernel design rather than a secondary implementation detail (Taka et al., 15 Dec 2025).

5. Attention mapping, folding, and transformer execution

Attention is treated as a particularly difficult workload for AMD NPUs because it is memory-bound, has low operational intensity, large activation tensors, reduction-heavy softmax, and frequent element-wise and data-reordering operations. The standard NPU mapping computes

$4 \times 5$1

then applies optional bias and mask,

$4 \times 5$2

then

$4 \times 5$3

and finally

$4 \times 5$4

If this chain is implemented layer by layer, each stage reads from and writes to host memory, producing repeated DRAM round trips that Zen-Attention identifies as the primary latency bottleneck (Deshmukh et al., 25 Aug 2025).

Zen-Attention’s answer is a hardware-aware graph optimizer plus a tiler. The graph optimizer inspects the ONNX attention subgraph and tries to fold the canonical chain into one folded attention node, first checking whether the operator chain fits in $4 \times 5$5 memory and recording a folding_level. The paper defines folding_level = 3 as fully folded attention, folding_level = 2 as folding that excludes the final $4 \times 5$6 matmul because of $4 \times 5$7 capacity limits, and folding_level = 1 as essentially unfolded attention. The optimizer also batches heads to classify the operation as MHA, MQA, or GQA, and handles optional mask and bias addition (Deshmukh et al., 25 Aug 2025).

The tiler chooses subvolumes $4 \times 5$8 for $4 \times 5$9 that fit in $4 \times 8$0 under kernel-specific buffer-allocation constraints. It generates all valid tilings, sorts them in decreasing order to maximize $4 \times 8$1, and searches for a tiling that fits. A key decision is whether $4 \times 8$2 and $4 \times 8$3 are pinned to $4 \times 8$4; if so, $4 \times 8$5 can be unrolled across cores in the same column. If $4 \times 8$6, $4 \times 8$7, and $4 \times 8$8 are spatially split, the batch size becomes temporally iterated and spatial reduction is used across columns through cascade streams (Deshmukh et al., 25 Aug 2025).

Zen-Attention also addresses transpose, padding, and masking in ways intended not to destroy folding opportunities. Its “Folding-Preserving Transpose” uses DMA-based block transposes in $4 \times 8$9 and an $L1$0 specialized kernel called Transposed-MatMul. Because DMA has a minimum stride of 4 bytes, it cannot perform a true arbitrary transpose, so the data is transposed in $L1$1 blocks and rearranged with SHUFFLE intrinsics at register level before matmul. Padding uses XDNA2’s $L1$2 MM2S DMA support across $L1$3; if DMA padding is not possible, the framework tries to have the previous layer generate padded output so that no separate Pad operator is needed (Deshmukh et al., 25 Aug 2025).

The reported effects are substantial on representative transformer models evaluated on an AMD Ryzen AI 9 HX 370 system with a 32-core NPU in a $L1$4 grid delivering 50 TOPS. Zen-Attention achieves up to $L1$5 lower latency for the attention block relative to unfolded baselines and up to 32% lower end-to-end network latency. Even when attention is not the dominant bottleneck, the folded version still gives about 8% latency reduction and lowers DRAM bandwidth utilization; for one BERT configuration the end-to-end improvement is only 1.4%, which the paper uses to illustrate that folding benefits are largest when attention is itself the dominant bottleneck (Deshmukh et al., 25 Aug 2025).

6. Quantized and end-to-end LLM deployment on XDNA2

The most direct LLM-oriented XDNA2 kernel library is TileFuse, a close-to-metal mixed-precision library for quantized transformer linear layers on Ryzen AI laptops. It treats XDNA2 as the NPU backend for the two phases of LLM inference: prefill, dominated by large GEMMs, and token generation, where many linear layers become GEMV-like because batch size is typically 1. TileFuse supports AWQ-style $L1$6 and $L1$7 directly. The paper states the quantization formulas as

$L1$8

for symmetric quantization, and

$L1$9

for asymmetric quantization. In its implementation, $L2$0 uses AWQ-style per-group asymmetric quantization with group size 128, while $L2$1 uses per-channel symmetric quantization (Pang et al., 9 Jun 2026).

TileFuse performs offline pre-tiling of weights, appending scales and zero-points so that compute tiles can fetch weights and metadata from one contiguous stream. It also introduces an interleaved column-major pre-tiling layout to avoid the DMA stride limits that arise when output tiles are assigned round-robin across the eight compute columns, extending support to matrices with dimensions up to 32K rather than the baseline’s effective limit of 8K without multiple invocations. Kernel-side, it fuses unpacking, dequantization, GEMM or GEMV, and accumulation into one flow; for GEMV, it redesigns data movement so that all 32 compute cores in the $L2$2 array are used rather than only one compute row (Pang et al., 9 Jun 2026).

At kernel level, TileFuse reports improvements of up to 121.6% for GEMM and 281% for GEMV over full-precision baselines, along with more than 2× performance and energy-efficiency gains over strong iGPU baselines on GEMM. End-to-end, it reports up to 2.0× lower prefill latency with more than 64.6% lower energy consumption. The same source is explicit that the practical deployment strategy is hybrid: the runtime is sequential rather than concurrent, the NPU handles linear layers, and the iGPU executes FlashAttention-2, softmax, residuals, layernorm, and other lightweight operators. Short prompts and batch-1 GEMVs are often less favorable for the NPU because dispatch and reconfiguration overheads are not fully amortized (Pang et al., 9 Jun 2026).

A broader systems view appears in the paper on autonomous deployment, which uses the AMD XDNA 2 NPU as the target for complete decoder-only LLM inference rather than isolated kernels. The hardware target is a Ryzen AI 9 HX 370, and the software stack is MLIR-AIR with IRON as the lower-level interface. The deployment methodology proceeds from a human-guided reference implementation of Llama-3.2-1B to an eight-phase agent skill system with strict numerical gates, covering CPU-oracle construction, kernel validation, single-block validation, full-model validation, prefill optimization, decode optimization, finalization, and independent evaluation. The reference deployment achieves 2.2× speedup on prefill and 4.0× on decode over the hand-optimized baseline, with TTFT = 1.3 s and decode throughput = 10.8 tokens/s at sequence length 2048 (Li et al., 27 May 2026).

The same skill system is then used to deploy eight additional models—Llama-3.2-3B, SmolLM2-1.7B, Qwen2.5-{0.5B, 1.5B, 3B}, and Qwen3-{0.6B, 1.7B, 4B}—end to end on the AMD XDNA 2 NPU using the open-source compiler stack. Each deployment completes in about 0.5–4 hours of agent wall time with almost no human guidance, and three of the eight match or exceed the sustained performance of the Llama-3.2-1B reference deployment across both prefill and decode (Li et al., 27 May 2026).

7. Bottlenecks, misconceptions, and emerging research directions

A common misconception is that XDNA2 behavior can be understood primarily through TOPS figures. The performance studies instead emphasize that XDNA2 is highly sensitive to memory traffic and orchestration. In GEMM, the wider array places more pressure on effective DRAM bandwidth than first-generation XDNA, and the paper reports that column-major $L2$3 gives on average 19.1%, 25.2%, and 8.7% higher performance than row-major $L2$4 for int8-int8, int8-int16, and bf16-bf16, respectively. The same study identifies BD reconfiguration at the ShimTile level as another critical bottleneck: because each ShimTile has only 16 buffer descriptors, careful overlap of data movement with BD retirement and reconfiguration is required, and disabling this overlap reduces XDNA2 int8-int16 performance from 30.77 TOPS to 22.21 TOPS, a 28% decrease (Taka et al., 15 Dec 2025).

A second misconception is that end-to-end LLM deployment is simply a matter of obtaining fast kernels. The deployment work stresses that kernel dispatch overhead can exceed kernel execution time, that host-side context setup and BufferObject reuse materially affect performance, and that merging multiple kernels into a single dispatch is often necessary. In the reported prefill path, the 8-kernel post-attention block was merged into one dispatch, the 6-kernel pre-attention block was merged into one dispatch, and per-layer prefill dispatches were reduced from 15 to 3 (Li et al., 27 May 2026).

A third misconception is that XDNA2 should replace the iGPU uniformly across all LLM phases. TileFuse is explicit that the best practical use is a hybrid strategy: prefilling on the NPU, token generation on the iGPU. This division follows from the workload structure itself: large GEMMs amortize NPU dispatch and reconfiguration overhead, whereas short, latency-sensitive GEMVs often do not (Pang et al., 9 Jun 2026).

The open research agenda described in the cited work remains broad. The autonomous deployment paper explicitly lists future work in kernel fusion, quantization, dataflow optimization, and support for sliding-window attention, Mixture-of-Experts, and Multi-head Latent Attention (Li et al., 27 May 2026). The recurring conclusion across the architecture, compiler, and workload papers is that XDNA2 is most effective when software explicitly exploits $L2$5 and $L2$6 residency, minimizes DRAM round trips, and schedules communication and computation as first-class objects rather than as incidental by-products of a generic execution model (Deshmukh et al., 25 Aug 2025, Wang et al., 16 Oct 2025).