---
title: 'AMD XDNA2 NPU: Deep Learning Accelerator'
url: https://www.emergentmind.com/topics/amd-xdna-2-npu
type: topic
---

# AMD XDNA2 NPU: Deep Learning Accelerator

AMD XDNA 2 NPU, often written XDNA2, is the second current generation of AMD’s Ryzen AI NPU family and functions as the dedicated deep-learning accelerator in Ryzen AI processors alongside CPUs and an integrated GPU. In the cited literature, it is described as a tiled spatial accelerator with explicit data movement, software-managed scratchpads, and a \(4 \times 8\) array of 32 compute cores. Its practical significance is defined less by nominal peak compute than by the ability of compilers and runtimes to control tiling, memory placement, DMA orchestration, fusion, and dispatch on a device that shares DRAM with the rest of the system [2512.13282][2508.17593].

## 1. Generational position within AMD’s Ryzen AI NPU family

The most explicit generational comparison presents XDNA2 as a materially more capable version of the original XDNA NPU rather than a minor refresh. XDNA is described as the earlier generation with **20 cores organized as a \(4 \times 5\) array**, while XDNA2 expands that to **32 cores in a \(4 \times 8\) array**. Both generations use the same basic modular architecture—compute tiles, memory tiles, and shim tiles—but XDNA2 is cited as offering **up to 50 TOPS** versus **10 TOPS** for XDNA. The same source also states that XDNA2 adds **hardware support for bfp16 (block floating-point)**, where “a block of eight numbers shares one common exponent,” in addition to native support for int8, int16, and bf16 [2512.13282].

The continuity with earlier client NPUs is also explicit in prior bare-metal work targeting the first-generation Phoenix XDNA platform rather than XDNA2. That work already described a tiled accelerator with **64 KB** local memory per compute core, **512 KB** per memory core, host-facing shim cores, and command-processor-driven reconfiguration, indicating that XDNA2 inherits a programming model centered on explicit orchestration rather than cache transparency [2504.03083].

| Feature | XDNA | XDNA2 |
|---|---:|---:|
| Compute-core array | \(4 \times 5\) | \(4 \times 8\) |
| Core count | 20 | 32 |
| Advertised peak capability | 10 TOPS | up to 50 TOPS |
| Datatype note | int8, int16, bf16 | int8, int16, bf16, bfp16 |

## 2. Spatial organization, memory hierarchy, and data movement

Across the architecture-focused papers, XDNA2 is consistently described as an explicit data-movement architecture rather than a dynamically scheduled one. The NPU is a 2D array of compute tiles with local **L1 memory**, a row of memory tiles providing **L2 memory**, and shim tiles that connect to DRAM; an on-chip command processor orchestrates transfers and kernel execution. Only shim tiles can initiate transactions to SoC memory, while compute tiles communicate through local memories and the streaming interconnect. Every tile has a DMA engine, and the performance model therefore depends on matching work partitioning to local storage and communication resources rather than relying on hardware caches [2512.13282][2510.14871].

The end-to-end deployment work instantiates this model for **Ryzen AI 300/400 Series processors**, describing the AMD XDNA 2 NPU as a **4 × 8 tile array** in which each compute tile has a **64 KB L1 scratchpad**, each column has a **512 KB memory tile (L2)** shared by its four compute tiles, a **shim tile** connects to host DDR, and tiles communicate through **streaming interconnects** and **cascade connections** [2606.07586]. TileFuse gives an equivalent hardware view in terms of **32 AIE compute cores**, **8 memory cores**, and **8 shim cores** on Ryzen AI laptops [2606.11357].

The memory hierarchy is central because XDNA-family client NPUs use software-managed memory rather than hardware caches. Zen-Attention describes a three-level hierarchy consisting of \(L1\) core-tile memory, \(L2\) shared “memory-tile” scratchpad, and \(L3\) host system memory or DRAM shared with the CPU and GPU. On an AMD Ryzen AI 9 HX platform, the NPU is allocated about \(60\ \mathrm{GB/s}\), versus \(130\ \mathrm{GB/s}\) for the GPU, making DRAM traffic a first-order constraint for transformer execution [2508.17593].

Several XDNA2-specific memory-system features recur in optimization papers. The GEMM study highlights **independent core execution**, **broadcast capability** in which A tiles are broadcast across rows and B tiles across columns, **3D addressing** in CompTiles and ShimTiles, **4D addressing** in MemTiles, **neighboring MemTile memory access** on XDNA2, and **six MM2S and six S2MM channels in MemTiles** for staging output tiles [2512.13282]. Zen-Attention adds the explicit architectural note that on **AMD-XDNA2 NPUs, \(L2\) MM2S read DMA channels support padding across dimensions \(\{D0,D1,D2\}\)**, which directly affects irregular transformer shapes [2508.17593].

## 3. Programming model and compiler infrastructure

The programming model exposed by XDNA2 is low level by accelerator standards. The deployment paper emphasizes that spatial NPUs expose explicit **tile placement**, explicit **data movement scheduling**, explicit **scratchpad / memory tile management**, explicit **kernel dispatch**, and hard constraints on **DMA channels**, **L1/L2 capacity**, **BufferObject descriptor limits**, and runtime, firmware, and driver overheads [2606.07586]. This differs from GPU or CPU execution models in which locality and scheduling are more extensively mediated by hardware.

Three open compiler layers recur in the literature. At the lowest level, **IRON** exposes compute tiles, memory tiles, shim tiles, ObjectFifos, and DMA tasks; earlier work describes it as a bare-metal toolflow in which a Python script defines NPU layout, routing, DMA behavior, and synchronization, while compute-core code is written in C++ with the AIE API and compiled into a static configuration and runtime instruction stream [2504.03083]. TileFuse is compiled through AMD’s open close-to-metal **IRON / MLIR-AIE** flow, using explicit control over compute tiles, memory layout, and data movement [2606.11357].

A more structured abstraction is provided by **MLIR-AIR**, whose AIR dialect introduces `air.launch`, `air.segment`, and `air.herd` for spatial scheduling; `air.memcpy` and `air.channel.put/get` for locality and communication; and `air.token` plus explicit dependency lists for synchronization. The compiler lowers high-level loop and tensor programs into explicit spatial schedules, overlaps communication with computation, and maps workloads onto AMD NPUs without relying on ad hoc runtime coordination. In matrix multiplication, MLIR-AIR reports up to **78.7%** compute efficiency and performance almost identical to hand-optimized MLIR-AIE code, while a fused LLaMA 2 multi-head attention prototype is expressed in approximately **150 lines of high-level MLIR** and reduces latency from **834 μs** to **373 μs**, a **2.24×** speedup [2510.14871].

The ecosystem study **NPUEval** shows how difficult efficient kernel creation remains even with open tooling. It evaluates **102 common operators** using **LLVM-AIE**, **MLIR-AIE**, and **IRON** on actual AMD NPU hardware, defines vectorization score as the percentage of cycles spent executing vector instructions, and reports that the average score across the dataset remains roughly **10%** even with compiler feedback and vectorized kernel examples. The same study stresses that efficient AMD NPU programming requires vector intrinsics, careful data layout, and compiler-aware pragmas rather than scalar C++ loops [2507.14403].

## 4. Dense linear algebra and the XDNA2 performance model

General matrix multiplication is the most systematic performance case study for XDNA2. The cited methodology is unified across XDNA and XDNA2 and is built around **multi-level tiling** plus an **output-stationary mapping**. Four tiling levels are defined: the innermost single-core tile \(r \times s \times t\), a core-local tile \(m_{ct} \times k_{ct} \times n_{ct}\), an array-level tile covering the whole NPU, and the outer matrix dimensions \(M \times K \times N\). Parallelization is performed across the \(M\) and \(N\) dimensions, while the reduction over \(K\) is done in time within each core, so all cores compute independently on different data and avoid cross-core partial-sum communication [2512.13282].

The output-stationary scheme keeps each output \(C\) tile resident in a core’s L1 memory while accumulation over \(K\) proceeds. Input tiles \(A\) and \(B\) are **double-buffered** at L1 and L2 to overlap DMA with computation, while \(C\) is kept as a **single buffer** to free L1 space for larger tile choices. Performance tuning is explicitly formulated around the balanced point where array compute time and DRAM time are comparable, summarized in the paper as \(T_{\text{comp}} \approx T_{\text{mem}}\) [2512.13282].

This method yields large gains on XDNA2 because the architecture can exploit its full **\(4 \times 8\)** array, unlike first-generation mappings that sometimes used only a symmetric \(4 \times 4\) subset. The best reported full-array XDNA2 throughputs are **38.05 TOPS** for **int8-int8**, **31.52 TOPS** for **int8-int16**, **25.31 TOPS** for **int8-int32**, and **14.71 TOPS** for **bf16-bf16**. Representative balanced kernels include **\(160 \times 64 \times 144\)** for int8-int8, reaching **36.13 TOPS**, and **\(112 \times 48 \times 96\)** for bf16-bf16, reaching **14.52 TOPS** in one reported case and **14.71 TOPS** in the roofline sweep [2512.13282].

A recurrent lesson is that XDNA2 performance is not determined by peak core compute alone. The same paper states that XDNA2 is more sensitive to DRAM access patterns than XDNA and that the difference between **column-major** and **row-major** \(B\) is larger on XDNA2, because column-major \(B\) enables larger contiguous accesses via \(k_{mt}\). This makes data layout and DMA address generation part of the kernel design rather than a secondary implementation detail [2512.13282].

## 5. Attention mapping, folding, and transformer execution

Attention is treated as a particularly difficult workload for AMD NPUs because it is memory-bound, has low operational intensity, large activation tensors, reduction-heavy softmax, and frequent element-wise and data-reordering operations. The standard NPU mapping computes
\[
A = Q \cdot K^T
\]
then applies optional bias and mask,
\[
A = A + B + M,
\]
then
\[
SM_{out} = \mathrm{SoftMax}(A),
\]
and finally
\[
Z = SM_{out} \cdot V.
\]
If this chain is implemented layer by layer, each stage reads from and writes to host memory, producing repeated DRAM round trips that Zen-Attention identifies as the primary latency bottleneck [2508.17593].

Zen-Attention’s answer is a hardware-aware graph optimizer plus a tiler. The graph optimizer inspects the ONNX attention subgraph and tries to fold the canonical chain into one folded attention node, first checking whether the operator chain fits in \(L1\) memory and recording a `folding_level`. The paper defines `folding_level = 3` as fully folded attention, `folding_level = 2` as folding that excludes the final \(SM \cdot V\) matmul because of \(L1\) capacity limits, and `folding_level = 1` as essentially unfolded attention. The optimizer also batches heads to classify the operation as MHA, MQA, or GQA, and handles optional mask and bias addition [2508.17593].

The tiler chooses subvolumes \(\{S_q,S_k,S_v,S_b,S_m\}\) for \(Q,K,V,B,M\) that fit in \(L1\) under kernel-specific buffer-allocation constraints. It generates all valid tilings, sorts them in decreasing order to maximize \(S_q\), and searches for a tiling that fits. A key decision is whether \(K\) and \(V\) are pinned to \(L1\); if so, \(S_q\) can be unrolled across cores in the same column. If \(K\), \(V\), and \(M\) are spatially split, the batch size becomes temporally iterated and spatial reduction is used across columns through cascade streams [2508.17593].

Zen-Attention also addresses transpose, padding, and masking in ways intended not to destroy folding opportunities. Its “Folding-Preserving Transpose” uses DMA-based block transposes in \(L2\) and an \(L1\) specialized kernel called **Transposed-MatMul**. Because DMA has a minimum stride of 4 bytes, it cannot perform a true arbitrary transpose, so the data is transposed in \(8 \times 8\) blocks and rearranged with **SHUFFLE intrinsics** at register level before matmul. Padding uses XDNA2’s \(L2\) MM2S DMA support across \(\{D0,D1,D2\}\); if DMA padding is not possible, the framework tries to have the previous layer generate padded output so that no separate Pad operator is needed [2508.17593].

The reported effects are substantial on representative transformer models evaluated on an AMD Ryzen AI 9 HX 370 system with a **32-core NPU** in a **\(4 \times 8\)** grid delivering **50 TOPS**. Zen-Attention achieves up to **\(4\times\)** lower latency for the attention block relative to unfolded baselines and up to **32%** lower end-to-end network latency. Even when attention is not the dominant bottleneck, the folded version still gives about **8%** latency reduction and lowers DRAM bandwidth utilization; for one BERT configuration the end-to-end improvement is only **1.4%**, which the paper uses to illustrate that folding benefits are largest when attention is itself the dominant bottleneck [2508.17593].

## 6. Quantized and end-to-end LLM deployment on XDNA2

The most direct LLM-oriented XDNA2 kernel library is **TileFuse**, a close-to-metal mixed-precision library for quantized transformer linear layers on Ryzen AI laptops. It treats XDNA2 as the NPU backend for the two phases of LLM inference: **prefill**, dominated by large GEMMs, and **token generation**, where many linear layers become GEMV-like because batch size is typically 1. TileFuse supports **AWQ-style \(W4A16\)** and **\(W8A16\)** directly. The paper states the quantization formulas as
\[
\hat{w} = s_g q
\]
for symmetric quantization, and
\[
\hat{w} = s_g \bigl(q - z_g\bigr)
\]
for asymmetric quantization. In its implementation, \(W4A16\) uses AWQ-style **per-group asymmetric quantization** with group size **128**, while \(W8A16\) uses **per-channel symmetric quantization** [2606.11357].

TileFuse performs **offline pre-tiling** of weights, appending scales and zero-points so that compute tiles can fetch weights and metadata from one contiguous stream. It also introduces an **interleaved column-major pre-tiling layout** to avoid the DMA stride limits that arise when output tiles are assigned round-robin across the eight compute columns, extending support to matrices with dimensions up to **32K** rather than the baseline’s effective limit of **8K** without multiple invocations. Kernel-side, it fuses unpacking, dequantization, GEMM or GEMV, and accumulation into one flow; for GEMV, it redesigns data movement so that all **32** compute cores in the \(4 \times 8\) array are used rather than only one compute row [2606.11357].

At kernel level, TileFuse reports improvements of up to **121.6% for GEMM** and **281% for GEMV** over full-precision baselines, along with more than **2×** performance and energy-efficiency gains over strong iGPU baselines on GEMM. End-to-end, it reports up to **2.0× lower prefill latency** with more than **64.6% lower energy consumption**. The same source is explicit that the practical deployment strategy is hybrid: the runtime is sequential rather than concurrent, the NPU handles linear layers, and the iGPU executes **FlashAttention-2, softmax, residuals, layernorm, and other lightweight operators**. Short prompts and batch-1 GEMVs are often less favorable for the NPU because dispatch and reconfiguration overheads are not fully amortized [2606.11357].

A broader systems view appears in the paper on autonomous deployment, which uses the AMD XDNA 2 NPU as the target for complete decoder-only LLM inference rather than isolated kernels. The hardware target is a Ryzen AI 9 HX 370, and the software stack is **MLIR-AIR** with **IRON** as the lower-level interface. The deployment methodology proceeds from a human-guided reference implementation of **Llama-3.2-1B** to an **eight-phase** agent skill system with strict numerical gates, covering CPU-oracle construction, kernel validation, single-block validation, full-model validation, prefill optimization, decode optimization, finalization, and independent evaluation. The reference deployment achieves **2.2×** speedup on prefill and **4.0×** on decode over the hand-optimized baseline, with **TTFT = 1.3 s** and **decode throughput = 10.8 tokens/s** at sequence length **2048** [2606.07586].

The same skill system is then used to deploy eight additional models—**Llama-3.2-3B**, **SmolLM2-1.7B**, **Qwen2.5-\{0.5B, 1.5B, 3B\}**, and **Qwen3-\{0.6B, 1.7B, 4B\}**—end to end on the AMD XDNA 2 NPU using the open-source compiler stack. Each deployment completes in about **0.5–4 hours** of agent wall time with almost no human guidance, and three of the eight match or exceed the sustained performance of the Llama-3.2-1B reference deployment across both prefill and decode [2606.07586].

## 7. Bottlenecks, misconceptions, and emerging research directions

A common misconception is that XDNA2 behavior can be understood primarily through TOPS figures. The performance studies instead emphasize that XDNA2 is highly sensitive to memory traffic and orchestration. In GEMM, the wider array places more pressure on effective DRAM bandwidth than first-generation XDNA, and the paper reports that column-major \(B\) gives on average **19.1%**, **25.2%**, and **8.7%** higher performance than row-major \(B\) for int8-int8, int8-int16, and bf16-bf16, respectively. The same study identifies **BD reconfiguration** at the ShimTile level as another critical bottleneck: because each ShimTile has only **16 buffer descriptors**, careful overlap of data movement with BD retirement and reconfiguration is required, and disabling this overlap reduces XDNA2 int8-int16 performance from **30.77 TOPS** to **22.21 TOPS**, a **28% decrease** [2512.13282].

A second misconception is that end-to-end LLM deployment is simply a matter of obtaining fast kernels. The deployment work stresses that kernel dispatch overhead can exceed kernel execution time, that host-side context setup and BufferObject reuse materially affect performance, and that merging multiple kernels into a single dispatch is often necessary. In the reported prefill path, the **8-kernel post-attention block** was merged into one dispatch, the **6-kernel pre-attention block** was merged into one dispatch, and per-layer prefill dispatches were reduced from **15 to 3** [2606.07586].

A third misconception is that XDNA2 should replace the iGPU uniformly across all LLM phases. TileFuse is explicit that the best practical use is a **hybrid strategy**: prefilling on the NPU, token generation on the iGPU. This division follows from the workload structure itself: large GEMMs amortize NPU dispatch and reconfiguration overhead, whereas short, latency-sensitive GEMVs often do not [2606.11357].

The open research agenda described in the cited work remains broad. The autonomous deployment paper explicitly lists future work in **kernel fusion**, **quantization**, **dataflow optimization**, and support for **sliding-window attention**, **Mixture-of-Experts**, and **Multi-head Latent Attention** [2606.07586]. The recurring conclusion across the architecture, compiler, and workload papers is that XDNA2 is most effective when software explicitly exploits \(L1\) and \(L2\) residency, minimizes DRAM round trips, and schedules communication and computation as first-class objects rather than as incidental by-products of a generic execution model [2508.17593][2510.14871].

Source: https://www.emergentmind.com/topics/amd-xdna-2-npu