Ryzen AI: AMD's Integrated Edge AI Subsystem

• Ryzen AI is AMD's edge AI subsystem featuring tiled NPUs integrated with CPU and GPU, optimized for explicit dataflow and communication.
• It leverages a spatial, tile-based architecture that balances compute and memory, achieving up to 50 TOPS on second-generation NPUs.
• The software stack offers both high-level compiler offload and bare-metal specialization, enabling dynamic heterogeneous scheduling across CPU, GPU, and NPU.

Searching arXiv for the provided Ryzen AI papers to ground the article and verify metadata. Ryzen AI is AMD’s edge AI subsystem integrated into Ryzen processors, alongside CPU and GPU, and centered on a dedicated Neural Processing Unit (NPU) derived from AMD’s AI Engine lineage (Taka et al., 15 Dec 2025). Across the literature, the term denotes both a hardware family—spanning first-generation XDNA and second-generation XDNA2 client NPUs—and a software stack that ranges from high-level framework integration to close-to-metal toolchains such as IRON, MLIR-AIE, XRT, and, in newer compiler work, MLIR-AIR (Rösti et al., 3 Apr 2025). The research record presents Ryzen AI not as a monolithic accelerator, but as a spatial, explicit-dataflow substrate whose behavior is governed by tile-local memories, memory tiles, shim-mediated DRAM access, DMA orchestration, and workload-dependent backend selection across CPU, GPU, and NPU (Karami et al., 19 Jul 2025).

1. Architectural organization and hardware model

Ryzen AI NPUs are described as tiled spatial architectures organized as a 2D array of compute tiles, memory tiles, and shim tiles (Taka et al., 15 Dec 2025). In the cross-generation GEMM study, both XDNA and XDNA2 are organized as a 2D array of compute tiles (CompTiles), each containing a VLIW SIMD core and a local 64 KB L1 memory; below them sits a row of memory tiles (MemTiles), each with 512 KB L2 memory; below that sits a row of shim tiles (ShimTiles) that connect the NPU to DRAM through the SoC fabric and NoC (Taka et al., 15 Dec 2025). XDNA has 20 cores arranged as $4 \times 5$, whereas XDNA2 has 32 cores arranged as $4 \times 8$; peak theoretical NPU throughput rises from about 10 TOPS on XDNA to 50 TOPS on XDNA2 (Taka et al., 15 Dec 2025).

The movement model is explicit rather than cache-coherent. DMAs exist in all tile types; transfers use MM2S/S2MM channels through configurable switches; and hardware locks synchronize producers and consumers (Taka et al., 15 Dec 2025). Each CompTile and ShimTile has two MM2S and two S2MM channels; each MemTile has six MM2S and six S2MM channels. Multi-dimensional DMA addressing is central: CompTiles and ShimTiles support 3D tensor addressing, while MemTiles support 4D addressing (Taka et al., 15 Dec 2025). The literature repeatedly treats this explicit routing and synchronization model as a defining property of Ryzen AI-class NPUs, including in low-level training work that emphasizes the absence of caches, fixed instruction latencies, and the need to program routes, DMAs, and semaphore locks directly (Rösti et al., 3 Apr 2025).

Datatype support is generation-sensitive. The GEMM study notes native support for int8, int16, bf16 on both XDNA and XDNA2, with XDNA2 additionally supporting bfp16 in hardware (Taka et al., 15 Dec 2025). By contrast, the Fortran-intrinsics study, which examines the Ryzen AI 7000-series AIE-ML substrate, states that int32 and float32 are not natively supported in hardware by the AIE-ML vector units, though they can be emulated, and that the best fits are workloads aligned with int16 and bfloat16 (Brown et al., 14 Feb 2025). This suggests that “Ryzen AI” names a family of related AMD AI-engine-based client accelerators rather than a single invariant microarchitecture.

A recurrent architectural consequence is that only part of the nominal array may be practical for a given mapping. In first-generation XDNA, because the last column lacks a ShimTile, the practical GEMM mapping uses a symmetric $4 \times 4$ subarray rather than all 20 cores; on XDNA2, the entire $4 \times 8$ array is used (Taka et al., 15 Dec 2025). The same regular $4 \times 4$ restriction appears in bare-metal GPT-2 fine-tuning, where only four columns have direct shim access to main memory, so the implementation adopts a regular $4 \times 4$ partition with full shim coverage (Rösti et al., 3 Apr 2025).

Generation / platform	Array organization	Noted hardware characteristics
XDNA	20 cores arranged as $4 \times 5$	Practical GEMM mapping uses $4 \times 4$
XDNA2	32 cores arranged as $4 \times 8$	Full $4 \times 8$ utilization
Ryzen AI 7940HS AIE-ML study	20 AIE-MLs arranged as five columns of four rows	Experiments use 16 AIEs

2. Software stacks, compilation flows, and programming abstractions

The software literature on Ryzen AI spans three distinct strata: transparent compiler offload, close-to-metal NPU programming, and intermediate compiler layers for spatial scheduling. In the Fortran work, the front end is Flang, which lowers Fortran through HLFIR and FIR into the broader MLIR ecosystem, preserving intrinsic semantics long enough to identify operations such as matmul, sum, product, maxval, and minval as accelerator-friendly patterns (Brown et al., 14 Feb 2025). Host-side control is generated through AMD XRT, mediated by a new MLIR dialect called xrt_wrapper (xrtw), while AIE-side programs are generated through AMD’s aie, aievec, and adf dialect ecosystem ported into xDSL (Brown et al., 14 Feb 2025). This flow is explicitly designed so that ordinary Fortran intrinsic calls can execute on Ryzen AI hardware without source modification.

At the low-level end, the client-side training work distinguishes between Ryzen AI Software, described as the higher-level production-oriented flow integrating with frameworks such as PyTorch and TensorFlow, and IRON, the low-level, open-source, bare-metal flow used to program the NPU directly (Rösti et al., 3 Apr 2025). With IRON, developers write C++ kernels for AI Engine cores using the AIE API, compile them with an open-source LLVM-based compiler or AMD’s proprietary xchessccc, describe layout, DMA behavior, locks, and routing in Python, and then generate MLIR for MLIR-AIE, producing a final.xclbin and insts.txt command stream for the command processor (Rösti et al., 3 Apr 2025). Runtime control again goes through XRT.

A third layer is introduced by MLIR-AIR, which is positioned between frontend tensor IRs and lower-level backends such as MLIR-AIE (Wang et al., 16 Oct 2025). AIR introduces air.launch, air.segment, and air.herd to represent offload regions, resource pools, and contiguous grids of concurrent workers; air.memcpy and air.channel to model explicit movement through the memory hierarchy; and air.async.token plus air.wait_all to encode an asynchronous control and dataflow graph directly in SSA form (Wang et al., 16 Oct 2025). The paper characterizes AIR as a structured representation for asynchronous scheduling, hierarchical resource partitioning, locality, and synchronization, precisely because general-purpose compiler abstractions hide the placement and dataflow structure that determines performance on AMD NPUs.

These stacks differ in ergonomics but converge on the same underlying model: explicit locality, explicit communication, and explicit orchestration. A common misconception is that Ryzen AI is accessible only through fixed inference libraries. The research record contradicts that in two ways. First, the Fortran study demonstrates transparent compiler-driven offload for selected scientific intrinsics (Brown et al., 14 Feb 2025). Second, the bare-metal training paper shows that the NPU can be programmed deeply enough for custom GPT-2 fine-tuning kernels (Rösti et al., 3 Apr 2025). A plausible implication is that future Ryzen AI software will continue to combine high-level semantic preservation with lower-level spatial compilation rather than collapsing onto a single programming interface.

3. GEMM as the dominant kernel and the central optimization problem

The most detailed performance model in the literature treats Ryzen AI optimization as a balance between on-chip compute efficiency and off-chip DRAM bandwidth (Taka et al., 15 Dec 2025). GEMM is central because modern deep learning inference is dominated by matrix multiplications, and improving GEMM directly improves the throughput of layers in transformers and other DNNs (Taka et al., 15 Dec 2025). The cross-generation study explicitly targets end-to-end inference-style GEMM on edge devices, not isolated single-core kernels, and adopts an output-stationary mapping in which each core owns a tile of $4 \times 8$0, repeatedly loads tiles of $4 \times 8$1 and $4 \times 8$2 across the reduction dimension $4 \times 8$3, accumulates into the local $4 \times 8$4 tile, and writes $4 \times 8$5 out only after finishing the full reduction (Taka et al., 15 Dec 2025).

The single-core design is constrained by the 63 KB effective L1 budget after reserving 1 KB for stack. The study defines compute and communication costs as

$4 \times 8$6

$4 \times 8$7

$4 \times 8$8

with the compute-bound constraint

$4 \times 8$9

and the L1 capacity constraint

$4 \times 4$0

(Taka et al., 15 Dec 2025).

At full-array scale, the key thesis is that the fastest single-core kernel is often not the best end-to-end kernel. The derived DRAM traffic formulas

$4 \times 4$1

$4 \times 4$2

$4 \times 4$3

lead to system-level time models

$4 \times 4$4

$4 \times 4$5

and the empirical optimum occurs near

$4 \times 4$6

rather than at maximum local MAC density (Taka et al., 15 Dec 2025). This “strike the balance” principle is the paper’s central contribution.

Quantitatively, the balanced end-to-end results reach up to 6.76 TOPS on XDNA and 38.05 TOPS on XDNA2 for int8, and up to 3.14 TOPS and 14.71 TOPS respectively for bf16 (Taka et al., 15 Dec 2025). The paper reports effective DRAM bandwidth of about 15 GB/s on XDNA and 50 GB/s on XDNA2, and attributes major performance gains to on-the-fly DMA tensor transformations, larger contiguous $4 \times 4$7-based transfers, column-major storage of $4 \times 4$8, and overlap of buffer-descriptor reuse with ongoing DMA (Taka et al., 15 Dec 2025). One particularly sharp result is that without overlapped buffer-descriptor reconfiguration, int8-int16 GEMM on XDNA2 drops from 30.77 TOPS to 22.21 TOPS, a 28% decrease; XDNA sees a similar 27% drop (Taka et al., 15 Dec 2025).

The same themes recur in adjacent AMD AI-engine work. The Versal AIE-ML GEMM study emphasizes local-memory fit, exact buffer-bank placement, reduction topology, pack size, and routing-aware mapping, and shows that compiler defaults can leave substantial efficiency on the table (Mhatre et al., 13 Apr 2025). Although that work targets Versal rather than Ryzen AI directly, it reinforces a Ryzen AI-specific lesson already visible in XDNA/XDNA2: matrix performance depends on treating the device as a distributed array with explicit locality and communication constraints, not as a black-box TOPS number (Mhatre et al., 13 Apr 2025).

4. Scientific computing, transparent offload, and client-side training

Ryzen AI is not presented solely as an inference substrate for pre-packaged models. In scientific computing, the Fortran study demonstrates compiler-driven offload of intrinsics such as sum, product, maxval, minval, transpose, and matmul onto the AIEs in Ryzen AI CPUs, with no source changes required by the programmer (Brown et al., 14 Feb 2025). The evaluated platform is an AMD Ryzen AI 7940HS with 32 GB DRAM, 20 AIE-ML tiles, GCC 13.2, Vitis 2023.1, XRT 2.18.0, and Flang 20.0.0, and all results are averaged over ten runs with AIE timings including data transfer overhead (Brown et al., 14 Feb 2025).

The results show both opportunity and constraint. For repeated reductions on a 1D array of 262,144 elements, first-run NPU runtimes are much worse than CPU runtimes because startup overhead is substantial; the paper attributes most of this to nearly constant startup cost incurred when run is first called (Brown et al., 14 Feb 2025). After amortization, the picture improves: for sum, int16 improves from CPU 606 µs to NPU 221 µs, about 2.74× faster, whereas sum int32 remains slower on the NPU, going from CPU 296 µs to NPU 373 µs (Brown et al., 14 Feb 2025). matmul is the strongest case: for 256×256 by 256×512 inputs producing 256×512 output, the paper reports NPU subsequent runtimes of 1353 µs for int16, 1503 µs for int32, **1357 µs* for bfloat16, and 1471 µs for float32, with large speedups relative to CPU baselines (Brown et al., 14 Feb 2025). The asterisk matters: for **int32* and float32 cases, the output type is int32/float32 but the internal algorithm uses reduced-precision counterparts (int16 and bfloat16) for inputs (Brown et al., 14 Feb 2025).

A more aggressive claim appears in client-side training. The GPT-2 fine-tuning paper studies the first-generation XDNA architecture inside an AMD Ryzen 9 7940HS system and offloads only GEMM from LLM.c while leaving the rest of training on the CPU (Rösti et al., 3 Apr 2025). The fixed compile-time tile sizes are

$4 \times 4$9

and each of the 16 compute cores accumulates one $4 \times 8$0 output tile at a time, repeating over $4 \times 8$1 input-tile pairs (Rösti et al., 3 Apr 2025). The NPU path uses bfloat16 inputs and float32 accumulation/output, whereas the CPU baseline uses float32 for both inputs and outputs (Rösti et al., 3 Apr 2025).

This distinction corrects another common misunderstanding: the work does not perform training entirely inside the NPU. Rather, it performs fine-tuning entirely on the client device, with the NPU accelerating the matrix multiplications in both the forward and backward passes (Rösti et al., 3 Apr 2025). For GPT-2 small (124M parameters), forward-pass GEMMs are on average 3.1× faster on the NPU and backward-pass GEMMs 2.8× faster; end-to-end fine-tuning throughput improves by $4 \times 8$2 on mains power and $4 \times 8$3 on battery power, while FLOPS/Ws on battery improve by $4 \times 8$4 (Rösti et al., 3 Apr 2025). The paper also reports mean relative divergence < 0.06\%, standard deviation 0.03\%, and maximum deviation 0.1\% at size $4 \times 8$5 (Rösti et al., 3 Apr 2025).

Taken together, these studies indicate that Ryzen AI supports both transparent compiler-driven acceleration and expert-level bare-metal specialization. The practical boundaries are equally clear: one-off fine-grained intrinsic calls may not amortize startup cost; unsupported datatypes such as float32 and int32 can be unfavorable; and end-to-end training gains remain limited when only GEMM is offloaded (Brown et al., 14 Feb 2025).

5. Heterogeneous scheduling across CPU, GPU, and NPU

A broader systems view of Ryzen AI emerges in work on real-time generative AI workloads, where the platform is treated as a heterogeneous SoC integrating CPU, GPU, and NPU (Karami et al., 19 Jul 2025). On the evaluated AMD Ryzen AI 9 365 system, the NPU is XDNA 2, 50 TOPS, the GPU is AMD Radeon 880M, 16 TOPS, and system memory is 24 GB (Karami et al., 19 Jul 2025). The study uses AMD’s open-source RyzenAI SW stack, with ONNX Runtime for non-LLM models, ONNX Runtime Generative AI for LLMs, Vitis AI Execution Provider for NPU, and DirectML Execution Provider for GPU (Karami et al., 19 Jul 2025).

The central observation is that backend preference is phase- and input-dependent rather than model-invariant. For the evaluated LLM, Llama3-1B prefill is 64.7% faster on NPU, while decode is 81.5% faster on GPU; later profiling states that prefill on NPU gives an average 3.0× speedup over GPU, whereas decode on GPU gives an average 7.5× speedup over NPU (Karami et al., 19 Jul 2025). The RAG encoder also changes preference with sequence length: at sequence length 16, NPU is 31.9% lower latency than GPU; at 32, NPU is 10.9% lower latency; at 1024, GPU is 22.3% lower latency than NPU (Karami et al., 19 Jul 2025). The paper interprets this as evidence that static ahead-of-time mapping is intrinsically brittle on Ryzen AI.

The same heterogeneity appears in real-time vision models. For RCAN super-resolution, NPU average latency is 0.59 ms, reducing latency by 81.9% versus GPU and 84.5% versus CPU; for YOLOv3 object detection, NPU average latency is 5.23 ms, with reductions of 42.9% versus GPU and 62.7% versus CPU; for PointPainting segmentation, GPU is 10.2% faster than NPU (Karami et al., 19 Jul 2025). Hence a mixed workload produces an immediate packing problem: SR, OD → NPU-favored, Seg → GPU-favored, while LLM prefill and decode prefer opposite engines (Karami et al., 19 Jul 2025).

The paper evaluates five policies—FCFS-AOT, FCFS-DYN, EDF-AOT, EDF-DYN, and FTF—and reports that scheduling decisions lead to a 41.7% average difference in deadline violation rates (Karami et al., 19 Jul 2025). In a video-conference scenario combining RAG, SR, segmentation, and object detection, FCFS-AOT yields 66.5% deadline violations with TTFT 1577.9 ms and TPT 84.0 ms, whereas FTF reduces deadline violations to 20.8% but worsens TPT to 339.9 ms (Karami et al., 19 Jul 2025). In gaming and conferencing scenarios, EDF eliminates deadline misses but starves the LLM completely, while FCFS preserves LLM responsiveness but can devastate real-time QoS (Karami et al., 19 Jul 2025).

The implication is not merely that dynamic scheduling helps, but that Ryzen AI requires workload-aware dynamic heterogeneous scheduling. The paper is explicit that backend dynamism without workload awareness may merely shift QoS failure from one model to another (Karami et al., 19 Jul 2025). This suggests that “using the NPU” is not, in itself, a sufficient deployment strategy for Ryzen AI-class client workloads.

6. Limitations, misconceptions, and research directions

Several limitations recur across the literature. First, Ryzen AI is not an all-purpose accelerator whose benefits are independent of datatype, kernel size, or invocation pattern. The Fortran study states directly that int32 and float32 are not natively supported in hardware by AIE-ML vector units, and its results show that startup overhead can dominate one-off intrinsic calls (Brown et al., 14 Feb 2025). Second, close-to-metal access does not imply frictionless usability. The GPT-2 fine-tuning work requires explicit control over tile layouts, DMA choreography, command streams, and buffer copies, and it identifies CPU-side copies, transposes, and driver synchronization as major remaining overheads (Rösti et al., 3 Apr 2025).

Third, high compute efficiency alone is not sufficient. The cross-generation GEMM study argues that the best end-to-end Ryzen AI GEMM occurs near the balanced point where compute and memory times meet, not at the maximum-MAC single-core kernel (Taka et al., 15 Dec 2025). This point is reinforced by compiler work on MLIR-AIR, which shows that explicit scheduling, resource grouping, and overlap of communication with computation are essential on AMD’s tiled NPUs (Wang et al., 16 Oct 2025). AIR-generated matrix multiplication reaches up to 78.7% compute efficiency and performance “almost identical” to hand-optimized MLIR-AIE code, while a fused multi-head attention block for LLaMA 2 is expressed in 155 lines of high-level MLIR and reduces latency from 834 µs unfused to 373 µs fused, a speedup of $4 \times 8$6 (Wang et al., 16 Oct 2025). This suggests that compiler structure preservation, rather than only kernel micro-optimization, is becoming central to Ryzen AI programmability.

Fourth, operator coverage remains incomplete in adjacent AMD AI-engine compiler frameworks. AIE4ML, although aimed at Versal AIE-ML rather than Ryzen AI directly, focuses mainly on quantized linear layers with fused bias and ReLU, and explicitly leaves extension to additional operators and model classes for future work (Danopoulos et al., 17 Dec 2025). A plausible implication is that end-to-end Ryzen AI compilers will need broader operator support, stronger graph-level placement, and better memory-residency planning if they are to generalize beyond GEMM- and MLP-dominated workloads.

The literature also corrects two opposing misconceptions. One is that Ryzen AI is merely an “AI PC” branding layer with limited technical substance. Multiple papers instead treat it as a serious, programmable accelerator substrate for GEMM, scientific kernels, and even client-side fine-tuning (Rösti et al., 3 Apr 2025). The opposite misconception is that Ryzen AI can be treated like a generic opaque accelerator. The research record consistently rejects that view: performance depends on tile-local memory fit, DMA tensor transformations, layout of $4 \times 8$7, buffer-descriptor scheduling, reuse of tuned designs across many problem sizes, and, at system scale, dynamic scheduling across CPU, GPU, and NPU (Taka et al., 15 Dec 2025).

In synthesis, Ryzen AI is best understood as a family of AMD client-side spatial accelerators whose practical performance emerges from the interaction of explicit data movement, local-memory-constrained kernel design, compiler-visible synchronization, and heterogeneous runtime scheduling. The most stable insight across generations and software stacks is not a single API or benchmark figure, but a systems principle: Ryzen AI performs best when computation, communication, placement, and orchestration are co-optimized rather than treated as separable layers.