AMD XDNA Neural Processing Units

• AMD XDNA NPUs are specialized accelerators featuring a spatial array architecture that enables high-throughput deep learning inference and training.
• They employ explicit hardware preemption, programmable data movement, and multi-tenant support to ensure low-latency and robust execution.
• Advanced scheduling, tiling strategies, and reliability-aware quantization drive significant improvements in latency, throughput, and energy efficiency.

AMD XDNA Neural Processing Units (NPUs) are a class of specialized accelerators designed to execute deep learning inference and, increasingly, training workloads with high throughput and energy efficiency. Their distinguishing architectural characteristics include a spatially arranged grid of compute cores, explicit hardware mechanisms for preemption and multi-tenancy, programmable data movement, and extensive support for hardware–software co-design. The XDNA architecture is engineered to address challenges found in both cloud datacenter consolidation and edge deployment, balancing requirements for latency, throughput, resource isolation, virtualization, memory bandwidth, and workload flexibility.

1. Architectural Fundamentals and Hardware Features

AMD XDNA NPUs leverage a spatial computing paradigm comprising AI Engines (compute cores), memory cores, and interface (“shim”) cores, interconnected via a configurable switchbox fabric. Each AI Engine is typically a VLIW processor capable of executing high-throughput, vectorized operations (including fused multiply–add) with explicit control over local memory (e.g., 64 KB per compute tile) and scratchpad banks (512 KB for memory tiles) (Rösti et al., 3 Apr 2025).

A characteristic feature is the direct interface from shim cores to shared system memory (L3), the programmable command processor for runtime configuration, and support for bare-metal programming via toolchains such as IRON (Rösti et al., 3 Apr 2025, Hunhoff et al., 25 Apr 2025). This allows fine-grained control over compute core placement, memory tiling, and DMA scheduling. The integrated memory management unit, when present (see NeuMMU (Hyun et al., 2019)), decouples the accelerator’s virtual and physical memory spaces, supporting system-wide memory sharing and oversubscription essential for multi-model and multi-tenant execution.

Key aspects of the architecture include:

• Spatial array topology for concurrent execution.
• Explicit software management of L1/L2 caches (scratchpad).
• High-throughput GEMM and convolution kernels mapped to vector engines.
• Support for operator fusion, flexible tiling, and dynamic buffer allocation for advanced layer types (e.g., folded attention (Deshmukh et al., 25 Aug 2025)).
• Hardware context tables for preemption, runtime reconfiguration, and low-latency switching between inference tasks (Choi et al., 2019).

2. Preemption, Multi-Tenancy, and Virtualization

Preemption and multi-tenancy are critical for sharing NPU resources between competing inference tasks while maintaining low tail latency and high service-level objective (SLO) satisfaction. Hardware-supported preemption mechanisms—such as CHECKPOINT, KILL, and DRAIN—allow for explicit interruption of running tasks (Choi et al., 2019). The preferred CHECKPOINT strategy saves the context at predictable tile boundaries, with minimal overhead, while DRAIN allows tasks near completion to finish, and KILL provides for immediate task termination at the expense of wasted computation.

PREMA (Predictive Multi-task Scheduling Algorithm) combines token-based priority scheduling with runtime inference time estimation (Choi et al., 2019). This enables the scheduler to select candidates based on fairness, normalized turnaround time (NTT), and estimated completion time, using formulas such as:

$$\text{ANTT} = \frac{1}{n}\sum_{i=1}^{n}\frac{C_i^\text{multi}}{C_i^\text{single}}, \quad \text{STP} = \sum_{i=1}^{n}\frac{C_i^\text{single}}{C_i^\text{multi}}$$

The Neu10 framework generalizes multi-tenant support with a virtual NPU (vNPU) abstraction. Here, compute units are partitioned across tenants, with allocations tuned by precise operator profiling and dynamic harvesting of idle engines (Xue et al., 7 Aug 2024). Scheduling at the micro-Tensor-Operator (uTop) level achieves fine-grained load balancing and tail latency reductions of up to 4.6× over standard sharing.

3. Memory Subsystem and Address Translation

XDNA NPUs depend on high-throughput scratchpad memory and must frequently move large tensor tiles from off-chip DRAM. Conventional GPU-centric MMUs, relying on spatial/temporal locality for TLB hits, are ill-suited for the rapid bursts of translations needed for tile-based NPUs (Hyun et al., 2019). NeuMMU introduces:

• Pending Request Merging Buffers (PRMB): Deduplicates translation requests in flight.
• Throughput-centric scaling of Page Table Walkers (PTWs): Achieves up to 128 parallel walks for high-bandwidth translation.
• Translation Path Registers (TPreg): Caches page walk paths to skip repeated accesses.

This enables near-ideal performance, incurring on average only 0.06% overhead, and supports memory oversubscription and direct NUMA-style memory access between accelerators and host CPU (Hyun et al., 2019).

A typical formula for intrinsic translation demand is:

$$N_\text{translations} = \frac{T_\text{tile}}{P_\text{size}}$$

where, for a 5 MB tile and 4 KB page size, $N_\text{translations} \approx 1250$.

4. Scheduling, Tiling, and Compiler Strategies

Efficient use of compute and memory resources relies on advanced scheduling and tiling techniques. Fluid Batching provides per-layer batch reshaping to match matrix dimensions to hardware tile sizes, maximizing engine utilization even under highly variable batch sizes and stochastic networks with early-exit paths (Kouris et al., 2022):

$$\hat{R}^{(l)} = B_R^{(l,B_\text{act})} \cdot R^{(l)}, \quad \hat{P}^{(l)} = (B_\text{act} - B_R^{(l,B_\text{act})} + 1) \cdot (P^{(l)} + P^{(l)} \mod T_P)$$

Stackable Processing Elements (PEs) facilitate runtime configuration of MAC units to avoid underutilization for layers or batches not aligned with the hardware tile shape.

TSO (Tensor Slicing Optimization) uses DRAM burst analysis to select tile shapes for convolutions that minimize transaction count and optimize parallelism (Sousa et al., 2023):

$$T_{\text{tile}} = \frac{\text{tile\_size}}{BW} + n_{\text{bursts}} \times CAS$$

$T_{CONV} = T_{MAC} + T_{DRAM} + T_{SW}$

Slicing and scheduling at both task and micro-operation levels allow multiple model instances and multi-tenant inference at scale (Ham et al., 12 Jun 2024).

Zen-Attention extends compiler-level optimization to dynamic folding of attention blocks, minimizing DRAM accesses through fusing MatMul, bias, mask, SoftMax, and output MatMul into a single kernel as permitted by scratchpad capacity (Deshmukh et al., 25 Aug 2025).

5. Reliability, Quantization, and Longevity

NPUs are vulnerable to transistor aging, affecting delay and reliability. Reliability-aware quantization mitigates this by compressing activations and weights over time, reducing bit-width dynamically in response to increased delay (Salamin et al., 2021). Instead of a fixed guardband, parameters α and β control bit-width reduction, with “LSB” and “MSB” zero-padding implemented as:

• Compressed MAC: $$F_\text{shifted} = (Bias + \sum_j(A_j \times W_j)) \times 2^{\alpha+\beta}$$, with post-compute shifting by $(\alpha+\beta)$.
• Bit-width intervals: Activations in $[0,2^{8-\alpha})$, Weights in $[0,2^{8-\beta})$.

This technique removes the 23% performance delay penalty imposed by fixed guardbands, achieving only 3% average accuracy loss over 10 years and 46% energy reduction.

6. Kernel Development, Programming Interfaces, and Evaluation

AMD XDNA NPUs support both bare-metal and higher-level kernel programming via toolkits such as IRON (Hunhoff et al., 25 Apr 2025), which balance low-level control and developer productivity through abstractions for placement (Placer), dataflow (ObjectFifo, DMA), and transformation (taplib for tiling and stride calculation). The toolkit reduces code size by 26% and decreases error-prone duplication, while preserving full control over DMA and synchronization.

Automated kernel generation and evaluation, as demonstrated in NPUEval (Kalade et al., 18 Jul 2025), test large-language-model-driven code against benchmarks of 102 ML operators using compiler feedback and vectorization metrics. Vectorization efficiency—a key metric for AMD XDNA NPUs—is typically modest (10% average), with select models achieving 50%+ when guided by code retrieval and compiler feedback.

7. Performance, Efficiency, and Future Directions

Measured improvements due to hardware preemption and predictive scheduling: up to 7.8× lower latency, 1.4× improved throughput, and 4.8× higher SLA satisfaction (Choi et al., 2019). Compiler-guided stacking and batching (Kouris et al., 2022), as well as dynamic folding (Deshmukh et al., 25 Aug 2025), boost hardware utilization and reduce latency. Simulation tools such as ONNXim (Ham et al., 12 Jun 2024) facilitate rapid, accurate modeling of multi-tenant and multi-core scheduling policies, DRAM/NoC contention, and their impact on inference latency under real workloads.

Context-driven operator selection for long-context inference shows quadratic attention to be severely memory-bound (pipeline stalls >95%, cache efficiency <8%) at scale, while linear and structured alternatives (Toeplitz, SSM) achieve latency and throughput gains, albeit with distinct bottlenecks on vector cores (Gupta et al., 29 Sep 2025).

eIQ Neutron NPU architecture (Bamberg et al., 17 Sep 2025) emphasizes that peak TOPS is not predictive of real-world efficiency; maximizing compute utilization through constraint-programming-based compilation, on-chip dataflow, persistent tiling, and adaptive memory banking achieves up to 3.3× better performance-per-cost versus competitors.

Tables

Mechanism / Framework	Primary Function	Impact / Metric
PREMA (Choi et al., 2019)	Predictive preemptive scheduling	7.8× latency, 1.4× throughput, 4.8× SLA
NeuMMU (Hyun et al., 2019)	Throughput-centric MMU	0.06% perf. overhead, 16× energy reduction
Fluid Batching (Kouris et al., 2022)	Dynamic batch reshaping	1.97× latency, 6.7× SLO improvement
Zen-Attention (Deshmukh et al., 25 Aug 2025)	Attention folding, tiling	4× attention block, 32% end-to-end latency
Reliability-Aware Quant. (Salamin et al., 2021)	Aging compensation	23% perf. gain, 3% accuracy loss, 46% energy
NPUEval (Kalade et al., 18 Jul 2025)	LLM kernel benchmarking	Average 10% vectorization, 50%+ select kernels

Conclusion

AMD XDNA NPUs exemplify contemporary accelerator design for machine learning workloads, integrating explicit hardware mechanisms for multi-tenancy and preemption, advanced memory and scheduling subsystems, reliability-aware quantization, and extensible programming models. Progressive strategies in scheduling, tiling, and kernel generation—substantiated by empirical and analytical evidence across the literature—enable substantial improvements in latency, throughput, and SLA satisfaction while safeguarding longevity and efficiency. The ongoing co-design of hardware and software, as demonstrated by open toolkits and evaluation benchmarks, will further advance the practical capabilities of XDNA NPUs in datacenter consolidation, edge deployment, and client-side machine learning.