AMD XDNA Architecture: Tile-based AI Accelerator

• AMD XDNA Architecture is a tile-based neural processing unit framework that replaces conventional caches with explicitly managed scratchpad SRAM to accelerate deep learning on client devices.
• It organizes processing into a two-dimensional array of compute and memory tiles, achieving up to 4 TFLOP/s peak performance and over 90% utilization for GEMM and attention workloads.
• The design leverages programmable vector engines and compiler-driven tools like Zen-Attention to optimize dataflow, tiling, and memory management for both inference and training.

AMD XDNA Architecture defines a spatially structured, tile-based neural processing unit (NPU) microarchitecture implemented in recent AMD Ryzen AI product lines, designed for high-throughput and energy-efficient acceleration of modern deep learning workloads on client devices. XDNA replaces conventional hardware-managed caches with explicitly managed scratchpad SRAM, leverages a programmable vectorized datapath within each tile, and orchestrates dataflow using finely controlled multi-channel DMA engines—enabling both inference and training workloads to achieve high utilization relative to the stringent bandwidth and power budgets found at the client edge (Deshmukh et al., 25 Aug 2025, Rösti et al., 3 Apr 2025, Taka et al., 15 Dec 2025).

1. Architectural Organization and Tile Array Structure

XDNA deploys a two-dimensional array of "compute tiles" (CompTiles) overlaid atop memory tiles (MemTiles) and interface shims (ShimTiles), forming a scalable matrix of tightly coupled compute and memory resources. In its first- and second-generation instantiations:

• XDNA features 20 CompTiles (4 rows × 5 columns), while XDNA2 expands this to 32 CompTiles (4 × 8 array) (Taka et al., 15 Dec 2025).
• Each CompTile incorporates a VLIW processor, SIMD datapath, 64 KB L1 scratchpad SRAM, vector register file, and two (or more) DMA engines for both memory-to-stream and stream-to-memory operations (Deshmukh et al., 25 Aug 2025, Rösti et al., 3 Apr 2025).
• Tiles in the same column access a shared 512 KB L2 (MemTile) scratchpad for intermediate buffer pools, supporting larger working sets and facilitating K-dimension reductions in GEMM/attention operations (Deshmukh et al., 25 Aug 2025, Taka et al., 15 Dec 2025).
• Interface ShimTiles handle all traffic to host DRAM (L3); a 60 GB/s bidirectional bus connects the NPU to system memory, shared with CPU and GPU (Deshmukh et al., 25 Aug 2025).

Tile-to-tile communication exploits two primary on-chip networks: a general-purpose interconnect mesh for peer exchange and dedicated "cascade" streams per column, the latter supporting spatial reductions (e.g., Softmax sum across tiles) and facilitating efficient implementation of SPMD-style reduction patterns (Deshmukh et al., 25 Aug 2025).

2. Programmable Vector Engines and Pipelined Dataflow

Each CompTile includes:

• VLIW logic capable of issuing multiple operations per cycle, supporting vectorized arithmetic (VMAC for dot-products, VMUL for elementwise multiply, VSHUFFLE/VPERMUTE for fine-grained reordering) (Rösti et al., 3 Apr 2025, Taka et al., 15 Dec 2025).
• Datatype support: int8, int16, and bf16 (brain-float); XDNA2 adds bfp16 pipelines (Taka et al., 15 Dec 2025).
• MAC bandwidth: Each core achieves up to 256 FLOP/cycle (bfloat16, 1 GHz), implying 4 TFLOP/s peak on a full 16-core array. Int8 throughput reaches 10 TOPS (XDNA) and 50 TOPS (XDNA2) under optimal conditions (Rösti et al., 3 Apr 2025, Taka et al., 15 Dec 2025).
• Deep pipelining and four-accumulator design enable efficient unrolled inner loops and scheduling of successive VMACs without pipeline stalls (Rösti et al., 3 Apr 2025).

Instruction streams are compiler-scheduled (no out-of-order hardware), so hazard avoidance and loop unrolling are determined by static analysis and explicit compiler passes. Importantly, there is no hardware caching or scoreboarding—locality and hazard avoidance are strict responsibilities of software and toolchain (Rösti et al., 3 Apr 2025).

3. Explicitly Managed Memory Hierarchy and DMA Scheduling

XDNA omits all hardware-managed caches, demanding all locality be exposed and managed by explicit tiling, double-buffering, and DMA orchestration (Deshmukh et al., 25 Aug 2025, Rösti et al., 3 Apr 2025, Taka et al., 15 Dec 2025):

Level	Description	Size per Tile
L1	Core scratchpad SRAM	64 KB
L2	Shared memory tile SRAM	512 KB
L3	Off-chip host DRAM (DDR5)	Platform-dependent

DMA engines (per CompTile, MemTile, ShimTile) are fully programmer exposed via bare-metal IRON or compiler frameworks such as Zen-Attention. Host-to-NPU data transfers and synchronization are mediated by the XRT stack and support future "zero-copy" buffer sharing (Rösti et al., 3 Apr 2025).

Memory accesses are multi-level: L3 (host DRAM) → L2 (MemTile) → L1 (CompTile), with each DMA stage programmable as to strides, tile sizes, and synchronization. Double-buffering is used to overlap compute and transfer phases. No hardware prefetch; all staging of tile data into L1/L2 is explicitly orchestrated by the toolchain (Deshmukh et al., 25 Aug 2025).

4. Dataflow Mapping, Tiling, and GEMM Kernel Optimization

GEMM (general matrix multiply) and attention mechanisms are realized by output-stationary mappings using up to four explicit tile levels (Taka et al., 15 Dec 2025):

• Level 1: Micro-tile (r×s×t) mandated by single-core API, fits within register file.
• Level 2: Per-core tile (m_ct×k_ct×n_ct), fits into 64 KB L1.
• Level 3: Array-wide "native" tile, spatially mapped across cores for M and N, temporally across K.
• Level 4: Full problem, composed by iterating native tile.

On-chip multi-dimensional DMA streams provide real-time multiway data reordering (transposes, layout transforms, padding) at each level. The IRON toolchain (bare-metal) and Zen-Attention (graph-level layer folding and tiling engine) automate these mappings for client ML workloads (Deshmukh et al., 25 Aug 2025, Rösti et al., 3 Apr 2025).

Mapping is optimized to maximize L1/L2 utilization and minimize host DRAM bandwidth. Theoretical and empirical performance is bound by:

$$\mathrm{TOPS}(N) = \min\left(\mathrm{peak\_TOPS},\; \mathrm{DRAM\_BW}\times \frac{2N^3}{N^2(\operatorname{ty}(A)+\operatorname{ty}(B))+N^2\operatorname{ty}(C)}\right)$$

where $\operatorname{ty}(\cdot)$ denotes bytes per element (1 for int8, 2 for bf16) (Taka et al., 15 Dec 2025).

Typical achieved core efficiency exceeds 94% of per-core peak for large (roofline-saturating) GEMMs and >90% of maximum array performance (Taka et al., 15 Dec 2025).

5. Compiler-Level Techniques: Zen-Attention and Folding

Zen-Attention is a compiler framework that automates the optimal folding, tiling, and DMA scheduling of transformer attention layers onto XDNA NPUs (Deshmukh et al., 25 Aug 2025). It functions by:

• Pattern-matching graph fragments representing $A=QK^T$, addition of bias $B$ and mask $M$, Softmax normalization, and $Z=SM_{out}V$.
• Computing a fusion "folding level" $f$ (3 = fully folded, all four ops; 2 = partial, up through Softmax; 1 = unfolded baseline).
• Enumerating all valid tiling subvolumes $\{S_q, S_k, S_v, S_b, S_m\}$ subject to L1 footprint constraints, solving for maximum $S_q$ reuse and assignment across tiles.
• Handling transpose by fusing block-transpose into DMA (in $8 \times 8$ subtiles at L2), followed by in-kernel register shuffling before VMAC call—minimizing DRAM trips and L1 buffering.
• Padding via DMA hardware or producing pre-padded outputs upstream.

Folding all attention stages eliminates redundant host DRAM round trips (e.g., Q, K, V, SMout), effectively halving bandwidth demands for large sequence dimensions.

Empirically, Zen-Attention demonstrates up to $\operatorname{ty}(\cdot)$0 lower attention block latency (e.g., ViT-base-patch-16 block from $\operatorname{ty}(\cdot)$1 to $\operatorname{ty}(\cdot)$2), with end-to-end inference speedups up to 32% on attention-dominant models (ViT, CLIP) and measurable DRAM traffic reduction even on compute-bound models (e.g., modest ~1.4% gain on BERT) (Deshmukh et al., 25 Aug 2025).

6. Programming, Toolflow, and Training Workloads

Bare-metal programming of XDNA is supported via the IRON toolchain, which orchestrates core DMAs, switch boxes, and kernel schedules at a fine level through Python-driven configuration scripts and compiler flows (AIE-MLIR dialect to xclbin + insts.txt loadable by XRT) (Rösti et al., 3 Apr 2025).

Typical workload structure:

• Forward: Offload all high-intensity matrix multiplies to CompTiles, orchestrating three-level DMA staging. Double-buffering ensures compute/transfer overlap.
• Backward: Same tiling strategies with matching data movement.
• Example: Edge fine-tuning of GPT-2-124M, offloading GEMM to NPU, achieves $\operatorname{ty}(\cdot)$3 speedup on forward, $\operatorname{ty}(\cdot)$4 on backward over CPU-only. Battery power throughput gain $\operatorname{ty}(\cdot)$5, FLOPS/W efficiency $\operatorname{ty}(\cdot)$6 better than CPU (Rösti et al., 3 Apr 2025).

Limitations include non-trivial overheads from host–XRT buffer copies and data layout transforms. Numerical fidelity is preserved in bfloat16/float32 modes; mean relative error remains $\operatorname{ty}(\cdot)$7 versus CPU references (Rösti et al., 3 Apr 2025).

7. Performance Characteristics and System-Level Bottlenecks

XDNA's performance is governed by a combination of array compute efficiency and achievable DRAM bandwidth, as summarized below:

Generation	TOPS (int8, peak)	TOPS (bf16, peak)	Typ. GEMM Attained (int8->int8, bf16->bf16)	DRAM BW (est.)
XDNA	10	4	6.76, 3.14	~15 GB/s
XDNA2	50	~15	38.05, 14.71	~50 GB/s

Performance bottlenecks occur when working-set dimensions are too small for array-wide tiling or too large for on-chip buffering, transitioning GEMM from compute-bound to DRAM-bound. DMA task overlap and careful firmware BD management are necessary for peak sustainable throughput (Taka et al., 15 Dec 2025).

XDNA2 introduces architectural mitigations: neighbor-sharing spillover in MemTiles, enhanced array size, expanded DMA parallelism, and more aggressive double-buffering.

8. Context, Significance, and Application Domains

AMD XDNA's tile-based programmable approach offers a distinct alternative to classical CNN accelerators or black-box NPUs relying on deep hardware caching. Its explicit scratchpad hierarchy and fine-grained DMA scheduling are particularly advantageous for workloads characterized by large, performance-critical matrix multiplies under tight power and bandwidth regimes—transformer attention, on-client LLM inference and fine-tuning, and vision transformer models (Deshmukh et al., 25 Aug 2025, Rösti et al., 3 Apr 2025).

Applications demonstrated include both inference and client-side training (GPT-2 fine-tuning), as well as bandwidth-optimized attention folding via compiler frameworks such as Zen-Attention. The architecture's ability to approach theoretical peak utilization at the per-core and array level (>90–94%) in realistic workloads suggests continued adoption for client device AI inferencing (Taka et al., 15 Dec 2025).

A plausible implication is that future client hardware will increasingly adopt similar explicit scratchpad and software-managed dataflow architectures to balance perf/watt, latency, and flexibility requirements of evolving AI workloads.