---
title: 'AMD XDNA Architecture: Tile-based AI Accelerator'
url: https://www.emergentmind.com/topics/amd-xdna-architecture
type: topic
---

# AMD XDNA Architecture: Tile-based AI Accelerator

AMD XDNA Architecture defines a spatially structured, tile-based neural processing unit (NPU) microarchitecture implemented in recent AMD Ryzen AI product lines, designed for high-throughput and energy-efficient acceleration of modern deep learning workloads on client devices. XDNA replaces conventional hardware-managed caches with explicitly managed scratchpad SRAM, leverages a programmable vectorized datapath within each tile, and orchestrates dataflow using finely controlled multi-channel DMA engines—enabling both inference and training workloads to achieve high utilization relative to the stringent bandwidth and power budgets found at the client edge [2508.17593, 2504.03083, 2512.13282].

## 1. Architectural Organization and Tile Array Structure

XDNA deploys a two-dimensional array of "compute tiles" (CompTiles) overlaid atop memory tiles (MemTiles) and interface shims (ShimTiles), forming a scalable matrix of tightly coupled compute and memory resources. In its first- and second-generation instantiations:

- XDNA features 20 CompTiles (4 rows × 5 columns), while XDNA2 expands this to 32 CompTiles (4 × 8 array) [2512.13282].
- Each CompTile incorporates a VLIW processor, SIMD datapath, 64 KB L1 scratchpad SRAM, vector register file, and two (or more) DMA engines for both memory-to-stream and stream-to-memory operations [2508.17593, 2504.03083].
- Tiles in the same column access a shared 512 KB L2 (MemTile) scratchpad for intermediate buffer pools, supporting larger working sets and facilitating K-dimension reductions in GEMM/attention operations [2508.17593, 2512.13282].
- Interface ShimTiles handle all traffic to host DRAM (L3); a 60 GB/s bidirectional bus connects the NPU to system memory, shared with CPU and GPU [2508.17593].

Tile-to-tile communication exploits two primary on-chip networks: a general-purpose interconnect mesh for peer exchange and dedicated "cascade" streams per column, the latter supporting spatial reductions (e.g., Softmax sum across tiles) and facilitating efficient implementation of SPMD-style reduction patterns [2508.17593].

## 2. Programmable Vector Engines and Pipelined Dataflow

Each CompTile includes:

- VLIW logic capable of issuing multiple operations per cycle, supporting vectorized arithmetic (VMAC for dot-products, VMUL for elementwise multiply, VSHUFFLE/VPERMUTE for fine-grained reordering) [2504.03083, 2512.13282].
- Datatype support: int8, int16, and bf16 (brain-float); XDNA2 adds bfp16 pipelines [2512.13282].
- MAC bandwidth: Each core achieves up to 256 FLOP/cycle (bfloat16, 1 GHz), implying 4 TFLOP/s peak on a full 16-core array. Int8 throughput reaches 10 TOPS (XDNA) and 50 TOPS (XDNA2) under optimal conditions [2504.03083, 2512.13282].
- Deep pipelining and four-accumulator design enable efficient unrolled inner loops and scheduling of successive VMACs without pipeline stalls [2504.03083].

Instruction streams are compiler-scheduled (no out-of-order hardware), so hazard avoidance and loop unrolling are determined by static analysis and explicit compiler passes. Importantly, there is no hardware caching or scoreboarding—locality and hazard avoidance are strict responsibilities of software and toolchain [2504.03083].

## 3. Explicitly Managed Memory Hierarchy and DMA Scheduling

XDNA omits all hardware-managed caches, demanding all locality be exposed and managed by explicit tiling, double-buffering, and DMA orchestration [2508.17593, 2504.03083, 2512.13282]:

| Level | Description                | Size per Tile    |
|-------|----------------------------|------------------|
| L1    | Core scratchpad SRAM       | 64 KB            |
| L2    | Shared memory tile SRAM    | 512 KB           |
| L3    | Off-chip host DRAM (DDR5)  | Platform-dependent|

DMA engines (per CompTile, MemTile, ShimTile) are fully programmer exposed via bare-metal IRON or compiler frameworks such as Zen-Attention. Host-to-NPU data transfers and synchronization are mediated by the XRT stack and support future "zero-copy" buffer sharing [2504.03083].  

Memory accesses are multi-level: L3 (host DRAM) → L2 (MemTile) → L1 (CompTile), with each DMA stage programmable as to strides, tile sizes, and synchronization. Double-buffering is used to overlap compute and transfer phases. No hardware prefetch; all staging of tile data into L1/L2 is explicitly orchestrated by the toolchain [2508.17593].

## 4. Dataflow Mapping, Tiling, and GEMM Kernel Optimization

GEMM (general matrix multiply) and attention mechanisms are realized by output-stationary mappings using up to four explicit tile levels [2512.13282]:

- Level 1: Micro-tile (r×s×t) mandated by single-core API, fits within register file.
- Level 2: Per-core tile (m_ct×k_ct×n_ct), fits into 64 KB L1.
- Level 3: Array-wide "native" tile, spatially mapped across cores for M and N, temporally across K.
- Level 4: Full problem, composed by iterating native tile.

On-chip multi-dimensional DMA streams provide real-time multiway data reordering (transposes, layout transforms, padding) at each level. The IRON toolchain (bare-metal) and Zen-Attention (graph-level layer folding and tiling engine) automate these mappings for client ML workloads [2508.17593, 2504.03083].

Mapping is optimized to maximize L1/L2 utilization and minimize host DRAM bandwidth. Theoretical and empirical performance is bound by:

$$
\mathrm{TOPS}(N) = \min\left(\mathrm{peak\_TOPS},\; \mathrm{DRAM\_BW}\times \frac{2N^3}{N^2(\operatorname{ty}(A)+\operatorname{ty}(B))+N^2\operatorname{ty}(C)}\right)
$$

where $\operatorname{ty}(\cdot)$ denotes bytes per element (1 for int8, 2 for bf16) [2512.13282].

Typical achieved core efficiency exceeds 94% of per-core peak for large (roofline-saturating) GEMMs and >90% of maximum array performance [2512.13282].

## 5. Compiler-Level Techniques: Zen-Attention and Folding

Zen-Attention is a compiler framework that automates the optimal folding, tiling, and DMA scheduling of transformer attention layers onto XDNA NPUs [2508.17593]. It functions by:

- Pattern-matching graph fragments representing $A=QK^T$, addition of bias $B$ and mask $M$, Softmax normalization, and $Z=SM_{out}V$.
- Computing a fusion "folding level" $f$ (3 = fully folded, all four ops; 2 = partial, up through Softmax; 1 = unfolded baseline).
- Enumerating all valid tiling subvolumes $\{S_q, S_k, S_v, S_b, S_m\}$ subject to L1 footprint constraints, solving for maximum $S_q$ reuse and assignment across tiles.
- Handling transpose by fusing block-transpose into DMA (in $8 \times 8$ subtiles at L2), followed by in-kernel register shuffling before VMAC call—minimizing DRAM trips and L1 buffering.
- Padding via DMA hardware or producing pre-padded outputs upstream.

Folding all attention stages eliminates redundant host DRAM round trips (e.g., Q, K, V, SMout), effectively halving bandwidth demands for large sequence dimensions.

Empirically, Zen-Attention demonstrates up to $4\times$ lower attention block latency (e.g., ViT-base-patch-16 block from $\approx8\,\mathrm{ms}$ to $\approx2\,\mathrm{ms}$), with end-to-end inference speedups up to 32% on attention-dominant models (ViT, CLIP) and measurable DRAM traffic reduction even on compute-bound models (e.g., modest ~1.4% gain on BERT) [2508.17593].

## 6. Programming, Toolflow, and Training Workloads

Bare-metal programming of XDNA is supported via the IRON toolchain, which orchestrates core DMAs, switch boxes, and kernel schedules at a fine level through Python-driven configuration scripts and compiler flows (AIE-MLIR dialect to xclbin + insts.txt loadable by XRT) [2504.03083].

Typical workload structure:

- Forward: Offload all high-intensity matrix multiplies to CompTiles, orchestrating three-level DMA staging. Double-buffering ensures compute/transfer overlap.
- Backward: Same tiling strategies with matching data movement.
- Example: Edge fine-tuning of GPT-2-124M, offloading GEMM to NPU, achieves $3.1\times$ speedup on forward, $2.8\times$ on backward over CPU-only. Battery power throughput gain $1.2\times$, FLOPS/W efficiency $1.4\times$ better than CPU [2504.03083].

Limitations include non-trivial overheads from host–XRT buffer copies and data layout transforms. Numerical fidelity is preserved in bfloat16/float32 modes; mean relative error remains $<0.06\%$ versus CPU references [2504.03083].

## 7. Performance Characteristics and System-Level Bottlenecks

XDNA's performance is governed by a combination of array compute efficiency and achievable DRAM bandwidth, as summarized below:

| Generation | TOPS (int8, peak) | TOPS (bf16, peak) | Typ. GEMM Attained (int8->int8, bf16->bf16) | DRAM BW (est.)   |
|------------|-------------------|-------------------|----------------------------------------------|------------------|
| XDNA       | 10                | 4                 | 6.76, 3.14                                   | ~15 GB/s         |
| XDNA2      | 50                | ~15               | 38.05, 14.71                                 | ~50 GB/s         |

Performance bottlenecks occur when working-set dimensions are too small for array-wide tiling or too large for on-chip buffering, transitioning GEMM from compute-bound to DRAM-bound. DMA task overlap and careful firmware BD management are necessary for peak sustainable throughput [2512.13282].

XDNA2 introduces architectural mitigations: neighbor-sharing spillover in MemTiles, enhanced array size, expanded DMA parallelism, and more aggressive double-buffering.

## 8. Context, Significance, and Application Domains

AMD XDNA's tile-based programmable approach offers a distinct alternative to classical CNN accelerators or black-box NPUs relying on deep hardware caching. Its explicit scratchpad hierarchy and fine-grained DMA scheduling are particularly advantageous for workloads characterized by large, performance-critical matrix multiplies under tight power and bandwidth regimes—transformer attention, on-client LLM inference and fine-tuning, and vision transformer models [2508.17593, 2504.03083].

Applications demonstrated include both inference and client-side training (GPT-2 fine-tuning), as well as bandwidth-optimized attention folding via compiler frameworks such as Zen-Attention. The architecture's ability to approach theoretical peak utilization at the per-core and array level (>90–94%) in realistic workloads suggests continued adoption for client device AI inferencing [2512.13282].

A plausible implication is that future client hardware will increasingly adopt similar explicit scratchpad and software-managed dataflow architectures to balance perf/watt, latency, and flexibility requirements of evolving AI workloads.

Source: https://www.emergentmind.com/topics/amd-xdna-architecture