AMD Instinct MI300X GPU Architecture

Updated 13 November 2025

AMD Instinct MI300X GPUs are multi-chiplet accelerators featuring eight XCD dies with 192 GB HBM3 memory designed for leadership-class AI and HPC applications.
Key performance metrics include up to 2,209 TFLOPS theoretical FP8 performance and an aggregate memory bandwidth of 5.3 TB/s, achieved through NUMA-aware and topology-optimized designs.
Innovative features such as Infinity Fabric interconnects and mixed-precision computation enable efficient scaling and algorithmic adaptations for complex matrix and transformer workloads.

The AMD Instinct MI300X GPU is a multi-chiplet, high-bandwidth memory accelerator designed for leadership-class AI and HPC workloads, featuring eight XCD compute dies, 192 GB HBM3 memory, and architectural support for mixed-precision computation, advanced interconnects, and NUMA-aware software optimization. Its deployment spans large model inference, matrix-vector HPC workflows, and transformer-based AI computation, with tuning strategies and algorithmic adaptations required to leverage its hierarchical and non-uniform memory system.

1. Architecture and Microarchitectural Features

The MI300X employs a CDNA-3 design, integrating eight XCD (eXascale Compute Die) chiplets per package. Each XCD contains 38–96 Compute Units (CUs, depending on reporting details) with integrated Matrix-Core Engines and vector ALUs. CUs feature 64 stream processors, supporting vectorized FP32/FP64 operations and matrix engines for FP8/FP16/BF16 GEMM. Vector width per CU is expanded to eight lanes of 32 bits, increasing arithmetic density.

Each XCD connects to its own HBM3 stack (24 GB per stack, totaling 192 GB per MI300X), local L2 cache (4 MiB per die, 32 MiB device aggregate), and supports a per-stack bandwidth of 662.4 GB/s. Aggregate memory bandwidth reaches 5.3 TB/s. The device exposes eight NUMA domains, with local (intra-XCD) memory access bandwidth $B_{\text{HBM,local}} \approx 0.66$ TB/s and remote access bandwidth $B_{\text{HBM,remote}}\approx0.30$ TB/s, mediated by Infinity Fabric links at 128 GB/s bidirectional each.

Interconnect topology connects XCDs in a mesh (128 GB/s bidirectional per link) and links multiple MI300X units peer-to-peer, using AMD’s proprietary Infinity Fabric mesh without central switching. This differs from NVLink-based bisection topologies in NVIDIA flagship devices.

2. Compute and Memory Performance

In matrix and kernel microbenchmarks, the MI300X delivers high theoretical throughput, though observed efficiency is capped by current software maturity. For large square GEMMs ( $M,N,K\geq4096$ ):

FP8 peak: 2,209 TFLOPS theoretical, measured 995–1,790 TFLOPS (efficiency 45–81%)
FP16/BF16 peak: 1,105 TFLOPS theoretical, measured 500–940 TFLOPS (45–85%)
FP32: peak throughput observed at 9.6 TFLOP/s (single-GPU FFTMatvec)
FP64: observed at 4.8 TFLOPS (single-GPU FFTMatvec)

Software efficiency in microkernels using aligned problem sizes achieves up to 81–85% (FP8–FP16), but larger, system-level benchmarks are limited to $\sim$ 45% of theoretical due to ROCm kernel and compiler performance, dynamic frequency and power throttling, and lack of full topology-aware collective algorithms (Ambati et al., 31 Oct 2025).

Memory bandwidth saturates at $\sim$ 4.3 TB/s for arrays $\geq$ 64 MiB, with copy kernels measured at 81% of theoretical bandwidth, trailing NV H100, H200, and B200, which reach 86–90% at higher absolute GB/s.

In FFT-based block-triangular Toeplitz matvec (FFTMatvec) workloads:

FP64: peak sustained bandwidth 840 GB/s (84% of 1 TB/s roofline)
FP16: peak arithmetic throughput 19.2 TFLOPS, bandwidth 940 GB/s (94%)
Mixed precision modes: 10–14 TFLOPS, BW utilization 78–88%, efficiency 65–72% (higher than FP16 due to reduced error overhead) (Venkat et al., 13 Aug 2025)

3. Hierarchical Memory, NUMA Effects, and Scheduling

The MI300X’s eight-way NUMA architecture fundamentally alters memory access semantics for large workloads. Each XCD hosts private L2 (4 MiB), a local HBM3 controller, and remote accesses traverse Infinity Fabric with higher latency ( $L_{\text{IF}}\approx100$ cycles versus $L_{\text{L2}}\approx 50$ ), and reduced bandwidth.

Naïve kernel launches in multi-head attention and transformer inference stripe workgroups across NUMA domains, splintering spatial locality and destroying intra-domain cache reuse. This leads to L2 hit rates dropping below 1% at scale ( $H\ge64$ , $N\ge32$ K), and redundant cross-chiplet HBM fetches.

Swizzled Head-first Mapping (Editor's term): By mapping all compute blocks (Attention Compute Cluster, ACC) of a single attention head onto a single NUMA domain (XCD), cache reuse is maximized. The assignment is formalized as:

$x = h \bmod N_X, \quad \mathrm{wid}' = x (H B) + \left\lfloor h / N_X \right\rfloor B + b$

Pseudocode:

wid = program_id(0)
heads_per_xcd = H // N_X
B = ceil(N_CTX/M)
head = (wid // B) % H
block = wid % B
xcd  = head % N_X
head_group = head // N_X
new_wid = xcd*(heads_per_xcd*B) + head_group*B + block

This algorithm enables a 2–3× reduction in per-head data movement cost, with sustained L2 hit rates of 80–97% (Choudhary et al., 3 Nov 2025).

4. Multi-GPU Scaling, Interconnect Bottlenecks, and Collective Operations

On large installations (Frontier, OLCF), MI300X scales to 2,048 units with key observations:

Strong scaling ( $n=2^{22}$ ): 1–512 GPUs, efficiency $\simeq$ 90%; 512–2048 GPUs, slides to $\simeq$ 75% (wall time drops from 0.55 s to 0.15 s)
Communication (RCCL All-to-All) cost grows $O\left(\frac{p-1}{p} n_\text{subvec}\right)$ ; post-1000 ranks, communication accounts for 20–30% of time
Overlapping communication phases with local FFT and transpose recovers $\sim$ 5% efficiency at scale (Venkat et al., 13 Aug 2025)
All-reduce (collective) bandwidth is limited: 512 GB/s theoretical ($8$ GPUs), 448 GB/s measured (88% utilization) vs. NVIDIA H100's 3,600 GB/s/3,060 GB/s (85%). Point-to-point latency for 64 KiB is ~5 µs (MI300X) vs. 2.5 µs (H100) (Ambati et al., 31 Oct 2025).

5. Mixed Precision, Error–Time Tradeoffs, and Kernel Optimization

Mixed-precision arithmetic is native to MI300X, enabling dynamically configured block-wise precision in FFTMatvec and LLM inference. Pareto front analysis sweeps all $2^5=32$ assignments of five FFT phases between FP16 and higher precision. Performance and error are characterized by:

High-fidelity ( $\leq 10^{-7}$ ): FP32 everywhere, 9.6 TFLOP/s, $E \simeq 5 \times 10^{-8}$
Mid-tier ( $\leq 10^{-4}$ ): config “d s s s d” (FP64 in 1,5; FP16 in 2–4), 12.7 TFLOP/s
Low-precision ( $\leq 10^{-2}$ ): FP16 only, 18.9 TFLOP/s, $E \simeq 3 \times 10^{-3}$

Theoretical error upper bound: $E_{\max}(\mathit{cfg}) = \|\widehat{y} - y\|_\infty \leq \gamma_k n \log n,\quad \gamma_k = \frac{k\varepsilon}{1-k\varepsilon}$

RocBLAS kernel optimizations for gfx94-class include:

Transpose-GEMV kernel: tile size 256×32 per work-group
Wavefront: 64 threads (2 × 32 lanes)
LDS double-buffering (64 KiB per CU)
Vectorized FP16 loads/stores (16 bytes width)
Yields up to 2.5× kernel speedup for $M,N\geq 1024,4096$ (Venkat et al., 13 Aug 2025)

6. Benchmarking in AI, HPC, and LLM Inference Applications

In LLM workloads, MI300X exhibits strong architectural potential but falls short of NVIDIA H100/B200 measured utilization (45–81% vs. 90–93%), primarily due to compiler/microkernel and collective (RCCL) software stack maturity (Ambati et al., 31 Oct 2025).

For Llama 3.1-70B FP8/FP16 inference:

MI300X achieves 49–66% of H100/H200 throughput depending on prefill or decode regime and improves to 56–80% in bandwidth-bound settings (large working sets).
Weak scaling for tensor-parallel workloads matches 70–88% efficiency from 2–8 GPUs.
Multi-head attention mapped via Swizzled Head-first delivers up to 50% higher throughput and near-perfect L2 hit rates (80–97%) versus naïve block/head-first (Choudhary et al., 3 Nov 2025).

7. Deployment Guidelines, Limitations, and Future Directions

Key deployment strategies include alignment of kernel problem size to CU count for clock stability, use of large batch sizes (≥64 MiB) to saturate HBM bandwidth, pipelined tensor-parallelism overlapping computation and communication, and explicit NUMA-aware remapping of kernel workgroups for memory-dense workloads.

Current bottlenecks center on:

Sustained clock-rate down-throttling for heavy GEMM
Incomplete ROCm kernel/compiler optimization
Non-topology-aware RCCL collectives for multi-GPU mesh
Software layered optimization lag relative to NVIDIA’s mature stack

A plausible implication is that as chiplet NUMA architectures become pervasive, kernel, scheduler, and communication stacks must evolve to provide explicit head/domain mapping primitives, and algorithm designers must default to spatially-aware scheduling to extract full device efficiency. The MI300X establishes a maximum for die-stack capacity and theoretical throughput, but closing the gap to realized performance will require advances in ROCm stack maturity and topology-integrated collective communication.

Summary Table: AMD Instinct MI300X Key Metrics

Feature	MI300X Value	Context/Significance
Chiplets	8 XCD (38–96 CUs/die)	NUMA, multi-domain memory
HBM3 Memory	192 GB, 5.3 TB/s	Largest per-GPU capacity
Matrix Engines/CU	4 (FP8/FP16/BF16 GEMM)	Tensor compute, mixed-precision
Measured FP16 GEMM	500–1,790 TFLOPS	45–81% peak efficiency
Memory BW (measured)	4.3 TB/s	81% peak, competitive
L2 Cache (total)	32 MiB	NUMA locality, L2 hit critical
Interconnect	Infinity Fabric mesh, 128 GB/s/link	Bottleneck for collectives
Cache Hit Rate	80–97% (Swizzled Head-first)	Sustained with domain mapping
Multi-GPU Scaling	75–90% (1–2048 GPUs)	Wall time 0.55→0.15 s

The AMD Instinct MI300X embodies the current state-of-the-art in chiplet-based GPU acceleration, demanding NUMA-aware software, precision-tunable algorithm design, and topology-sensitive multi-GPU orchestration to approach the device’s raw potential in AI and HPC deployments.