AMD MI300X GPUs: Architecture & Performance

• AMD MI300X GPUs are flagship datacenter accelerators built on CDNA3 architecture, combining large HBM3 capacity, high memory bandwidth, and chiplet-based design for scalable HPC and AI.
• They feature 192GB HBM3 and 5.3TB/s theoretical memory bandwidth with 1,216 matrix cores and Infinity Fabric interconnect, enabling efficient large model inference and training.
• Their practical performance relies on a mature software stack and detailed simulation support, addressing challenges like DVFS overhead and kernel scheduling for improved throughput.

Searching arXiv for papers on AMD MI300X GPUs and related microarchitecture, performance, and software studies. AMD MI300X GPUs are AMD’s flagship datacenter GPUs in the CDNA 3 family, designed for high-performance computing and AI workloads rather than gaming. In the literature, MI300X is characterized by a combination of large on-package HBM capacity, high memory bandwidth, matrix-core acceleration, and AMD’s Infinity Fabric interconnect, and it is studied both as a deployment target for LLM inference and training and as a reference platform for software, compiler, and simulator research (Ambati et al., 31 Oct 2025, Kurzynski et al., 9 Dec 2025, Kurzynski et al., 30 Jan 2025).

1. Architectural profile and device position

The reported hardware profile of MI300X centers on memory capacity, memory bandwidth, chiplet organization, and matrix acceleration. One performance study describes MI300X as having 192 GB of HBM3 memory, 5.3 TB/s theoretical memory bandwidth, Matrix Cores, Infinity Fabric with up to 128 GB/s bidirectional bandwidth per link, and a CDNA 3 chiplet-based design with up to 8 vertically stacked accelerator compute dies (XCDs) (Ambati et al., 31 Oct 2025). A training characterization on an eight-GPU node further reports 192 GB HBM, 5.3 TB/s HBM bandwidth, peak 1.3 PFLOPS BF16, and 1,216 matrix cores per MI300X, together with a fully connected eight-GPU Infinity Fabric topology in which every pair of GPUs is connected by a 128 GB/s bidirectional link (Kurzynski et al., 9 Dec 2025).

Attribute	Reported value
Product position	AMD’s flagship datacenter GPU
Microarchitecture family	CDNA 3
HBM capacity	192 GB HBM3
Theoretical HBM bandwidth	5.3 TB/s
Matrix cores	1,216
Peak BF16	1.3 PFLOPS
Interconnect	Infinity Fabric, 128 GB/s bidirectional per link
Chiplet organization	up to 8 vertically stacked XCDs

These features matter because the literature repeatedly frames MI300X as a system in which compute throughput, memory bandwidth, memory capacity, and interconnect bandwidth must all be considered simultaneously. In inference studies, the large HBM capacity is treated as important for fitting larger models and KV caches, while in training studies the same capacity enables comparatively simple parallelism choices, including data parallelism with ZeRO-1 only in some large-scale pretraining configurations (Ambati et al., 31 Oct 2025, Anthony et al., 21 Nov 2025).

Earlier cross-platform AMD GPU studies predate MI300X and evaluate MI100 and MI250 rather than MI300X. Those results are explicitly described as only partially generalizable to MI300X, but they provide useful context on ROCm, HIP, CU and Matrix Core terminology, and the sensitivity of AMD GPU performance to compiler and operator selection (Peng et al., 2023).

2. Matrix cores, MFMA semantics, and execution behavior

A central technical feature of MI300X-class GPUs is AMD’s Matrix Core Engine execution of Matrix Fused Multiply Add instructions. In gem5 support for AMD MI200 and especially MI300/MI300X-class GPUs, MFMA is modeled as executing on dedicated Matrix Core Engines (MCEs) located inside each SIMD unit of a compute unit. The model assumes one MCE per SIMD unit and, because the modeled AMD GPUs have 4 SIMD units per CU, this yields 4 MCEs per CU. The MCE is treated as a distinct functional unit from transcendental units, vector ALUs, vector load/store logic, and scalar memory operations, so unrelated work may proceed while an MFMA is in flight, subject to dependencies and scheduling constraints (Kurzynski et al., 30 Jan 2025).

The paper describes the MFMA operation class as

$D = C + A * B$

with $D$ and $C$ as $4 \times 4$ matrices, $A$ as $4 \times 1$, and $B$ as $1 \times 4$. It also gives the ISA naming convention

$$\texttt{V\_MFMA\_[output type]\_[M]x[N]x[K]\_[B]B\_[input type]}$$

where $M$, $D$0, and $D$1 describe matrix dimensions and $D$2 is the number of blocks operated on. The gem5 latency model follows the cycle counts in the MI300 ISA manual, and the MI300/MI300X generation is treated as architecturally distinct from MI200: some instructions were added, some removed, and some became faster. The paper specifically notes a new 2-block variant of v_mfma_f32_32x32x4_bf16 with the same cycle count as the MI200 1-block version, and the removal of v_mfma_f32_32x32x2_bf16 and v_mfma_i32_16x16x16i8 (Kurzynski et al., 30 Jan 2025).

Dependency semantics are treated conservatively. The same work emphasizes that compilers insert independent work or s_nop instructions between dependent MFMAs rather than treating them as fully pipelined within a single wavefront. In gem5 this behavior is enforced through the per-SIMD NRDY_MATRIX_CORE availability field, and validation against hardware reports average MFMA-latency error of about 1.332% for MI300 (Kurzynski et al., 30 Jan 2025). For architectural studies of MI300X-like systems, this is significant because it makes matrix-core timing a first-class simulation concern rather than an omitted accelerator detail.

A related execution-centric CDNA3 study on MI300A is presented as the closest directly measured guidance for MI300X-class GPUs. It reports that FP8 matrix execution is highly occupancy-sensitive, requiring about 256+ active wavefronts to approach peak behavior, compared with 192 for FP16 and 128 for FP32 in that characterization. The same study identifies $D$3 as the preferred FP8 MFMA tile shape and shows that lower-occupancy FP8 execution can become memory-latency-bound rather than compute-saturated (Jarmusch et al., 10 Feb 2026). This suggests that MI300X matrix-core performance depends not only on ISA support and nominal peak throughput, but also on wavefront-level parallelism and kernel shape.

3. Memory system, topology, and interconnect

MI300X is consistently presented as a memory-rich accelerator whose system behavior depends strongly on both HBM and interconnect. In inference benchmarking, the device is reported to achieve about 81% of its 5.3 TB/s theoretical memory-bandwidth peak, saturating around 4.3 TB/s and plateauing around 64–128 MiB working sets in the measured bandwidth scaling results (Ambati et al., 31 Oct 2025). The same study treats this memory subsystem as strong and competitive, particularly for decode-heavy inference, while still distinguishing theoretical bandwidth from effective end-to-end performance.

For intra-node and multi-GPU communication, two strands of evidence are important. First, MI300X communication microbenchmarks report measured collective bandwidth scaling from 64 GB/s with 2 GPUs, to 192 GB/s with 4 GPUs, to 448 GB/s with 8 GPUs, reaching about 70% of theoretical peak bandwidth in the tested AMD communication stack (Ambati et al., 31 Oct 2025). Second, a full-stack AMD pretraining study gives an explicit xGMI bandwidth rule:

$D$4

with $D$5 GB/s for MI300X xGMI and $D$6 GB/s per xGMI link. The practical guidance drawn there is that AMD parallelism schemes should ideally span a full node, or avoid partial-node collectives when possible (Anthony et al., 21 Nov 2025).

Topology-discovery work adds another layer of detail. MT4G explicitly validates on an AMD Instinct MI300X VF platform and states that it works on all AMD CDNA GPUs, but it also notes two MI300X-specific caveats: the CU id / sharing benchmark could not be executed on MI300X because of a virtualized environment, and AMD CDNA3 L3 support is still incomplete in the sense that MT4G can retrieve L3 size via API but is currently not able to obtain load latency and fetch granularity for AMD L3 caches (Vanecek et al., 8 Nov 2025). In other words, the existence of a CDNA3 L3 layer is acknowledged, but the published MI300X topology characterization remains partial.

The convergence of these studies indicates that MI300X is best understood as a device whose usable performance depends on how kernels traverse HBM, caches, and collective fabrics, rather than as a processor for which nominal FLOP rate alone is decisive.

4. Empirical performance in LLM inference and training

The most direct MI300X inference study evaluates compute throughput, memory bandwidth, communication, and end-to-end serving for Llama 3.1 70B under FP8 and FP16. It reports that MI300X has 1.5× higher theoretical compute capacity than H100, yet achieves only about 45% of its theoretical peak FLOPs on average across FP8, BF16, and FP16, whereas H100 and B200 sustain up to about 93% of peak. In end-to-end inference, MI300X delivers about 50% or less of H100/H200 throughput in FP8 prefill-dominated settings, improves from 49% of H100 at output length 1 to 66% at output length 2048 in FP8 decode-dominated settings, and improves from about 56% to 80% of H100 throughput in FP16 decode-dominated settings (Ambati et al., 31 Oct 2025).

The same paper attributes the gap between theoretical capability and realized inference performance primarily to software stack maturity, dynamic frequency scaling / power management, and interconnect efficiency rather than to raw hardware limits alone. A common misconception is therefore corrected directly by measurement: MI300X’s large HBM and strong theoretical compute do not automatically translate into superior LLM inference throughput (Ambati et al., 31 Oct 2025).

Training studies present a different picture. Chopper’s multi-granularity characterization of Llama 3 8B training on an eight-MI300X node under FSDP and FSDPv2 finds that MI300X can sustain strong training performance, but that the main gap between theoretical and observed throughput is frequency overhead from DVFS/power management. The paper reports that FSDPv2 achieves approximately 20% higher GPU and memory clock frequency, with significantly less variation, and also states that average GPU frequency is approximately 25% higher with no change in power relative to FSDPv1. It further concludes that frequency overhead exceeds the impact of MFMA utilization loss, communication/computation overlap, and kernel launch overheads (Kurzynski et al., 9 Dec 2025).

The same training study reports that b2s4 and b4s4 achieve the highest throughput, while b1s4 and b1s8 achieve the lowest, with batch size 1 showing about 30% lower throughput. It also identifies backward FlashAttention as unusually inefficient at batch size 1, noting that it can take less time at batch size 2 than at batch size 1 despite larger FLOP count (Kurzynski et al., 9 Dec 2025). This suggests that MI300X training efficiency is strongly shaped by framework execution order, memory-allocation determinism, and batch-sensitive kernel implementations.

5. Software stack, programmability, and performance portability

MI300X is studied not only as hardware, but also as a software target. Matrix-core dependence in the AMD software stack is explicit: modern AMD ML frameworks and libraries, including ROCm, MIOpen, rocBLAS, PyTorch, and TensorFlow, rely on MFMA instructions for high performance, which is why adding MCE/MFMA support to gem5 was necessary for realistic MI300/MI300X-class workload simulation (Kurzynski et al., 30 Jan 2025).

Performance-portable HPC work shows MI300X as a reference AMD platform for mixed-precision and library optimization. The FFTMatvec study reports that its performance-portable, mixed-precision application runs on MI250X, MI300X, and MI355X, that MI300X support uses ROCm v6.4.1 and typically amd-clang++, and that reproducing the rocBLAS transpose-GEMV benchmark requires at least one AMD Instinct MI300X with architecture flag -a gfx942. The paper’s mixed-precision framework evaluates 32 possible test configurations encoded as -prec xxxxx, compares their error to double precision, and chooses the configuration that gives the greatest speedup subject to a desired error tolerance (Venkat et al., 13 Aug 2025).

AI-assisted kernel generation work places MI300X in a different software trajectory. GEAK, an AMD framework for generating Triton kernels for MI300X and MI250, reports on MI300 that it correctly generated 101 / 184 kernels on TritonBench-revised, reached 54.89% execution accuracy, and achieved 2.59× average speedup; on a more realistic ROCm benchmark it generated 19 / 30 correct kernels with 63.33% execution accuracy and 0.92× average speedup (Wang et al., 31 Jul 2025). This indicates that MI300X-targeted kernel generation is feasible, but that realistic ROCm workloads remain substantially harder than curated benchmark kernels.

Large-scale pretraining on a pure AMD platform extends the same point to the system level. That work uses a stack centered on ROCm, HIP, RCCL, Composable Kernel, AITER, and a modified Megatron-LM, and it reports that custom HIP kernels were implemented for fused LayerNorm / RMSNorm, Muon optimizer routines, and symmetric matrix multiplication in the Newton–Schulz step (Anthony et al., 21 Nov 2025). A plausible implication is that MI300X programmability is now broad enough for production-scale training, but that best-in-class performance still depends on explicit kernel and communication engineering.

6. System behavior, simulation, and operational pathologies

MI300X has also become a research platform for studying node-level effects that do not appear in single-kernel benchmarks. The Lit Silicon study examines two eight-MI300X nodes running Llama 3.1 8B and Mistral 7B v0.1 under FSDP and FSDP2, and identifies a coupling between thermal imbalance and concurrent computation/communication overlap. It reports that the highest temperature and frequency were 1.155× and 1.062× the lowest values, respectively, and that leader GPUs can reach overlap ratios as high as 52.7% while a straggler can remain near 29.6%. Across its evaluated settings, the paper reports up to 6% performance improvement and 4% power improvement from node-level power-cap adjustment, with GPU-Red reducing average power by 4%, GPU-Realloc improving throughput by 3%, and CPU-Slosh improving throughput by 4% while increasing GPU power by 3% (Kurzynski et al., 13 Nov 2025).

This result corrects another common misconception: identical work assignment across GPUs does not imply identical execution time on MI300X nodes. Thermal state, DVFS response, and overlap behavior can interact to create persistent leaders and stragglers (Kurzynski et al., 13 Nov 2025).

Simulation and profiling infrastructure reinforce the same shift from peak-centric to behavior-centric analysis. gem5’s MI300/MI300X-class MFMA support allows cycle-accurate exploration of instruction-specific matrix-core latency, scheduling rules, and hypothetical latency scaling via the --mfma-scale parameter, while Chopper aligns runtime traces and hardware counters across kernels, operations, layers, phases, iterations, and GPUs to expose the relative importance of launch overhead, overlap overhead, MFMA utilization, and frequency overhead (Kurzynski et al., 30 Jan 2025, Kurzynski et al., 9 Dec 2025).

A final technical caution comes from the related MI300A / CDNA3 execution study. There, FP8 matrix execution, asynchronous execution, and 2:4 structured sparsity are shown to be strongly context-dependent: ACE concurrency improves throughput but can sharply reduce fairness at high stream counts, and structured sparsity is nearly break-even in isolated execution yet beneficial under concurrency (Jarmusch et al., 10 Feb 2026). For MI300X-class systems, this suggests that FP8, overlap, and sparsity should be treated as scheduling- and workload-dependent mechanisms rather than uniformly favorable features.

Taken together, these studies portray MI300X as a large-memory CDNA 3 accelerator whose practical behavior emerges from the interaction of matrix-core execution, HBM and cache behavior, collective communication, compiler and library maturity, framework scheduling, and node-level power management. The published literature therefore treats MI300X not merely as a high-specification GPU, but as a full-stack research object spanning microarchitecture, runtime systems, distributed training, inference serving, and simulation.