---
title: 'AMD MI300X GPUs: Architecture & Performance'
url: https://www.emergentmind.com/topics/amd-mi300x-gpus
type: topic
---

# AMD MI300X GPUs: Architecture & Performance

Searching arXiv for recent papers on AMD MI300X GPUs and related microarchitecture, performance, and software studies.
AMD MI300X GPUs are AMD’s flagship datacenter GPUs in the CDNA 3 family, designed for high-performance computing and AI workloads rather than gaming. In the literature, MI300X is characterized by a combination of large on-package HBM capacity, high memory bandwidth, matrix-core acceleration, and AMD’s Infinity Fabric interconnect, and it is studied both as a deployment target for large language model inference and training and as a reference platform for software, compiler, and simulator research [2510.27583] [2512.08242] [2501.18113].

## 1. Architectural profile and device position

The reported hardware profile of MI300X centers on memory capacity, memory bandwidth, chiplet organization, and matrix acceleration. One performance study describes MI300X as having **192 GB of HBM3** memory, **5.3 TB/s theoretical memory bandwidth**, **Matrix Cores**, **Infinity Fabric** with up to **128 GB/s bidirectional bandwidth per link**, and a CDNA 3 chiplet-based design with up to **8 vertically stacked accelerator compute dies (XCDs)** [2510.27583]. A training characterization on an eight-GPU node further reports **192 GB HBM**, **5.3 TB/s HBM bandwidth**, **peak 1.3 PFLOPS BF16**, and **1,216 matrix cores** per MI300X, together with a fully connected eight-GPU Infinity Fabric topology in which every pair of GPUs is connected by a **128 GB/s bidirectional** link [2512.08242].

| Attribute | Reported value |
|---|---|
| Product position | AMD’s flagship datacenter GPU |
| Microarchitecture family | CDNA 3 |
| HBM capacity | 192 GB HBM3 |
| Theoretical HBM bandwidth | 5.3 TB/s |
| Matrix cores | 1,216 |
| Peak BF16 | 1.3 PFLOPS |
| Interconnect | Infinity Fabric, 128 GB/s bidirectional per link |
| Chiplet organization | up to 8 vertically stacked XCDs |

These features matter because the literature repeatedly frames MI300X as a system in which compute throughput, memory bandwidth, memory capacity, and interconnect bandwidth must all be considered simultaneously. In inference studies, the large HBM capacity is treated as important for fitting larger models and KV caches, while in training studies the same capacity enables comparatively simple parallelism choices, including data parallelism with ZeRO-1 only in some large-scale pretraining configurations [2510.27583] [2511.17127].

Earlier cross-platform AMD GPU studies predate MI300X and evaluate MI100 and MI250 rather than MI300X. Those results are explicitly described as only partially generalizable to MI300X, but they provide useful context on ROCm, HIP, CU and Matrix Core terminology, and the sensitivity of AMD GPU performance to compiler and operator selection [2311.04417].

## 2. Matrix cores, MFMA semantics, and execution behavior

A central technical feature of MI300X-class GPUs is AMD’s Matrix Core Engine execution of Matrix Fused Multiply Add instructions. In gem5 support for AMD MI200 and especially MI300/MI300X-class GPUs, MFMA is modeled as executing on dedicated **Matrix Core Engines (MCEs)** located inside each SIMD unit of a compute unit. The model assumes **one MCE per SIMD unit** and, because the modeled AMD GPUs have **4 SIMD units per CU**, this yields **4 MCEs per CU**. The MCE is treated as a distinct functional unit from transcendental units, vector ALUs, vector load/store logic, and scalar memory operations, so unrelated work may proceed while an MFMA is in flight, subject to dependencies and scheduling constraints [2501.18113].

The paper describes the MFMA operation class as
$$
D = C + A * B
$$
with \(D\) and \(C\) as \(4 \times 4\) matrices, \(A\) as \(4 \times 1\), and \(B\) as \(1 \times 4\). It also gives the ISA naming convention
$$
\texttt{V\_MFMA\_[output type]\_[M]x[N]x[K]\_[B]B\_[input type]}
$$
where \(M\), \(N\), and \(K\) describe matrix dimensions and \(B\) is the number of blocks operated on. The gem5 latency model follows the cycle counts in the MI300 ISA manual, and the MI300/MI300X generation is treated as architecturally distinct from MI200: some instructions were added, some removed, and some became faster. The paper specifically notes a new **2-block variant of `v_mfma_f32_32x32x4_bf16`** with the same cycle count as the MI200 1-block version, and the removal of **`v_mfma_f32_32x32x2_bf16`** and **`v_mfma_i32_16x16x16i8`** [2501.18113].

Dependency semantics are treated conservatively. The same work emphasizes that compilers insert independent work or `s_nop` instructions between dependent MFMAs rather than treating them as fully pipelined within a single wavefront. In gem5 this behavior is enforced through the per-SIMD **`NRDY_MATRIX_CORE`** availability field, and validation against hardware reports average MFMA-latency error of about **1.332% for MI300** [2501.18113]. For architectural studies of MI300X-like systems, this is significant because it makes matrix-core timing a first-class simulation concern rather than an omitted accelerator detail.

A related execution-centric CDNA3 study on MI300A is presented as the closest directly measured guidance for MI300X-class GPUs. It reports that FP8 matrix execution is highly occupancy-sensitive, requiring about **256+ active wavefronts** to approach peak behavior, compared with **192** for FP16 and **128** for FP32 in that characterization. The same study identifies **\(16 \times 16 \times 32\)** as the preferred FP8 MFMA tile shape and shows that lower-occupancy FP8 execution can become memory-latency-bound rather than compute-saturated [2602.10262]. This suggests that MI300X matrix-core performance depends not only on ISA support and nominal peak throughput, but also on wavefront-level parallelism and kernel shape.

## 3. Memory system, topology, and interconnect

MI300X is consistently presented as a memory-rich accelerator whose system behavior depends strongly on both HBM and interconnect. In inference benchmarking, the device is reported to achieve about **81%** of its **5.3 TB/s** theoretical memory-bandwidth peak, saturating around **4.3 TB/s** and plateauing around **64–128 MiB** working sets in the measured bandwidth scaling results [2510.27583]. The same study treats this memory subsystem as strong and competitive, particularly for decode-heavy inference, while still distinguishing theoretical bandwidth from effective end-to-end performance.

For intra-node and multi-GPU communication, two strands of evidence are important. First, MI300X communication microbenchmarks report measured collective bandwidth scaling from **64 GB/s** with **2 GPUs**, to **192 GB/s** with **4 GPUs**, to **448 GB/s** with **8 GPUs**, reaching about **70% of theoretical peak bandwidth** in the tested AMD communication stack [2510.27583]. Second, a full-stack AMD pretraining study gives an explicit xGMI bandwidth rule:
$$
B_{\text{per-GPU}} = 
\begin{cases}
B_{\max} & \text{NVSwitch (NVIDIA)} \\
(n-1)\cdot B_{\text{link}} & \text{xGMI (AMD)}
\end{cases}
$$
with \(B_{\max} \approx 450\) GB/s for MI300X xGMI and \(B_{\text{link}} = 64\) GB/s per xGMI link. The practical guidance drawn there is that AMD parallelism schemes should ideally span a full node, or avoid partial-node collectives when possible [2511.17127].

Topology-discovery work adds another layer of detail. MT4G explicitly validates on an **AMD Instinct MI300X VF** platform and states that it works on all AMD CDNA GPUs, but it also notes two MI300X-specific caveats: the **CU id / sharing benchmark** could not be executed on MI300X because of a virtualized environment, and AMD CDNA3 **L3** support is still incomplete in the sense that MT4G can retrieve **L3 size** via API but is **currently not able to obtain load latency and fetch granularity for AMD L3 caches** [2511.05958]. In other words, the existence of a CDNA3 L3 layer is acknowledged, but the published MI300X topology characterization remains partial.

The convergence of these studies indicates that MI300X is best understood as a device whose usable performance depends on how kernels traverse HBM, caches, and collective fabrics, rather than as a processor for which nominal FLOP rate alone is decisive.

## 4. Empirical performance in LLM inference and training

The most direct MI300X inference study evaluates compute throughput, memory bandwidth, communication, and end-to-end serving for **Llama 3.1 70B** under **FP8** and **FP16**. It reports that MI300X has **1.5× higher theoretical compute capacity than H100**, yet achieves only about **45% of its theoretical peak FLOPs on average** across FP8, BF16, and FP16, whereas **H100** and **B200** sustain up to about **93%** of peak. In end-to-end inference, MI300X delivers about **50% or less** of H100/H200 throughput in FP8 prefill-dominated settings, improves from **49% of H100** at output length 1 to **66%** at output length 2048 in FP8 decode-dominated settings, and improves from about **56% to 80% of H100** throughput in FP16 decode-dominated settings [2510.27583].

The same paper attributes the gap between theoretical capability and realized inference performance primarily to **software stack maturity**, **dynamic frequency scaling / power management**, and interconnect efficiency rather than to raw hardware limits alone. A common misconception is therefore corrected directly by measurement: MI300X’s large HBM and strong theoretical compute do not automatically translate into superior LLM inference throughput [2510.27583].

Training studies present a different picture. Chopper’s multi-granularity characterization of **Llama 3 8B** training on an eight-MI300X node under **FSDP** and **FSDPv2** finds that MI300X can sustain strong training performance, but that the main gap between theoretical and observed throughput is **frequency overhead from DVFS/power management**. The paper reports that **FSDPv2 achieves approximately 20% higher GPU and memory clock frequency**, with **significantly less variation**, and also states that average GPU frequency is **approximately 25% higher** with **no change in power** relative to FSDPv1. It further concludes that frequency overhead exceeds the impact of **MFMA utilization loss**, **communication/computation overlap**, and **kernel launch overheads** [2512.08242].

The same training study reports that **b2s4** and **b4s4** achieve the highest throughput, while **b1s4** and **b1s8** achieve the lowest, with batch size 1 showing about **30% lower throughput**. It also identifies backward FlashAttention as unusually inefficient at batch size 1, noting that it can take **less time at batch size 2 than at batch size 1** despite larger FLOP count [2512.08242]. This suggests that MI300X training efficiency is strongly shaped by framework execution order, memory-allocation determinism, and batch-sensitive kernel implementations.

## 5. Software stack, programmability, and performance portability

MI300X is studied not only as hardware, but also as a software target. Matrix-core dependence in the AMD software stack is explicit: modern AMD ML frameworks and libraries, including **ROCm**, **MIOpen**, **rocBLAS**, **PyTorch**, and **TensorFlow**, rely on MFMA instructions for high performance, which is why adding MCE/MFMA support to gem5 was necessary for realistic MI300/MI300X-class workload simulation [2501.18113].

Performance-portable HPC work shows MI300X as a reference AMD platform for mixed-precision and library optimization. The FFTMatvec study reports that its performance-portable, mixed-precision application runs on **MI250X, MI300X, and MI355X**, that MI300X support uses **ROCm v6.4.1** and typically **`amd-clang++`**, and that reproducing the rocBLAS transpose-GEMV benchmark requires at least one **AMD Instinct MI300X** with architecture flag **`-a gfx942`**. The paper’s mixed-precision framework evaluates **32 possible test configurations** encoded as `-prec xxxxx`, compares their error to double precision, and chooses the configuration that gives the greatest speedup subject to a desired error tolerance [2508.10202].

AI-assisted kernel generation work places MI300X in a different software trajectory. GEAK, an AMD framework for generating Triton kernels for **MI300X** and **MI250**, reports on MI300 that it correctly generated **101 / 184** kernels on TritonBench-revised, reached **54.89%** execution accuracy, and achieved **2.59×** average speedup; on a more realistic ROCm benchmark it generated **19 / 30** correct kernels with **63.33%** execution accuracy and **0.92×** average speedup [2507.23194]. This indicates that MI300X-targeted kernel generation is feasible, but that realistic ROCm workloads remain substantially harder than curated benchmark kernels.

Large-scale pretraining on a pure AMD platform extends the same point to the system level. That work uses a stack centered on **ROCm**, **HIP**, **RCCL**, **Composable Kernel**, **AITER**, and a modified **Megatron-LM**, and it reports that custom HIP kernels were implemented for fused LayerNorm / RMSNorm, Muon optimizer routines, and symmetric matrix multiplication in the Newton–Schulz step [2511.17127]. A plausible implication is that MI300X programmability is now broad enough for production-scale training, but that best-in-class performance still depends on explicit kernel and communication engineering.

## 6. System behavior, simulation, and operational pathologies

MI300X has also become a research platform for studying node-level effects that do not appear in single-kernel benchmarks. The **Lit Silicon** study examines two eight-MI300X nodes running **Llama 3.1 8B** and **Mistral 7B v0.1** under **FSDP** and **FSDP2**, and identifies a coupling between thermal imbalance and concurrent computation/communication overlap. It reports that the highest temperature and frequency were **1.155×** and **1.062×** the lowest values, respectively, and that leader GPUs can reach overlap ratios as high as **52.7%** while a straggler can remain near **29.6%**. Across its evaluated settings, the paper reports up to **6% performance improvement** and **4% power improvement** from node-level power-cap adjustment, with **GPU-Red** reducing average power by **4%**, **GPU-Realloc** improving throughput by **3%**, and **CPU-Slosh** improving throughput by **4%** while increasing GPU power by **3%** [2511.09861].

This result corrects another common misconception: identical work assignment across GPUs does not imply identical execution time on MI300X nodes. Thermal state, DVFS response, and overlap behavior can interact to create persistent leaders and stragglers [2511.09861].

Simulation and profiling infrastructure reinforce the same shift from peak-centric to behavior-centric analysis. gem5’s MI300/MI300X-class MFMA support allows cycle-accurate exploration of instruction-specific matrix-core latency, scheduling rules, and hypothetical latency scaling via the `--mfma-scale` parameter, while Chopper aligns runtime traces and hardware counters across kernels, operations, layers, phases, iterations, and GPUs to expose the relative importance of launch overhead, overlap overhead, MFMA utilization, and frequency overhead [2501.18113] [2512.08242].

A final technical caution comes from the related MI300A / CDNA3 execution study. There, FP8 matrix execution, asynchronous execution, and 2:4 structured sparsity are shown to be strongly context-dependent: ACE concurrency improves throughput but can sharply reduce fairness at high stream counts, and structured sparsity is nearly break-even in isolated execution yet beneficial under concurrency [2602.10262]. For MI300X-class systems, this suggests that FP8, overlap, and sparsity should be treated as scheduling- and workload-dependent mechanisms rather than uniformly favorable features.

Taken together, these studies portray MI300X as a large-memory CDNA 3 accelerator whose practical behavior emerges from the interaction of matrix-core execution, HBM and cache behavior, collective communication, compiler and library maturity, framework scheduling, and node-level power management. The published literature therefore treats MI300X not merely as a high-specification GPU, but as a full-stack research object spanning microarchitecture, runtime systems, distributed training, inference serving, and simulation.

Source: https://www.emergentmind.com/topics/amd-mi300x-gpus