Gem5-AcceSys: Full-System Accelerator Simulation

• Gem5-AcceSys is a full-system simulation framework that extends gem5 to co-design interconnects, DMA engines, PCIe paths, and heterogeneous memory hierarchies.
• It models realistic system behaviors, demonstrating up to 400× GEMM speedups and achieving nearly 78% of device-memory performance via optimized host memory implementations.
• The platform enables end-to-end evaluation of transformer workloads by accurately capturing both GEMM and Non-GEMM phases under NUMA-related constraints.

Gem5-AcceSys is a full-system simulation framework that extends gem5 to enable principled, system-level exploration of standard interconnects and heterogeneous memory hierarchies for novel accelerators. It was motivated by the data-movement intensity of modern machine-learning and image workloads, in which interconnects and memory often bound performance more than raw compute. The framework models realistic PCIe paths, DMA engines, virtual memory via SMMU, caches, and both host- and device-side memories, and couples them with RTL or C++ accelerators. In the reported case study, optimized host-side interconnects reached around 78% of device-side-memory performance and, under transformer workloads with substantial Non-GEMM components, even surpassed device-side memory because of NUMA-related penalties in the device memory path (Liu et al., 17 Feb 2025).

1. Purpose and problem setting

Gem5-AcceSys was introduced against a backdrop in which hardware accelerators such as GPUs and Data Streaming Accelerators are increasingly used to offload machine learning and image-processing computation from the CPU, yet the resulting systems remain dominated by data movement among CPU, accelerator, and memory. The framework therefore targets a specific systems question: when host-side memory over PCIe is “good enough,” which interconnect parameters matter most, and when device-side memory is necessary (Liu et al., 17 Feb 2025).

The motivating workloads are transformer-like pipelines, including Vision Transformer variants, in which large matrix multiplications are interleaved with Non-GEMM phases such as softmax, layer norms, and data packing. This decomposition matters because a memory or interconnect configuration that is favorable for GEMM kernels in isolation need not be optimal for end-to-end inference. A common misconception is that device-side memory necessarily dominates host-memory designs. Gem5-AcceSys was explicitly built to test that assumption under full-system conditions rather than under an idealized accelerator-local model (Liu et al., 17 Feb 2025).

The framework also responds to limitations in prior gem5-based accelerator simulators, which often rely on simplistic buses and lack standard interconnects such as PCIe, NUMA behavior, realistic DMA and SMMU support, and comprehensive memory modeling. In that sense, Gem5-AcceSys shifts the design problem from accelerator microarchitecture alone toward interconnect-memory co-design (Liu et al., 17 Feb 2025).

2. System architecture and integration with gem5

The modeled system contains a CPU cluster, a MemBus and memory controller, a PCIe stack, an accelerator wrapper, an SMMU, and a configurable memory hierarchy. In the reported configuration, the CPU cluster uses ARM cores at 1 GHz with a 32 kB L1 instruction cache, a 64 kB L1 data cache, a 2 MB LLC, and a 32 kB IOCache. The MemBus connects CPU, caches, and host memory and can integrate with Ramulator2, DRAMsim3, and DRAMSys5 for detailed DRAM timing (Liu et al., 17 Feb 2025).

The PCIe stack includes a Root Complex, a PHY or link layer, and a Switch. The Root Complex latency is modeled as 150 ns and the Switch as 50 ns. The accelerator wrapper includes a PCIe endpoint interface, an accelerator controller, a multi-channel DMA engine, a local memory buffer, and a device memory controller. The accelerator can be instantiated either as a C++ model or through Verilator-generated C++ using a child-process model with shared memory syscalls. Between the MemBus and PCIe components sits an SMMU, which provides accelerator-friendly IOMMU address translation and virtual-memory mappings (Liu et al., 17 Feb 2025).

Gem5-AcceSys exposes three accelerator access modes. In Direct Cache (DC) mode, coherence between the accelerator’s cache and the CPU cache is supported. In Direct Memory (DM) mode, the accelerator accesses host memory directly. In DevMem mode, the accelerator bypasses PCIe and uses device-side memory. The framework also supports a configurable sub-memory architecture, including device-side memory, local buffers, and an optional device-side cache (Liu et al., 17 Feb 2025).

Compared to prior gem5-based frameworks, the paper states that Gem5-AcceSys is the only one to combine PCIe, SMMU, external memory simulators, multi-channel DMA, device-side memory, and full-system simulation. This suggests that its primary contribution is not a single new component but the joint modeling of components that are usually studied in isolation (Liu et al., 17 Feb 2025).

3. Interconnect and memory modeling

A central feature of Gem5-AcceSys is packet- and bandwidth-aware PCIe modeling. The baseline configuration uses PCIe 2.0 with 4 Gb/s per lane and 4 lanes, but the framework sweeps lane counts from 2 to 16 and per-lane speeds from 2 to 64 Gbps, reaching aggregate throughput up to 64 GB/s. Transaction-level requests are generated by accelerator-initiated DMA transfers with configurable request or payload sizes from 64 B to 4096 B, queue and doorbell structures in the controller, and coalescing policies for burst formation (Liu et al., 17 Feb 2025).

The paper uses the standard throughput model

$$T_{\mathrm{PCIe}} = R_{\mathrm{link}} \times N_{\mathrm{lanes}} \times \eta_{\mathrm{payload}},$$

with

$$\eta_{\mathrm{payload}} = \frac{S_{\mathrm{payload}}}{S_{\mathrm{payload}} + S_{\mathrm{overhead}}}.$$

For bulk transfers of size $B$, the transfer time is described as

$$t_{\mathrm{transfer}} \approx t_{\mathrm{setup}} + \frac{B}{T_{\mathrm{PCIe}}} + t_{\mathrm{latency}}.$$

Larger payloads improve payload efficiency by amortizing header and framing overheads, but the simulations show that very large requests can disrupt pipeline flow through the Root Complex and Switch, creating stalls that offset those gains (Liu et al., 17 Feb 2025).

The memory hierarchy is equally configurable. Host-side DRAM can be instantiated as DDR3, DDR4, DDR5, GDDR6, or HBM2 via Ramulator2, and the same technologies can also be modeled as device-side memory.

Memory type	Configuration	Peak bandwidth
DDR3	1 channel, 64-bit bus, 1600 MT/s	12.8 GB/s
DDR4	1 channel, 64-bit bus, 2400 MT/s	19.2 GB/s
DDR5	2 channels, 32-bit per channel, 3200 MT/s	25.6 GB/s
GDDR6	2 channels, 64-bit per channel, 2000 MT/s	32 GB/s
HBM2	2 channels, 128-bit per channel, 2000 MT/s	64 GB/s

The framework’s roofline analysis uses the standard operational-intensity formulation

$$I = \frac{\mathrm{FLOPs}}{\mathrm{Bytes}}, \qquad P \le \min\!\bigl(P_{\mathrm{peak}},\, I \cdot B_{\mathrm{mem}}\bigr),$$

where $B_{\mathrm{mem}}$ is the sustained bandwidth along the active path, either host memory via PCIe or device-side DRAM. For the tested GEMM, the roofline study identifies the transition between memory-bound and compute-bound regimes around 1500 ns (Liu et al., 17 Feb 2025).

4. Accelerator case study and experimental methodology

The main case study is MatrixFlow, described in the original paper as a loosely coupled systolic-array accelerator tailored for transformer GEMM. Its microarchitecture contains a 16×16 MAC array with integer input and output, and a controller that orchestrates streams from PCIe and DMA to a local buffer and to device or host memory. Matrices are tiled to match the array dimensions, and representative sizes such as 1024×1024 and 2048×2048 are used to span compute- and memory-bound regimes. For GEMM kernels, the reported speedups reach up to 400× versus a single-threaded CPU (Liu et al., 17 Feb 2025).

The workload set includes Vision Transformer variants: ViT_base with hidden dimension 768, ViT_large with hidden dimension 1024, and ViT_huge with hidden dimension 1280, using 12 or 16 attention heads. The study separates GEMM from Non-GEMM components in order to quantify how end-to-end performance depends on workload composition rather than on GEMM throughput alone (Liu et al., 17 Feb 2025).

The experimental setup uses gem5 full-system simulation with the ARM 1 GHz host, the cache hierarchy noted above, DDR3_1600_8×8 4 GB as the baseline host memory, and MatrixFlow in DC mode for GEMM runs unless otherwise stated. Metrics include normalized execution time and speedup, GEMM versus Non-GEMM breakdown, PCIe link utilization, stall cycles due to the Root Complex and Switch, memory bandwidth utilization, cache hit and miss rates in DC mode, address-translation overheads, and roofline-bound classification. Energy was not reported, and no direct hardware calibration or error margins were provided (Liu et al., 17 Feb 2025).

A subsequent system-accelerator co-design study retained Gem5-AcceSys and MatrixFlow but emphasized paged streaming dataflows and explicit overlap of compute and transfer. In that later formulation, MatrixFlow used page-aligned 4 KB tiles, a small on-chip buffer of about 20 KB, and a pipelined schedule of DMA, compute, and DMA-out to study transformer inference under realistic system constraints, including BERT and ViT across multiple data types (Liu et al., 19 Mar 2026).

5. Quantitative findings and design interpretation

The PCIe bandwidth sweep for GEMM 2048×2048 shows that increasing lane count and per-lane speed reduces execution time until compute-bound saturation at 16 lanes. The highest-bandwidth configuration outperforms the lowest by up to 1109.9%, but the gains diminish once the system crosses into compute-bound territory (Liu et al., 17 Feb 2025).

Request size exhibits a convex performance curve. Execution time improves from 64 B toward an optimum around 256 B, then degrades at 4096 B because of pipeline stalls at the Root Complex and Switch. Relative to the 256 B optimum, 64 B packets incur about 12% overhead and 4096 B packets about 36% overhead. This result directly counters the simplistic intuition that larger packets are always better as long as header overhead is amortized (Liu et al., 17 Feb 2025).

The comparison between host-side and device-side memory is more nuanced than a pure bandwidth ranking. Device-side memory consistently outperforms host-side memory across DDR4, GDDR5, and HBM in isolated comparisons because PCIe overhead is avoided. However, host-side memory with a 64 GB/s PCIe link reaches around 78% of device-side-memory performance. For mixed transformer workloads, optimized host interconnects can even outperform device memory because the device-memory path incurs NUMA-related Non-GEMM penalties (Liu et al., 17 Feb 2025).

Bandwidth dominates latency sensitivity for GEMM-heavy workloads. Increasing HBM-equivalent bandwidth up to about 50 GB/s yields about a 60% reduction in normalized execution time, while moving from 50 to 256 GB/s yields only about 1.7% further improvement. By contrast, increasing DRAM latency from 1 ns to about 36 ns raises execution time by only about 4.9% (Liu et al., 17 Feb 2025).

Address translation is non-negligible at larger problem sizes. For matrices up to size 2048, translation overhead peaks at 6.49%. The mean translation time rises to 54.38 cycles at size 2048, while PTW mean time grows to 368.14 cycles, associated with a footprint of 12,288 pages. The paper accordingly notes that optimizing page sizes or pinning can reduce these costs (Liu et al., 17 Feb 2025).

The transformer-level analysis provides threshold conditions for when DevMem is preferable. Using the paper’s performance model

$$\text{Time}_{\text{overall}} = T_{\text{other}} + \frac{W_{\text{GEMM}}}{P_{\text{GEMM}}} + \frac{W_{\text{NonGEMM}}}{P_{\text{NonGEMM}}},$$

the reported thresholds are as follows: DevMem is preferable if $W_{\text{GEMM}} > 34.31\%$ at PCIe-2 GB/s, greater than $10.16\%$ at PCIe-8 GB/s, and greater than $4.27\%$ at PCIe-64 GB/s. For full-model ViT runs, host memory over PCIe-64 GB/s achieves 2.5×–3.4× speedups over PCIe-2 GB/s, while DevMem with HBM2 performs slightly worse than PCIe-64 GB/s despite superior GEMM performance, because Non-GEMM overheads become dominant. In that decomposition, DevMem incurs up to 500% overhead in Non-GEMM relative to PCIe systems, and NUMA overhead contributes about 40% degradation in ViT_large (Liu et al., 17 Feb 2025).

6. Scope, limitations, and later developments

Gem5-AcceSys models PCIe at a level intended for interconnect and packet-size studies, but the paper explicitly notes abstractions in detailed TLP credit-based flow control and some transaction-layer behavior. Energy modeling is not reported. Coherent interconnects such as CXL.cache, NVLink with coherence, and CCIX are not yet modeled, driver-level overheads are approximated, and hardware calibration plus error margins are not provided. Cache contention and CPU-accelerator coherence traffic are modeled functionally, but deeper contention and QoS effects are identified as expansion points (Liu et al., 17 Feb 2025).

The stated extension path includes support for CXL, NVLink, CCIX, richer PCIe transaction and flow-control models, enhanced NUMA placement modeling, and expanded driver and runtime models. Reproducibility is supported by integration with Ramulator2, DRAMsim3, and DRAMSys5, by Verilator or C++ accelerator plug-ins, and by configuration scripts that sweep PCIe lanes, per-lane rates, and payload sizes while selecting among DC, DM, and DevMem modes (Liu et al., 17 Feb 2025).

In a later bandwidth-wall study, Gem5-AcceSys was used as the system substrate for page-aligned streaming transformer inference. That work reported up to 22x end-to-end speedup over a CPU-only baseline and 5x to 8x gains over state-of-the-art loosely and tightly coupled accelerators. It also reiterated the earlier systems conclusion that a standard PCIe-based host-memory design can achieve about 80 percent of the performance of on-device HBM, and argued that paged streaming and pipeline overlap, rather than large local SRAMs, are the most effective levers under realistic system constraints (Liu et al., 19 Mar 2026).

Taken together, these studies define Gem5-AcceSys less as a single accelerator simulator than as a system-level methodology for evaluating accelerators in the presence of standard interconnects, virtual-memory translation, heterogeneous memory placement, and full software stacks. This suggests that its enduring significance lies in shifting accelerator evaluation from isolated kernel throughput toward end-to-end system behavior (Liu et al., 17 Feb 2025).