Hybrid gem5–Verilator Co-Simulation

• The paper presents an innovative method that integrates gem5’s full-system simulation with Verilator’s cycle-accurate RTL models for validating cache-coherent memory subsystems.
• It employs per-core dual FIFOs and a scoreboard mechanism with a two-phase handshake protocol to synchronize and verify correctness against the gem5 Ruby model.
• Experimental results demonstrate reduced simulation slowdown with increasing core counts, offering a cost-effective, high-fidelity alternative to FPGA emulation and commercial RTL tools.

Hybrid gem5–Verilator co-simulation is an advanced methodology for system-level validation of register-transfer-level (RTL) cache-coherent memory subsystems within multi-core system-on-chip (SoC) architectures. It leverages the strengths of gem5—a cycle-accurate, extensible full-system simulator—and Verilator, an open-source tool that translates synthesizable RTL (Verilog/SystemVerilog) into cycle-accurate C++ models. This approach enables simultaneous execution of cycle-accurate RTL hardware models and realistic workloads, including actual operating systems, providing both functional correctness and high-fidelity performance evaluation. The Rhea framework epitomizes this integration, establishing a unified environment for both rapid design and comprehensive system-level validation of highly parameterizable cache-coherent subsystems in scalable multicore scenarios (Zoni et al., 5 Aug 2025).

1. System Architecture of gem5–Verilator Integration

In Rhea's hybrid infrastructure, the Device Under Test (DUT) is a Verilated C++ model of the RTL cache-coherent memory subsystem, while gem5 implements the remainder of the system (including CPUs, interconnects, main memory, and operating system support). Communication and synchronization are performed on each gem5 simulation cycle using a tightly coordinated interface built from per-core FIFO request and response queues and a minimal two-phase handshake protocol. A dedicated co-simulation wrapper module in gem5 tracks per-CPU requests and responses, and maintains a scoreboard for golden-model checking against the Ruby memory system.

The cycle-wise workflow is algorithmically summarized in the following pseudocode:

$L_{\rm sync} = L_{roe} + L_{wri} + 2L_{eval}$2

A block diagram (see also LaTeX/TikZ LaTeX code provided in (Zoni et al., 5 Aug 2025)) clarifies the data flow between gem5, the co-simulation wrapper, and the Verilated DUT.

Critical features include:

• Dual FIFOs per CPU for request and response tracking.
• Scoreboard mechanism for packet-level correctness checking against the gem5 Ruby memory reference model.
• Conditional clock gating; Verilator's eval() is skipped on idle simulation cycles to minimize computation.
• Multithreaded Verilator execution is supported, enabling parallel evaluation of RTL internal logic on many-core hosts.

2. Cycle-Level Synchronization and Timing Model

Synchronization between gem5 and Verilator is achieved at the granularity of the gem5 simulation cycle, typically corresponding to a 1 ns period at 1 GHz. At each synchronization step, cost components include reading outputs from the DUT (denoted $L_{roe}$), writing new inputs ($L_{wri}$), and invoking two Verilator eval() calls for clock edge toggling ($2L_{eval}$). The aggregate synchronization latency per active cycle is thus:

$L_{\rm sync} = L_{roe} + L_{wri} + 2L_{eval}$

When the model is idle and no outstanding requests exist, evaluation is skipped entirely, reducing cost to simple polling ($O(1)$ per idle cycle). Over the course of $N_{co}$ total gem5 cycles with $N_{active}$ active cycles, the total wall-clock co-simulation time is:

$$T_{\rm co} = T_{\rm gem5} + N_{\rm active} L_{\rm sync} + T_{\rm DUT}$$

Where:

• $T_{\rm gem5}$: time spent simulating high-level gem5 components
• $N_{active}$: count of synchronization-active cycles
• $L_{wri}$0: time consumed by full C++ RTL evaluation (excluding synchronization overhead)

This model formalizes the trade-off between high-level event scheduling in gem5 and fine-grained, cycle-level RTL evaluation.

3. Performance Overhead Metric and Core-Count Scaling

The principal performance metric is the co-simulation slowdown relative to a purely gem5-based simulation using its MI protocol, denoted $L_{wri}$1, for a given core-count $L_{wri}$2:

$L_{wri}$3

where $L_{wri}$4 represents total co-simulation time and $L_{wri}$5 the time for an MI-only gem5 run at matching core count. Empirical measurement involves running a suite of 22 representative multithreaded benchmarks (from PARSEC and Splash-3) and calculating the geometric mean of slowdown ratios across all applications.

TABLE: Co-Simulation Slowdown vs. Core Count

cores $L_{wri}$6	2	4	8	16
single-level MSI	2.3×	1.9×	1.7×	1.7×
two-level MSI	2.7×	2.1×	1.6×	1.6×

A key observation is the monotonic reduction of slowdown as $L_{wri}$7 increases, reflecting the amortization of synchronization and I/O overhead across more concurrent memory traffic per cycle.

4. Experimental Methodology

The evaluation platform comprises a 16-core Intel Xeon Gold 6326 @ 2.9 GHz with 128 GB DDR4 and Rocky Linux 8.10. The toolchain incorporates gem5 v23.1 (timing CPUs at 100 MHz) with Ruby configured for MI, MESI, and MOESI protocols, and Verilator 4.104 for RTL execution. Rhea-generated MSI-based DUTs with both single-level and two-level private cache hierarchies are validated, scaling up to $L_{wri}$8 cores. The detailed configurations involve L1 caches of 8 kB (4-way associative) and L2 caches of 256 kB (8-way associative, with two banks in two-level subsystems).

The workload suite comprises 22 multithreaded applications from the PARSEC and Splash-3 benchmark suites, each executed with a matching $L_{wri}$9 thread count. Evaluation metrics comprise:

• Functional correctness via packet-for-packet comparison between RTL and Ruby models.
• Performance metrics: normalized application execution speedup (RTL versus Ruby MI) and simulation-time slowdown $2L_{eval}$0.

5. Quantitative Performance Results

Evaluation of RTL subsystems using Rhea's hybrid flow quantifies both hardware performance and simulation-time overhead.

A. Application speedup ($2L_{eval}$1) for RTL against Ruby-MI:

cores $2L_{eval}$2	2	4	8	16
Ruby-MESI	2.1×	3.5×	7.3×	11.8×
Ruby-MOESI	2.0×	4.1×	7.6×	12.0×
RTL 1-level MSI	1.4×	2.2×	4.1×	5.7×
RTL 2-level MSI	1.5×	2.6×	5.5×	8.1×

B. Co-simulation slowdown ($2L_{eval}$3): The geometric-mean slowdown reduces from 2.7× at $2L_{eval}$4 cores to 1.6× at $2L_{eval}$5, indicating favorable scaling where synchronization overhead is amortized as concurrency increases.

6. Trade-offs, Limitations, and Prospects

The core overhead model is formalized as:

$2L_{eval}$6

With increasing $2L_{eval}$7, the number of active synchronization cycles ($2L_{eval}$8) grows sublinearly while compute efficiency improves, yielding a net reduction in relative overhead.

Principal trade-offs:

• Hybrid co-simulation achieves higher functional and timing fidelity (real RTL subsystem) at the expense of a moderate (1.6×–2.7×) simulation slowdown relative to pure MI-based gem5.
• FPGA emulation offers greater speed at higher engineering and hardware deployment cost; hybrid co-simulation is software-only, enabling rapid iteration without custom hardware.
• Compared to pure RTL simulation using commercial tools (such as VCS/Xcelium), the methodology is orders-of-magnitude faster due to cycle skipping for idle periods and parallelized execution within Verilator.

Limitations include:

• Fixed-cycle synchronization precludes the detection of sub-cycle RTL hazards, though protocols such as MSI remain cycle-accurate within the existing framework.
• Scoreboard and queue memory consumption grows with core-count but remains practical for $2L_{eval}$9.
• Host-side resource pressures increase for large designs with high Verilator thread counts, potentially leading to CPU saturation.

Demonstrated scalability extends to 16 cores while maintaining slowdown below 2×. Proposed future enhancements include adaptive synchronization (allowing batched cycles per evaluation), zero-copy communication channels to reduce I/O latency, and finer-grained hybrid scheduling to further reduce $L_{\rm sync} = L_{roe} + L_{wri} + 2L_{eval}$0 and $L_{\rm sync} = L_{roe} + L_{wri} + 2L_{eval}$1.

These findings document the effectiveness and practical scalability of hybrid gem5–Verilator co-simulation as implemented in Rhea, providing a reproducible and extensible foundation for research in efficient RTL memory subsystem validation (Zoni et al., 5 Aug 2025).