CXL-ClusterSim: CXL.mem Memory Simulator

• CXL-ClusterSim is a full-system simulation framework that models disaggregated CXL.mem clusters, focusing on memory pooling and sharing with detailed host and remote memory modeling.
• It integrates gem5’s full-system host simulation with SST’s parallel multi-node orchestration, achieving high fidelity in replicating Linux-visible memory behaviors.
• The simulator supports realistic OS-level features like NUMA policies, DAX mappings, and checkpointed execution to analyze bandwidth, latency, and contention effects.

CXL-ClusterSim is a full-system modeling and simulation framework for CXL.mem-based disaggregated memory clusters that combines gem5 for detailed host modeling with the Structural Simulation Toolkit (SST) for parallel simulation of multiple nodes and a remote memory blade. It is designed to study memory pooling and memory sharing under realistic operating-system behavior, including NUMA policies, DAX mappings, TLBs, and huge pages, while scaling beyond a single host through SST’s MPI-based orchestration. The framework targets the gap between hardware prototypes that are difficult to modify and lightweight simulators that omit kernel-visible memory-management effects (Goswami et al., 26 May 2026).

1. Scope, terminology, and design objectives

CXL-ClusterSim uses a specific vocabulary for the cluster it models. A system node is a CXL host; a remote memory node is CXL device memory or a memory blade; local memory is memory attached to the host; and remote memory is host-managed device memory. Within that model, the framework is intended to support both memory pooling, where remote capacity is partitioned and assigned to hosts, and memory sharing, where multiple hosts access the same remote-memory range (Goswami et al., 26 May 2026).

Its design goals are explicitly twofold. The first is full-system fidelity: each host runs in gem5 full-system mode, boots an unmodified Ubuntu-based Linux distribution, and exposes remote memory through mechanisms that preserve OS-visible behavior. The second is cluster-scale exploration: SST acts as the global simulation driver, advances multiple gem5 instances, and models the shared remote memory node in parallel. This combination lets the framework investigate remote-memory latency, bandwidth limits, NUMA placement, shared-memory contention, and heterogeneous-host configurations, including mixed ISAs (Goswami et al., 26 May 2026).

The framework is deliberately scoped to CXL.mem-based disaggregated memory rather than the entire CXL protocol stack. The paper is explicit that it does not model CXL.io configuration and initialization, CXL.cache, detailed switch internals, or full host-to-host cache-line coherence for sharing. This suggests that CXL-ClusterSim is optimized for system-level exploration of pooled and shared remote memory, not for low-level protocol-pipeline validation or coherent accelerator studies (Goswami et al., 26 May 2026).

2. Simulator composition and execution model

The simulator is partitioned between gem5 and SST. Each system node is a separate gem5 instance containing CPUs, caches, local DRAM, an operating system, and an application. The remote memory node is modeled in SST. Requests to a host’s remote-memory address range are forwarded out of gem5 through ExternalMemory, wrapped in the stdlib as ExternalRemoteMemory. This component is based on gem5’s OutgoingRequestBridge, but instead of exposing SST as an I/O range, it exposes SST as an AbstractMemory SimObject. The backing shared range is then simulated in SST using memHierarchy with a malloc-based backing store (Goswami et al., 26 May 2026).

SST provides the global event and synchronization engine. The integration uses SST’s parallel event queue as the main simulation queue, and SST advances all gem5 instances cycle by cycle. This gives a single coordinated simulation timeline across hosts and the remote memory node. The framework’s central engineering problem is that full-system initialization is expensive and distributed functional memory is difficult to maintain across ranks. To address this, CXL-ClusterSim adopts a two-phase simulation model with checkpointing. In Phase 1, each node is booted independently in standalone gem5 using AtomicSimpleCPU or KVM CPU, and a checkpoint is taken at the region of interest. In Phase 2, the checkpoints are restored in the integrated gem5+SST environment, gem5 switches to timing-accurate O3 CPU and timing memory models, and SST coordinates synchronized detailed execution (Goswami et al., 26 May 2026).

The default evaluated host configuration is an 8-core OoO ARM system at 4 GHz with 32 KiB I-cache + 32 KiB D-cache per core, 512 KiB L2 per core, 8 MiB shared L3, 1 local DDR4-2400 channel, and 8 GB local memory. The software stack is Ubuntu 22.04.4 with Linux kernel 5.4.49. The remote memory node uses DDR4 DRAM with 4 remote channels and variable capacity per experiment (Goswami et al., 26 May 2026).

3. Pooling and sharing semantics

CXL-ClusterSim represents pooled and shared remote memory through two different OS-visible mechanisms. For pooling, remote memory appears to Linux as a CPU-less NUMA node. On x86, the framework exposes this capacity through E820 entries and SRAT; on ARM/RISC-V, it uses Device Tree Blob (DTB). Linux then allocates from the remote pool using standard NUMA mechanisms such as numactl, allowing experiments with local-only, remote-only, interleaved, and preferred-local placement policies (Goswami et al., 26 May 2026).

For sharing, the framework does not expose remote memory as ordinary NUMA DRAM. Instead, it maps the shared remote range as a DAX character device (/dev/dax), following the FAMFS style. The reason given is that if the kernel treated the range as regular system memory, it might zero the pages and destroy shared contents. The implemented sharing model is therefore single-writer, multiple-reader: one designated allocator host populates the shared data structure in remote memory, other hosts map the same physical contents read-only, and correctness is maintained by software discipline rather than hardware coherence (Goswami et al., 26 May 2026).

The representation of CXL concepts is intentionally simplified. The framework is explicitly scoped to CXL.mem and does not model the Host Physical Address (HPA) / Device Physical Address (DPA) distinction in detail. Instead, it uses a simplified identity mapping with fixed hardcoded ranges per system node. Likewise, although CXL 3.0 sharing supports back-invalidate-based coherence, the simulator does not model host-to-host cache-line coherence or detailed coherence transactions for shared memory. This suggests that the current sharing mode is appropriate for read-mostly data structures and contention studies, but not for a full coherent shared-memory implementation (Goswami et al., 26 May 2026).

4. Calibration, validation, and methodology

The remote memory model is calibrated first in isolation. The SST side models a DDR4-2400, 4-channel remote memory node with a peak theoretical bandwidth of 76.8 GB/s. Using synthetic read traffic from gem5 traffic generators into SST, the paper measures a sustained practical baseline of 59.6 GB/s, corresponding to 77.5% of theoretical peak. This value becomes the reference bandwidth for later validation and experiments (Goswami et al., 26 May 2026).

Validation uses STREAM with a single system node accessing only remote memory. At the gem5-side external memory interface, reported bandwidth matches the expected baseline to within less than 1.0%; at the SST memory controller, the deviation is only 0.1%. The paper summarizes this as baseline and reported bandwidth being within 0.99%. This validation is intentionally narrow: it establishes correctness of the gem5–SST forwarding path and the remote-memory bandwidth model, rather than full protocol fidelity (Goswami et al., 26 May 2026).

The evaluation then uses three workload classes. STREAM studies bandwidth scaling, NUMA placement, and latency sensitivity with 64 MiB arrays and the kernels copy, scale, add, and triad. NAS Parallel Benchmarks (NPB), class D are used for pooling studies, with working-set sizes of 11 GiB (bt), 17 GiB (cg), 1 GiB (ep), 85 GiB (ft), 27 GiB (mg), 12 GiB (sp), and 8 GiB (ua); these experiments use 7 system nodes and simulate 500 ms in full-system mode. GAPBS studies sharing, with six kernels—bfs, bc, cc, cc_sv, pr, and tc—running on separate hosts against one shared graph image stored once in remote memory (Goswami et al., 26 May 2026).

5. Quantitative behavior and system-level implications

The most direct bandwidth result appears in the 8-node STREAM experiment. With local-only placement, traffic remains almost entirely at local memory controllers. With remote-only placement, traffic is directed solely to the remote node. With interleaving across local and remote memory, effective per-node bandwidth peaks at only 6.45 GiB/s, compared to 11.4 GiB/s for local-only execution. The remote node supplies 46.04 GiB/s aggregate across all 8 nodes, against the calibrated practical maximum of 59.6 GB/s, so average remote bandwidth per node is only 5.75 GiB/s. The paper interprets this as the interleaved configuration being bottlenecked by the shared remote node (Goswami et al., 26 May 2026).

Latency sensitivity is quantified in a 4-node remote-only STREAM study. Relative to the no-extra-latency case, adding 170 ns of remote path latency reduces bandwidth by 8.95%; adding 250 ns reduces bandwidth by 29%; and 250 ns causes a further 22% dip relative to 170 ns. This shows that remote-memory latency is not merely a fixed additive penalty: it directly reduces usable bandwidth for memory-intensive code paths (Goswami et al., 26 May 2026).

The pooling case study with NPB makes the capacity–performance tradeoff explicit. In the No-NUMA baseline, each node has 128 GiB of local memory. In the NUMA-Local-Preferred configuration, each node has 8 GiB local memory plus a 160 GiB shared pool. Performance degrades as a larger fraction of the working set is accessed remotely. The clearest example is mg, whose working set is 27 GiB: 52% of its data is accessed in remote memory, its relative IPC falls to 0.38, and the resulting slowdown is 62%. At the same time, because mg uses only 27 GiB out of 128 GiB in the baseline, the paper reports that 79% of stranded local memory is saved in the NUMA-Local-Preferred setup. By contrast, workloads with little or no remote footprint such as ep and ua show little IPC loss (Goswami et al., 26 May 2026).

The sharing case study with GAPBS shows a distinct access pattern. A single host allocates and populates the shared graph in remote memory, and six other hosts read it concurrently. Across the six kernels, the average fraction of total memory instructions served from remote memory is 31.8%. IPC declines relative to the private-memory baseline because of the added 250 ns remote-access latency and contention at the shared remote memory controller. The paper notes that irregular pointer-chasing kernels such as bfs and pr are more sensitive than more compute-heavy kernels. A heterogeneous-ISA experiment with one ARM host and one RISC-V host reports similar qualitative pooling behavior, while the RISC-V node exploits remote bandwidth 31% more than the ARM node in that setup (Goswami et al., 26 May 2026).

The framework’s own scalability is bounded by the remote-memory rank. As the number of system nodes grows from 1 to 16, memory usage on the simulation host increases roughly linearly. For 8 system nodes, the paper reports 25.3 GiB RSS per system node and about 73.9 GiB global RSS; for 16 system nodes, it reports 145.96 GiB global RSS. Parallel efficiency declines from 0.38 in a 2-node cluster to 0.06 in a 16-node cluster, but at 17 host processes—16 nodes + 1 remote rank—the framework still achieves about 1.09× speedup over the serial version. This suggests that the current implementation is practically useful for moderate node counts, but increasingly bottlenecked by the centralized remote-memory component (Goswami et al., 26 May 2026).

6. Position in the CXL simulation landscape, limitations, and extensions

CXL-ClusterSim occupies a specific position among recent CXL simulators. CXL-DMSim is an open-source, full-system, gem5-based simulator for single-host CXL disaggregated memory, focused on Type-3 memory expansion and validated against FPGA- and ASIC-based devices, but it does not model multi-host switching, memory sharing, or cluster orchestration (Wang et al., 2024). CXLRAMSim is a gem5-integrated full-system simulator for CXL memory expander cards that emphasizes correct placement of CXL devices on the I/O bus and support for unmodified Linux kernels, but its scope remains node-level rather than cluster-level (Pathak et al., 31 Mar 2026). By contrast, CXL-ClusterSim is explicitly multi-host and full-system, but narrows its focus to CXL.mem-based pooling and sharing with SST-mediated remote memory rather than detailed protocol modeling (Goswami et al., 26 May 2026).

This division of labor is reinforced by broader fabric-oriented work. ESF targets multi-level switching, port-based routing, non-tree topologies, and device-managed coherence, making it closer to a protocol- and topology-oriented CXL fabric simulator than to CXL-ClusterSim’s OS-visible pooling/sharing framework (An et al., 2024). CXL-ClusterSim instead prioritizes NUMA behavior, DAX mapping, checkpointed full-system execution, and multi-host contention at a shared remote memory blade. A plausible implication is that the two approaches are complementary: ESF-like models are suited to switch and routing studies, whereas CXL-ClusterSim is suited to end-to-end studies of Linux-visible disaggregated memory behavior.

The limitations of CXL-ClusterSim are explicit. It models only CXL.mem; it does not include detailed CXL.io or CXL.cache support. It does not implement full coherence for sharing, instead restricting the current sharing model to single-writer, multiple-reader. Address ranges are statically partitioned and hardcoded, with no dynamic memory hot-plugging or fabric-manager-driven rebinding. It also does not include a detailed switch or fabric model, detailed routing or credit-based flow control, or a full HPA↔DPA mapping. The paper identifies future work in dynamic memory management via hot-plugging, incorporation of CXL coherence specification details, CXL switch/fabric modeling, richer CXL table structures, improved parallel efficiency, and possible extension to CXL.io- and CXL.cache-based systems (Goswami et al., 26 May 2026).