---
title: 'CXL-ClusterSim: CXL.mem Memory Simulator'
url: https://www.emergentmind.com/topics/cxl-clustersim
type: topic
---

# CXL-ClusterSim: CXL.mem Memory Simulator

CXL-ClusterSim is a full-system modeling and simulation framework for **CXL.mem-based disaggregated memory clusters** that combines **gem5** for detailed host modeling with the **Structural Simulation Toolkit (SST)** for parallel simulation of multiple nodes and a remote memory blade. It is designed to study **memory pooling** and **memory sharing** under realistic operating-system behavior, including **NUMA policies**, **DAX mappings**, **TLBs**, and **huge pages**, while scaling beyond a single host through SST’s MPI-based orchestration. The framework targets the gap between hardware prototypes that are difficult to modify and lightweight simulators that omit kernel-visible memory-management effects [2605.27745].

## 1. Scope, terminology, and design objectives

CXL-ClusterSim uses a specific vocabulary for the cluster it models. A **system node** is a CXL **host**; a **remote memory node** is CXL **device memory** or a **memory blade**; **local memory** is memory attached to the host; and **remote memory** is host-managed device memory. Within that model, the framework is intended to support both **memory pooling**, where remote capacity is partitioned and assigned to hosts, and **memory sharing**, where multiple hosts access the same remote-memory range [2605.27745].

Its design goals are explicitly twofold. The first is **full-system fidelity**: each host runs in gem5 full-system mode, boots an **unmodified Ubuntu-based Linux distribution**, and exposes remote memory through mechanisms that preserve OS-visible behavior. The second is **cluster-scale exploration**: SST acts as the global simulation driver, advances multiple gem5 instances, and models the shared remote memory node in parallel. This combination lets the framework investigate remote-memory latency, bandwidth limits, NUMA placement, shared-memory contention, and heterogeneous-host configurations, including **mixed ISAs** [2605.27745].

The framework is deliberately scoped to **CXL.mem-based disaggregated memory** rather than the entire CXL protocol stack. The paper is explicit that it does **not** model **CXL.io** configuration and initialization, **CXL.cache**, detailed switch internals, or full host-to-host cache-line coherence for sharing. This suggests that CXL-ClusterSim is optimized for system-level exploration of pooled and shared remote memory, not for low-level protocol-pipeline validation or coherent accelerator studies [2605.27745].

## 2. Simulator composition and execution model

The simulator is partitioned between gem5 and SST. Each **system node** is a separate gem5 instance containing CPUs, caches, local DRAM, an operating system, and an application. The **remote memory node** is modeled in SST. Requests to a host’s remote-memory address range are forwarded out of gem5 through **ExternalMemory**, wrapped in the stdlib as **ExternalRemoteMemory**. This component is based on gem5’s **OutgoingRequestBridge**, but instead of exposing SST as an I/O range, it exposes SST as an **AbstractMemory SimObject**. The backing shared range is then simulated in SST using **memHierarchy** with a **malloc-based backing store** [2605.27745].

SST provides the global event and synchronization engine. The integration uses **SST’s parallel event queue as the main simulation queue**, and SST advances all gem5 instances cycle by cycle. This gives a single coordinated simulation timeline across hosts and the remote memory node. The framework’s central engineering problem is that full-system initialization is expensive and distributed functional memory is difficult to maintain across ranks. To address this, CXL-ClusterSim adopts a **two-phase simulation model with checkpointing**. In **Phase 1**, each node is booted independently in standalone gem5 using **AtomicSimpleCPU** or **KVM CPU**, and a checkpoint is taken at the region of interest. In **Phase 2**, the checkpoints are restored in the integrated gem5+SST environment, gem5 switches to **timing-accurate O3 CPU** and timing memory models, and SST coordinates synchronized detailed execution [2605.27745].

The default evaluated host configuration is an **8-core OoO ARM** system at **4 GHz** with **32 KiB I-cache + 32 KiB D-cache per core**, **512 KiB L2 per core**, **8 MiB shared L3**, **1 local DDR4-2400 channel**, and **8 GB local memory**. The software stack is **Ubuntu 22.04.4** with **Linux kernel 5.4.49**. The remote memory node uses **DDR4 DRAM** with **4 remote channels** and variable capacity per experiment [2605.27745].

## 3. Pooling and sharing semantics

CXL-ClusterSim represents pooled and shared remote memory through two different OS-visible mechanisms. For **pooling**, remote memory appears to Linux as a **CPU-less NUMA node**. On **x86**, the framework exposes this capacity through **E820** entries and **SRAT**; on **ARM/RISC-V**, it uses **Device Tree Blob (DTB)**. Linux then allocates from the remote pool using standard NUMA mechanisms such as `numactl`, allowing experiments with local-only, remote-only, interleaved, and preferred-local placement policies [2605.27745].

For **sharing**, the framework does not expose remote memory as ordinary NUMA DRAM. Instead, it maps the shared remote range as a **DAX character device** (`/dev/dax`), following the **FAMFS** style. The reason given is that if the kernel treated the range as regular system memory, it might **zero the pages** and destroy shared contents. The implemented sharing model is therefore **single-writer, multiple-reader**: one designated allocator host populates the shared data structure in remote memory, other hosts map the same physical contents read-only, and correctness is maintained by software discipline rather than hardware coherence [2605.27745].

The representation of CXL concepts is intentionally simplified. The framework is explicitly scoped to **CXL.mem** and does not model the **Host Physical Address (HPA)** / **Device Physical Address (DPA)** distinction in detail. Instead, it uses a **simplified identity mapping with fixed hardcoded ranges** per system node. Likewise, although CXL 3.0 sharing supports back-invalidate-based coherence, the simulator does **not** model host-to-host cache-line coherence or detailed coherence transactions for shared memory. This suggests that the current sharing mode is appropriate for read-mostly data structures and contention studies, but not for a full coherent shared-memory implementation [2605.27745].

## 4. Calibration, validation, and methodology

The remote memory model is calibrated first in isolation. The SST side models a **DDR4-2400, 4-channel** remote memory node with a peak theoretical bandwidth of **76.8 GB/s**. Using synthetic read traffic from gem5 traffic generators into SST, the paper measures a sustained practical baseline of **59.6 GB/s**, corresponding to **77.5%** of theoretical peak. This value becomes the reference bandwidth for later validation and experiments [2605.27745].

Validation uses **STREAM** with a single system node accessing only remote memory. At the gem5-side external memory interface, reported bandwidth matches the expected baseline to within **less than 1.0%**; at the SST memory controller, the deviation is only **0.1%**. The paper summarizes this as baseline and reported bandwidth being within **0.99%**. This validation is intentionally narrow: it establishes correctness of the gem5–SST forwarding path and the remote-memory bandwidth model, rather than full protocol fidelity [2605.27745].

The evaluation then uses three workload classes. **STREAM** studies bandwidth scaling, NUMA placement, and latency sensitivity with **64 MiB** arrays and the kernels **copy**, **scale**, **add**, and **triad**. **NAS Parallel Benchmarks (NPB), class D** are used for pooling studies, with working-set sizes of **11 GiB** (`bt`), **17 GiB** (`cg`), **1 GiB** (`ep`), **85 GiB** (`ft`), **27 GiB** (`mg`), **12 GiB** (`sp`), and **8 GiB** (`ua`); these experiments use **7 system nodes** and simulate **500 ms** in full-system mode. **GAPBS** studies sharing, with six kernels—`bfs`, `bc`, `cc`, `cc_sv`, `pr`, and `tc`—running on separate hosts against one shared graph image stored once in remote memory [2605.27745].

## 5. Quantitative behavior and system-level implications

The most direct bandwidth result appears in the **8-node STREAM** experiment. With local-only placement, traffic remains almost entirely at local memory controllers. With remote-only placement, traffic is directed solely to the remote node. With interleaving across local and remote memory, effective per-node bandwidth peaks at only **6.45 GiB/s**, compared to **11.4 GiB/s** for local-only execution. The remote node supplies **46.04 GiB/s** aggregate across all 8 nodes, against the calibrated practical maximum of **59.6 GB/s**, so average remote bandwidth per node is only **5.75 GiB/s**. The paper interprets this as the interleaved configuration being bottlenecked by the shared remote node [2605.27745].

Latency sensitivity is quantified in a **4-node** remote-only STREAM study. Relative to the no-extra-latency case, adding **170 ns** of remote path latency reduces bandwidth by **8.95%**; adding **250 ns** reduces bandwidth by **29%**; and **250 ns** causes a further **22%** dip relative to **170 ns**. This shows that remote-memory latency is not merely a fixed additive penalty: it directly reduces usable bandwidth for memory-intensive code paths [2605.27745].

The pooling case study with **NPB** makes the capacity–performance tradeoff explicit. In the **No-NUMA** baseline, each node has **128 GiB** of local memory. In the **NUMA-Local-Preferred** configuration, each node has **8 GiB** local memory plus a **160 GiB shared pool**. Performance degrades as a larger fraction of the working set is accessed remotely. The clearest example is `mg`, whose working set is **27 GiB**: **52%** of its data is accessed in remote memory, its relative IPC falls to **0.38**, and the resulting slowdown is **62%**. At the same time, because `mg` uses only **27 GiB** out of **128 GiB** in the baseline, the paper reports that **79% of stranded local memory is saved** in the NUMA-Local-Preferred setup. By contrast, workloads with little or no remote footprint such as `ep` and `ua` show little IPC loss [2605.27745].

The sharing case study with **GAPBS** shows a distinct access pattern. A single host allocates and populates the shared graph in remote memory, and six other hosts read it concurrently. Across the six kernels, the average fraction of total memory instructions served from remote memory is **31.8%**. IPC declines relative to the private-memory baseline because of the added **250 ns** remote-access latency and contention at the shared remote memory controller. The paper notes that irregular pointer-chasing kernels such as `bfs` and `pr` are more sensitive than more compute-heavy kernels. A heterogeneous-ISA experiment with one **ARM** host and one **RISC-V** host reports similar qualitative pooling behavior, while the **RISC-V** node exploits remote bandwidth **31% more** than the ARM node in that setup [2605.27745].

The framework’s own scalability is bounded by the remote-memory rank. As the number of system nodes grows from **1** to **16**, memory usage on the simulation host increases roughly linearly. For **8 system nodes**, the paper reports **25.3 GiB RSS per system node** and about **73.9 GiB** global RSS; for **16 system nodes**, it reports **145.96 GiB** global RSS. Parallel efficiency declines from **0.38** in a **2-node cluster** to **0.06** in a **16-node cluster**, but at **17 host processes**—**16 nodes + 1 remote rank**—the framework still achieves about **1.09× speedup** over the serial version. This suggests that the current implementation is practically useful for moderate node counts, but increasingly bottlenecked by the centralized remote-memory component [2605.27745].

## 6. Position in the CXL simulation landscape, limitations, and extensions

CXL-ClusterSim occupies a specific position among recent CXL simulators. **CXL-DMSim** is an open-source, full-system, gem5-based simulator for **single-host CXL disaggregated memory**, focused on Type-3 memory expansion and validated against FPGA- and ASIC-based devices, but it does **not** model multi-host switching, memory sharing, or cluster orchestration [2411.02282]. **CXLRAMSim** is a gem5-integrated full-system simulator for **CXL memory expander cards** that emphasizes correct placement of CXL devices on the **I/O bus** and support for **unmodified Linux kernels**, but its scope remains node-level rather than cluster-level [2603.29483]. By contrast, CXL-ClusterSim is explicitly multi-host and full-system, but narrows its focus to **CXL.mem-based** pooling and sharing with SST-mediated remote memory rather than detailed protocol modeling [2605.27745].

This division of labor is reinforced by broader fabric-oriented work. **ESF** targets **multi-level switching**, **port-based routing**, **non-tree topologies**, and **device-managed coherence**, making it closer to a protocol- and topology-oriented CXL fabric simulator than to CXL-ClusterSim’s OS-visible pooling/sharing framework [2411.08312]. CXL-ClusterSim instead prioritizes **NUMA behavior**, **DAX mapping**, **checkpointed full-system execution**, and **multi-host contention at a shared remote memory blade**. A plausible implication is that the two approaches are complementary: ESF-like models are suited to switch and routing studies, whereas CXL-ClusterSim is suited to end-to-end studies of Linux-visible disaggregated memory behavior.

The limitations of CXL-ClusterSim are explicit. It models **only CXL.mem**; it does **not** include detailed **CXL.io** or **CXL.cache** support. It does **not** implement full coherence for sharing, instead restricting the current sharing model to **single-writer, multiple-reader**. Address ranges are **statically partitioned** and **hardcoded**, with no dynamic memory hot-plugging or fabric-manager-driven rebinding. It also does **not** include a detailed switch or fabric model, detailed routing or credit-based flow control, or a full HPA↔DPA mapping. The paper identifies future work in **dynamic memory management via hot-plugging**, incorporation of **CXL coherence specification details**, **CXL switch/fabric modeling**, richer **CXL table structures**, improved **parallel efficiency**, and possible extension to **CXL.io**- and **CXL.cache**-based systems [2605.27745].

Source: https://www.emergentmind.com/topics/cxl-clustersim