CXL-ClusterSim: Modeling CXL-based Disaggregated Memory Cluster for Pooling and Sharing using gem5 and SST

Abstract: Large-scale AI training and inference require hundreds of gigabytes to terabytes of DRAM with high peak to average utilization ratios, resulting in overprovisioning. In cloud computing, DRAM constitutes a significant share of the cost. Yet, as shown by recent articles, DRAM is heavily under utilized. Memory disaggregation is a solution to both these problems. With the advent of the CXL protocol, there is renewed interest in designing and optimizing computing systems with disaggregated memory. However, at present, there are limited simulation tools available for exploring the design space and evaluating the performance tradeoffs in computer systems with disaggregated memory. In this paper, we propose CXL-ClusterSim, a full-system modeling and simulation framework by combining the gem5 simulator for fidelity, with the Structural Simulation Toolkit (SST) for parallel simulation. We outline the challenges in creating this simulation infrastructure and present a design that is scalable, flexible, and reasonably fast to help computer architects to explore the design space of CXL-based disaggregated memory and identify new opportunities for hardware/software codesign and performance optimization.

• The paper introduces a full-system simulation framework using gem5 and SST that models dynamic pooling and sharing in CXL-based disaggregated memory clusters.
• It validates performance tradeoffs by measuring NUMA effects, remote memory bandwidth, injected latency impacts, and cross-ISA behaviors.
• The framework supports hardware/software co-design exploration, enabling optimization of memory stranding, pooling policies, and datacenter efficiency.

CXL-ClusterSim: Full-System Simulation of CXL-Based Disaggregated Memory Clusters

Motivation and Context

The proliferation of memory-intensive workloads in AI and scientific computing has driven infrastructure providers to overprovision DRAM resources, incurring significant operational cost and power inefficiency. Compute Express Link (CXL), leveraging a PCIe-based open standard, enables memory disaggregation wherein compute nodes can dynamically pool and share remote memory resources, mitigating stranded DRAM and optimizing hardware utilization. However, the architecture and system-level behavior of CXL-based disaggregated memory present nontrivial performance and engineering tradeoffs, yet the research community has lacked high-fidelity, scalable full-system simulators for rigorous evaluation. The paper introduces CXL-ClusterSim (2605.27745) as a novel simulation framework combining gem5 and Structural Simulation Toolkit (SST) to address these gaps.

Simulation Infrastructure

CXL-ClusterSim integrates gem5 for detailed CPU, cache, memory, and OS-level modeling, and SST for parallel execution of multi-node clusters including remote memory blades. gem5 models hosts (termed "system nodes"), each with independent cores, caches, OS, and local/remote memory mappings. SST orchestrates cluster-wide simulation via MPI, allowing scalable parallel event scheduling and simulation across distributed nodes.

Figure 1: Overview of the CXL-ClusterSim infrastructure showing multi-node host and remote memory blade simulation via gem5 and SST.

Crucially, CXL-ClusterSim exposes remote memory slices as CPU-less NUMA nodes or as DAX devices (character regions) according to pooling or sharing mode, respectively. The simulator supports heterogeneous ISAs (ARM, RISC-V, x86), precise NUMA placement, and end-to-end application and OS execution for hardware/software codesign experimentation.

Figure 2: System-level view of remote memory ranges, demonstrating NUMA and DAX device exposure in guest OS environments.

Modeling and Methodology

To address the dual challenges of robust initialization (exposing memory topology at boot and runtime) and feasible simulation time under full-system constraints, CXL-ClusterSim employs functional fast-forwarding and checkpointing in gem5 prior to region-of-interest (ROI) timing-accurate simulation with SST. Memory ranges are statically partitioned via E820 (x86) or DTB (ARM/RISC-V), with NUMA affinity defined for OS-level management.

The framework allows flexible configuration of local and remote memory, simulation of injected CXL path latencies (170–250 ns typical), and scaling of system nodes up to 16 hosts with quantitative profiling of cluster bandwidth, memory footprint, and parallel efficiency.

Experimental Validation and Results

Extensive validation is performed using STREAM, NAS Parallel Benchmarks (NPB), and GAPBS workloads, providing both synthetic and real-world coverage.

STREAM: Memory Bandwidth Calibration and Scalability

Single-node bandwidth is measured at remote memory controllers, achieving 59.6 GB/s—77.5% of theoretical DRAM bandwidth—serving as baseline for subsequent experiments.

Figure 3: Baseline and observed memory bandwidth from single-node STREAM runs, validating gem5/SST integration fidelity (<1% deviation).

Multi-node experiments with 8 hosts quantify NUMA local/interleaved/remote pinning effects. Interleaving is heavily constrained by shared remote node bandwidth and link-level contention, with effective bandwidth throttled to 6.45 GiB/s per node in interleaved mode.

Figure 4: Total bandwidth measured at local and remote memory controllers for 8-node cluster under various NUMA policies.

Link latency injection further demonstrates bandwidth sensitivity: increasing CXL latency from 0 to 250 ns leads to a 29% bandwidth drop, mapping performance envelope to realistic device numbers.

Figure 5: STREAM kernel bandwidth as a function of varied injected CXL path latencies across four system nodes.

Parallel scaling analysis for host system resources shows linear increase in global RSS with node count. Parallel efficiency deteriorates at larger cluster sizes due to serialization at remote memory controller, with opportunities for future improvement in SST-level parallelism and congestion management.

Figure 6: Host system statistics (memory footprint, parallel efficiency) for cluster size scaling (1–16 nodes).

Notably, experiments combining ARM and RISC-V hosts confirm the remote memory node's ISA-agnostic behavior, opening avenues for heterogeneous architecture evaluation and comparative analysis.

Figure 7: Bandwidth reported for ARM and RISC-V hosts pooling remote memory, illustrating ISA-independent operation.

NPB: Memory Pooling and Stranding Effects

Simulations of seven NPB workloads on 8-core hosts with varying local/remote memory configurations highlight the tradeoff between IPC (performance) and reduction of stranded DRAM. NUMA-local-preferred setup (8 GiB local, 160 GiB pooled remote) leads to significant IPC drops when >50% of the workload resides in remote memory (e.g. 62% degradation for mg benchmark). However, memory stranding is substantially reduced, demonstrating effective mitigation of wastage and inefficiency.

Figure 8: Relative IPC for NPB workloads versus fraction of working set accessed in remote memory, correlating remote access to performance degradation.

GAPBS: Memory Sharing and Cross-Host Contention

Memory sharing is evaluated using GAPBS kernels accessing a shared synthetic graph via single-writer, multiple-reader DAX-mapped region in the remote memory node. 31.8% of instructions are served from remote memory on average, with performance degradation proportional to workload access patterns and CXL path latency.

Figure 9: Proportion of memory instructions served by local vs. remote memory (normalized per kernel) under multi-host memory sharing.

Performance benchmarks show pointer-chasing kernels (bfs, pr) to be most sensitive to remote access latency and contention. The results demonstrate the framework's capability to accurately capture fabric-level interference and multi-host sharing semantics.

Figure 10: GAPBS kernel IPC in baseline (private memory) vs. DAX-mapped shared memory mode (CXL-ClusterSim), quantifying performance penalty.

Implications and Future Directions

CXL-ClusterSim enables exploration of hardware/software codesign across pooling and sharing modes, memory management policies, NUMA configurations, and heterogeneous host architectures. The platform facilitates rigorous study of system-level effects (OS, application, microarchitecture), remote access optimization, and fabric-level contention previously inaccessible to the community.

Limitations include lack of dynamic memory hot-plugging, incomplete CXL coherence specification integration, and omission of detailed switch/fabric modeling, which are slated for future development. The open-source release strategy (integration with gem5/SST) positions CXL-ClusterSim as a critical asset for architectural exploration and benchmarking.

Potential future directions include integration with hardware-level silicon validation (as in CXL-DMSim [cxl-dmsim]), fabric topology modeling, credit-based flow control, advanced page allocation (Caption), and comparison with industry deployments (e.g., TMTS [tmts], Pond [pond], StarNUMA [starnuma]). The simulator serves as a foundation for evaluation of novel memory tiering, pooling policies, and performance optimization for next-generation cloud and AI systems.

Conclusion

CXL-ClusterSim provides a scalable, full-system modeling framework for CXL-based disaggregated memory clusters offering high-fidelity simulation of pooling, sharing, and host configuration effects. Through integration of gem5 and SST, the platform demonstrates accurate characterization of bandwidth, latency, memory stranding, and performance tradeoffs across synthetic and real workload profiles. By enabling multi-node, multi-ISA, and multi-policy evaluation, CXL-ClusterSim addresses a critical gap in architectural research, with broad potential for guiding hardware-software co-design and datacenter optimization.