Memory Disaggregation via CXL.mem

• Memory disaggregation via CXL.mem is a paradigm that decouples physical memory from compute nodes, creating shared, coherent, byte-addressable pools over PCIe links.
• It enables dynamic pooling and interleaving of memory resources using NUMA abstractions and cache-coherent transaction protocols for efficient load/store operations.
• This architecture improves scalability and performance for applications like machine learning and in-memory databases while addressing latency, bandwidth, and security challenges.

Memory disaggregation via Compute Express Link (CXL.mem) refers to the architectural paradigm in which physical memory resources are decoupled from individual compute nodes and exposed as shared, byte-addressable pools over a coherent interconnect. This model enables dynamic, fine-grained composition of memory capacity, bandwidth, and performance isolation at the datacenter and rack scale, supporting a range of emerging applications including large-scale machine learning, high-performance computing, in-memory databases, and composable infrastructure. CXL.mem is realized through the CXL protocol stack layered on contemporary PCIe physical links, providing low-overhead, cache-coherent, load/store semantics for host CPUs and accelerators attached to memory expander cards, switches, or other CXL-compliant devices (Pathak et al., 31 Mar 2026).

1. Architectural Principles of CXL.mem Disaggregation

CXL.mem disaggregation is architected around modular hardware components and an integrated protocol hierarchy. The physical address space of a host is extended transparently such that both on-chip DRAM (typically via DIMMs directly controlled by the CPU) and remote CXL.mem-backed regions are simultaneously accessible by the CPU’s MMU and cache coherence engine. At the hardware level, a CXL Root Complex (RC), placed downstream of the PCIe hierarchy, packetizes memory operations from the CPU and forwards them over dedicated CXL.mem links to End Point (EP) devices—typically memory expander cards holding clusters of DRAM or persistent memory chips (Pathak et al., 31 Mar 2026, Jain et al., 2024).

To the operating system, CXL memory regions can appear as CPU-less NUMA nodes (“zNUMA”), which allows fine-grained, page-level interleaving between host and CXL-attached memory. BIOS, ACPI, and kernel-level support is required for discovery, enumeration, and affinity mapping (e.g., via MCFG and DSDT tables, SRAT) but commodity Linux kernels (≥ 6.14) and user-level NUMA libraries remain unmodified (Pathak et al., 31 Mar 2026). The protocol stack comprises:

• PCIe PHY and data link (physical transport)
• CXL link and arbitration layer (flitization, credit-based flow control)
• CXL.mem transaction layer (load/store semantics, coherence enforcement across hosts and devices)

These collectively ensure that remote memory can be addressed at cache-line granularity, while maintaining coherence with local CPU caches (Pathak et al., 31 Mar 2026, Jain et al., 2024).

2. CXL.mem Transaction Protocols and Performance Modeling

The CXL.mem protocol utilizes a structured set of packets to support memory semantics over the PCIe fabric. The main transaction types include:

• M2S (Memory-to-Slave) load and store requests
• S2M (Slave-to-Memory-Controller) data responses (DRS)
• S2M No-Data Responses (NDR) for write acknowledgments

The transaction pipeline models performance using:

• Bandwidth: For a link with width $W$, rate $f$, and encoding efficiency $\eta$, $B = W \times f \times \eta$ (e.g., $\eta \approx 128/130$ for PCIe Gen5) (Pathak et al., 31 Mar 2026, Jain et al., 2024).
• Latency: The end-to-end memory access latency is decomposed as:

$$L_{\text{total}} = L_{\text{packet}} + L_{\text{link}} + L_{\text{queue}} + L_{\text{DRAM}} + L_{\text{depacket}}$$

where $L_{\text{link}}$ is proportional to physical propagation and serialization, and $L_{\text{queue}}$ reflects resource contention. For representative systems, one-way CXL.mem latency is approximately 200 ns (vs. 80 ns for local DRAM), and bandwidth is $\sim80$ GB/s (vs. 120 GB/s for host DRAM) (Pathak et al., 31 Mar 2026). Additional penalties are observed for higher contention or when device internal DRAM differs in speed (Wang et al., 2024).

CXLRAMSim, a gem5-based simulator, exposes all protocol latencies and provides architectural positioning of devices, enabling realistic comparisons/calibration to hardware (Pathak et al., 31 Mar 2026). CXL-DMSim targets cycle-accurate modeling with empirically bounded error (4.1%), capturing full round-trip timing and the effects of queueing and resource contention (Wang et al., 2024).

3. Pooling, Interleaving, and Software Abstractions

Memory disaggregation with CXL.mem supports both capacity and bandwidth pooling. This is realized by exposing memory expander devices as NUMA domains and allowing the OS’s page allocator to stripe allocations between local DRAM and CXL-attached memory at configurable ratios (e.g., 70/30, 50/50). Stripe-based interleaving smooths the performance transition as hot pages migrate, avoiding performance cliffs typical of tiered systems with abrupt transitions (Pathak et al., 31 Mar 2026, Wahlgren et al., 2022). The equations describing effective average latency and bandwidth are:

$$L_{\text{avg}}(f) = (1-f) L_{\text{local}} + f L_{\text{remote}}$$

$f$0

where $f$1 is the remote memory fraction (Wahlgren et al., 2022).

Enablers for memory pooling include hardware support for interleavening (NUMA/zNUMA), ACPI tables that annotate remote regions, and userspace tools (NDCTL, CXL-CLI, SMDK, UMF, etc.) running unchanged atop kernel CXL drivers (Pathak et al., 31 Mar 2026). Software frameworks may extend this via custom allocation/migration policies, user-space APIs (e.g., emucxl_alloc(), migrate()), and support for hybrid policies like “migrate on access” (Gond et al., 2024).

For distributed, multi-host fabrics, OpenSHMEM/PGAS models and custom runtime software provide primitives for symmetric allocation, Remote Memory Access (RMA), atomics, and collectives built atop CXL.mem (Jain et al., 2024).

4. Performance, Scalability, and Microarchitectural Challenges

Quantitative evaluation across simulation and prototype hardware consistently identifies:

Metric	Local DRAM	CXL.mem Only	50/50 Interleave
Bandwidth (GB/s)	115	75	95
One-way Latency (ns)	80	200	–
LLC Miss Rate (%)	5	18	11–12 (linear)

CXL.mem access is %%%%12$\eta$13%%%% higher in latency and $f$430\% lower in peak bandwidth versus local DRAM. Under high concurrent access, queuing at the Root Complex and increased LLC pollution from remote lines further degrade throughput (Pathak et al., 31 Mar 2026, Wang et al., 2024, Sun et al., 2023). Bandwidth for CXL-attached DRAM is empirically 45–83% that of local DRAM, with pointer-chasing and bandwidth-bound kernels most affected.

Pool-sharing among multiple hosts introduces additional performance variability due to head-of-line blocking, as confirmed by pool bandwidth falling from $f$5 GB/s (1 host) to $f$6 GB/s (3 hosts), and calls for hardware or OS-enforced per-host bandwidth reservations (Wahlgren et al., 2022).

Scalability of CXL-based memory pools is governed by switch topology (e.g., leaf/spine, mesh/ring), port/radix of CXL switches and expanders, and the efficiency of hardware and software coherence mechanisms. Higher-tier CXL 3.0/3.1 fabrics with multi-hop, dynamic address mapping, and hardware Back Invalidation support can scale to rack-level and beyond, although snoop filter size and topology-induced contention remain research challenges (Jain et al., 2024).

5. Use Cases, Applications, and Advanced Deployment Models

Memory disaggregation via CXL.mem finds adoption in several high-impact use cases:

• LLM training and inference: CXL.mem enables scale-up systems to avoid memory stranding and overflow beyond on-chip DRAM capacity, providing transparent memory for large embeddings and intermediate data (Pathak et al., 31 Mar 2026). ScalePool demonstrates up to 1.84× acceleration of LLM training compared to RDMA, due to lower latency remote pooling (Woo et al., 16 Oct 2025).
• Fault-tolerant training: TRAININGCXL illustrates persistent memory disaggregation for GPU-addressable PMEM, leveraging CXL Type-2 devices, near-data logging, and relaxed checkpoint sequencing to achieve 5.2× speedup, 76% energy reduction (Kwon et al., 2023).
• Persistent memory and storage convergence: Architectures transforming PCIe SSDs into CXL.mem Type-3 endpoints (“CXL-SSD”) achieve near-DRAM performance under high locality and offer instruction-level semantic annotations for persistence and determinism (Kwon et al., 18 Jun 2025). FPGA-based CXL.mem endpoints prototype practical persistence models with benchmarks showing CXL-attached DDR4 exceeding Optane DCPMM bandwidth by 2–3× (Fridman et al., 2023).
• Tiered memory and resource control: Dynamic Memory Request Control (MIKU) prioritizes DDR requests and throttles CXL traffic to ensure fair bandwidth and low tail latency for hybrid workloads, restoring >90% of DDR throughput even under heavy CXL contention (Yang et al., 22 Mar 2025).
• Emulation and simulation frameworks: Emucxl, CXLRAMSim, CXL-DMSim, and CXLMemSim enable principled design-space exploration, policy development, and performance evaluation prior to widespread hardware availability (Pathak et al., 31 Mar 2026, Wang et al., 2024, Gond et al., 2024, Yang et al., 2023).

6. Future Directions, Challenges, and Security

Research identifies several ongoing and open challenges in CXL.mem-enabled disaggregation:

• Hardware-software coherence scaling: Directory state explosion and snoop filter scalability motivate hybrid hardware-software approaches (e.g., precise tracking for synchronized regions, software replication for bulk data) (Jain et al., 2024).
• Placement and migration: Dynamic, latency-aware, or workload-driven allocation of hot pages to DRAM and cold pages to CXL regions is essential. Policies such as Caption (dynamic interleave adjustment) deliver up to 24% throughput improvements in bandwidth-bound applications (Sun et al., 2023).
• Security and isolation: The absence of process-level isolation in pooled memory spaces leaves provisioning vulnerable to intra-host attacks. Space-Control addresses this with fine-grained, hardware-managed tagging, per-process authentication, and permission caching, achieving only 3.3% runtime overhead (Goswami et al., 6 Mar 2026).
• Programming models and durability: Novel models (e.g., CXL0) formalize operational semantics for concurrent memory accesses, define transformations for algorithmic crash-consistency, and provide a foundation for software correctness under CXL’s partially asynchronous, disaggregated failure domains (Assa et al., 2024).
• Cross-rack and multi-fabric deployment: Layering CXL.mem on Ethernet or integrating with in-rack fabrics supports low-latency, rack-spanning pools with latencies of 1.97 μs (uncached) and 415 ns (cache hits), outperforming state-of-the-art RDMA while requiring no application changes (Wang et al., 2023).
• Multi-tenant fairness: Queuing, dynamic bandwidth partitioning, and policy-compliant mapping are needed for co-scheduling memory-bound jobs, extending current scheduler and OS abstractions to include memory pool awareness (Wahlgren et al., 2022).

7. Summary Table: Primary CXL.mem Disaggregation Attributes

Attribute	Characteristic	Reference
Protocol Layer	PCIe physical/data-link, CXL.mem transaction	(Pathak et al., 31 Mar 2026)
Coherence	MESI (hardware or hybrid), page/directory tracking	(Pathak et al., 31 Mar 2026, Jain et al., 2024)
DRAM vs CXL.mem (lat/BW)	80 ns/120 GB/s vs. 200 ns/80 GB/s (typical, programmable)	(Pathak et al., 31 Mar 2026, Wang et al., 2024)
Pooling Unit	Type-3 endpoint, multi-headed/switch, NUMA exposure	(Jain et al., 2024, Berger et al., 15 Jan 2025)
Software Model	NUMA node, zNUMA, OS-local, library API, OpenSHMEM/PGAS	(Pathak et al., 31 Mar 2026, Gond et al., 2024)
Security/Isolation	Host-level (default), process-level via Space-Control	(Goswami et al., 6 Mar 2026)

Widely, CXL.mem-based memory disaggregation enables flexible, performance-scalable, and software-transparent extension of memory resources, with a growing ecosystem of hardware, emulators, and OS/runtime support. As system researchers move toward deploying next-generation scale-up and scale-out architectures, CXL.mem is central to balancing utilization, elasticity, and isolation in disaggregated infrastructure (Pathak et al., 31 Mar 2026, Jain et al., 2024, Wang et al., 2024).