Memory Disaggregation in Data Centers

• Memory disaggregation is the decoupling of compute and memory resources, allowing elastic sharing across nodes in data centers.
• It employs high-speed interconnects like RDMA, CXL, and OpenCAPI to provide transparent load/store operations with low latency.
• This approach enhances resource utilization, scalability, and performance in HPC, cloud computing, and big data analytics.

Memory disaggregation is the architectural and system-level decoupling of compute and memory resources in modern data centers and high-performance computing (HPC) environments. By separating DRAM from CPU sockets and enabling compute nodes to access remote memory pools over high-speed interconnects, memory disaggregation addresses memory underutilization, resource stranding, and inflexible scaling. Through hardware, architectural, and software advances, this paradigm transforms memory from a statically-attached, per-server resource into a shared, elastic pool available to any node within a cluster or rack (Abrahamse et al., 2022, Wang et al., 26 Mar 2025, Puri et al., 2023, Ding et al., 2023).

1. Architectural Foundations and System Models

The core principle of memory disaggregation is the creation of logically or physically separated pools for compute (CPUs/GPUs) and memory (DRAM, NVRAM), connected via a high-speed fabric capable of supporting load/store semantics at cache-line or page granularity (Wang et al., 26 Mar 2025, Abrahamse et al., 2022). There are two canonical deployment models:

• Software-disaggregated clusters: Commodity servers expose their local unused DRAM as part of a cluster-wide pool, accessed via kernel/user-level software (e.g., Infiniswap, Memtrade, FluidMem) (Yelam, 2022, Maruf et al., 2021, Caldwell et al., 2017).
• Hardware-disaggregated architectures: Dedicated memory "blades" or memory nodes offer large banks of DRAM accessed directly by compute nodes via protocols such as RDMA, OpenCAPI, CXL, or Gen-Z (Wang et al., 26 Mar 2025, Wang et al., 2023).

Interconnects are critical:

• RDMA enables zero-copy, one-sided memory operations with ~1–2 μs latency.
• CXL and OpenCAPI provide cache-coherent, byte-addressable access with sub-microsecond latencies (e.g., 200–500 ns in CXL 3.0) and PCIe-level throughput (Wang et al., 2023, Yang et al., 22 Mar 2025, Wang et al., 26 Mar 2025).
• EDM demonstrates an ultra-low-latency Ethernet fabric for remote memory by implementing network protocols at the PHY layer, reducing remote-read access to ~300 ns (Su et al., 2024).

Addressing schemes allow remote DRAM to be mapped as local NUMA nodes, supporting transparent load/store semantics (Wang et al., 2023, Wang et al., 26 Mar 2025). This design is extensible from rack-scale (single ToR switch) to pod-scale (multi-rack) deployments, with hop-count and bandwidth scaling considerations.

2. Memory Disaggregation Interfaces and Programming Models

Memory disaggregation exposes remote memory at multiple layers:

• Block-level pooling: Remote memory as fast block devices (e.g., NVMe-over-Fabrics). Efficient for capacity scaling, but incurs block I/O and lacks fine-grained load/store (Wang et al., 26 Mar 2025).
• Byte-addressable (far) memory: Direct load/store semantics via RDMA, CXL.mem, or OpenCAPI, supporting cache-line or page-level accesses mapped into the global address space (Abrahamse et al., 2022, Wang et al., 2023).

Exposure to software is varied:

• Page-based interfaces: E.g., swap-device integration (Infiniswap, FluidMem), allowing transparent extension of virtual memory (Caldwell et al., 2017, Yelam, 2022).
• Object-level APIs: Extended key-value stores or in-memory object stores (Plasma over ThymesisFlow) that span local and distributed memory pools (Abrahamse et al., 2022).
• Application-aware and custom APIs: Systems like DOLMA and Farview operate at the data-object or buffer-cache level and expose explicit prefetch or placement controls (Zheng et al., 2 Dec 2025, Korolija et al., 2021).

System software may be required to handle page faults, manage allocation, and support backward compatibility, while advanced frameworks enable direct user-space steering of remote memory (e.g., via custom mmaps or ioctls) (Guo et al., 2021, Yelam, 2022, Abrahamse et al., 2022).

3. Performance Models, Bottlenecks, and Empirical Results

Memory disaggregation introduces increased access latency and reduced bandwidth for remote memory. Core quantitative performance models decompose remote-access cost into:

$$T_{\mathrm{remote}} = T_{\mathrm{prop}} + T_{\mathrm{trans}} + T_{\mathrm{queue}} + T_{\mathrm{service}}$$

where $T_{\mathrm{prop}}$ is propagation delay, $T_{\mathrm{trans}}$ transmission, $T_{\mathrm{queue}}$ queuing, and $T_{\mathrm{service}}$ DRAM access (Puri et al., 2023, Wang et al., 26 Mar 2025, Yang et al., 22 Mar 2025).

Illustrative empirical results:

• ThymesisFlow (POWER9+OpenCAPI): Local bandwidth ~6.5 GiB/s vs. remote ~5.75 GiB/s (~11.5% penalty). Remote read latency increases from ~75 μs (local) to ~2.6–5.0 ms (remote), especially for small objects due to metadata and gRPC overhead (Abrahamse et al., 2022).
• Rack-scale simulation: Smart-Idle pool selection reduces mean remote-access latency to 200–350 ns, cutting long tail events (>1 µs) by >90% compared to naïve random allocation (Puri et al., 2023).
• CXL disaggregation: Remote memory can achieve 1.97 μs uncached access over Ethernet, with local FPGA caching reducing this to 415 ns (Wang et al., 2023).
• EDM: Implements in-PHY protocol, demonstrating ≈300 ns unloaded read/write access over commodity Ethernet, approaching on-board CXL performance (Su et al., 2024).

Bottlenecks include:

• Limited hardware-level parallelism on CXL devices (fewer banks than DDR), leading to increased queuing latency under multicore concurrency (Yang et al., 22 Mar 2025).
• Unfair queuing in processor uncore (CHA/CCX) request tables can cause DDR bandwidth to drop by up to 81% when contended by CXL (Yang et al., 22 Mar 2025).
• Tail latency and bandwidth sensitivity to allocation/pool selection policies: Load balancing and interference-aware algorithms (Smart-Idle, DOLMA’s MRC-driven object placement) are crucial to avoid hotspotting and achieve acceptable application performance (Zheng et al., 2 Dec 2025, Puri et al., 2023, Wang et al., 26 Mar 2025).
• Cache coherence and protocol overhead when sharing memory among multiple compute blades, with coherence transitions (multicast invalidations) impacting latency (Lee et al., 2021).

4. System Software, Page Management, and Orchestration Strategies

Disaggregated memory subsystems require multi-level management, combining OS kernels, user-space marshaling, and fabric-wide resource orchestrators:

• Metadata and allocation: Systems like Plasma over ThymesisFlow maintain cluster-wide unique object IDs and object directories synchronized via lightweight RPC/gRPC. Consistency and lookup latency depend on directory protocol design (Abrahamse et al., 2022).
• Page migration and locality optimization: INDIGO implements network-aware page migration, using per-page telemetry (access frequency and burst duration) and a learning-based (contextual bandit) policy to minimize migration costs under variable network congestion. This reduces application runtime by 50–70% compared to conventional page migration (Patke et al., 23 Mar 2025).
• Object and buffer-level management: DOLMA formulates and solves an optimization for local-vs-remote object allocation using measured hit-rate curves and solves for local DRAM allocation per object to minimize overall memory-access time given performance constraints (Zheng et al., 2 Dec 2025).
• Cloud elasticity: FluidMem and Memtrade harvest idle or underutilized DRAM from VMs, exposing it as either page-level swap (FluidMem using userfaultfd) or as a trusted marketplace in which VMs rent/lease remote memory under performance-aware, brokered arrangements (Memtrade) (Caldwell et al., 2017, Maruf et al., 2021).

Orchestration frameworks such as LegoOS (splitkernel design), Memtrade (marketplace), and HyFarM (hybrid far memory scheduler) enable cluster-wide memory allocation, migration, and job scheduling that is sensitive to interference and workload characteristics (Wang et al., 26 Mar 2025, Maruf et al., 2021).

Security and isolation are addressed using hardware-hardened access checks (FPGA-based, page-granular ACLs and owners in TDMem), encrypted transmission (AES-GCM), and randomized remapping for oblivious access-patterns (Heo et al., 2021).

5. Application Domains and Workload Characterization

Memory disaggregation is evaluated and adopted in multiple domains:

• Big Data Analytics: Distributed in-memory object stores (Arrow Plasma over ThymesisFlow) extend familiar APIs (e.g., Spark RDDs over Arrow), enabling large-scale data shuffling and wide-join operators without redundant network copies (Abrahamse et al., 2022).
• High-Performance Computing (HPC): Disaggregated memory reduces underutilization in clusters, improves packing efficiency, and enables rack-scale pooling. Frameworks such as DOLMA leverage known memory access patterns (sequential, strided) to prefetch and pipeline remote chunks for stencil, FFT, and SpMV kernels, with <16% slowdown for up to 63% local DRAM reduction (Zheng et al., 2 Dec 2025, Wahlgren et al., 2023, Ding et al., 2023).
• Datacenter Storage and Databases: Smart memory buffer caches with operator offloading via FPGA (Farview) demonstrate competitive and even superior performance to local caches for analytical queries, especially under high data-reduction (predicate/selectivity) (Korolija et al., 2021).
• Cloud Provisioning: Elastic VM memory provisioning, secure memory marketplaces, and serverless function cold-start acceleration are enabled by providing “memory-as-a-service” built on underlying disaggregated pools (Caldwell et al., 2017, Maruf et al., 2021, Wang et al., 26 Mar 2025).

6. Design Trade-offs, Challenges, and Future Directions

Trade-offs in memory disaggregation design include:

• Latency vs. Transparency: Page-faulted or swap-based approaches maximize backward compatibility but suffer high per-access overhead. Byte-granular, load/store semantics (CXL, OpenCAPI) minimize latency but may require code adaptation or new device drivers (Wang et al., 2023, Yang et al., 22 Mar 2025).
• Capacity scaling vs. management complexity: Unlimited capacity is theoretically accessible with multi-rack memory blades, but requires distributed metadata, global allocation, and efficient failure isolation (Wang et al., 26 Mar 2025).
• Coherence and consistency: Strong hardware cache coherence over disaggregated memory (e.g., via MSI protocol in MIND) imposes directory/TCAM/SRAM scaling limits and can bottleneck writes under contention (Lee et al., 2021, Wang et al., 26 Mar 2025).

Key ongoing challenges:

• Global metadata management, failover, and consistency protocols become central at scale (Wang et al., 26 Mar 2025, Lee et al., 2021).
• Performance isolation and multi-tenancy are unsolved problems in public cloud scenarios and must contend with real-world adversarial tenants (Maruf et al., 2021, Heo et al., 2021).
• Congestion and bandwidth management remain essential, with dynamic control schemes (MIKU) and scheduling in the data-plane or PHY layer (EDM) demonstrating path forward (Yang et al., 22 Mar 2025, Su et al., 2024).
• Cache-coherent, multi-tiered architectures (local/HBM + remote/DDR + NVM) necessitate new queueing models and orchestrators, with hardware support for per-tier flow control (Yang et al., 22 Mar 2025, Wang et al., 2023).
• Security and confidential computation in untrusted or semi-trusted pools are addressed via per-page hardware isolation and encryption protocols (Heo et al., 2021).

Research opportunities include photonic/NVM co-design for sub-μs disaggregated memory, composable accelerators with unified allocators, edge datacenter disaggregation, and memory-centric scheduling for AI/HPC clusters (Wang et al., 26 Mar 2025, Puri et al., 2023).

7. Comparative Summary and Practical Recommendations

The table below summarizes key architectures and system characteristics:

System / Framework	Access Model	Latency (remote)	Bandwidth	Security / Isolation	Application Domain
ThymesisFlow+Plasma	Load/store, object API	~2.6–5.0 ms	5.75 GiB/s (remote)	FPGA-based hardware, gRPC	Big-data, distributed memory
DOLMA	Data-object, dual buf	<16% slowdown vs local	>85 Gb/s RDMA	User-level, profiling/model	HPC, stencils, FFTs
INDIGO	Page migration	up to 70% faster	Adaptive, reduced	OS-kernel, telemetry+RL	General-purpose, cloud/HPC
FluidMem	Page-fault, swap	34.9–86.2 μs	Limited by backend	Userfaultfd, KVM isolation	Cloud VMs, general
CXL over Ethernet	Load/store, cacheline	1.97 μs (remote)	100 Gbps (measured)	Hardware, FPGA cache+ctrl	Datacenter, pod-scale
EDM	PHY-embedded protocol	0.30 μs (remote)	Up to line rate	Hardware scheduler	Datacenter, ultra-low latency
TDMem	Page-based, hardware	4–6 μs	~95% of FastSwap	Per-page FPGA, AES-GCM	Cluster, security-critical

Practical recommendations:

• Match remote memory fraction and bandwidth to application needs: for many AI/HPC codes, remote memory can provide capacity without penalty if working set and access ratio constraints are met (Wahlgren et al., 2023, Ding et al., 2023).
• Use interference-aware placement policies and object-level steering (e.g., DOLMA, Smart-Idle) to avoid contention and queue buildup (Zheng et al., 2 Dec 2025, Puri et al., 2023).
• Leverage hardware prefetch and dual-buffering for predictable access patterns to hide remote latency (Zheng et al., 2 Dec 2025).
• Adopt network-aware, dynamic throttling (e.g., MIKU) in multi-tier systems with CXL/DDRx to optimize local bandwidth (Yang et al., 22 Mar 2025).
• Address security and isolation with hardware access validation, encryption, and randomized remapping in multi-tenant environments (Heo et al., 2021, Maruf et al., 2021).

Memory disaggregation is emerging as a core cross-layer architectural strategy for the next generation of datacenters, HPC clusters, and public clouds, uniting advances from hardware interconnects and protocol design through orchestration frameworks, security primitives, and application-aware memory management (Wang et al., 26 Mar 2025, Abrahamse et al., 2022, Zheng et al., 2 Dec 2025).