---
title: 'Memory Disaggregation: Decoupling Memory & Compute'
url: https://www.emergentmind.com/topics/memory-disaggregation-md
type: topic
---

# Memory Disaggregation: Decoupling Memory & Compute

Memory disaggregation (MD) denotes the architectural separation of compute and memory resources in datacenters, decoupling DRAM from CPUs and exposing memory as a shared, network-accessible pool. This paradigm responds to persistent challenges in resource fragmentation and under-utilization inherent in monolithic server designs and is increasingly enabled by advances in high-performance interconnects such as CXL, RDMA, and FPGA-based network fabrics. MD delivers the elasticity to dynamically allocate memory independent of CPU provisioning, accommodates memory-intensive applications that exceed local DRAM capacity, and improves overall utilization and resource efficiency. However, it introduces trade-offs in the form of increased access latency, complexity in coherence and consistency, and additional costs in power and infrastructure.

## 1. Architectural Foundations and System Models

Memory disaggregation is architected through a variety of system fabrics, classified broadly as follows [2503.20275, 2202.02223]:

- **Single-Node, Single-Pool:** Memory expanders (e.g., CXL-attached DIMMs, OpenCAPI) connect one compute node to a single memory blade, typically via a cache-coherent protocol and PCIe class link.
- **Rack-Scale (Multi-Compute, Multi-Memory):** A mesh of compute nodes and memory pools communicate over high-throughput, low-latency networks (e.g., Ethernet, Infiniband, Gen-Z, CXL fabrics), with local DRAM acting as a cache for the disaggregated memory pool [2303.06420, 2305.09977]. Interconnects may employ standard ToR switches with PFC, custom FPGA packetization, or even in-network logic [2411.08300, 2107.00164].
- **Component Blocks:** System-level modules include:
  - Host CPUs/SoCs with CXL/PCIe interfaces
  - Network-attached memory nodes (FPGAs, DDR/NVM banks)
  - In-network managers for translation, protection, and scheduling
  - Software or hardware global managers for allocation and policy [2303.06420, 2302.08055]

Typical access flows, as in CXL-over-Ethernet designs, combine PCIe-based CXL endpoints in compute FPGAs, custom packet managers that encapsulate memory requests and responses in Ethernet payloads, and dedicated translation/CAM units on memory-side FPGAs for address mapping and response formulation [2302.08055].

## 2. Performance, Latency, and Modeling

Critical to MD adoption are end-to-end memory access latencies, throughput ceilings, and their decomposition [2503.20275, 2305.03943]. Measured results from current hardware primitives include:

| Stack/Mechanism               | Remote Read Latency | Throughput            |
|-------------------------------|---------------------|-----------------------|
| CXL-over-Ethernet + FPGA [2302.08055] | 1.97 μs avg / 415 ns (FPGA cache hit) | 100 Gbps/FPGA          |
| RDMA over RoCEv2 [2411.08300] | ≈2.03 μs            | ≈8 Greq/sec at 25 Gbps |
| EDM/PHY-based Ethernet [2411.08300]   | ≈0.30 μs            | ≈22 Greq/sec at 25 Gbps |

Latency decompositions in MD systems typically sum:
\[
T_\text{total} = T_\text{host-FPGA} + T_\text{fpga-proc} + T_\text{net-fwd} + T_\text{remote-mem} + T_\text{net-back} + T_\text{fpga-host}
\]
where each term covers serialization, protocol logic, network transit, access, and response [2302.08055].

Performance is tightly bound to:
- Cache/miss ratios in local DRAM/L3
- Network link serialization & queuing (e.g., 800 ns two-way for MAC+PHY+switch [2302.08055])
- Queueing in DRAM controllers on memory nodes (modeled as M/M/1 systems [2303.06420])
- Allocation/policy designs that avoid hot-spotting in both interconnects and memory-microqueues

Cache-optimized MD architectures demonstrate significant reductions in remote access delays (e.g., 415 ns cache-hit, 1.97 μs miss on FPGA [2302.08055]), outperforming RDMA-based or unoptimized systems by up to 37%.

## 3. Resource Management and Allocation Policies

Memory management spans global and pool-local policies, addressing not only function (load/store access) but also resource balancing [2303.06420, 2202.02223]. Notable points:

- **Address Translation:** TLBs or content-addressable memory (CAM) in FPGA/ASICs manage high-throughput CXL or RDMA memory translations [2302.08055].
- **Allocation Algorithms:** 'Smart-Idle' pool-selection [2303.06420] uses recent access frequencies and allocation counts to select the least-congested set of pools (formally: minimize Allocₚ over a subset m = ⎡log₂n⎤ with smallest access factors Afₚ).
- **Local-First (LF) vs. Alternate (LR):** LF policies exhaust local DRAM before pulling remote, leading to sharp 'latency cliffs'; LR interleaves local/remote pages to flatten latency ramps and improve tail metrics [2303.06420].
- **Congestion Control:** Hardware and software rate limiting (token-buckets, PFC, additive increase/multiplicative decrease) and retransmission protocols (Go-Back-N/Selective-ACK) are deployed in custom packet managers to maximize sustainable link utilization without incurring excessive 'pause' events [2302.08055].

Empirical results indicate that intelligent pool selection (Smart-Idle) can consistently halve average as well as 99th-percentile access latencies (e.g., to 250–350 ns per access, >80% reduction in >1 μs tail events [2303.06420]).

## 4. Software, Hardware, and Cross-Layer Optimizations

Modern MD systems leverage a full cross-stack integration for both programmability and efficiency [2503.20275]:

- **FPGA/SmartNIC Offload:** CXL endpoint IPs and soft logic on FPGAs enable full hardware data-paths for load/store and protocol management, isolating CPU overhead and supporting in-hardware MESI cache coherence [2302.08055, 2108.03492].
- **On-FPGA Caching:** Low-latency, set-associative caches (e.g., 32 KB, 4-way, MESI) with LRU per set and write-allocate policies deliver <60 ns hit times [2302.08055].
- **Event-based & Cycle-accurate Simulation:** Models such as DRackSim [2305.09977] and detailed hybrid models [2303.06420] integrate event-based interconnects, cycle-accurate DRAM modeling, and realistic queue/buffer sizes for system design space exploration (e.g., queueing, migration, and hot-page policies).
- **Congestion-Aware Scheduling:** In switch-based or PHY-based fabrics (EDM [2411.08300]), parallel iterative matching (PIM) algorithms in hardware can schedule slot-reservations for memory traffic, eliminating queuing delay and maintaining close to line-rate utilization even at 90% loading.

Performance modeling consistently shows that remote DRAM is 4–10× slower than local (cacheline scale), but can be effectively hidden for workloads with high spatial/temporal locality or when hot/cold sets are managed in local tiers.

## 5. Security, Isolation, and Practical Deployment

Security in MD systems requires both fine-grained protection and performance-aware validation [2108.11507]:

- **Access Control:** Hardware-enforced per-page permission tables (e.g., in FPGA HBM) replace RDMA's region-level rkey facilities, preventing kernel or hypervisor attacks from escalating privileges or leaking data across tenants [2108.11507].
- **Confidentiality & Integrity:** AES-GCM encryption is implemented CPU-side, with keys managed by trusted hardware, ensuring that even if physical DRAM or host software is compromised, memory content remains unreadable and unmodifiable [2108.11507].
- **Oblivious Access:** Randomized page remapping on every swap-out/in by hardware (FPGA-level randomization) breaks static access-pattern leakage, supporting weaker forms of obliviousness without full ORAM-level guarantees.

Overheads for fine-grained secure MD remain modest (≈10–20% in bandwidth or latency), with multi-tenant scaling and hardware resource usage (LUT/BRAM) supporting future extensibility [2108.11507].

## 6. Comparative Analysis, Impact, and Open Problems

Memory disaggregation is shown to deliver substantial improvements over fixed, local DRAM allocation in both utilization and average performance [2302.08055, 2303.06420, 2305.09977]. Key findings include:

- **Latency reductions** by more than 30% compared to RDMA-disaggregation baseline, and median sub-microsecond (415 ns) access for cache hits [2302.08055].
- **Tail-latency control** via optimized allocation and proactive congestion-management policies [2303.06420].
- **Scalability** to multiple racks and network fabrics, contingent on FPGA resource sizing and switching fabric support [2302.08055, 2411.08300].
- **Trade-offs** in cache size, aggressiveness of allocation/pooling, and hardware complexity, with portable FPGA-based deployment favored for rapid system adoption [2302.08055].

Open questions remain in integrating MD into commodity CXL host ASICs, generalizing hardware prefetch to minimize cache-miss penalties, co-designing network and memory management for larger topologies (e.g., CXL 3.0’s multi-root support [2302.08055]), and extending security primitives for denial-of-service resilience and stronger access pattern hiding [2108.11507].

## 7. Future Directions and Research Challenges

Research advances in MD target lower latency fabrics (e.g., PHY-layer scheduling [2411.08300]), more intelligent tiering and hot/cold management [2305.09977], improved policy for allocation and migration under contention [2303.06420], and expanded support for cross-tenant multitenancy and isolation. Integration with composable data center architectures, further hardware/software co-design (e.g., PIM, in-network management), and adaptive, real-time policy frameworks will likely mature MD platforms for mainstream, large-scale deployments. As design, deployment, and operational experience accumulate, the interplay between hardware scalability, network architecture, and resource policy will shape the next generation of elastic, efficient datacenter memory systems.

Source: https://www.emergentmind.com/topics/memory-disaggregation-md