Distributed Persistence Domain for Persistent Memory Pooling

Abstract: Compute Express Link (CXL) enables memory pooling over disaggregated memory, offering the potential to improve resource utilization in persistent memory systems. However, integrating persistence semantics into CXL-based memory pooling introduces substantial latency, which limits system scalability. This overhead arises because persist operations must traverse the entire CXL fabric, including switches, links, and protocol layers, before reaching remote persistent memory. To this end, we argue that extending CXL switches with persistence support is a promising direction for improving the scalability of persistent memory pooling. However, moving persistence support into the network breaks the traditional correctness assumptions of centralized persistence domains. In particular, enabling persistence within distributed structures, such as CXL switches, can introduce stale reads and writes if not carefully coordinated. In this paper, we propose Distributed Persistence Domain (DPD), a new abstraction for persistent memory pooling that enables persistence support at the CXL switch level. We first formalize the concept of a distributed persistence domain and use DPD as a framework to identify the correctness hazards that arise when persistence structures are distributed across the CXL fabric. Based on this analysis, we derive the design requirements needed to guarantee correctness. Building on these insights, we present Persistent CXL Switch, a CXL switch architecture that incorporates persistence support to significantly reduce persist latency, enable read forwarding, and coalesce writes, while preserving correctness and crash consistency. We evaluated our system design using both SPLASH-4 and YCSB benchmarks. Simulation results show an average speedup of 33% over volatile CXL switches, and up to 36% speedup with read forwarding optimization across all workloads.

• The paper introduces the Distributed Persistence Domain (DPD) abstraction to decouple persist latency in disaggregated persistent memory systems.
• It proposes architectural extensions to CXL switches, enabling adaptive read forwarding and write coalescing to achieve up to 36% performance improvements.
• The design rigorously ensures crash consistency and fault tolerance through coordinated persist protocols, dynamic drain policies, and hardware–software co-design.

Distributed Persistence Domain for Persistent Memory Pooling

Introduction and Motivation

The increasing adoption of CXL (Compute Express Link) as a memory interconnect standard is fundamentally changing the architecture of disaggregated memory systems, facilitating resource pooling and high utilization for both HPC and large-scale data center workloads. The integration of persistent memory (PM) in such environments presents new challenges, particularly for ensuring both the correctness and performance of persist operations, as traditional persistence models incur high latency due to multi-hop CXL switching across the network. The paper "Distributed Persistence Domain for Persistent Memory Pooling" (2606.07159) introduces the Distributed Persistence Domain (DPD) abstraction, decoupling persist latency from system-scale and proposing architectural extensions to CXL switches to directly support distributed persistence semantics.

System Architecture and the Distributed Persistence Domain

The traditional persistence domain model assumes a centralized, contiguous, and local-to-endpoint architecture boundary. In the context of disaggregated persistent memory, this design is inadequate, as persist operations must traverse CXL links, resulting in increased latency and scalability bottlenecks as memory is physically distributed. The paper advocates for supporting persistence directly at the CXL switches, thereby introducing the concept of a distributed persistence domain, in which persistence structures (e.g., switches and controllers) collectively participate in guaranteeing durability and crash consistency along the fabric path.

Figure 1: Comparison of persistence semantics in (a) volatile CXL switch-based pooling and (b) the proposed persistent CXL switch–based pooling.

Figure 2: Illustration of persist write/read in conventional volatile pooling, PCS-based persist, and PCS-based read forwarding modes.

This system permits selective extension of the persistence domain to any subset of network switches, improving performance while retaining flexibility. The design further supports non-volatile or battery-backed persistent buffers at each switch, mitigating persistent operation latency and providing protocol support for read forwarding and write coalescing.

Correctness and Consistency Hazards in Distributed Persistence

Introducing persistence capabilities in CXL switches leads to multi-versioned data and the threat of stale reads or write-reordering errors if directed naively. The distributed persistence domain requires rigorous mechanisms to (1) ensure updated values are observable network-wide, (2) propagate persists in proper causal order, and (3) cleanly handle data draining and recovery events. Failure to do so would violate crash consistency and data correctness, especially under concurrent access and process migration scenarios.

Figure 3: Data correctness hazards, including stale data propagation and write reordering, when persistent switches are not carefully coordinated.

Figure 4: DPD topology—correctness depends strictly on persistence structures along the active path for any given host.

DPD is formalized as the set of persistence structures along the path that a request traverses. This allows for correct and efficient crash-consistent operation, provided software and hardware protocols (e.g., draining mechanisms) are invoked judiciously during events such as process migration or topology changes.

Persistent CXL Switch Architecture

The core architectural change is to endow each CXL switch with dedicated persist buffers (PBs), a persist buffer controller (PBC), and a PBC selector (PBCS). The PB itself can be NVM-based or utilize a powerful backup unit, and is split into tightly managed tables to support rapid lookup and status tracking for high-concurrency access. Write requests are buffered, acknowledged early (masking remote persist latency), and drained or forwarded according to system state. The PBC ensures all correctness constraints are met by strictly managing the life cycle of each persist buffer entry.

Figure 5: Persistent buffer architecture—PBC Selector operates in the control logic, PB and PBC operate off the switch’s critical path.

Read forwarding is achieved by serving data directly from the buffer when present; write coalescing is naturally supported by PB granularity and careful replacement policy. System-level software (or OS) support is required for “DrainPath” operations, which ensure the draining of all inflight persists prior to events such as process migration, thereby guaranteeing there are no stranded or stale versions in the distributed buffers.

Correctness, Crash Recovery, and Fault Tolerance

The PCS design maintains strict tracking of data versions and guarantees any freed entry was already propagated to the downstream persistent structure, ensuring that, after a crash or migration, the latest version is available on the correct path. OS-coordinated “DrainPath” messages orchestrate draining so that, after topology reconfiguration or recovery, the system state remains globally consistent.

Figure 6: Operating system–coordinated process migration issues DrainPath commands to ensure correctness under DPD.

Correctness after either processor- or memory-side crashes is preserved using these mechanisms, so long as draining invariants are followed strictly. The approach permits scalable and fault-tolerant operation, where DPDs can be recomposed even in the presence of partial fabric failures.

Performance Evaluation

Extensive simulation using gem5 demonstrates the architectural efficacy. CXL switch–level persistence support produces a 33% mean speedup over a baseline volatile switch design, and up to 36% improvement with adaptive read forwarding. The evaluation utilizes both SPLASH-4 and YCSB persistent-memory–backed workloads, demonstrating improvement in both computation-intensive and data-serving scenarios.

Figure 7: Speedup for SPLASH-4 and YCSB workloads under various PCS/DPD scheme configurations.

Notably, applications with high persist latency sensitivity (e.g., FFT, LU) benefit most. Write-heavy and mixed workloads, such as YCSB-A and F, also realize tangible gains. The benefit is less pronounced for read-only or persist-insensitive workloads (e.g., YCSB-C, RAYTRACE).

Commit-stage analysis reveals that flush/fence stalls are dramatically reduced, and PCS write failure rates are controlled effectively with proper buffer sizing and drain policies.

Figure 8: Commit stall breakdown illustrating impact on persist/fence-induced stalls.

Adaptive schemes outperform static draining due to the dynamic nature of bursty writes and read-locality, with further analyses presented on write persistence failures, read buffering hit rates, and write coalescing behavior.

Figure 9: PCS buffer write failure rates are manageable and minimized with aggressive draining policies.

Figure 10: Read hit rates at PCS, which benefit from lazy draining.

Figure 11: Write coalescing efficacy at PCS, especially substantial for access patterns with temporal locality.

Drain policy adaptivity (Figure 12), buffer sizing sensitivity (Figure 13), and scaling across multi-switch hops (Figure 14) are also analyzed, confirming design robustness and scalability.

Figure 12: Adaptive drain thresholds yield best results, tuning persist buffer effectiveness to workload characteristics.

Figure 13: Splash-4 speedup sensitivity to number of persist buffer entries at PCS.

Figure 14: DPD maintains scalability as switch hop count increases, with persistent switches masking remote latency.

Implications and Future Directions

This work significantly advances the practical and theoretical foundation for persistent memory pooling in disaggregated, fabric-attached architectures. By supporting distributed persistence semantics and formalizing correctness requirements, the architecture eliminates a critical barrier in adopting large-scale, persistent-capable, memory pools. Practically, the reduced latency and improved scalability enable broader adoption of persistent memory for both HPC and large-scale online data serving.

The framework's flexibility suggests further enhancements, such as intelligent fabric-level coordination, cross-domain draining optimizations, and adaptive policies driven by workload or ML-based predictions. The hardware–software co-design techniques demonstrated in this system will likely become essential as CXL-based composable resources are deployed at datacenter scale.

Conclusion

The proposed Distributed Persistence Domain and persistent CXL switch architecture define a scalable, high-performance framework for memory pooling with strong persistence guarantees in a disaggregated system. The implementation achieves significant speedups on real-world workloads, all while formalizing correctness and ensuring robust crash consistency. This design establishes a solid foundation for integrating next-generation persistent memory at scale within CXL-based architectures, underscoring the increasing convergence of computer architecture, systems software, and networking in memory-centric compute fabrics.