--- title: Distributed Shared Memory (DSM) url: https://www.emergentmind.com/topics/distributed-shared-memory-dsm type: topic --- # Distributed Shared Memory (DSM) Distributed Shared Memory (DSM) provides the abstraction of a single, coherent memory address space over a physically distributed set of computing nodes. Under DSM, each node in a distributed or parallel system participates in what appears to be a globally shared memory, but accesses may, in fact, cause communication over a network. DSM enables the decoupling of program logic from the physical memory layout and was originally motivated by the desire to preserve the programming simplicity of shared-memory models while scaling to distributed systems and clusters. ## 1. Formal DSM Models and Semantics DSM is defined by the mapping of a global address space onto the physically distributed memory of a set of processes or nodes. Each process \( P_i \) may access both private (local) and public (remotely accessible) memory. A global address typically takes the form (process_name, local_address) or, in PGAS systems, (locale, offset). The addressing abstraction admits both software and hardware implementations; in software DSM, mapping and coherence are the responsibility of the runtime or communication library [1101.4193][2112.00068]. DSM systems are characterized by: - **Memory consistency models:** which define the allowed visibility and ordering of reads and writes. Key models include sequential consistency, linearizability, causal consistency, and eventual consistency [1608.02442][2208.02411][2102.03124]. - **Coherence protocols:** which guarantee that writes to bytes or objects are observed consistently, often via locks, invalidation protocols, or, in scalable designs, logical clocks or version tags [1501.04504][2406.02803][1101.4193]. ## 2. Consistency Models and Coherence Protocols DSM implementations must specify, enforce, and expose a well-defined consistency model: - **Sequential Consistency (SC):** All operations appear in some global total order respecting program order. Achievable via centralized or quorum-based timestamping and update protocols, as in SC-ABD [1608.02442]. - **Linearizability:** Stronger than SC; requires respecting real-time orders of non-overlapping operations [2208.02411]. - **Causal Consistency:** Ensures visibility of causally related updates; implemented via vector clocks or edge-based share-graph timestamps in full or partial replication regimes [1703.05424][0909.2704]. - **Eventual Consistency:** Only requires eventual convergence of replicas; ordering is loose except for per-node FIFO [2208.02411]. Coherence can be maintained via: - **Directory-based Protocols:** E.g., Tardis replaces traditional directories and broadcast invalidations with logical timestamps (wts, rts) per block, requiring only O(log N) metadata per cacheline and no multicast [1501.04504]. - **NIC-supported locking:** Per-datum exclusive access on one-sided RDMA-enabled NICs, supporting fine-grained lock/unlock around gets and puts [1101.4193]. - **Version tagging with language guarantees:** DRust exposes the Rust ownership model (SWMR enforced by the type system) to drastically simplify coherence logic. All writes are mediated by unique mutable borrows, and per-pointer version (color) tags invalidate stale copies on modification [2406.02803]. - **Home-based protocols and chunking:** E.g., S-DSM in SAT provides multi-protocol coherence at chunk granularity, using home-based MESI with scope or release consistency [2009.01507]. ### Table: DSM Consistency Models and Representative Implementations | Consistency Model | Key Mechanism | Example Protocol | |---------------------------|---------------------|-------------------------| | Linearizability | Total order, tmcast | Attiya-Welch [2208.02411] | | Sequential consistency | Quorums, timestamps | SC-ABD [1608.02442], Tardis [1501.04504] | | Causal consistency | Vector/timestamp | Share-graph algorithm [1703.05424] | | Eventual consistency | FIFO, convergence | COPS-style [2208.02411] | ## 3. Algorithmic Frameworks and Storage Efficiency DSM’s communication and storage cost depend strongly on the chosen algorithms and data models: - **Erasure-coded atomic memory:** Storage-efficient DSM can be achieved by parameterizing erasure codes with concurrency bound ν, reducing per-server storage to \( N/\lceil (N-2f)/\nu \rceil \) units for an N-server, f-fault-tolerant system. Atomicity is preserved by encoding versions and supporting multi-writer, multi-reader extensions via a combination of replication and coding, with message complexity of O(1) rounds per operation [1803.01098]. - **Compositional correctness proofs for consistency:** The SC-ABD algorithm demonstrates a write operation in 1 round and read in 2 rounds, ensuring sequential consistency via a register-by-register compositional proof structure; intersection of majorities ensures propagation of the latest value [1608.02442]. - **Non-blocking algorithms and reclamation:** In PGAS-based DSMs, scalable atomic operations leverage RDMA pointer compression and epoch-based memory reclamation for non-blocking data structures at scale [2002.03068]. - **Partial replication and metadata lower bounds:** Causally consistent DSM under partial replication requires each replica to track only those directed edges in the share graph along which causal information can flow; the bit-complexity is minimized accordingly [1703.05424]. ## 4. Architectural Realizations and Programming Models DSM spans a variety of hardware and software platforms: - **Network-on-Chip DSM:** The Epiphany many-core RISC array provides a flat, memory-mapped global address space over a 2D mesh NoC, with sequential consistency and zero hardware cache; all sharing is explicit and incurs mesh latency. Transparent DSM programming is achieved via C++ TMP-generated accessor code [1704.08343]. - **Cache-Coherent NUMA and Partitioned Shared Memory:** Systems like JArena partition the global heap into per-thread or per-node regions, pinning physical pages to NUMA nodes and eliminating false sharing and remote access [1902.07590]. - **Heterogeneous, event-driven S-DSM:** SAT integrates chunk-based allocation, multiple coherence protocols, and a hybrid event-driven/shared-memory API suited to clusters of CPUs, GPUs, and FPGAs, with energy-optimized idle polling [2009.01507]. - **Disaggregated memory and edge DSM:** Emerging systems combine DSM abstractions with memory-disaggregation and routable byte-addressable NVM pools, even across volatile edge environments [2207.03027][2102.03124]. The programming model flexibility is evident: - **PGAS (Partitioned Global Address Space):** Exposes global view with explicit or implicit locality [2112.00068][2002.03068]. - **Language-guided DSM:** DRust integrates Rust’s ownership and lifetimes to present a transparent, zero-overhead DSM interface, achieving strong consistency and high performance on modern RDMA [2406.02803]. ## 5. Performance, Scalability, and Trade-Offs DSM solutions must balance communication, metadata, and synchronization overhead with scalability and transparency: - **Metadata scaling:** Tardis achieves O(log N) per-block state versus O(N) for traditional directories, supporting thousands of cores [1501.04504]. Vector or edge-based timestamp mechanisms scale with replica count and sharing topology [1703.05424]. - **Communication patterns:** Directoryless or logical-clock-based protocols (e.g., DRust, Tardis) eliminate broadcast invalidations and allow point-to-point messaging; renewed leases or color tags invalidate stale copies only as needed [2406.02803][1501.04504]. - **Performance isolation:** Partitioned heap managers in NUMA-aware systems deliver near-linear scaling on manycore servers, up to 4.3× over standard allocators [1902.07590]. - **Checkpointing and rollback:** Stronger consistency simplifies global state capture (linearizability allows O(1) image save), while weaker models (causal/eventual) require vector clocks, deltas, and more complex marker protocols, increasing checkpoint and rollback cost [2208.02411]. - **Storage efficiency:** Erasure-coded DSM matches lower bounds when concurrency is limited, with trade-offs between storage, liveness, and latency [1803.01098]. ## 6. Advanced Topics: Fault Tolerance, Computability, and Special-Purpose DSM DSM systems address crash and recovery: - **Fault-tolerant protocols:** SC-ABD and erasure-coded DSMs tolerate up to \( f