Memory-Oriented Distributed Computing (MODC)

• MODC is a computing paradigm that treats memory as the primary global resource, emphasizing visibility, residency, and movement across distributed systems.
• It organizes computation, synchronization, and placement around memory semantics rather than explicit message passing, enabling efficient fault tolerance and resilience.
• Applications range from resilient task runtimes and distributed shared memory to memory-centric machine learning and in-network computing architectures.

Searching arXiv for papers on MODC and DistZO2 to ground the article in current literature. Memory-Oriented Distributed Computing (MODC) is a systems and programming paradigm that treats memory as the primary distributed resource and, in several formulations, as the primary global abstraction exposed to applications. In the literature, MODC appears both as a specific framework for resilience on disaggregated memory architectures and as a broader design orientation spanning atomic shared memory, software-distributed shared memory, compiler-managed locality, in-network memory management, and memory-centric machine learning systems. Across these variants, the recurring principle is that computation, synchronization, and placement are organized around memory visibility, residency, and movement rather than around explicit message exchange or bulk checkpoint/restart (Keeton et al., 2021, Nicolaou et al., 19 Jan 2026, Wang et al., 3 Jul 2025).

1. Conceptual foundations and historical scope

The original MODC formulation targets rack-scale systems in which a large pool of byte-addressable non-volatile memory is exposed over a low-latency fabric and shared among decentralized compute nodes. In that setting, compute nodes have local private DRAM, while the disaggregated memory pool is shared and byte-addressable through load/store, one-sided get/put, and atomic fabric operations. The resulting system has independent failure domains: compute nodes may crash while disaggregated memory remains available, and updates written before a crash remain visible to surviving nodes. MODC is defined there as a programming and runtime framework that exploits this persistence and shared visibility to achieve resilience without relying on global checkpoint/restart or application-level data replication (Keeton et al., 2021).

Subsequent work generalizes the term. In decentralized shared memory, MODC is described as a paradigm that treats memory—rather than servers or logs—as the primary, global abstraction exposed to distributed applications, with strong consistency semantics such as atomic or linearizable read/write. In that formulation, memory is not merely a storage substrate; it is the composition boundary for higher-level objects and protocols, including multi-object atomic shared memory, membership services, and Byzantine-tolerant operation (Nicolaou et al., 19 Jan 2026).

A more recent systems interpretation appears in large-model training. DistZO2 is explicitly presented as exemplifying MODC because it makes memory placement and movement the first-class design concerns, then orchestrates distributed parallelism around those memory decisions. Its design keeps the master parameter state in CPU RAM, moves one transformer block at a time to GPU, overlaps upload, compute, and offload, and only then scales across multiple GPUs with communication patterns shaped by that memory organization (Wang et al., 3 Jul 2025).

This suggests that MODC is best understood not as a single architecture, but as a family of designs in which memory placement, memory movement, and memory semantics determine the distributed execution model.

2. Programming models and consistency semantics

In the original MODC framework, applications are expressed as idempotent, side-effect-free tasks operating on named data objects resident in disaggregated memory. Runtime state—including task graph metadata, per-worker queues, and progress counters—is itself stored in lock-free data structures placed in the disaggregated pool so that any surviving worker can observe and modify it. Jobs impose control dependencies across collections of tasks, futures allow consumer tasks to be scheduled before their inputs exist, and correctness relies on atomic state transitions in task descriptors together with task idempotence and named outputs (Keeton et al., 2021).

A distinct but related line of work casts MODC as atomic shared memory. OPTIMUM-DERAM implements a decentralized, reconfigurable, atomic read/write shared memory in which objects are placed on a consistent hashing ring and stored as coded fragments across object-specific clusters. Values are encoded by Random Linear Network Codes: if $x \in \mathrm{GF}(q)^k$ is the data vector and $a_i \in \mathrm{GF}(q)^k$ are random coefficients, coded fragments are

$y_i = a_i^\top x,\qquad i=1,\ldots,n,$

and decoding from $k$ linearly independent fragments solves

$x = A_k^{-1} y_k.$

The design gives a storage overhead factor of $\frac{n}{k}$ and uses Byzantine quorum constraints

$$b < \frac{|C|-k}{3},\qquad |Q| = \frac{2|C|+k}{3},\qquad |Q_1 \cap Q_2| \ge b+k,$$

to preserve atomicity under adversarial faults (Nicolaou et al., 19 Jan 2026).

Software-distributed shared memory offers another MODC realization. SAT concatenates distributed physical memories into a logical address space, decomposes memory into atomic chunks, and supports scope consistency through operations such as READ, WRITE, READWRITE, and RELEASE. Its default coherence protocol is a 4-state home-based MESI, and it extends memory management with event subscriptions so that updates to chunks can trigger handlers asynchronously. In this model, MODC is both an address-space abstraction and a coherence-management problem (Cudennec, 2020).

Regional Consistency provides a formal memory model for non-cache-coherent systems that is likewise aligned with MODC. It separates code into consistency regions, delimited by lock acquire and release, and ordinary regions, delimited by barriers. Consistent stores are those executed within a consistency region; ordinary stores are those outside such regions. The model then gives distinct visibility rules for lock-protected and barrier-separated execution, allowing a runtime to use fine-grained updates for critical sections and page-based invalidations for ordinary regions while preserving familiar thread-based shared-memory semantics (Ramesh et al., 2013).

Across these formulations, MODC does not imply a single consistency model. Rather, it denotes a class of designs in which memory visibility and memory semantics are made explicit and programmable.

3. Placement, movement, and topology-aware orchestration

A central MODC theme is that placement and movement are optimization variables, not afterthoughts. In stencil and convolution workloads, Locally-Oriented Programming encodes this directly in the language. A pure CONCURRENT procedure may write only the local element $U(0,0)$ while reading a declared halo such as $U(-1,0)$ or $U(0,+1)$. Halo sizes are part of the array type, and HALO_TRANSFER propagates ghost regions across images and subimages. The compiler and runtime therefore infer halo sizes, schedule exchanges, and place computation across host DRAM, device memory, and distributed processes using a locality-first abstraction (Rasmussen et al., 2015).

Molly applies a related principle through polyhedral compilation. Programmers declare distributed arrays as fields, the compiler interprets loop nests as polyhedral iteration spaces, assigns an owner-computes mapping, and automatically synthesizes asynchronous MPI communication. Distribution, layout transformation, and communication placement are first-class compiler concerns, and halo-like transfers are aggregated into chunks according to dependence structure and schedule points (Kruse, 2014).

DOLMA pushes MODC into memory disaggregation for HPC by managing memory at data-object granularity rather than page granularity. It identifies and offloads data objects to remote memory, provides quantitative analysis to choose a suitable local memory size, and exploits predictable access patterns through remote memory prefetch via a dual-buffer design. Its analysis introduces quantities such as the memory reduction ratio

$a_i \in \mathrm{GF}(q)^k$0

and models runtime as

$a_i \in \mathrm{GF}(q)^k$1

The selection policy prefers objects that are large, predictable, and amenable to overlap between remote transfer and compute (Zheng et al., 2 Dec 2025).

DistZO2 gives a modern MODC exemplar in machine learning systems. It starts from a block-wise CPU–GPU offload pipeline in which parameters live on the host and only one transformer block is moved to GPU at a time. Dual forward passes compute the central-difference scalar projected gradient

$a_i \in \mathrm{GF}(q)^k$2

followed by the block-wise update

$a_i \in \mathrm{GF}(q)^k$3

A dynamic scheduler overlaps upload $a_i \in \mathrm{GF}(q)^k$4, compute $a_i \in \mathrm{GF}(q)^k$5, and offload $a_i \in \mathrm{GF}(q)^k$6 on asynchronous CUDA streams. In the distributed version, parameter blocks are sliced at the CPU into $a_i \in \mathrm{GF}(q)^k$7 pieces, sent over PCIe to $a_i \in \mathrm{GF}(q)^k$8 GPUs, and reconstructed by peer-to-peer redistribution over NVLink, with effective upload time modeled as

$a_i \in \mathrm{GF}(q)^k$9

The design preserves the single-block memory bound while shaping distributed parallelism around the offload pattern (Wang et al., 3 Jul 2025).

These systems differ in hardware and domain, but they share a common MODC pattern: determine what must reside where, minimize the critical memory footprint, and organize execution so that communication follows those memory decisions.

4. Resilience, elasticity, and protection

Resilience is a primary motivation in the original MODC framework. Because queues, descriptors, and application datasets already reside in disaggregated non-volatile memory, failure recovery reduces to detecting failure, re-executing the single last RUNNING task, and continuing the job. Workers monitor heartbeat counters in shared memory, modify barrier membership using 128-bit CAS when failures are detected, and steal queued READY tasks as well as the failed worker’s RUNNING task. On the PageRank workload, MODC’s execution time with one injected failure was 23.3 s, with less than 1% degradation relative to failure-free execution, whereas an MPI baseline with checkpointing was 1.06×–1.35× slower than failure-free and 19%–51% slower than MODC’s resilient execution (Keeton et al., 2021).

Elasticity and protection appear in rack-scale memory disaggregation through MIND. MIND relocates memory management logic—address translation, protection, allocation, and cache coherence—into a programmable switch ASIC acting as an in-network MMU. It provides transparent resource elasticity because applications continue to use familiar virtual memory APIs while compute threads can span multiple blades and share a single address space backed by disaggregated memory. Translation is range-based, protection uses protection domains and permission classes enforced by TCAM range matches, and coherence is provided by an in-network MSI directory. Measured coherence transitions without invalidation are approximately 9 $y_i = a_i^\top x,\qquad i=1,\ldots,n,$0s, transitions requiring sequential invalidation and fetch from an owner are approximately 18 $y_i = a_i^\top x,\qquad i=1,\ldots,n,$1s, and read-only shared workloads reach 1–2×10^6 IOPS across 8 blades (Lee et al., 2021).

OPTIMUM-DERAM addresses another resilience and protection frontier: Byzantine faults in decentralized shared memory. Digital signatures are attached to $y_i = a_i^\top x,\qquad i=1,\ldots,n,$2 pairs, clients verify returned fragments, and a blockchain/SMR oracle maintains a total order of join and leave events. The result is atomic read/write shared memory with dynamic membership and Byzantine quorum intersection, rather than mere crash tolerance (Nicolaou et al., 19 Jan 2026).

Security and isolation also appear in software DSM. SAT can place different chunks under different coherence protocols, uses a home-node function

$y_i = a_i^\top x,\qquad i=1,\ldots,n,$3

and provides an ABE-based security layer as a transparent add-on. Here, memory-centric design extends beyond performance to include naming, policy, and event-driven synchronization (Cudennec, 2020).

A plausible implication is that MODC broadens the conventional fault-tolerance question. It is not limited to durable data storage or process restart; it also includes where metadata lives, who can observe it after failure, and which protection and membership mechanisms preserve correctness when memory is globally visible.

5. Representative systems and application domains

MODC has been instantiated in resilient task runtimes, distributed shared memory, HPC memory disaggregation, big-data object stores, database architecture, in-network computing, and distributed machine learning. The systems below illustrate how the same memory-centric premise leads to different mechanisms.

System	MODC mechanism	Representative result
DistZO2	CPU-resident parameters, block-wise GPU offload, NVLink redistribution	508 tokens/s on OPT-175B with ZO2+PertP+DDP
DOLMA	Data-object-level offload and dual-buffer remote prefetch	local memory usage reduced by up to 63%, on average
Plasma + ThymesisFlow	Remote in-memory object access through disaggregated memory	remote throughput stabilizes at approximately 5.75 GiB/s
NetDAM	NIC-attached memory with programmable in-memory/in-network ISA	preliminary 0.4 s for MPI Allreduce
LOCO	Channel objects over RDMA with explicit locality and ordering	distributed map has better read and comparable write performance

In big-data systems, a memory-disaggregated Plasma object store extends Apache Arrow Plasma with ThymesisFlow so that clients can produce and consume immutable in-memory objects across nodes. For larger objects, local throughput stabilizes at approximately 6.5 GiB/s and remote throughput stabilizes at approximately 5.75 GiB/s, indicating what the paper describes as a modest performance penalty for remote disaggregated memory access as opposed to local (Abrahamse et al., 2022).

For databases, memory disaggregation with RDMA motivates distributed shared-memory databases in which data is stored primarily in the distributed shared-memory layer on memory nodes, while compute nodes cache hot data locally. The paper frames this as a re-opening of the “shared what” question and points to RDMA hardware such as Mellanox ConnectX-6 with 0.8 $y_i = a_i^\top x,\qquad i=1,\ldots,n,$4s latency and 200 Gb/s throughput as the enabling substrate for memory pooling, one-sided access, and elastic scaling of compute and memory (Wang et al., 2022).

NetDAM pushes MODC into NIC-centric memory pooling and in-memory computing. Memory is directly attached to the Ethernet controller, requests carry both instruction and data, and SIMD-capable ALUs execute operations such as READ, WRITE, MEMCOPY, CAS, ADD, MIN, and MAX near memory. In an MPI Allreduce case study on 536,870,912 × float32 elements, native MPI Allreduce took 2.8 s, a ring-based RoCEv2 baseline took 2.1 s, and preliminary NetDAM time was 0.4 s. The prototype also reports wire-to-wire SIMD read latency of 618 ns on average, with 39 ns jitter and 920 ns maximum (Fang et al., 2021).

LOCO explores an object-based programming model for network memory. Its “channel objects” store state across the network and expose methods implemented with RDMA verbs, explicit fences, and explicit placement control. The distributed map uses a local replicated index, remote reads to owner slots, ticket locks via RDMA atomics, and metadata such as checksum, counter, and valid bit. The paper states that its distributed map channel has better read and comparable write performance to a state-of-the-art custom RDMA solution, while using well-encapsulated and reusable primitives (Hodgkins et al., 25 Mar 2025).

DistZO2 shows that the same memory-centric logic applies even when the resource bottleneck is GPU memory rather than rack-scale DRAM. On OPT-175B with batch size 4, sequence length 2048, and 4× H20 96GB, ZO2 achieved 166 tokens/s, ZO2+PertP achieved 331 tokens/s, ZO2+DDP achieved 290 tokens/s, and ZO2+PertP+DDP achieved 508 tokens/s, while preserving a per-GPU memory footprint in the 18–20 GB range (Wang et al., 3 Jul 2025).

6. Limitations, misconceptions, and open problems

A common misconception is that MODC implies a single shared-memory style or that it necessarily hides locality. The literature shows the opposite. LOCO explicitly argues that channel objects do not hide memory complexity and instead rely on the programmer to use NUMA-like techniques to explicitly manage each object. LOPe and Molly encode locality into types and schedules. DistZO2 and DOLMA likewise expose the centrality of placement and movement rather than abstracting it away (Hodgkins et al., 25 Mar 2025, Rasmussen et al., 2015, Wang et al., 3 Jul 2025, Zheng et al., 2 Dec 2025).

Another misconception is that memory pooling alone eliminates communication bottlenecks. DistZO2 states that without NVLink the peer-to-peer stage becomes limited by PCIe and throughput gains may be modest, and that communication can dominate when forward kernels become highly optimized. MIND shows that high write-sharing reduces throughput by about 10× because many transitions become invalidation-heavy. OPTIMUM-DERAM reduces bandwidth and storage relative to replication, but read paths still incur $y_i = a_i^\top x,\qquad i=1,\ldots,n,$5 decoding cost and require quorum replies sized to preserve Byzantine intersection. DOLMA is effective when access patterns are predictable and compute can hide remote transfers, but irregular graph traversals or pointer chasing reduce prefetch hit ratio and increase uncovered latency (Wang et al., 3 Jul 2025, Lee et al., 2021, Nicolaou et al., 19 Jan 2026, Zheng et al., 2 Dec 2025).

Resilience assumptions are also system-specific. The original MODC framework assumes hardware redundancy and software replication to survive failures that render an entire memory node unavailable and does not directly address correlated failures or network partitions. SAT does not describe fault-tolerance mechanisms for failures or partitions. LOCO assumes reliable, static networks of known IP peers and leaves resilience to higher layers. These boundaries matter because memory-centric visibility can simplify recovery only when the memory substrate itself remains available and trustworthy (Keeton et al., 2021, Cudennec, 2020, Hodgkins et al., 25 Mar 2025).

Open problems recur across the literature. The original MODC paper highlights integrating memory-side replication and offload, extending correctness guarantees across broader crash models, adaptive task sizing, and richer metadata services in disaggregated memory. DOLMA points toward CXL-based pooling. The database work raises questions about DSM APIs, durability, availability, software coherence, buffer management, and RDMA-conscious index design. MIND leaves open larger directory capacities, weaker consistency models than TSO, and multi-rack scaling. DistZO2 suggests broader applicability of memory-first orchestration beyond LLM fine-tuning, but also makes clear that topology-aware mapping and overlap are prerequisites rather than optional optimizations (Keeton et al., 2021, Zheng et al., 2 Dec 2025, Wang et al., 2022, Lee et al., 2021, Wang et al., 3 Jul 2025).

Taken together, these results indicate that MODC is neither a synonym for distributed shared memory nor a mere implementation trick for disaggregation. It is a broad systems stance: place memory at the center of the design, expose or exploit its semantics directly, and derive scheduling, coherence, resilience, and parallelization policies from that memory organization.