LOCO: Rethinking Objects for Network Memory

Abstract: In this work, we explore an object-based programming model for filling the space between shared memory and distributed systems programming. We argue that the natural representation for resources distributed across a memory network (e.g. RDMA or CXL) is the traditional shared memory object. This concurrent object (which we call a "channel" object) exports traditional methods, but, especially in an incoherent or uncacheable memory network, stores its state in a distributed fashion across all participating nodes. In a sense, the channel object's state is stored "across the network". Based on this philosophy, we introduce the Library of Channel Objects (LOCO), a library for building multi-node objects on RDMA. Channel objects are composable and designed for both the strong locality effects and the weak consistency of RDMA. Unlike prior work, channel objects do not hide memory complexity, instead relying on the programmer to use NUMA-like techniques to explicitly manage each object. As a consequence, our channel objects have performance similar to custom RDMA systems (e.g. distributed maps), but with a far simpler programming model. Our distributed map channel has better read and comparable write performance to a state-of-the-art custom RDMA solution, using well-encapsulated and reusable primitives.