--- title: In-Memory Distributed Object Store url: https://www.emergentmind.com/topics/in-memory-distributed-object-store type: topic --- # In-Memory Distributed Object Store An in-memory distributed object store is a system that manages and exposes collections of arbitrary data objects across a cluster of nodes, maintaining all or most object data in main memory to provide low latency access, high throughput, and resilience to typical datacenter-scale or cloud-scale failures. Such stores provide efficient sharing, replication, versioning, and consistency for a wide range of workloads, from transactional key-value access and high-performance analytics to collaborative and versioned data applications. Modern designs span disaggregated, memory-centric datacenter fabrics, integrate with non-volatile memory (NVM), and often provide cloud elasticity, programmable indexing, and durability through erasure coding or replication. ## 1. Architectural Taxonomy and Design Paradigms In-memory distributed object stores have evolved through several major architectural paradigms, each addressing system-level trade-offs in elasticity, resilience, locality, and interface expressiveness: - **Disaggregated Memory Stores**: Outback [2502.08982] exemplifies designs decoupling compute and memory into separate node pools—compute nodes run transactional and indexing logic entirely in user space, memory nodes provide object address spaces with no CPU intervention, and all communication is via one-sided (RDMA) or two-sided verbs. This separation enhances resource utilization and enables independent scaling. - **Memory-Disaggregated Hardware Backends**: The framework built on ThymesisFlow [2204.12889] demonstrates full memory disaggregation at the hardware level. Here, DRAM regions across nodes are exported as a single globally addressable region, and the object store (an extended Plasma) can transparently allocate buffers on local or remote nodes via FPGA-accelerated OpenCAPI. - **Serverless/Microservices-Based Stores**: InfiniStore [2209.01496] introduces "ServerlessMemory," a logically continuous memory space composed of many FaaS (Function-as-a-Service) instances, coupled with a persistent, inexpensive cloud object store for durability. Elasticity and resilience are achieved by leveraging the autoscaling, stateless, and ephemeral properties of serverless computing. - **In-Memory with Versioned and Rich Semantics**: UStore [1702.02799] provides an object store with inherent immutability, sharing, and security, exposing a fully versioned DAG abstraction per object, with rich branch/merge capabilities, authenticated history, and tamper-evident storage, while using memory-resident indexing for high throughput. - **RDMA-Accelerated Channel-Based Object Models**: LOCO [2503.19270] presents a compositional, object-oriented interface for building distributed objects ("channel objects") with their state partitioned and distributed across network memory, exposing the underlying weak coherence and locality of fabrics like RDMA or CXL directly to the programming model. - **All-Encoding, Small-Object Optimized Schemes**: MemEC [1701.08084] demonstrates how encoding small objects into fixed-size, erasure-coded chunks, alongside decentralized and coordinated I/O pipelines, minimizes redundancy and transition overhead during failures, specifically for small-object-dominated workloads. - **NVM-Centric Distributed Object Stores**: DAOS [2409.18682] leverages byte-addressable SCM/NVM devices, with all metadata in DRAM/scalable class memory, and implements lock-free, epoch-based atomic updates in user space, bypassing kernel bottlenecks for high-throughput I/O. ## 2. Core Mechanisms: Memory Management, Metadata, and Object Access ### Memory and Data Placement - **Segmented Allocators and Remote Access**: Systems such as the ThymesisFlow-based object store [2204.12889] allocate objects in contiguous DRAM chunks accessible via an FPGA-mediated address translation mechanism. Clients write objects locally or via proxy handles, with mmapped pointers forwarded transparently to remote memory. - **Distributed Hash Tables and Indices**: Outback [2502.08982] establishes a two-level, dynamic minimal perfect hashing index, minimizing both CPU load at storage (memory-side) and communication overhead. Compute nodes maintain subtree-locators and seeds for bucket assignment, incurring ~5 bits per key, while memory nodes keep dense, collision-free bucket maps. - **All-Encoded Chunks**: MemEC [1701.08084] packs key, value, and metadata into 4 KB chunks, seals chunks when full, and assigns each to a fixed (n, k) MDS erasure code stripe, optimizing for small-object packing density. - **Persistent and Elastic Layering**: InfiniStore [2209.01496] splits object store duties—short-lived in-memory pools absorb hot objects and provide fast path read/write, while background processes asynchronously flush and reconstruct data from persistent block/object stores to provide durability and recovery. ### Metadata Management - **In-Memory Manifests and Hash Maps**: DAOS [2409.18682] maintains all object metadata—including manifests, attribute trees, and lookup indices—in DRAM or NVM, with per-container structures persisted atomically at transaction boundaries. - **gRPC/rPC-Based Directory Services**: Metadata location in systems such as the ThymesisFlow-enhanced Plasma [2204.12889] is tracked via gRPC lookups, with distributed hash maps coupling object IDs to buffer addresses and lengths. - **Client-Side Caching and Version Tables**: InfiniStore [2209.01496] and Outback [2502.08982] employ lightweight, client-library resident metadata caches to minimize repeated lookups and maintain index consistency under workload migration or elastic scaling. ### Object Access Protocols - **RDMA Primitives**: One-sided RDMA (read/write/atomics) enables compute nodes to fetch object data or update state without remote CPU involvement [2502.08982, 2503.19270]. Two-sided RDMA (SEND/RECV) is leveraged for single-round-trip object RPCs. - **Consistent Hashing and Placement Rules**: In DAOS, a consistent hash on the object ID distributes object shards or replicas across storage targets, while UStore partitions object version DAGs using a hash of key and version-specific nonce [2409.18682, 1702.02799]. ## 3. Consistency, Concurrency, and Fault Tolerance - **Lock-Free, Epoch-Based Transactions**: DAOS [2409.18682] uses epoch numbers to group I/O calls and commits changes across multiple targets atomically at epoch close. This provides transactional crash-consistency and eliminates the need for central locking or consensus for normal operations. - **Linearizability and Explicit Fencing**: LOCO [2503.19270] exposes a fence-based programming model, where global, per-thread, or per-pair fences enforce ordering among RDMA verbs, allowing linearizable execution of distributed object methods. All state modifications can be forced into global visibility via explicit fences and per-object locking protocols. - **Decentralized vs Coordinated I/O Modes**: MemEC [1701.08084] supports decentralized, low-latency I/O when all servers are healthy, and transitions to a coordinated, redirect-based mode upon failures. State transitions (normal → intermediate → degraded) are orchestrated by a coordinator using atomic broadcasts. - **Replication, Erasure Coding, and Recovery**: InfiniStore and MemEC both leverage erasure codes to provide fault tolerance with lower storage overhead than replication [2209.01496, 1701.08084]. Upon loss of memory capacity (e.g., ephemeral FaaS reclaim or node failure), objects are restored from persistent backups or reconstructed in parallel using coding redundancies. - **Immutability and Versioned Authenticated Storage**: UStore [1702.02799] ensures all object versions are immutable, and cryptographically binds each version's payload, parent links, and metadata via content-addressed hashes and signatures, providing tamper-evident state and enabling efficient, lock-free multi-version concurrency. ## 4. Performance and Scalability ### Throughput and Latency - **RDMA-Accelerated Designs**: Outback [2502.08982] achieves 6.0 Mops for get-only YCSB-C workloads, superior to RACE, MICA, and Cluster, with peak latencies ~10 μs at scale. LOCO’s channel-based key-value store achieves 1.5 Mops (uniform) and 1.2 Mops (Zipfian) in 8-node clusters, outstripping hand-tuned custom RDMA systems by 10–30% [2503.19270]. - **Large-Scale Object Access**: DAOS, on 16-node, 0.24 PiB clusters, sustained 60 GiB/s writes and 90 GiB/s reads under 1 MiB I/O [2409.18682]. Plasma over ThymesisFlow [2204.12889] observed 5.75 GiB/s remote (6.5 GiB/s local) bandwidth, with per-object latencies 1.9–5 ms for small objects and ~0.075–2.6 ms for larger objects. - **Serverless Elasticity**: InfiniStore maintained linear throughput and sub-150 ms p90 latencies for 10–100 MB objects during workload scaling, and recovered 3 GB lost memory in ~1.2 s with 20 parallel recovery operations [2209.01496]. ### Overhead, Redundancy, and Transition Costs - **Erasure Coding Redundancy**: MemEC cuts redundancy by up to 58% versus triple replication, with overhead $r = n/k$, e.g., (10,8) MDS code yields $r = 1.25$ [1701.08084]. - **Transition Overhead**: MemEC’s state transitions from normal to degraded and back occur in ≤5 ms and ~600 ms, respectively, under load [1701.08084]. In degraded mode, set/update/get latencies increase by 11.6%, 50.9%, and 36.9% respectively. - **Elasticity Cost Model**: InfiniStore’s cost is $C_{\rm total} = C_{\rm mem} + C_{\rm req}$, where $C_{\rm mem}$ minimizes the memory in use and $C_{\rm req}$ scales with the number of FaaS invocations [2209.01496]. ### Comparative Analysis | Store/Framework | Peak Throughput | Storage Overhead | Fault Tolerance Mechanism | Reference | |:----------------:|:---------------:|:----------------:|:----------------------------------------:|:-----------------:| | Outback | 6.0 Mops (get) | ~7 bits/key | compute-side index + end-to-end hashing | [2502.08982] | | DAOS | 90 GiB/s (read) | k+m, tunable | epoch-based txns, EC/replication | [2409.18682] | | MemEC | 800 Kops/s (get)| 1.25× (10,8 EC) | all-encoding, chunked EC, coordinator | [1701.08084] | | InfiniStore | 95% SMS hit ratio| EC, minimal | erasure codes, snapshotted log | [2209.01496] | | Plasma/Thymesis | 5.75 GiB/s remote| N/A | currently none, possible replication EC | [2204.12889] | | LOCO | 1.5 Mops (kv) | explicit mgmt | channel fencing, programmable semantics | [2503.19270] | | UStore | 430 Kops/s | versioned | immutable DAG, R+W > N consensus | [1702.02799] | ## 5. Interface Expressiveness, Applications, and Semantics - **Rich Object Abstractions**: UStore [1702.02799] exposes a full version DAG per object with per-version access control, authenticated history, and efficient version scan APIs, enabling direct support for Git-like systems, relational collaboration, weakly-consistent transactions, and blockchain ledgers. LOCO [2503.19270] provides composable, network-resident channel objects (registers, queues, key-value maps). - **API Multiplicity and Compatibility**: DAOS [2409.18682] offers native object, POSIX-lite, and FUSE-based interfaces, supporting both high-performance custom applications and legacy POSIX software with varying degrees of fidelity and performance. - **Elasticity for Cloud and Serverless**: InfiniStore [2209.01496] and similar architectures natively absorb variable working sets and provide fine-grained scaling and pay-per-access models, suitable for cloud registries, caching, and as a bridge between fast memory and durable, slow storage layers. - **Small vs. Large Object Optimization**: MemEC [1701.08084] is designed for workloads with small objects, optimizing packing density, redundancy, and recovery overhead. Outback [2502.08982] and Plasma/ThymesisFlow [2204.12889] are object-size agnostic, but throughput of disaggregated memory rises for ≥1 MB objects. ## 6. Limitations, Trade-offs, and Open Challenges - **Consistency and Incoherence**: Systems leveraging low-level RDMA or uncacheable memory (LOCO [2503.19270], Outback [2502.08982]) avoid hardware coherence overhead by exposing explicit fencing and placement to the application, trading ease of use for sharper performance/correctness control. - **Metadata Overhead for Small Objects**: For high object counts with small object sizes, gRPC lookup latency and metadata mapping overhead become significant relative to data transfer [2204.12889]. Caching and batching reduce, but do not eliminate, this penalty. - **Fault Tolerance and Recovery**: Serverless stores depend on fast, parallel rehydration from persistent layers (InfiniStore [2209.01496]), but as lost working set size or degree of FaaS churn increases, background traffic and cold start overhead become dominant. - **Mutable Object Support and Write Consistency**: Most systems, e.g. Plasma/Thymesis, expose only immutable sealed objects to avoid write races, complicating certain update-heavy applications [2204.12889]. - **Scaling Coordination and State Transition**: MemEC’s reliance on a single coordinator (potential single point of failure) and the need for atomic broadcast during state transitions may impose overhead at scale [1701.08084]. Extensions to replicated coordinators are suggested. - **End-to-End Security**: While UStore [1702.02799] deeply integrates version hash authentication and per-version signatures, most operator- or network-facing object store APIs do not provide object-level encryption or MACs by default. ## 7. Future Directions - **Programmable SmartNIC and In-Network Offload**: Outback proposes leveraging SmartNICs (e.g., PRISM-style extended RDMA operations) to offload indexing and data motion, achieving storage-node CPU elimination and potentially improved latency [2502.08982]. - **Memory Disaggregation Hardware Evolution**: Migration to new architectures such as IBM POWER10 Memory Inception and global memory fabrics with improved cache coherence or persistent memory tiers is an ongoing area [2204.12889]. - **Weak and Strong Consistency Models**: Further investigation into combining explicit fence-based (LOCO [2503.19270]) and optimistic transaction (DAOS [2409.18682]) programming models with application-specific consistency requirements. - **Multi-Object Transactions and Advanced Indexing**: Ongoing work in embedding transactional protocols and secondary indices, including learned indexes and hybrid B+-tree/perfect hash data planes, will expand the applicability of RDMA and memory-disaggregated stores to broader classes of workloads [2502.08982]. - **Cross-Store Metadata Management**: Enhanced caching, publish/subscribe, and lightweight distributed consensus for in-use tracking and eviction across large object stores are critical for large-scale, multi-tenant deployments [2204.12889]. - **Application Integration**: End-to-end benchmarking on production analytics frameworks and AI/ML platforms (e.g., Spark, Dask) remains a central need to measure the real-world impact and trade-offs of emerging in-memory distributed object store designs. --- For further empirical details, design rationale, and code references, see the cited works: Outback [2502.08982], DAOS [2409.18682], InfiniStore [2209.01496], MemEC [1701.08084], ThymesisFlow/Plasma [2204.12889], LOCO [2503.19270], and UStore [1702.02799]. Source: https://www.emergentmind.com/topics/in-memory-distributed-object-store