Distributed Asynchronous Object Storage (DAOS)

• Distributed Asynchronous Object Storage (DAOS) is an open-source, scale-out object store that leverages non-volatile memory and asynchronous, transactional I/O to optimize high-performance computing tasks.
• DAOS organizes data into pools and containers, eliminating a global namespace to enable fault tolerance and efficient handling of metadata-heavy, small-object workloads.
• Its modular design, featuring user-space storage engines and diverse interface layers like libdaos and DFuse, delivers low latency and scalable throughput in demanding HPC environments.

Distributed Asynchronous Object Storage (DAOS) is an open-source, user-space, scale-out object store for high-performance computing, designed to exploit byte-addressable non-volatile memory, including Intel Optane DCPMM, backed by NVMe SSDs and accessed over high-speed fabrics through asynchronous, transactional I/O. In the DAOS literature, it is presented as an alternative to conventional distributed POSIX file systems for workloads in which excessive consistency assurance, centralized metadata services, and prescriptive file-system metadata structures become limiting factors, especially under small-object, metadata-heavy, or mixed read/write conditions (Manubens et al., 2024, Manubens et al., 2022).

1. Design rationale and conceptual scope

Early DAOS work framed the system as part of the exascale I/O transition: an object-based storage substrate intended to provide straightforward functionality, fault tolerance, and efficiency for NVRAM- and NVMe-backed storage tiers close to compute nodes (Breitenfeld et al., 2017). Later studies sharpened that rationale by contrasting DAOS with POSIX-style parallel file systems such as Lustre and GPFS, whose block-oriented design, kernel-level data paths, centralized metadata servers, and lock-based coherence mechanisms can become bottlenecks under contention or for small I/O (Manubens et al., 2024).

A central distinction is that DAOS is not organized as a conventional global directory tree. Data is instead arranged into pools and containers, with application-visible objects accessed through native object APIs or higher-level adapters. A 2026 study of DAOS in Numerical Weather Prediction states this difference explicitly: there is no global namespace or directory tree; data is organized into pools and containers; and there are no dedicated metadata servers, because placement and metadata decisions are made algorithmically by clients (Gil, 19 Feb 2026).

This design does not imply that DAOS replaces POSIX in all circumstances. The literature repeatedly treats it as a storage substrate that can coexist with POSIX systems. The same 2026 study argues that object stores such as DAOS can be deployed side-by-side with existing POSIX file systems, while several interface papers emphasize DFuse, libdfs, MPI-IO, and HDF5 connectors as migration paths for legacy software (Gil, 19 Feb 2026, Jackson et al., 2023).

2. Storage engines, pools, containers, and objects

The core DAOS server abstraction is the storage “engine,” typically one per CPU socket, each managing local storage targets and attached media. Across the literature, engines are described as user-space services that manage SCM and NVMe, export multiple targets for concurrency, and communicate with clients through low-latency RPCs over libfabric/OFI or related RDMA-capable transports (Manubens et al., 2022, Manubens et al., 2022). On NEXTGenIO deployments, for example, DAOS commonly used two engines per node and 12 targets per engine, with affinity controlled so that engines and network interfaces shared the same NUMA socket (Manubens et al., 2022, Manubens et al., 2024).

DAOS organizes storage as a hierarchy:

Pool $\rightarrow$ Container $\rightarrow$ Object $\rightarrow$ dkey/akey $\rightarrow$ Value.

A pool aggregates capacity across targets; containers are transactional address spaces within pools; and objects, identified by 128-bit object identifiers, are exposed as key-value entities or arrays. Several papers describe the low-level addressing model as a two-level key scheme in which the distribution key, or dkey, determines placement and co-location, while the attribute key, or akey, names a value or extent within the object (Allen et al., 10 Sep 2025, Manubens et al., 2024). Object classes such as OC_S1, OC_S2, OC_SX, replicated classes, and erasure-coded layouts determine striping, replication, or coding behavior across targets (Manubens et al., 2022, Gil, 19 Feb 2026).

The internal implementation is commonly described as a log-structured key-value engine for both metadata and data. A 2023 interface paper states that metadata such as object indices and container metadata is held in Optane PMem or in DRAM backed by NVMe, while client writes are batched into RPCs on the client side and merged into persistable transactions on the server side before being flushed to backing media (Jackson et al., 2023). Other studies emphasize zero-copy I/O to SCM for low latency and the decoupling of metadata and data paths (Manubens et al., 2022, Manubens et al., 2024).

Aurora provides the largest deployment description in the cited corpus. There, DAOS is deployed across 1,024 dual-socket storage nodes, giving 2,048 DAOS engines, each node equipped with 16 NVMe SSDs, 16 Intel Optane DIMMs, and two 200 Gb/s Cassini NICs on the Slingshot fabric (Allen et al., 10 Sep 2025).

3. Transactions, MVCC, and resilience

DAOS exposes a fully asynchronous I/O model. Native calls in libdaos and higher-level APIs are nonblocking, with completion signaled through event handles or event queues. The Aurora paper illustrates this with explicit daos_event_t usage, in which an application issues an operation such as daos_obj_open(...), waits on the event, and finalizes it only after completion (Allen et al., 10 Sep 2025). The design goal is to permit hundreds of concurrent requests without blocking compute threads, thereby overlapping computation and storage activity (Jackson et al., 2023).

Consistency is typically described through epochs, transactions, and multiversion concurrency control. In the ECMWF NWP work, writes never overwrite in place; instead, each write appends a new version that becomes visible only when the associated epoch is committed. An epoch is a client-assigned 64-bit identifier, and all operations tagged with the same epoch form a transaction; daos_epoch_commit makes them visible atomically, while daos_epoch_abort rolls them back (Manubens et al., 2024). Earlier DAOS descriptions used the same container-level epoch model and stressed that a committed epoch is immutable and durable, with no partial visibility of a transaction (Breitenfeld et al., 2017).

Fault tolerance and integrity are integrated into the object model. Object classes encode replication or erasure coding, and metadata is protected by a small replicated consensus group. Recent papers describe metadata consistency as RAFT-based, while the Aurora paper refers to a small Paxos/Raft group maintaining the pool and container map (Jackson et al., 2023, Allen et al., 10 Sep 2025). Aurora also gives a concrete erasure-coded layout, EC_16P2GX, in which each 64 KiB block is striped across 18 engines with 16 data and 2 parity stripes (Allen et al., 10 Sep 2025). Earlier work additionally emphasized end-to-end checksums for both data and metadata, validated at send, receipt, and storage time (Breitenfeld et al., 2017).

The literature also makes clear that lock avoidance is a server-side property of the object model rather than a blanket absence of serialization. MVCC removes the need for distributed POSIX-style locks on shared extents, but POSIX-facing layers can still serialize updates in their own translation logic. For example, libdfs is reported to use per-object locks on the client side to serialize concurrent updates against the same file (Manubens et al., 2024, Jackson et al., 2023). This distinction is important in understanding why DAOS can simplify application logic relative to manual locking in Lustre while still imposing ordering within specific interface layers (Manubens et al., 2022).

4. Interface layers and middleware integration

DAOS exposes several interface families. At the lowest level is the native libdaos API, including calls such as daos_pool_create, daos_cont_open, daos_obj_open, daos_obj_update, daos_array_write, daos_array_read, daos_kv_put, and daos_kv_get (Manubens et al., 2024, Allen et al., 10 Sep 2025). Above that, libdfs provides a POSIX-like abstraction implemented on top of DAOS objects, while DFuse mounts a DAOS container through FUSE and translates system calls such as open, read, write, getattr, and readdir into libdfs or lower-level object operations (Jackson et al., 2023, Manubens et al., 2024).

Higher-level scientific interfaces map their native abstractions onto pools, containers, and DAOS objects. The HDF5 DAOS VOL plugin maps each HDF5 file to a DAOS container; groups and datasets become DAOS objects identified by HDF5 object IDs; and attributes become small key-value objects in the same container (Jackson et al., 2023). MPI-IO support is provided through a ROMIO DAOS driver that maps file handles to DAOS objects and performs collective epoch commits (Jackson et al., 2023). Earlier HDF5 integration work emphasized that this arrangement preserves the HDF5 API while replacing the conventional byte-stream file format with DAOS objects and key-value metadata stores (Breitenfeld et al., 2017).

Domain-specific storage layers have used DAOS in more specialized ways. ECMWF’s FDB backend maps archive(), flush(), retrieve(), and list() onto DAOS pool/container operations, array objects for field payloads, and key-value objects for indexes (Manubens et al., 2024). ROOT’s RNTuple backend maps columnar pages and clusters onto DAOS objects, dkeys, and akeys. In its improved locality-driven mapping, pages belonging to the same cluster-column pair are colocated in one object:

$$\phi_{\text{locality-driven}}(Cluster_i, Column_j, Page_k) \mapsto \langle oid = \mathrm{hash}(Cluster_i, Column_j),\ dkey = Cluster_i,\ akey = Page_k \rangle,$$

so that DAOS can serve batched, co-located I/O rather than isolated page fetches (Miotto et al., 2023).

Interface choice measurably affects performance. In a 2023 DAOS interface study, MPI-IO on DFuse stayed within 3–5% of native DFS for large sequential I/O, whereas HDF5 via DFuse incurred the highest overhead, approximately 20–25%, owing to collective metadata operations and extra copy steps through the FUSE page cache (Jackson et al., 2023). The same study reported that DFuse’s single-threaded FUSE event loop can limit per-client throughput to about 6 GiB/s and that small-file creation stalls at approximately 10 K files/s when multiple clients contend on the same directory object (Jackson et al., 2023). A later cloud study found that for 1 MiB transfers, libdaos, libdfs, DFUSE, and DFUSE+IL all reached 57–59 GiB/s write and 87–90 GiB/s read on 16 DAOS servers, but for 1 KiB operations DFUSE+IL improved IOPS by about $4\times$ over plain DFUSE (Manubens et al., 2024).

5. Performance characteristics and scaling behavior

The DAOS literature usually reports throughput as

$$\mathrm{BW} = \frac{\text{Total bytes transferred}}{\text{Elapsed time}},$$

and IOPS as

$$\mathrm{IOPS} = \frac{\text{Number of I/O operations}}{\text{Elapsed time}}.$$

Some studies distinguish IOR’s “synchronous bandwidth” from “global timing bandwidth,” the latter defined over the interval from the first operation start to the last completion across all processes (Manubens et al., 2022, Manubens et al., 2022).

For bulk, large-file I/O, DAOS is generally reported as comparable to Lustre on identical hardware. On NEXTGenIO, one comparative study found that Lustre scaled at about 6 GiB/s write per node and 9 GiB/s read per node, while DAOS scaled at about 5 GiB/s write per node and 7.5 GiB/s read per node for IOR “segments” mode. The same study concluded that for bulk large-file I/O, “DAOS $\approx$ Lustre on identical hardware, with linear scaling,” while noting that DAOS required only $2\times$ clients to saturate I/O at larger server counts, compared with approximately $\rightarrow$0 for Lustre (Manubens et al., 2022). An earlier NWP-oriented study on the same platform also reported almost linear IOR scaling, with per-engine rates of about 2.5 GiB/s write and 3.75 GiB/s read, exceeding 80 GiB/s write and 120 GiB/s read on 16 nodes even over TCP (Manubens et al., 2022).

The distinctive DAOS advantage appears in object-style and metadata-heavy workloads. For 1 MiB field-style objects on NEXTGenIO, the DAOS-versus-Lustre comparison reports that DAOS outperformed Lustre by factors of 3–5$\rightarrow$1 for object-style workloads involving 1–10 MiB objects and many metadata operations (Manubens et al., 2022). In Pattern B, where half the processes wrote while half read concurrently, DAOS in “no-containers” mode sustained aggregate throughput of about 50 GiB/s at 8 server nodes, compared with about 20 GiB/s for Lustre in full mode (Manubens et al., 2022). ECMWF’s later FDB backend study similarly found that under write-plus-read contention DAOS write and read throughput both scaled near-linearly to roughly 35–40 GiB/s at 12 servers, whereas Lustre degraded sharply beyond 4 servers to about 25 GiB/s (Manubens et al., 2024).

Object size and mapping strategy matter. The 2022 NWP benchmark found that performance climbed steeply as array size increased, peaking around 5–10 MiB per array before flattening beyond 20 MiB (Manubens et al., 2022). The ROOT RNTuple backend reached 4.5 $\rightarrow$2 0.1 GB/s write and 6.3 $\rightarrow$3 0.1 GB/s read on a single-client, two-server testbed once locality-driven mapping, vector I/O, and 1 MiB scatter–gather batches were used; the paper reports up to $\rightarrow$4 write and 4.3$\rightarrow$5 read improvements depending on compression settings and page sizes (Miotto et al., 2023).

At larger scale, Aurora provides the most extensive published DAOS deployment in this corpus. Its DAOS tier peaks at approximately 31 TB/s aggregate bandwidth and about 400 M IOPS, matching exascale compute injection bandwidth. A simple model in that paper writes total throughput as $\rightarrow$6, yielding approximately $\rightarrow$7 GiB/s $\rightarrow$8 TiB/s. The paper also reports near-linear scaling of small writes up to 660 servers in IO500, an IO500 score of 32,165.90, and microbenchmark latencies below 500 $\rightarrow$9s for 4 KiB reads and below 1 ms for writes at the 95th percentile across 512 engines (Allen et al., 10 Sep 2025).

6. Scientific use, deployment practice, and open trade-offs

DAOS has been integrated into several scientific and data-intensive workflows. In operational Numerical Weather Prediction, ECMWF implemented DAOS backends for both the storage and catalogue components of the FDB, preserving the high-level archive, flush, retrieve, and list interface while substituting DAOS arrays and key-value stores underneath (Manubens et al., 2024). In high-energy physics, ROOT RNTuple added a DAOS backend so that analyses can select DAOS by changing a storage URI; later work redesigned the mapping and vectorization strategy to better saturate the network and storage link (López-Gómez et al., 2021, Miotto et al., 2023). Aurora’s Early Science Program used DAOS for checkpointing and data analytics, and the CANDLE Pilot Uno workflow is reported to have saved hundreds of thousands of small model snapshots to DAOS with aggregate bandwidths greater than 10 GB/s per rack (Allen et al., 10 Sep 2025).

Practical deployment guidance in the literature is explicit. Recommended practices include maintaining at least twice as many client nodes as server nodes to saturate network and storage; matching process count to engine count, for example about 24–36 tasks per engine or 16–32 processes per client node depending on the transport; minimizing container open/close frequency in metadata-heavy codes; pre-allocating OIDs; and selecting object classes according to workload, such as OC_S1 for 1 MiB objects in some NWP cases and SX for high-contention writes in others (Manubens et al., 2022, Manubens et al., 2024, Jackson et al., 2023). Process and engine pinning are also critical: one study reports up to 50% throughput loss on servers and up to 90% on clients under suboptimal pinning (Manubens et al., 2022).

The main trade-off is that DAOS is not simply a faster POSIX file system. Several studies show that for large sequential I/O, DAOS often matches rather than exceeds a well-configured Lustre deployment, while interface layers such as DFuse or HDF5 can reintroduce serialization, metadata overhead, or page-cache effects (Manubens et al., 2022, Jackson et al., 2023). Porting applications may require redesigning indexes, choosing collocation keys to avoid hot spots, and rethinking assumptions tied to POSIX operations such as O_APPEND or directory-centric metadata (Gil, 19 Feb 2026). This suggests that DAOS is most advantageous where applications can either use native object semantics directly or map domain abstractions onto DAOS arrays and key-value objects without inheriting unnecessary file-system behavior.

Current research extends DAOS into new environments rather than altering its server architecture. ROS2, for example, relocates the DAOS DFS client onto an NVIDIA BlueField-3 SmartNIC while leaving the DAOS I/O engine unchanged on the storage server. In that design, RDMA on the DPU preserves near-host end-to-end throughput, whereas TCP on the DPU lags host performance; the motivation is to enable multi-tenant isolation and inline services close to the NIC, with GPU-direct placement left as future work (Zhu et al., 17 Sep 2025).