OffloadDB: RocksDB Offloading in Disaggregated Storage

• OffloadDB is a RocksDB variant that offloads MemTable flush and compaction to a disaggregated NVMe-oF node, harnessing idle CPU and memory resources.
• It employs initiator-centric metadata management and selective offloading with Log Recycling to avoid distributed locks and minimize network traffic.
• Performance evaluations indicate up to 3.36× throughput gains and improved write/read efficiencies by shifting I/O-intensive tasks to the storage node.

Searching arXiv for papers related to OffloadDB and database offloading. OffloadDB is a RocksDB variant built on top of OffloadFS that enables RocksDB to offload MemTable flush and compaction operations to a disaggregated NVMe-oF storage node, while leaving WAL reads/writes and MANIFEST updates on the initiator. It is presented as a near-data processing design for I/O-intensive background work in LSM-tree key-value stores, motivated by the observation that NVMe-oF target nodes already possess CPU and memory resources that are often underutilized beyond transport handling, DMA buffering, and volume management (Moon et al., 15 Apr 2026). By specializing for selectively offloaded tasks from a logically single-initiator application, OffloadDB avoids the distributed metadata management and distributed locking associated with conventional shared-disk file systems, and instead relies on initiator-centric metadata ownership and block-level authorization.

1. System setting and design objective

OffloadDB is situated in a disaggregated storage environment in which NVMe over fabrics exposes remote NVMe SSDs as low-overhead block devices and allows multiple initiators to share a target volume at block-address granularity. Within that setting, the storage node’s CPU and memory are treated as execution resources for near-data computation rather than as passive support for I/O delegation alone (Moon et al., 15 Apr 2026).

The immediate target is RocksDB. The paper identifies four main RocksDB I/O activities: WAL logging for each write request, flushing MemTables into L0 SSTables, compaction or merge-sorting of SSTables, and updating the MANIFEST file. OffloadDB offloads the two background I/O operations—MemTable flush and compaction—while retaining WAL and MANIFEST handling on the initiator. This division is explicit: WAL and MANIFEST are accessed frequently by foreground threads, so keeping them local avoids contention and preserves fast request handling.

A common misconception is to treat OffloadDB as a general shared-disk file-system design. The underlying model is narrower. OffloadFS is specialized for selective offloading in applications that are logically single-initiator, and this specialization is what lets it avoid distributed lock management. The target node is not a peer metadata owner; it is an execution site given bounded authority over already allocated blocks.

2. OffloadFS substrate and initiator-centric consistency

OffloadDB inherits its execution model from OffloadFS, described as an initiator-centric, user-level file system. The initiator owns file-system metadata, including the inode table, extents, and extent allocation and defragmentation. On the initiator side, OffloadFS provides an Extent Manager, which allocates extents, manages free space, and maintains per-file extent trees, and a Task Offloader, which decides when to offload and sends offload requests via gRPC. On the target side, it provides an Offload Engine, an Offload Cache, and SPDK-based offload_read() and offload_write() primitives used by target-side stub code (Moon et al., 15 Apr 2026).

Offloaded work is implemented as a gRPC stub/skeleton pair. The target node can read existing extents, update existing extents, and return results, but it cannot perform metadata-changing operations such as truncate, fallocate, stat, or ioctl. The initiator pre-allocates the needed blocks, passes block addresses and permissions for a specific task, and tracks which blocks are in use by offloaded work. It does not access those blocks until completion is reported back. The paper characterizes this as initiator-centric block management.

The cache-coherence model is likewise deliberately restricted. OffloadFS does not attempt conventional distributed cache coherence. The initiator side does not provide a cache through OffloadFS itself; the target’s Offload Cache may store blocks used by offloaded tasks; and coherence is pushed upward to the application or initiator. The initiator may explicitly decide whether to use cached blocks or bypass the cache, and it may pass file modified time so that the Offload Engine can bypass stale cache entries if the file has changed. This is coarse-grained by design.

Because a storage node may serve many initiators, OffloadFS includes two load-control mechanisms: threshold-based rejection, where the storage node rejects new offload requests if CPU usage exceeds a threshold, and token-based scheduling, where the storage node hands out tokens that allow a limited number of offload requests for a limited time. These mechanisms are directly relevant to OffloadDB because the storage node can otherwise become the bottleneck under multi-tenant offloading pressure.

3. RocksDB integration and execution path

OffloadDB maps RocksDB’s storage-engine workflow onto the OffloadFS execution substrate. The offloaded and non-offloaded activities are fixed by design.

RocksDB activity	Placement in OffloadDB
WAL logging for each write request	Initiator
Flushing MemTables into L0 SSTables	Offloaded
Compaction / merge-sorting SSTables	Offloaded
Updating the MANIFEST file	Initiator

The foreground write path remains initiator-local. When a client inserts a key-value pair, the record goes into the MemTable on the initiator, and the WAL is appended locally through SPDK if preallocated space is available; otherwise the Extent Manager allocates more blocks (Moon et al., 15 Apr 2026).

When a MemTable becomes immutable, the initiator’s background thread collects the file offsets of log entries and sends a compact offset array to the storage node. The Extent Manager allocates output SSTable space on the target, and the target-side Log Recycler reads the WAL directly and reconstructs the L0 SSTable in sorted order. This is the mechanism by which OffloadDB turns flush into a storage-side task.

Compaction is also initiated by the owner but executed remotely. The MANIFEST is examined to determine victim SSTables; the initiator chooses which SSTables to compact; the Extent Manager allocates output blocks for the resulting SSTables; and the block addresses and metadata for victim SSTables are sent to the target via RPC. The target-side Offload Engine reads the victim SSTables, merge-sorts them, and writes new SSTables; metadata about the new SSTables is returned to the initiator; and the initiator updates the MANIFEST, which acts as the commit point. The paper notes that if a crash occurs after output blocks are allocated but before the MANIFEST update, those blocks are garbage and can be reclaimed, matching RocksDB-style commit semantics.

4. Log Recycling, L0 caching, and cache-pollution control

The central OffloadDB-specific optimization is Log Recycling. Its purpose is to avoid sending the same key-value data twice over the network: once for the WAL and again when flushing to an SSTable. Instead of retransmitting the key-value payload, OffloadDB stores log-entry offsets in the MemTable; when the MemTable becomes immutable, the target node reads the WAL blocks and reconstructs the SSTable from those offsets. The flush communication is therefore small, because only offsets and block addresses are sent (Moon et al., 15 Apr 2026).

A second optimization is the L0 cache on the initiator. Immutable MemTables or L0 SSTables are kept locally for a short period until L0→L1 compaction deletes them from the MANIFEST. The paper’s sizing argument is concrete: with 64 MB MemTables, a 3 GB L0 cache can hold about 48 immutable MemTables, which is considered sufficient because RocksDB usually triggers L0→L1 compaction when L0 exceeds 10 SSTables. This design keeps data needed by foreground queries and background compaction close to the initiator while allowing actual SSTable reconstruction to be deferred.

These mechanisms are framed as responses to cache pollution. In vanilla RocksDB, compaction reads victim SSTables and writes new SSTables, and these blocks may occupy the page cache even though they are not useful to foreground queries. OffloadDB mitigates this by separating foreground and background caching and by using the storage node’s Offload Cache for background access while reserving the initiator’s L0 cache for short-lived immutable MemTables. In the reported cache-pollution experiment, OffloadDB reduces accesses to SSTables substantially and improves throughput by about 32%. The paper also reports that Log Recycling improves write throughput by up to 9% over compaction offloading without Log Recycling, and improves read throughput on the read-only workload by about 40%, helped by the L0 cache.

The broader claimed benefits are correspondingly specific: reduced write stall by moving compaction off the initiator, use of the target node’s idle CPU and memory, reduced network traffic through Log Recycling, reduced cache pollution because background compaction runs away from the initiator’s foreground cache, and additional gains when the Offload Cache is enabled.

5. Evaluation, performance, and operating envelope

The evaluation uses a 9-node cluster consisting of 8 compute nodes and 1 NVMe-oF storage node. The storage node has 2 Intel Xeon Silver 4215 CPUs, 128 GB DRAM, and 24 Samsung PM9A3 NVMe SSDs, and runs PoseidonOS for NVMe-oF management. Each compute node has 2 Intel Xeon Gold 5115 CPUs and 64 GB DRAM. For RocksDB and OffloadDB, the setup uses 32 client threads, 20 background threads, 64 MB SSTable size, a 200 GB database, and 24-byte keys and 1 KB values, with YCSB workloads including write-heavy and mixed cases (Moon et al., 15 Apr 2026).

The comparison set includes OCFS2, GFS2, vanilla RocksDB, SpanDB, and Hailstorm. The paper also distinguishes three OffloadDB configurations: ODB-LR-C, which enables compaction offloading only and disables both Log Recycling and Offload Cache; ODB-C, which adds Log Recycling but still disables Offload Cache; and ODB, which enables all designs including Offload Cache.

The headline result is that OffloadFS improves RocksDB performance by up to 3.36× compared to OCFS2. In the write-heavy RocksDB experiment, OCFS2 without offloading achieves its best performance at about 110 Kops/sec in the Local case, whereas OffloadFS or OffloadDB achieves about 379 Kops/sec when all compaction tasks are offloaded to the target or peer. Relative to plain RocksDB, compaction offloading alone gives about 1.51× higher throughput on the Load workload.

The performance gains are not uniform across workloads. OffloadDB is worse than vanilla RocksDB only for the scan-intensive YCSB Workload E, which the authors attribute to OffloadFS not being optimized for sequential scans. This limitation is central to interpreting the system: OffloadDB is optimized for background maintenance and I/O locality, not for scan-heavy data paths. The storage node may also become the bottleneck if too many initiators offload simultaneously. Under these conditions, the token policy performs slightly better than CPU-threshold rejection because it avoids repeated rejected requests, though it requires tuning of token lifetime.

The baselines expose additional trade-offs. SpanDB is sometimes worse than RocksDB in this setup because it flushes WAL aggressively in read-committed mode, increasing overhead. Hailstorm performs very poorly, largely due to its FUSE-based implementation and Akka-based communication overhead. These comparisons position OffloadDB not simply as a faster RocksDB variant, but as a specific point in the design space of disaggregated LSM execution.

6. Position within database offloading research

OffloadDB belongs to a broader class of database systems that relocate work toward storage, memory, or network-adjacent resources in order to reduce data movement or relieve host-side contention. Its distinguishing feature is that it offloads background LSM maintenance to an NVMe-oF storage node under an initiator-centric consistency model, rather than offloading general query operators or treating remote storage as a passive device (Moon et al., 15 Apr 2026).

In Farview, the offload target is not disaggregated storage but disaggregated memory implemented as an FPGA-based smart NIC with on-board DRAM and operator offloading capabilities. Farview exposes a remote buffer cache and supports offloading operators such as selection, projection, aggregation, regular expression matching, and encryption; it is reported as competitive with a local buffer cache solution for all workloads and better in a number of cases (Korolija et al., 2021). The contrast is instructive: Farview offloads analytical query operators over remote DRAM, whereas OffloadDB offloads storage-engine maintenance over NVMe-oF.

In cloud OLAP systems, Adaptive pushdown addresses a different problem: existing pushdown decisions are often static, while storage-layer computational capacity is shared and variable. That work proposes runtime pushback from storage to compute and a general criterion that pushdown-amenable operators should be local and bounded, reporting up to 1.9x speedup over both No pushdown and Eager pushdown baselines and up to 3.0x further acceleration from new pushdown operators (Yang et al., 2023). A plausible implication is that OffloadDB’s threshold-based rejection and token-based scheduling are storage-node admission-control mechanisms analogous in spirit, though applied to LSM maintenance rather than OLAP operator placement.

For LSM-tree systems specifically, O$^3$-LSM extends the offloading agenda beyond compaction by adding memtable Offloading and flush Offloading on top of compaction Offloading via shared Disaggregated Memory. It reports up to 4.5X write throughput improvement, up to 5.2X range-query throughput improvement, and up to 76% P99 latency reduction over Disaggregated-RocksDB, CaaS-LSM, and Nova-LSM (Lin et al., 5 Mar 2026). This suggests that OffloadDB occupies an earlier point in the same trajectory: it demonstrates that storage-node execution can accelerate flush and compaction without distributed locks, while later systems redesign a larger fraction of the mutable-data lifecycle.

A further extension appears in UDON, which studies offloading to general-purpose compute on CXL memory devices. For vector databases, UDON reports that offloading HNSW kernels can provide upto 6.87× performance improvement with under 10% offload overhead (Hermes et al., 2024). The common thread is not a shared implementation substrate but a shared premise: once a disaggregated component exposes both data proximity and general-purpose compute, it becomes a candidate execution site for functions whose dominant cost is data movement rather than arithmetic density.

Taken together, these systems place OffloadDB within the broader transition from storage disaggregation as a capacity mechanism to disaggregation as a computation-placement mechanism. In that landscape, OffloadDB’s specific contribution is to show that RocksDB’s MemTable flush and compaction can be offloaded to NVMe-oF storage nodes through initiator-centric block management, Log Recycling, and cache-separation policies, yielding substantial throughput gains while preserving RocksDB-style commit semantics.