---
title: 'OffloadDB: RocksDB Offloading in Disaggregated Storage'
url: https://www.emergentmind.com/topics/offloaddb
type: topic
---

# OffloadDB: RocksDB Offloading in Disaggregated Storage

Searching arXiv for recent papers related to OffloadDB and database offloading.
OffloadDB is a RocksDB variant built on top of OffloadFS that enables RocksDB to offload **MemTable flush** and **compaction** operations to a disaggregated NVMe-oF storage node, while leaving **WAL reads/writes** and **MANIFEST updates** on the initiator. It is presented as a near-data processing design for I/O-intensive background work in LSM-tree key-value stores, motivated by the observation that NVMe-oF target nodes already possess CPU and memory resources that are often underutilized beyond transport handling, DMA buffering, and volume management [2604.13743]. By specializing for selectively offloaded tasks from a logically single-initiator application, OffloadDB avoids the distributed metadata management and distributed locking associated with conventional shared-disk file systems, and instead relies on initiator-centric metadata ownership and block-level authorization.

## 1. System setting and design objective

OffloadDB is situated in a disaggregated storage environment in which NVMe over fabrics exposes remote NVMe SSDs as low-overhead block devices and allows multiple initiators to share a target volume at block-address granularity. Within that setting, the storage node’s CPU and memory are treated as execution resources for near-data computation rather than as passive support for I/O delegation alone [2604.13743].

The immediate target is RocksDB. The paper identifies four main RocksDB I/O activities: WAL logging for each write request, flushing MemTables into L0 SSTables, compaction or merge-sorting of SSTables, and updating the MANIFEST file. OffloadDB offloads the two background I/O operations—MemTable flush and compaction—while retaining WAL and MANIFEST handling on the initiator. This division is explicit: WAL and MANIFEST are accessed frequently by foreground threads, so keeping them local avoids contention and preserves fast request handling.

A common misconception is to treat OffloadDB as a general shared-disk file-system design. The underlying model is narrower. OffloadFS is specialized for **selective offloading** in applications that are logically single-initiator, and this specialization is what lets it avoid distributed lock management. The target node is not a peer metadata owner; it is an execution site given bounded authority over already allocated blocks.

## 2. OffloadFS substrate and initiator-centric consistency

OffloadDB inherits its execution model from OffloadFS, described as an **initiator-centric, user-level file system**. The initiator owns file-system metadata, including the inode table, extents, and extent allocation and defragmentation. On the initiator side, OffloadFS provides an **Extent Manager**, which allocates extents, manages free space, and maintains per-file extent trees, and a **Task Offloader**, which decides when to offload and sends offload requests via gRPC. On the target side, it provides an **Offload Engine**, an **Offload Cache**, and SPDK-based `offload_read()` and `offload_write()` primitives used by target-side stub code [2604.13743].

Offloaded work is implemented as a gRPC stub/skeleton pair. The target node can read existing extents, update existing extents, and return results, but it cannot perform metadata-changing operations such as `truncate`, `fallocate`, `stat`, or `ioctl`. The initiator pre-allocates the needed blocks, passes block addresses and permissions for a specific task, and tracks which blocks are in use by offloaded work. It does not access those blocks until completion is reported back. The paper characterizes this as **initiator-centric block management**.

The cache-coherence model is likewise deliberately restricted. OffloadFS does not attempt conventional distributed cache coherence. The initiator side does not provide a cache through OffloadFS itself; the target’s Offload Cache may store blocks used by offloaded tasks; and coherence is pushed upward to the application or initiator. The initiator may explicitly decide whether to use cached blocks or bypass the cache, and it may pass file modified time so that the Offload Engine can bypass stale cache entries if the file has changed. This is coarse-grained by design.

Because a storage node may serve many initiators, OffloadFS includes two load-control mechanisms: **threshold-based rejection**, where the storage node rejects new offload requests if CPU usage exceeds a threshold, and **token-based scheduling**, where the storage node hands out tokens that allow a limited number of offload requests for a limited time. These mechanisms are directly relevant to OffloadDB because the storage node can otherwise become the bottleneck under multi-tenant offloading pressure.

## 3. RocksDB integration and execution path

OffloadDB maps RocksDB’s storage-engine workflow onto the OffloadFS execution substrate. The offloaded and non-offloaded activities are fixed by design.

| RocksDB activity | Placement in OffloadDB |
|---|---|
| WAL logging for each write request | Initiator |
| Flushing MemTables into L0 SSTables | Offloaded |
| Compaction / merge-sorting SSTables | Offloaded |
| Updating the MANIFEST file | Initiator |

The foreground write path remains initiator-local. When a client inserts a key-value pair, the record goes into the MemTable on the initiator, and the WAL is appended locally through SPDK if preallocated space is available; otherwise the Extent Manager allocates more blocks [2604.13743].

When a MemTable becomes immutable, the initiator’s background thread collects the file offsets of log entries and sends a compact offset array to the storage node. The Extent Manager allocates output SSTable space on the target, and the target-side **Log Recycler** reads the WAL directly and reconstructs the L0 SSTable in sorted order. This is the mechanism by which OffloadDB turns flush into a storage-side task.

Compaction is also initiated by the owner but executed remotely. The MANIFEST is examined to determine victim SSTables; the initiator chooses which SSTables to compact; the Extent Manager allocates output blocks for the resulting SSTables; and the block addresses and metadata for victim SSTables are sent to the target via RPC. The target-side Offload Engine reads the victim SSTables, merge-sorts them, and writes new SSTables; metadata about the new SSTables is returned to the initiator; and the initiator updates the MANIFEST, which acts as the commit point. The paper notes that if a crash occurs after output blocks are allocated but before the MANIFEST update, those blocks are garbage and can be reclaimed, matching RocksDB-style commit semantics.

## 4. Log Recycling, L0 caching, and cache-pollution control

The central OffloadDB-specific optimization is **Log Recycling**. Its purpose is to avoid sending the same key-value data twice over the network: once for the WAL and again when flushing to an SSTable. Instead of retransmitting the key-value payload, OffloadDB stores log-entry offsets in the MemTable; when the MemTable becomes immutable, the target node reads the WAL blocks and reconstructs the SSTable from those offsets. The flush communication is therefore small, because only offsets and block addresses are sent [2604.13743].

A second optimization is the **L0 cache** on the initiator. Immutable MemTables or L0 SSTables are kept locally for a short period until L0→L1 compaction deletes them from the MANIFEST. The paper’s sizing argument is concrete: with **64 MB MemTables**, a **3 GB L0 cache** can hold about **48 immutable MemTables**, which is considered sufficient because RocksDB usually triggers L0→L1 compaction when L0 exceeds **10 SSTables**. This design keeps data needed by foreground queries and background compaction close to the initiator while allowing actual SSTable reconstruction to be deferred.

These mechanisms are framed as responses to cache pollution. In vanilla RocksDB, compaction reads victim SSTables and writes new SSTables, and these blocks may occupy the page cache even though they are not useful to foreground queries. OffloadDB mitigates this by separating foreground and background caching and by using the storage node’s **Offload Cache** for background access while reserving the initiator’s L0 cache for short-lived immutable MemTables. In the reported cache-pollution experiment, OffloadDB reduces accesses to SSTables substantially and improves throughput by about **32%**. The paper also reports that Log Recycling improves write throughput by up to **9%** over compaction offloading without Log Recycling, and improves read throughput on the read-only workload by about **40%**, helped by the L0 cache.

The broader claimed benefits are correspondingly specific: reduced write stall by moving compaction off the initiator, use of the target node’s idle CPU and memory, reduced network traffic through Log Recycling, reduced cache pollution because background compaction runs away from the initiator’s foreground cache, and additional gains when the Offload Cache is enabled.

## 5. Evaluation, performance, and operating envelope

The evaluation uses a **9-node cluster** consisting of **8 compute nodes** and **1 NVMe-oF storage node**. The storage node has **2 Intel Xeon Silver 4215 CPUs**, **128 GB DRAM**, and **24 Samsung PM9A3 NVMe SSDs**, and runs **PoseidonOS** for NVMe-oF management. Each compute node has **2 Intel Xeon Gold 5115 CPUs** and **64 GB DRAM**. For RocksDB and OffloadDB, the setup uses **32 client threads**, **20 background threads**, **64 MB SSTable size**, a **200 GB database**, and **24-byte keys and 1 KB values**, with YCSB workloads including write-heavy and mixed cases [2604.13743].

The comparison set includes **OCFS2**, **GFS2**, **vanilla RocksDB**, **SpanDB**, and **Hailstorm**. The paper also distinguishes three OffloadDB configurations: **ODB-LR-C**, which enables compaction offloading only and disables both Log Recycling and Offload Cache; **ODB-C**, which adds Log Recycling but still disables Offload Cache; and **ODB**, which enables all designs including Offload Cache.

The headline result is that OffloadFS improves RocksDB performance by up to **3.36× compared to OCFS2**. In the write-heavy RocksDB experiment, OCFS2 without offloading achieves its best performance at about **110 Kops/sec** in the Local case, whereas OffloadFS or OffloadDB achieves about **379 Kops/sec** when all compaction tasks are offloaded to the target or peer. Relative to plain RocksDB, compaction offloading alone gives about **1.51×** higher throughput on the Load workload.

The performance gains are not uniform across workloads. OffloadDB is worse than vanilla RocksDB only for the scan-intensive **YCSB Workload E**, which the authors attribute to OffloadFS not being optimized for sequential scans. This limitation is central to interpreting the system: OffloadDB is optimized for background maintenance and I/O locality, not for scan-heavy data paths. The storage node may also become the bottleneck if too many initiators offload simultaneously. Under these conditions, the **token** policy performs slightly better than CPU-threshold rejection because it avoids repeated rejected requests, though it requires tuning of token lifetime.

The baselines expose additional trade-offs. **SpanDB** is sometimes worse than RocksDB in this setup because it flushes WAL aggressively in read-committed mode, increasing overhead. **Hailstorm** performs very poorly, largely due to its FUSE-based implementation and Akka-based communication overhead. These comparisons position OffloadDB not simply as a faster RocksDB variant, but as a specific point in the design space of disaggregated LSM execution.

## 6. Position within database offloading research

OffloadDB belongs to a broader class of database systems that relocate work toward storage, memory, or network-adjacent resources in order to reduce data movement or relieve host-side contention. Its distinguishing feature is that it offloads **background LSM maintenance** to an NVMe-oF storage node under an initiator-centric consistency model, rather than offloading general query operators or treating remote storage as a passive device [2604.13743].

In **Farview**, the offload target is not disaggregated storage but disaggregated **memory** implemented as an FPGA-based smart NIC with on-board DRAM and **operator offloading** capabilities. Farview exposes a remote buffer cache and supports offloading operators such as selection, projection, aggregation, regular expression matching, and encryption; it is reported as competitive with a local buffer cache solution for all workloads and better in a number of cases [2106.07102]. The contrast is instructive: Farview offloads analytical query operators over remote DRAM, whereas OffloadDB offloads storage-engine maintenance over NVMe-oF.

In cloud OLAP systems, **Adaptive pushdown** addresses a different problem: existing pushdown decisions are often static, while storage-layer computational capacity is shared and variable. That work proposes runtime pushback from storage to compute and a general criterion that pushdown-amenable operators should be **local** and **bounded**, reporting up to **1.9x** speedup over both No pushdown and Eager pushdown baselines and up to **3.0x** further acceleration from new pushdown operators [2312.15405]. A plausible implication is that OffloadDB’s threshold-based rejection and token-based scheduling are storage-node admission-control mechanisms analogous in spirit, though applied to LSM maintenance rather than OLAP operator placement.

For LSM-tree systems specifically, **O\(^3\)-LSM** extends the offloading agenda beyond compaction by adding **memtable Offloading** and **flush Offloading** on top of compaction Offloading via shared Disaggregated Memory. It reports up to **4.5X** write throughput improvement, up to **5.2X** range-query throughput improvement, and up to **76%** P99 latency reduction over Disaggregated-RocksDB, CaaS-LSM, and Nova-LSM [2603.05439]. This suggests that OffloadDB occupies an earlier point in the same trajectory: it demonstrates that storage-node execution can accelerate flush and compaction without distributed locks, while later systems redesign a larger fraction of the mutable-data lifecycle.

A further extension appears in **UDON**, which studies offloading to general-purpose compute on **CXL memory** devices. For vector databases, UDON reports that offloading **HNSW** kernels can provide upto **6.87×** performance improvement with under **10%** offload overhead [2404.02868]. The common thread is not a shared implementation substrate but a shared premise: once a disaggregated component exposes both data proximity and general-purpose compute, it becomes a candidate execution site for functions whose dominant cost is data movement rather than arithmetic density.

Taken together, these systems place OffloadDB within the broader transition from storage disaggregation as a capacity mechanism to disaggregation as a **computation-placement** mechanism. In that landscape, OffloadDB’s specific contribution is to show that RocksDB’s MemTable flush and compaction can be offloaded to NVMe-oF storage nodes through initiator-centric block management, Log Recycling, and cache-separation policies, yielding substantial throughput gains while preserving RocksDB-style commit semantics.

Source: https://www.emergentmind.com/topics/offloaddb