OffloadPrep: Near-Data Image Preprocessing

• OffloadPrep is an image pre-processing library that offloads operations like decode, resize, and normalization to NVMeoF storage nodes using a gRPC-based, initiator-centric model.
• It minimizes compute-node bottlenecks by leveraging underutilized storage-node CPUs and bypassing traditional distributed lock management, optimizing ML data pipelines.
• The framework supports flexible offload placements—storage-only, peer-only, or hybrid—achieving up to 1.85× faster epoch times compared to conventional file systems.

OffloadPrep is an image pre-processing library built atop OffloadFS that offloads pre-processing pipelines to NVMe over Fabrics target nodes, and optionally to peer compute nodes, using OffloadFS’s gRPC-based task offloading. It is designed for read-heavy, I/O-intensive machine-learning data preparation on disaggregated storage, where remote NVMe SSDs are exposed as block devices to initiator nodes while storage-node CPU and memory often remain underutilized. Within this setting, OffloadPrep uses an initiator-centric near-data processing model that avoids distributed lock management and enables selective offload of operations such as decode, resize, crop, flip, rotate, and normalization. In the reported evaluation, storage-only offload with OffloadFS achieved approximately $1.85\times$ faster pre-processing than OCFS2, reducing epoch time from $28.18\,\mathrm{s}$ to $15.19\,\mathrm{s}$ (Moon et al., 15 Apr 2026).

1. Context within disaggregated storage systems

OffloadPrep is situated in a disaggregated storage architecture in which storage resources and compute resources are separated and scaled independently. In the underlying NVMeoF model, remote NVMe SSDs are presented to multiple initiator nodes over RDMA or TCP while bypassing the target’s kernel I/O stack. The motivating observation is that modern NVMeoF JBOF storage nodes can expose substantial unused CPU and memory even when the NVMe array is saturated with only a handful of reactor threads, because data movement is offloaded to DMA and the target avoids heavy operating-system paths (Moon et al., 15 Apr 2026).

This architectural asymmetry makes near-data processing attractive. Network bandwidth and latency, rather than SSD bandwidth, limit end-to-end I/O in NVMeoF. Offloading computation to the storage side can therefore reduce network traffic and host-side I/O pressure while exploiting otherwise idle target-side resources. OffloadPrep applies this idea to computer-vision training pipelines, where CPU-side pre-processing commonly includes decode, resize, crop, flip, rotate, and normalization. The motivating claim is that such pre-processing can dominate end-to-end GPU training time, with prior work reporting figures of up to $65\%$. OffloadPrep targets precisely this bottleneck by moving pre-processing closer to the data while avoiding the concurrency-control overheads typical of shared-disk or distributed file systems (Moon et al., 15 Apr 2026).

The design goals are correspondingly narrow and system-oriented: reduce I/O and CPU bottlenecks on compute nodes, exploit underutilized CPU and memory on storage nodes, avoid distributed lock management and metadata coordination, and minimize cache interference with the training process. A notable consequence is that Offload Cache is disabled by default for OffloadPrep, because training datasets are typically larger than the cache and are accessed in a streaming fashion across epochs.

2. Architecture and execution model

The OffloadPrep execution path is split between an initiator-side client and a target-side execution engine. On the compute node, the training job links against the OffloadPrep client library and uses OffloadFS’s Task Offloader to issue gRPC requests. OffloadFS on the initiator manages inode and extent maps, performs SPDK-backed local I/O when offload is not beneficial, and serializes block authorizations to avoid conflicts. On the disaggregated storage node, the Offload Engine serves gRPC requests, executes the pre-processing on CPU, and accesses NVMe volumes through SPDK using offload_read() and offload_write() restricted to caller-authorized physical block ranges (Moon et al., 15 Apr 2026).

For OffloadPrep specifically, the target-side path is read-only. The control path selects a subset of each minibatch for remote execution and transmits block addresses together with pipeline parameters such as crop and resize specifications. The data path then reads the blocks via offload_read(), decodes and transforms the images on the target CPU, and returns the pre-processed outputs to the initiator over RPC. If the target rejects offload because of load conditions, OffloadPrep falls back immediately to local pre-processing.

A central design choice is the absence of distributed lock management. OffloadFS uses initiator-centric block authorization rather than a shared-disk DLM or inter-node cache-coherence protocol. The initiator grants the target access only to specific extents for the duration of the task and abstains from conflicting access during that interval. For OffloadPrep, whose operations are read-only, this further simplifies correctness and reduces interference.

The implementation is entirely user-level and kernel-bypass oriented. OffloadFS is built on SPDK, uses gRPC for offload control, and operates over RDMA or TCP NVMeoF fabrics. The reported platform uses PoseidonOS v0.12.0 on the storage node. The paper does not specify exact image-processing libraries or SIMD/GPU acceleration on the storage node; the target-side implementation is CPU-based.

3. Pre-processing pipeline and training-path integration

OffloadPrep offloads standard computer-vision pre-processing operations: decode, resize, crop, flip, rotate, and normalization. The evaluation uses image classification on OpenImages, with a $10\,\mathrm{GB}$ dataset and minibatch size $64$. A fraction of each minibatch is offloaded, while the remainder is processed locally; the returned remote outputs are then merged with the local outputs before being handed to the GPU pipeline (Moon et al., 15 Apr 2026).

The design is therefore hybrid rather than all-or-nothing. The initiator chooses an offload ratio or policy, partitions the minibatch into local and remote subsets, submits remote requests through OffloadFS, and merges the results on completion. The description notes a typical transformation order of decode $\rightarrow$ resize/crop $\rightarrow$ augmentation $\rightarrow$ normalize, while also emphasizing that the focus is on offload effects rather than stage-specific image-processing algorithms.

Threading is explicitly provisioned on both sides. The initiator uses multiple pre-processing threads, with $4$ threads per training process per GPU following prior guidance to avoid data stalls. The target runs corresponding RPC worker threads in the Offload Engine. Because the workload is streaming and read-only, Offload Cache is disabled by default; this avoids long-lived storage-node caching of large datasets, coherence work, and cache pollution that would not be useful under uniform scans across epochs.

OffloadPrep can also offload to peer compute nodes when the same NVMeoF volume is mounted on multiple initiators. In that case, an idle peer with visibility into the same blocks can execute the same gRPC-based processing path. The same authorization discipline applies: the initiating node grants access only to the relevant blocks for the task’s duration.

4. Placement policies and multi-tenant scheduling

The system supports three principal placement modes for pre-processing work: storage-only, peer-only, and a split “both” mode using peer and storage resources concurrently. The reported results show that peer-only offload outperforms storage-only when peer CPUs are stronger than storage-node CPUs, while “both” often performs best by exploiting both CPU pools (Moon et al., 15 Apr 2026).

For multi-tenant control, the paper evaluates several request-admission policies for the target-side Offload Engine:

Policy	Rule	Reported behavior
AcceptAll	Accept all offload requests	Highest gains at low concurrency; overload at scale
RejectAll	Reject all requests	Used to measure fallback overhead; overhead is small
CPU-threshold	Accept only if target CPU utilization $28.18\,\mathrm{s}$0	Prevents overload once concurrency rises
Token-based	Fixed number of tokens with expiry, e.g. 4 tokens and 1 s expiry	Improves fairness and reduces reject-induced latency

The principal scaling observation is that target CPU saturation, not merely offload availability, governs performance. In the reported experiments, more than $28.18\,\mathrm{s}$1 concurrent pre-processing RPC threads pushed target CPU utilization beyond $28.18\,\mathrm{s}$2 and caused performance degradation. The token-based policy and the CPU-threshold policy were similar overall, with token-based admission approximately $28.18\,\mathrm{s}$3 better because it caused fewer rejected requests.

A useful derived model for choosing the offload fraction $28.18\,\mathrm{s}$4 is:

$28.18\,\mathrm{s}$5

This suggests that the most effective operating point balances local and remote completion times rather than maximizing offload fraction indiscriminately.

5. Correctness, isolation, and filesystem semantics

OffloadPrep inherits its correctness model from OffloadFS’s initiator-centric file-system design. The target-side RPC stubs are authorized only for specific physical block ranges, and directory-level or inode-table operations are disallowed on the target. Inode and extent metadata remain initiator-owned. For OffloadPrep, only read operations are used, which simplifies the consistency problem considerably (Moon et al., 15 Apr 2026).

The system’s failure behavior is deliberately simple. If offload is rejected because of overload or failure, the work is executed locally immediately. The description notes that, for pre-processing, idempotent retry semantics are straightforward, although no detailed retry mechanism is specified. This suggests that OffloadPrep emphasizes predictable fallback over elaborate remote recovery.

A common misconception is that OffloadPrep’s benefit in machine-learning pre-processing derives primarily from eliminating shared-file-system lock traffic. The data support a more qualified view. For read-only workloads, the penalties of OCFS2 and GFS2 are smaller than in write-heavy key-value store workloads because directory and file lock bottlenecks are less severe. The advantage of OffloadFS in OffloadPrep therefore stems mainly from SPDK kernel-bypass, lightweight metadata control, and the ability to execute pre-processing near the data without heavier shared-file-system machinery.

Another misconception is that any amount of offloading is beneficial. The reported design notes the opposite: if local CPU is faster than storage CPUs, if network overhead dominates, or if too many initiators compete for target CPU, offloading yields limited gains or can become harmful.

6. Evaluation, comparisons, and limitations

The reported experimental setup uses a $28.18\,\mathrm{s}$6-node cluster with $28.18\,\mathrm{s}$7 compute nodes and $28.18\,\mathrm{s}$8 storage node connected through a Mellanox FDR $28.18\,\mathrm{s}$9 InfiniBand switch. Each compute node has dual Intel Xeon Gold 5115 processors, $15.19\,\mathrm{s}$0 DRAM, and the storage node has dual Xeon Silver 4215 processors with $15.19\,\mathrm{s}$1 DRAM. Storage consists of $15.19\,\mathrm{s}$2 Samsung PM9A3 U.2 $15.19\,\mathrm{s}$3 NVMe SSDs managed by PoseidonOS NVMeoF. The software stack includes Ubuntu 18.04.4, PoseidonOS v0.12.0, OffloadFS, and OffloadPrep, with OCFS2 and GFS2 POSIX I/O variants used for comparison (Moon et al., 15 Apr 2026).

In single-initiator image pre-processing, storage-only OffloadFS achieved approximately $15.19\,\mathrm{s}$4 faster epoch time than OCFS2: $15.19\,\mathrm{s}$5 versus $15.19\,\mathrm{s}$6. Peer-only offload was faster than storage-only, and the split “both” mode was best. In multi-tenant experiments, NoOffload increased from roughly $15.19\,\mathrm{s}$7 at one instance to roughly $15.19\,\mathrm{s}$8 at eight instances. AcceptAll offloading, with one third of tasks offloaded, improved epoch time by up to $15.19\,\mathrm{s}$9 relative to NoOffload at four instances, but degraded to roughly $65\%$0 when eight instances competed for target CPU. RejectAll showed that the overhead of rejected requests was small. CPU-threshold and token policies prevented overload and maintained performance under heavier contention, with the token policy slightly ahead.

The comparison to traditional shared-disk file systems is correspondingly nuanced. For machine-learning pre-processing, OCFS2 and GFS2 incur smaller penalties than they do in write-heavy LSM workloads, because the pipeline is read-only. Even so, OffloadFS remains faster under storage-node offload. The paper also positions OffloadPrep relative to distributed data-pipeline systems such as tf.data.service, Cachew, Sophon, Pecan, and FusionFlow. The stated distinction is that OffloadPrep adds storage-node CPUs as an additional resource pool through OffloadFS’s near-data offload model rather than relying only on remote compute workers or shared file systems.

The limitations are explicit. The implementation is CPU-based on the target node, with no GPU acceleration on storage nodes. Exact image libraries, codecs, and SIMD choices are not specified. Benefits depend on target CPU capability, network conditions, and contention. Because Offload Cache is disabled, repeated reuse within an epoch is assumed to be minimal; the description notes that smaller datasets with strong locality might benefit from caching, but that would introduce additional coherence considerations. Stronger security isolation or multi-tenant QoS would require further policy tuning.

Taken together, these results position OffloadPrep as a systems mechanism for near-data machine-learning pre-processing on disaggregated NVMeoF storage rather than as a generic accelerator runtime. Its significance lies in showing that storage-node CPU cycles, usually incidental to I/O delegation, can be incorporated into the training input pipeline with lightweight filesystem semantics and measurable performance gains when placement and admission are controlled carefully.