Asynchronous Optimizer Host-Offload

Updated 4 July 2026

Asynchronous optimizer host-offload is an execution strategy that decouples CPU-side optimizer tasks from device computations to overlap processing and updates.
It employs dynamic scheduling, model partitioning, and overlapped data transfers to mitigate synchronization delays and balance resource use across CPU, GPU, and storage.
It enables faster training and reduced idle times in federated, large-model, and graphics applications by managing state updates asynchronously.

Asynchronous optimizer host-offload denotes a family of execution strategies in which optimization, update-state management, or offload-control work is decoupled from accelerator-side execution and performed on a host CPU or server in parallel with device computation. The concept appears in several distinct but technically related settings: integrated CPU–GPU loop scheduling, where a host thread continuously feeds the GPU while CPU cores execute other chunks; federated learning, where the server hosts and trains the offloaded part of a model asynchronously relative to devices; large-model training, where optimizer states are moved to host DRAM or deeper storage tiers and updated through overlapped CPU, GPU, and I/O pipelines; and host-offloaded graphics training, where only a sparse active subset of parameters is staged to the GPU while host-side optimizer work proceeds concurrently (Corbera et al., 2015, Zhang et al., 10 Mar 2025, Maurya et al., 2024).

1. Conceptual scope and problem setting

In integrated heterogeneous processors, the original motivation was that CPU cores and an integrated GPU share the on-chip memory system, which removes PCIe transfer overheads and makes fine-grained dynamic collaboration feasible. In that setting, the host thread is not merely a control thread: it is the entity that selects chunk ranges, enqueues host-to-device copy, kernel launch, and device-to-host copy, then blocks in clFinish until completion. Under oversubscription, the operating system’s ability to reschedule that host thread becomes a first-order performance determinant, because GPU completion does not automatically imply immediate availability of the next offload (Corbera et al., 2015).

In distributed and offloaded training systems, the term shifts from host-driven kernel dispatch to host-resident optimizer execution. FedOptima partitions a sequential DNN into device-side $M_d$ and server-side $M_s$ , then trains $M_s$ centrally on the server using queued activations while devices update $M_d$ locally through auxiliary heads. In this formulation, host-offload is not limited to storage placement; it is an asynchronous split-optimization regime in which the server optimizer proceeds independently of device-side timing (Zhang et al., 10 Mar 2025).

For large Transformer and LLM training, host-offload usually means that FP32 master parameters and optimizer states, notably Adam momentum $m$ and variance $v$ , live in host memory or even deeper tiers such as NVMe and parallel file systems. The GPU retains FP16 or BF16 parameters and activations for forward and backward, while the update phase is converted into an overlapped pipeline involving CPU update kernels, H2D prefetch, D2H flush, and sometimes opportunistic GPU-side updates for a subset of subgroups (Maurya et al., 2024, Maurya et al., 2024, Maurya et al., 2 Sep 2025).

A plausible implication is that asynchronous optimizer host-offload is best understood as an execution pattern rather than a single algorithm. Its common structure is the same across domains: the host owns some critical state or control responsibility, the device performs throughput-oriented computation, and the interface between them is scheduled so that communication, update, and execution overlap instead of serializing.

2. Architectural patterns across domains

A first architectural pattern is dynamic heterogeneous loop scheduling on integrated chips. The TBB-based scheduler extended by Navarro et al. uses a two-stage pipeline. Filter1 decides whether the next chunk goes to the GPU or a CPU core and carves the corresponding iteration range; Filter2 executes that chunk and records execution time so that device throughputs can be updated online. GPU chunk size $G$ is fixed by offline throughput training, whereas the CPU chunk at interval $t_i$ is chosen by the proportional-throughput rule

$C(t_i) = G \times \frac{\lambda_C(tC_{i-1})}{\lambda_G(tG_{i-1})},$

with $\lambda_G(tG_i)=G/T(tG_i)$ and $M_s$ 0 (Corbera et al., 2015).

A second pattern is asynchronous model partitioning in federated learning. FedOptima attaches an auxiliary network $M_s$ 1 to the last device-side layer so that each device can compute a local loss $M_s$ 2, update $M_s$ 3 and $M_s$ 4 with SGD, and push activations to the server only when allowed by flow control. The server maintains a single $M_s$ 5, trains it centrally on activations, and separately aggregates device-side models with staleness-aware weights $M_s$ 6, skipping updates with $M_s$ 7. The result is a split architecture in which task dependency and straggler dependency are simultaneously reduced (Zhang et al., 10 Mar 2025).

A third pattern appears in offloaded LLM training. Deep Optimizer States splits each rank’s ZeRO-3 partition into independent subgroups and schedules them across two update lanes: CPU-updated subgroups and GPU-updated subgroups. While the CPU updates $M_s$ 8 subgroups and downscales their updated parameters to FP16, the GPU prefetches FP32 $M_s$ 9 for one subgroup, performs the update, and flushes the updated FP32 state back to host memory. The schedule is governed by a closed-form CPU:GPU ratio

$M_s$ 0

where $M_s$ 1, $M_s$ 2, $M_s$ 3, and $M_s$ 4 denote CPU update throughput, GPU update throughput, CPU downscale throughput, and PCIe bandwidth, respectively (Maurya et al., 2024).

A fourth pattern extends host-offload beyond DRAM to multi-tier storage. MLP-Offload aggregates local NVMe and remote PFS/object storage into a virtual third-level tier, assigns optimizer subgroups across paths proportional to measured bandwidth via

$M_s$ 5

and overlaps prefetch, CPU update, and lazy write-back through a rolling window with one prefetched subgroup, one updating subgroup, and one flushing subgroup. The design targets the case where optimizer-state I/O, not compute, dominates iteration time (Maurya et al., 2 Sep 2025).

A fifth pattern is sparse active-set offload. GS-Scale stores all Gaussians’ canonical parameters and optimizer states on the host, but keeps only geometric parameters—mean, scale, and quaternion, 10 of 59 parameters per Gaussian—resident on the GPU. Non-geometric parameters are forwarded only for the visible set valid_ids of the current or next view, while a host-side deferred optimizer updates the rest lazily using counters and lookup tables. This converts offload from full-model movement to sparse, per-view staging (Lee et al., 19 Sep 2025).

3. Runtime mechanisms and scheduler design

The runtime substrate of asynchronous host-offload is usually a pipeline with explicit queueing and blocking only at carefully chosen points. In the integrated CPU–GPU scheduler, a single OpenCL command queue in profiling mode enqueues H2D, kernel, and D2H asynchronously, with clFinish providing completion synchronization. The important systems issue is that the OpenCL driver blocks the host thread to avoid busy-waiting. On Windows, the resulting Thread Dispatch delay becomes large under oversubscription, so host-thread priority elevation with SetThreadPriority is used to reduce $M_s$ 6; on Exynos/Linux, pinning the host thread to an A7 core yields small positive gains in high-thread-count configurations (Corbera et al., 2015).

FedOptima uses a server-side Task Scheduler with two queue classes: $M_s$ 7 for device model updates and per-device activation queues $M_s$ 8. Model updates are prioritized whenever $M_s$ 9 is non-empty. Otherwise, activation selection follows the fairness rule

$M_d$ 0

where $M_d$ 1 counts how many activations from device $M_d$ 2 have already been used, thereby preventing faster devices from dominating centralized training of $M_d$ 3. Activation flow control bounds memory as

$M_d$ 4

independent of the number of devices $M_d$ 5, because each per-device queue is capped at length $M_d$ 6 (Zhang et al., 10 Mar 2025).

In LLM offload runtimes, asynchronous behavior is implemented with pinned host buffers, CUDA streams, and subgroup-local state machines. Deep Optimizer States casts FP16 gradients to FP32 on the GPU, then transfers them D2H into pre-pinned FP32 buffers during backward. During update, CPU and GPU subgroups are interleaved so that async prefetch of a future GPU-updated subgroup, async flush of a previously GPU-updated subgroup, and H2D upload of downscaled parameters from CPU-updated subgroups all proceed concurrently with compute (Maurya et al., 2024).

MLP-Offload adds two scheduler policies specific to deeper storage hierarchies. First, it enforces tier-exclusive concurrency: only one worker process on a compute node accesses a given alternative storage tier at a time, which reduces per-process latency under contention. Second, it alternates subgroup order between ascending and descending IDs across successive iterations so that the tail of one iteration remains hot in host DRAM for the next. It also stores FP16 gradients in host memory and converts them to FP32 in place immediately before CPU update, rather than persisting FP32 gradients to disk (Maurya et al., 2 Sep 2025).

GS-Scale introduces parameter forwarding and deferred optimizer restoration. Before iteration $M_d$ 7, the CPU uses the latest available gradients to compute updated non-geometric parameters for the next visible set and forwards them to the GPU in 32 MB chunks. For Gaussians that received zero gradients, a 4-bit counter records how many deferred steps have elapsed. When a Gaussian becomes active again or the counter saturates at 15, the optimizer state is restored in closed form using precomputed lookup tables for $M_d$ 8 and $M_d$ 9, which keeps host memory traffic proportional to the active set rather than the full model (Lee et al., 19 Sep 2025).

4. Overheads, bottlenecks, and failure modes

The most explicit overhead taxonomy was developed for integrated heterogeneous chips. The measured host-offload terms are scheduling and partitioning overhead $m$ 0, host-to-device transfer $m$ 1, kernel launch overhead $m$ 2, device-to-host transfer $m$ 3, and thread-dispatch delay $m$ 4. In Barnes–Hut with 100k bodies, $m$ 5 was negligible; on Ivy Bridge and Haswell, $m$ 6 and $m$ 7 were each below 0.3%; on Exynos, $m$ 8 and $m$ 9 on average; average launch latency was 1.8 ms on Ivy, 1.0 ms on Haswell, and 3.6 ms on Exynos. The dominant pathology under oversubscription was $v$ 0: 22% on Ivy Bridge and 33% on Haswell under Windows in 4+1 configurations, versus less than 0.09% on Exynos/Linux (Corbera et al., 2015).

In host-offloaded Transformer training, the critical path often shifts from kernel launch to gradient flush and CPU-side update. The DeepSpeed study of LLaMA2-13B on a 4×H100-80GB node reported that the gradient-boundary backward phase $v$ 1 accounted for 60.92% of total iteration time, with measured D→H throughput collapsing to about 8 GB/s, far below the 55 GB/s pinned-memory peak. During update, CPU-only optimizer throughput was about 4.5 billion parameters per second, but effective throughput dropped to about 2.8 billion parameters per second because H→D uploads of updated parameters were blocking. The study attributed the worst D→H behavior to non-pinned staging and FP16→FP32 conversion semantics in the measured path (Maurya et al., 2024).

MLP-Offload exposed a deeper-storage variant of the same problem: the update phase becomes overwhelmingly I/O-bound. For a 40B model on 4×H100, the paper reported 0.6 s forward, 28 s backward, and 213 s update out of a 242 s iteration, with 99% of update time spent in SSD I/O despite asynchronous overlap. In a single-node comparison for a 20B LLaMA model, GPU-only training took 0.4 s per iteration, host offload took 3.7 s, and NVMe offload took 67 s, illustrating that offload feasibility and offload efficiency are distinct properties (Maurya et al., 2 Sep 2025).

ZenFlow framed the principal failure mode as GPU stalls induced by uniform CPU offload. For Llama2-7B on 4×A100, the reported per-iteration times were about 45 ms for forward, about 2000 ms for backward, about 4600 ms for CPU update, and about 1000 ms for two PCIe transfers, implying a net stall of 3600 ms even under ideal overlap. The system’s response was not to eliminate offload, but to decouple important and less-important parameters: the top 1% of gradients carried about 90% of gradient energy, so updating that set in place on the GPU removed the hard synchronization between all GPU progress and full-model CPU completion (Lan et al., 18 May 2025).

These measurements also correct two common misunderstandings. Integrated memory does not imply zero transfer cost, because OpenCL-managed copies can still be non-negligible on bandwidth-limited SoCs. Conversely, asynchronous offload does not automatically guarantee zero stalls: if host thread scheduling, PCIe staging, host-memory bandwidth, or deep-tier contention remain on the critical path, the accelerator can still idle substantially (Corbera et al., 2015, Maurya et al., 2024).

5. Representative systems and reported outcomes

The reported benefits of asynchronous optimizer host-offload vary strongly with the workload, hierarchy, and scope of offloaded state. The following systems are representative rather than directly comparable.

System	Mechanism	Reported outcome
Dynamic scheduler on integrated CPU–GPU (Corbera et al., 2015)	Host-thread priority on Windows; big.LITTLE utilization and pinning on Exynos	EDP $v$ 2 on Ivy Bridge, $v$ 3 on Haswell, and $v$ 4 on Exynos big.LITTLE
FedOptima (Zhang et al., 10 Mar 2025)	Server-side training of offloaded layers, auxiliary heads, asynchronous aggregation	1.9× to 21.8× faster training, up to 93.9% server idle-time reduction, up to 81.8% device idle-time reduction, and 1.1× to 2.0× higher throughput
Deep Optimizer States (Maurya et al., 2024)	Interleaved CPU/GPU subgroup updates with overlapped prefetch and flush	2.0×–2.5× faster iterations and about 70% higher update throughput than ZeRO-3
MLP-Offload (Maurya et al., 2 Sep 2025)	Multi-level, multi-path optimizer-state offload across DRAM, NVMe, and PFS	2.5× faster iterations; backward and update accelerated by 13.5× and 2.3×, respectively
ZenFlow (Lan et al., 18 May 2025)	In-place GPU updates for important parameters and asynchronous CPU accumulation for the rest	Up to 5× end-to-end speedup, 2× lower PCIe traffic, and GPU stalls reduced by over 85 percent
GS-Scale (Lee et al., 19 Sep 2025)	Selective geometry residency, parameter forwarding, deferred optimizer update	3.3–5.6× lower GPU memory demand; 4 million to 18 million Gaussians on an RTX 4070 Mobile GPU; 23–35% LPIPS improvement
Occamy MPSoC offload runtime (Colagrande et al., 9 May 2025)	Multicast NoC and Job Completion Unit for offload control and synchronization	Up to 2.3× runtime improvement and restoration of more than 70% of ideally attainable speedups

The Occamy results are especially notable because they concern control-plane offload overheads rather than optimizer-state movement. The paper reported average baseline overheads of 242 cycles at 1 cluster and up to 1146 cycles on 32-cluster Matmul, whereas multicast plus the Job Completion Unit reduced residual overhead to about 185 cycles with standard deviation about 18 cycles across cluster counts. This is a reminder that asynchronous host-offload includes the synchronization substrate as well as the optimizer or data path (Colagrande et al., 9 May 2025).

A further observation is that several systems achieve their gains not by maximizing host work, but by minimizing the amount of state that must be synchronized immediately. FedOptima confines staleness to device-side aggregation while keeping a single centrally trained $v$ 5; ZenFlow updates only an important subset every step on the GPU; GS-Scale keeps only geometry resident and forwards just the next active set; Deep Optimizer States updates only a fraction of subgroups on the GPU at a time. This suggests that selective freshness, rather than universal freshness, is a recurrent design principle.

6. Convergence theory, constraints, and unresolved issues

The strongest formal treatment of unbounded-delay asynchrony in the supplied literature is the totally asynchronous Nesterov method for convex optimization. In that model, agents update blocks of a constrained variable $v$ 6 under total asynchrony, with arbitrarily long delays but eventual communication. With

$v$ 7

and $v$ 8 from $v$ 9-diagonal dominance of the Hessian, the method achieves linear convergence in the number of completed operation cycles:

$G$ 0

The numerical study reported 28% fewer iterations than heavy ball and 61% fewer iterations than gradient descent under total asynchrony (Pond et al., 2024).

A broader survey of asynchronous parallel and distributed optimization gives the standard bounded-delay picture. In centralized delayed-gradient form,

$G$ 1

the convergence rate degrades with delay, with asynchronous mini-batch SGD achieving iteration complexity

$G$ 2

for $G$ 3-strongly convex and $G$ 4-smooth objectives. Hogwild!-style methods tolerate delay more readily when updates are sparse, whereas decentralized push-sum variants compensate for heterogeneous communication and update rates through mixing and reweighting (Assran et al., 2020).

System papers make clear, however, that convergence bounds do not settle the engineering question. On Windows, oversubscription can make the host thread itself the bottleneck unless priority is raised; on Linux, awakened I/O-bound host threads are favored, making blocking completion comparatively efficient (Corbera et al., 2015). FedOptima still requires tuning of $G$ 5, $G$ 6, $G$ 7, $G$ 8, and $G$ 9, and its device-side aggregation remains stale by design even though $t_i$ 0 is never stale because it is trained centrally on the host (Zhang et al., 10 Mar 2025). Deep Optimizer States depends on subgroup-based sharding and on sufficient PCIe and DRAM bandwidth to support its interleaved schedule (Maurya et al., 2024).

The storage-centric systems expose additional unresolved issues. MLP-Offload is sensitive to remote-tier bandwidth fluctuations, shows reduced caching benefit as model size grows, and explicitly notes that earlier one-step-delayed GPU/CPU overlap was removed because of model inconsistencies; safe asynchronous variants of that type remain open (Maurya et al., 2 Sep 2025). GS-Scale can underperform on small scenes, on very high-resolution images where activations dominate, or on hardware with slow PCIe and weak CPUs, because host memory bandwidth and scattered host-side optimizer access remain limiting factors (Lee et al., 19 Sep 2025).

Across these threads, the central constraint is consistent. Asynchrony improves utilization by removing rigid barriers, but it succeeds only when the resulting staleness, scheduling delay, and data movement are all constrained enough that overlap dominates drift. The modern literature therefore treats asynchronous optimizer host-offload less as a single optimization and more as a balancing act among freshness, memory capacity, bandwidth, fairness, and critical-path control.