Cooperative CPU-GPU Transformer Inference

• Cooperative CPU-GPU transformer inference is a heterogeneous execution approach that jointly utilizes CPU and GPU resources to optimize memory capacity, bandwidth, and compute load.
• It employs dynamic layer streaming, selective offloading, and cooperative scheduling to extend model capacity and improve throughput for large-scale transformers.
• Experimental results show substantial improvements, including up to 25× model scalability and significant latency reductions compared to GPU-only inference.

Searching arXiv for papers on cooperative CPU-GPU transformer inference and closely related heterogeneous inference systems. Cooperative CPU-GPU transformer inference denotes heterogeneous inference in which the CPU is not treated only as a launch host or a storage spillover tier, but as part of the execution substrate for transformer workloads. In recent systems, cooperation takes several concrete forms: CPU DRAM or NVMe can extend model memory while the GPU remains the execution engine; the CPU can execute selected expert or attention sub-operations while the GPU handles the most compute-intensive kernels; or layer boundaries can be switched between processors according to measured operator behavior. The unifying objective is to manage memory capacity, memory bandwidth, interconnect transfer, kernel-launch overhead, and workload imbalance jointly, rather than optimizing GPU kernels in isolation (Aminabadi et al., 2022, Son et al., 18 May 2026, Lin et al., 20 Apr 2026, Xu et al., 1 Jun 2026, Vellaisamy et al., 16 Apr 2025).

1. Core design space

In the cited systems, cooperation is organized around the resource that limits inference. When model weights do not fit in aggregate GPU memory, the CPU acts primarily as a memory extension and streaming source. When sparse Mixture-of-Experts (MoE) layers create unstable working sets and repeated expert movement, the CPU becomes both a memory tier and a compute peer. When decoding is dominated by KV-cache-heavy attention, the CPU can participate directly in attention computation, especially on systems with expanded tiered memory. On ARM-based HMPSoCs, the split is operator-specific: the CPU is better for memory-intensive layers, while the GPU is better for highly parallelizable, compute-intensive layers (Aminabadi et al., 2022, Son et al., 18 May 2026, Lin et al., 20 Apr 2026, Xu et al., 1 Jun 2026).

System	Cooperative mechanism	Reported result
ZeRO-Inference (Aminabadi et al., 2022)	Weights in CPU DRAM or NVMe; layer streaming to GPU	530B on a single A6000 GPU; 25× larger models; 84 TFLOPS
CoX-MoE (Son et al., 18 May 2026)	AMX-enabled CPU-GPU co-execution; coalesced experts; selective attention offloading	Up to 7.1x over FlexGen; up to 2.4x over MoE-Lightning
HybridGen (Lin et al., 20 Apr 2026)	CPU-GPU collaborative attention with tiered memory	1.41x–3.2x average speedup over six KV-cache methods
ARM-CL cooperative execution (Xu et al., 1 Jun 2026)	CPU for memory-intensive layers; GPU for compute-intensive layers	Up to 15.72% lower latency than best single-processor inference

This design space is not reducible to a single offloading pattern. Some systems are throughput-oriented and explicitly trade latency for capacity and batching. Others target single-request inference on memory-limited systems, or long-context decoding where KV caches dominate memory and bandwidth. A plausible implication is that “cooperative CPU-GPU transformer inference” is best understood as a family of heterogeneity-aware execution models rather than a single algorithm.

2. CPU memory as model extension: streamed dense-model inference

The canonical dense-model formulation appears in DeepSpeed Inference, whose heterogeneous mode is ZeRO-Inference. The system keeps only the tensors immediately needed for execution in GPU memory and places the rest of the model state in CPU DRAM or NVMe, streaming each layer to the GPU just in time. The paper explicitly states that, instead of pinning as many weights as possible in GPU memory, ZeRO-Inference pins the model weights in DRAM or NVMe and streams each layer into GPU memory for computation when needed. GPU memory is therefore prioritized for activations and working tensors rather than long-lived parameters, because larger batch sizes improve throughput and only a small fraction of a very large model can remain resident anyway (Aminabadi et al., 2022).

The runtime is organized as a streaming pipeline over layers. Before a layer is needed, ZeRO-Inference prefetches a configurable number of upcoming layers while the current layer is being computed. This overlaps host-device transfer with computation. In multi-GPU configurations, each GPU fetches only a partition of a layer, and those partitions are then aggregated over the faster GPU-GPU interconnect, thereby using the aggregate PCIe bandwidth across GPUs to shorten transfer time. The same logic is applied to activations: when allocated activation memory exceeds a threshold, some activations are offloaded from GPU to CPU memory while not in use. This is applied especially to the KV cache, which grows with the number of concurrently generated sequences.

The motivation is a bandwidth lower bound on inference latency:

$$\text{latency} \ge \frac{\text{model size}}{\text{achievable memory bandwidth}}.$$

The paper gives a concrete example: for a 40B parameter model on an NVIDIA A100 80GB GPU with about 2 TB/s memory bandwidth, the minimum latency is about

$40 \times 2 / 2000 \text{ sec} = 40 \text{ ms},$

and for MT-NLG 530B, with over 1 TB of fp16 weights, even 1.5 TB/s peak bandwidth yields about 700 ms latency. These estimates are used to argue that single-device GPU-only inference cannot satisfy both memory and latency constraints at very large scale.

The performance results define the operating regime clearly. ZeRO-Inference can infer a 530B parameter model on a single A6000 GPU, which is 25× larger than what a GPU-only solution could handle in the comparison. It achieves up to 84 TFLOPS, about 54% of theoretical peak on that hardware, and over 50% better throughput than the GPU-only solution even for models that already fit in GPU memory, because it can support larger batches. For models that fit only in CPU memory, it claims over 25× higher throughput than CPU-only inference. The paper positions this design as suitable for less latency-sensitive, resource-constrained applications.

3. MoE inference as cooperative scheduling, caching, and co-execution

MoE models make CPU-GPU cooperation more intricate because sparsity reduces arithmetic work per token but introduces large expert parameter footprints, unstable activation patterns, and repeated expert movement. Three recent systems illustrate distinct solutions: AMX-enabled co-execution, GPU expert caching with CPU miss handling, and dynamic hybrid scheduling with impact-driven prefetching (Son et al., 18 May 2026, Huang et al., 18 Dec 2025, Zhong et al., 8 Apr 2025).

CoX-MoE formulates throughput-oriented MoE inference as an AMX-enabled CPU-GPU collaborative system. Its decoder layer is partitioned into four units: $OP_0$ QKV projection, $OP_1$ attention, $OP_2$ output projection, and $OP_3$ expert FFN computation. The crucial departure from prior work is that CoX-MoE uses ordinary batch execution for expert computation instead of micro-batching: it coalesces all micro-batches into a single expert batch of size $B$ for $OP_3$, while still allowing micro-batching for the non-expert parts when beneficial. The motivation is empirical. For Qwen3-30B-A3B, expert computation accounts for over 97.5% of total latency, and halving the micro-batch size nearly doubles expert latency. The paper also reports that, in a GPU-attention baseline, intermediate data takes 84.6% of VRAM, leaving only 11.5% for expert weights; with CPU attention, expert-resident VRAM expands to 58.5%, and total latency drops by about 40%. AMX is the enabling CPU-side primitive: roughly 144 TFLOPs per socket for BF16 versus roughly 18 TFLOPs for AVX-512. Across Mixtral-8x7B-Instruct, DeepSeek-V2-Lite, and Qwen3-30B-A3B, CoX-MoE achieves 1.7–2.4× higher throughput than MoE-Lightning and 3.4–7.1× higher throughput than FlexGen.

The system’s latency model makes the cooperative split explicit:

$$T_{tot} = \sum_{l=1}^{N} T_l \ge N \times \min(T_l) = N \times T_{opt},$$

with

$$T_{opt} = \min \left( \sum_{i=0}^3 T(OP_i) \right).$$

This leads to a balanced allocation of expert work between CPU and GPU rather than a fixed offload policy. Static expert-aware stratification then pre-assigns frequently activated experts to the GPU, reducing dynamic PCIe transfers. In memory-constrained settings, the paper reports that this improves expert hit ratio by about 40% over random selection and yields up to 1.47–1.50× throughput improvement.

A second line of work targets memory-limited consumer systems. The framework in “Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems” treats the GPU as a small expert cache and uses the CPU’s multi-core compute to execute cache misses directly instead of waiting for GPU weight transfer (Huang et al., 18 Dec 2025). The motivating comparison is sharp: for Mixtral 8x7B expert FFN computation, GPU compute is about 0.25–0.3 ms, but transferring expert weights over PCIe costs about 28 ms; CPU compute falls from about 44.1 ms with one thread to 7.3 ms with 24 threads. For Phi-3.5-MoE, GPU expert compute is about 0.11 ms, GPU transfer is about 12.26 ms, and CPU compute falls from 22.73 ms to 3.36 ms across the same thread range. The cache is described as an $40 \times 2 / 2000 \text{ sec} = 40 \text{ ms},$0-index, $40 \times 2 / 2000 \text{ sec} = 40 \text{ ms},$1-way set-associative cache, with

$40 \times 2 / 2000 \text{ sec} = 40 \text{ ms},$2

Misses do not stall waiting for GPU weights; instead, the framework sends the intermediate hidden state to the CPU, computes the expert on the CPU, and fetches the missed weights asynchronously to the GPU for future reuse. The paper reports up to 4.4× speedup over the prior prefetching-based approach, up to 4.8 tokens/s for Mixtral 8x7B, up to 10.4 tokens/s for Phi-3.5-MoE, and about 1.6× better than Fiddler. It also reports that with 24 CPU cores, the method uses only about 29.9% of the energy of the prefetching method for Mixtral and 27.8% for Phi-3.5-MoE.

HybriMoE addresses the same MoE setting but emphasizes runtime instability rather than static locality. It adds a scheduling and memory-management layer on top of kTransformers and llama.cpp kernels, with dynamic intra-layer scheduling, impact-driven inter-layer prefetching, and a score-aware caching algorithm (Zhong et al., 8 Apr 2025). The scheduling objective is formulated as

$40 \times 2 / 2000 \text{ sec} = 40 \text{ ms},$3

Its cache policy, “Minus Recent Score” (MRS), updates a smoothed routing-priority score

$40 \times 2 / 2000 \text{ sec} = 40 \text{ ms},$4

The paper reports that MRS beats LRU by 6–8% at 25% cache capacity, and that the full system achieves an average speedup of 1.33× in the prefill stage and 1.70× in the decode stage compared to the state-of-the-art hybrid MoE inference framework.

4. Attention, KV caches, and long-context CPU-GPU collaboration

For long-context decoding, the main bottleneck often shifts from model weights to KV-cache growth. HybridGen addresses this directly by splitting decode-time attention across CPU and GPU while preserving exact attention semantics. The paper argues that neither GPU-only attention with KV offload nor CPU-only attention is sufficient: GPU-only approaches must fetch both $40 \times 2 / 2000 \text{ sec} = 40 \text{ ms},$5 and $40 \times 2 / 2000 \text{ sec} = 40 \text{ ms},$6 from CPU memory, with data traffic roughly $40 \times 2 / 2000 \text{ sec} = 40 \text{ ms},$7 per attention layer, while CPU-only attention becomes compute-bound and, at 64K tokens, CPU-side computation can be nearly 5× GPU-side work in the paper’s breakdown (Lin et al., 20 Apr 2026).

HybridGen’s key idea is attention logit parallelism. CPU and GPU compute logits for their resident KV segments independently; the GPU then concatenates the logits, restores token order, applies softmax over the full set, performs value aggregation, and completes the rest of the transformer block. The system adds a feedback-driven scheduler whose target is

$40 \times 2 / 2000 \text{ sec} = 40 \text{ ms},$8

so that CPU work is hidden behind GPU work. It also uses semantic-aware KV cache mapping: $40 \times 2 / 2000 \text{ sec} = 40 \text{ ms},$9 vectors are placed in CPU DRAM because CPU logit computation needs them, while $OP_0$0 vectors are placed in CXL-expanded memory once DRAM is full because $OP_0$1 is not used on the CPU and only needs to be moved to the GPU later. On three GPU platforms with a CXL-expanded memory, across three LLM models with eleven different sizes, HybridGen outperforms six state-of-the-art KV cache management methods by 1.41x–3.2x on average while maintaining superior accuracy, with only about a 0.02 average accuracy gap across tasks.

A related heterogeneous design replaces the CPU with a dedicated memory-intensive co-processor. The HPU system argues that attention during generation has low operational intensity and is therefore memory-bound rather than compute-bound. On Llama 2 7B profiled on an A100, GEMV-like attention kernels remain memory-bound regardless of batch size, while GEMM kernels improve with batch size and saturate around a batch size of 203, where the A100’s performance/bandwidth balance is matched (Rhee et al., 18 Apr 2025). The GPU-HPU split leaves GEMM-based layers on the GPU and offloads generation-stage attention together with KV-cache storage/access to the HPU. Although this is not a CPU-GPU design, it strengthens the same architectural claim: memory-bound attention and compute-bound linear layers need not share the same execution substrate. With 4 HPU prototypes, throughput improves over the GPU-only baseline by 1.9×, 2.9×, and 4.1× as batch size rises from 16 to 32 to 64, and GPU MFU rises to as high as 44% on L40S+HPU.

Taken together, these systems show that KV-cache management is not merely a storage problem. It is a joint compute-placement and bandwidth-management problem. This suggests that future CPU-GPU inference stacks will increasingly split attention by memory locality and value of on-device reuse, rather than by a uniform “attention on GPU” rule.

5. Coupling granularity, launch overhead, and embedded cooperative execution

The benefits of cooperation depend strongly on how the CPU and GPU are coupled. “Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures” studies loosely-coupled PCIe systems and closely-coupled GH200 systems and shows that closer coupling does not eliminate CPU-side bottlenecks (Vellaisamy et al., 16 Apr 2025). The paper introduces Total Kernel Launch and Queuing Time (TKLQT),

$OP_0$2

to isolate the cost of getting kernels from CPU-side framework execution into GPU execution. It finds that GH200 significantly outperforms loosely-coupled systems at large batch sizes, achieving 1.9x–2.7x faster prefill latency for Llama 3.2-1B, but GH200 remains CPU-bound up to 4x larger batch sizes than loosely-coupled systems. For encoder-only workloads, loosely-coupled systems transition around batch size 8 while GH200 transitions around batch size 32. The paper also reports nullKernel launch overheads of 2260.5 ns for AMD+A100, 2374.6 ns for Intel+H100, and 2771.6 ns for GH200, and argues that kernel fusion is especially attractive in this extended CPU-bound region.

At the opposite end of the deployment spectrum, “Fast Transformer Inference on ARM-Based HMPSoCs” shows that cooperative execution can also reduce latency on tightly resource-constrained embedded platforms. The work extends ARM Compute Library with native transformer kernels and then uses layer-wise performance characterization to decide which sublayers should run on the CPU versus the GPU (Xu et al., 1 Jun 2026). The CPU is better for Embedding, SDPA, and Add & Norm; the GPU is better for Attention Linear and Feed Forward. The implementation uses shared memory and an “upscale” approach so that the CPU can access shared buffers via OpenCL without explicit memcpy. On a Khadas VIM 3 BASIC with an Amlogic A331D HMPSoC, the cooperative design further reduces transformer inference latency by up to 15.72% compared to the best single-processor inference on ARM-CL, while the extended ARM-CL achieves up to three times faster transformer inference compared to state-of-the-art CPU/GPU implementations on the tested board.

Adjacent edge systems broaden the meaning of cooperation beyond a single CPU-GPU pair. Galaxy breaks the resource wall across heterogeneous edge devices with hybrid model parallelism and tile-based overlap of communication and computation, achieving up to 2.5x end-to-end latency reduction (Ye et al., 2024). CoFormer decomposes a pretrained transformer into smaller sub-models for distributed inference, reports up to 3.1× inference speedup, enables GPT2-XL with 1.6 billion parameters on edge devices with 76.3% lower memory requirements, and can reduce energy consumption by approximately 40%; however, the paper does not present a special CPU–GPU partitioning or hybrid CPU/GPU scheduling scheme inside a single device (Xu et al., 28 Aug 2025). These systems are better read as neighboring work rather than direct instances of intra-node CPU-GPU cooperative transformer inference.

6. Trade-offs, misconceptions, and recurring design principles

A common misconception is that cooperative inference is equivalent to naive weight offloading. The cited systems instead implement pipelined execution models in which transfer is overlapped with compute, GPU memory is reserved for the tensors that most improve utilization, and CPU work is scheduled to avoid blocking the critical path. ZeRO-Inference uses layer prefetching, aggregate PCIe bandwidth across GPUs, and architecture-aware offload scheduling; one explicit example is that odd-numbered GPUs offload activations for odd-numbered layers, while even-numbered GPUs offload even-numbered layers, to reduce PCIe contention (Aminabadi et al., 2022).

A second misconception is that the CPU is useful only as a slow fallback. In practice, the CPU can be the preferred executor for specific workloads. On consumer-grade MoE systems, computing missing experts on the CPU can outperform “GPU + transfer” in end-to-end latency because transfer dominates GPU compute (Huang et al., 18 Dec 2025). In CoX-MoE, AMX raises CPU-side BF16 throughput enough that offloaded attention or expert computation becomes feasible, allowing the CPU to act as a real compute partner rather than a storage tier (Son et al., 18 May 2026). On ARM-based HMPSoCs, the CPU wins on memory-intensive layers such as Embedding, SDPA, and Add & Norm (Xu et al., 1 Jun 2026). In HybridGen, the CPU handles logits for offloaded KV segments while the GPU handles the local segment and the final softmax/value aggregation (Lin et al., 20 Apr 2026).

A third misconception is that tighter coupling always lowers latency. The GH200 results complicate that view. Closer CPU-GPU coupling improves large-batch prefill, but GH200 can have higher latency at low batch sizes because the Grace CPU launch path remains the limiter and the CPU-bound region extends to larger batch sizes than on loosely-coupled systems (Vellaisamy et al., 16 Apr 2025). Similarly, heterogeneous offload helps most in its intended regime. The HPU prototype shows limited gain at small batches because the HPU is underutilized and offload overhead dominates (Rhee et al., 18 Apr 2025). ZeRO-Inference is explicitly a throughput-oriented system for less latency-sensitive, resource-constrained applications (Aminabadi et al., 2022).

Across these papers, several stable principles recur. First, operator heterogeneity matters: attention, FFN, expert FFN, normalization, and embedding stress different resources. Second, memory hierarchy is an algorithmic concern rather than a backend detail: CPU DRAM, NVMe, CXL memory, GPU HBM, and caches alter the optimal execution plan. Third, runtime policies matter as much as kernels: prefetching, cache management, batch formation, and launch reduction often determine whether heterogeneity helps or hurts. A plausible implication is that cooperative CPU-GPU transformer inference will continue to move away from static device assignment toward bottleneck-aware runtimes that co-optimize compute placement, memory residency, and orchestration overhead.