Asynchronous Cache Swapping in Modern Systems

• Asynchronous cache swapping is a set of mechanisms that transfers cached state (e.g., KV blocks, memory pages) without interrupting ongoing computation.
• It applies to diverse scenarios including dynamic GPU memory reallocation in LLM serving, predictive KV prefetching, and remote memory offloading, thereby optimizing system performance.
• Implementations like MorphServe and AsyncTLS demonstrate significant gains—such as a 92.45% reduction in SLO violations and up to 1.83× throughput boost—while managing consistency challenges.

Asynchronous cache swapping denotes a class of mechanisms in which cached state is moved, resized, promoted, replicated, or replaced while the primary computation continues, rather than through stop-the-world synchronization. In current systems literature, the phrase spans several distinct but technically related practices: runtime repartitioning of GPU memory between model weights and KV cache during ongoing decoding, predictive prefetching of future KV blocks into faster cache levels, CPU–GPU or local–remote migration of cached state under memory pressure, background promotion of semantically verified cache entries, and even adversarial mid-generation replacement of KV history blocks (Su et al., 24 May 2025, Dong et al., 8 Apr 2025, Hu et al., 9 Apr 2026, Zhong et al., 2021, Singh et al., 13 Feb 2026, Ganesh et al., 16 Nov 2025). This suggests an umbrella systems pattern centered on asynchronous state movement under active execution.

1. Conceptual scope and representative forms

The most concrete contemporary use of the term appears in LLM serving, where the cached object is typically the autoregressive KV cache and the asynchronous action is performed at token granularity. In MorphServe, asynchronous cache swapping means dynamically reallocating GPU memory between model weights and KV blocks, with operations issued on separate CUDA streams so that decoding continues while weight copies and KV-pool updates occur in the background (Su et al., 24 May 2025). In long-context attention systems such as AsyncTLS and AsyncSpade, the same idea appears as asynchronous prefetching or offloading: a resident working set is kept hot on GPU, while future or less-active KV blocks are selected, transferred, or compacted outside the critical path (Hu et al., 9 Apr 2026, Luo et al., 8 Oct 2025). At a lower systems level, LightSwap and Canvas apply the pattern to virtual memory and remote memory, turning swap-in and swap-out into user-space or RDMA-driven background activity rather than a globally blocking path (Zhong et al., 2021, Wang et al., 2022).

Setting	Cached object	Asynchronous action
LLM serving	KV blocks, model layers	resize KV pool, swap quantized layers, prefetch future KV
Memory systems	pages, microbatch KV	GPU–CPU or local–remote swap, replication
Semantic or agent caches	responses, plan transitions	background verification, promotion, correction
Security	active KV history	mid-generation overwrite of cache segments

A common misconception is that asynchronous cache swapping is only an optimization for latency hiding. The literature shows a broader picture. Some works use it for elasticity and throughput, some for isolation and fault tolerance, some for semantic reuse, and one line of work demonstrates that exactly the same capability—mid-stream cache replacement—can be weaponized as an integrity attack (Ganesh et al., 16 Nov 2025).

2. Runtime KV and weight reconfiguration in LLM serving

MorphServe is an explicit formulation of asynchronous cache swapping for production LLM serving under highly dynamic and bursty workloads, including Azure and BurstGPT traces, where short spikes can exhaust GPU memory for prefilling or decoding and drive TTFT SLO violations around $2$ seconds (Su et al., 24 May 2025). The system targets Vicuna 7B, LLaMA 2 7B, LLaMA 3 8B, and CodeLLaMA 34B on L4 24 GB and A100 80 GB GPUs, using a SwiftLLM engine with PagedAttention, FlashAttention, and continuous batching. Its core design combines two asynchronous, token-level, state-preserving mechanisms: quantized layer swapping (“LayerSwapper”) and pressure-aware KV cache resizing (“KVResizer”).

LayerSwapper precomputes an offline layer ordering using the Layer Importance Score

$$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$

then, at runtime, walks this sequence when pressure rises. All precision variants—FP16, INT8, INT4, and INT3—are loaded into pinned CPU memory at startup. GPU memory for each layer is preallocated and fixed, so runtime swapping consists of cudaMemcpyAsync from pinned CPU buffers into stable GPU addresses on dedicated CUDA streams. Because addresses do not change, the inference graph requires no pointer remapping. The system reports that an entire INT4 swap, including copy and reconstruction, is approximately $6$ ms per layer and is overlapped with decoding. KVResizer extends PagedAttention with block-level reallocation and remapping through dynamic memory registration, so that the mapping tables can be updated while attention kernels continue to run.

The runtime control loop is driven by a Serving Monitor and Morphing Controller. Signals include GPU memory utilization, KV-cache memory usage, queue length, queueing delay, TTFT, and TPOT. Thresholds such as KV-cache usage above $85\%$, queueing delay above $100$ ms, or inability to allocate new KV blocks trigger pressure handling. Under pressure, the controller swaps additional layers to lower precision, frees GPU memory, and converts the released capacity into new KV blocks. When pressure subsides, extra KV blocks are released and layers are restored toward FP16. Because these actions occur on separate CUDA streams and through updated PagedAttention mappings, the first $N$ tokens of a request may be produced by full-precision layers and later tokens by mixed-precision layers, with subsequent restoration inside the same request.

This design changes the practical meaning of “cache swapping.” Rather than evicting individual sequences or pushing live KV entries to host memory, MorphServe shifts the boundary between weight memory and KV memory. Its reported results are correspondingly system-level: average SLO violations are reduced by $92.45\%$ versus full precision, P95 TTFT improves by $2.2\times$–$3.9\times$, throughput under saturation rises by $1.6\times$–$$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$0, KV capacity expands by up to $$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$1 beyond full-precision capacity during peaks, KV memory utilization is $$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$2 better than static INT4, output quality is $$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$3 higher than static INT4, and P99 TPOT improves by up to $$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$4. Quality control is handled through LIS ordering, threshold hysteresis, and distinct accuracy and performance modes, with reported F1/Rouge-L degradation in accuracy mode of at most approximately $$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$5–$$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$6.

3. Predictive prefetching and hierarchical offloading for long-context inference

A second major interpretation of asynchronous cache swapping treats the KV cache as a working set that can be predicted, prefetched, filtered, or offloaded ahead of use. AsyncSpade removes the sequential dependence of query-aware sparse decoding by approximating the next-step query from a short window of recent queries and delegating KV filtering to a disaggregated “Cache Rank” that runs in parallel with an “Inference Rank” (Luo et al., 8 Oct 2025). The temporal-regressive module uses recent query states $$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$7 to predict $$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$8, and the Cache Rank uses that predicted query to compute token-level criticality scores, select the top-$$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$9 KV tokens, compact them into contiguous buffers, and return them before the next forward step. Under the condition that cache-management time plus P2P latency remains smaller than the forward time for a layer pack, KV operations are fully overlapped and TPOT approaches the theoretical optimum. Reported results on Qwen3-8B and Qwen3-32B show more than $6$0 TPOT reduction versus Quest and at least $6$1 TPOT reduction versus full attention, while matching or surpassing accuracy on AIME-24/25, GPQA-Diamond, and MATH-500.

AsyncTLS uses a different but related hierarchy: coarse block filtering, fine-grained token selection, and a CPU–GPU offloading engine keyed to temporal locality (Hu et al., 9 Apr 2026). At decoding step $6$2, the system computes a block set $6$3 and defines the transfer set

$6$4

where $6$5 is the current GPU-resident cache. The next resident set is then updated through asynchronous prefetch while the current sparse attention and FFN continue. The design deliberately introduces a one-step lag: token-level selection for step $6$6 uses $6$7, while block selection at $6$8 schedules the prefetch needed for $6$9. The analysis assumes high temporal locality, written as $85\%$0 with $85\%$1, so that only a small delta of blocks must move each step. On Qwen3 and GLM-4.7-Flash across GQA and MLA, AsyncTLS reports $85\%$2–$85\%$3 operator speedups and $85\%$4–$85\%$5 end-to-end throughput improvements on $85\%$6k–$85\%$7k contexts.

At a finer level in the GPU hierarchy, asynchronous cache swapping can occur between HBM and L2 rather than between host and device memory. An H20-focused prefetching method uses cp.async.bulk.prefetch.L2 to stage future KV blocks into L2 during active computation windows (Dong et al., 8 Apr 2025). For a KV block, the paper gives

$85\%$8

and for one iteration over all heads and batch elements,

$85\%$9

With Llama2-7B FP16 parameters $100$0, $100$1, $100$2, $100$3, and $100$4, this yields $100$5 KB and $100$6 KB at batch size $100$7, implying full L2 residency for up to $100$8 batches on a $100$9 MB L2. On H20, the method reports $N$0 improvement in attention kernel efficiency and up to $N$1 end-to-end throughput gain, with L2 load hit rate for Llama2-7B moving from $N$2 to $N$3 and Stall Long Scoreboard dropping from $N$4 cycles to $N$5.

Taken together, these works establish a continuum: asynchronous cache swapping may mean speculative query-conditioned token selection, hierarchical CPU–GPU block migration, or explicit HBM–L2 staging. In every case, the central systems move is the same: predicted future cache demand is serviced in parallel with current compute.

4. Integrity, correctness, and adversarial manipulation

The same mechanism that hides latency can also alter model behavior. The paper on “History Swapping” defines a block-level attack in which a contiguous segment of the active generation’s KV cache is replaced mid-generation with a precomputed cache from a different topic (Ganesh et al., 16 Nov 2025). At generation step $N$6, with current cache length $N$7, the attack chooses

$N$8

then overwrites the last $N$9 timesteps of the victim cache with the first $92.45\%$0 timesteps of an attacker topic cache. Across $92.45\%$1 evaluated configurations on Qwen3-32B, Qwen3-14B, Qwen3-8B, and Qwen3-4B-Instruct, topic-level hijacks occur only when every transformer layer is overwritten for the chosen timesteps; partial-layer overwrites do not cause deviations in any evaluated configuration. Among the $92.45\%$2 deviating configurations, all involve full-layer overwrites and exhibit three behaviors: immediate and persistent hijack, immediate hijack with partial recovery, or delayed hijack. The paper further argues that high-level structural plans are encoded early in generation, while local discourse structure is maintained especially by the final layers.

This result directly counters the benign-only reading of asynchronous cache swapping. In the attack model, the prompt and visible conversation remain unchanged; only the internal state is swapped asynchronously. The KV cache therefore functions as an integrity-sensitive interface, not merely an optimization buffer.

A distinct but related correctness problem appears in transactional caches updated asynchronously relative to a serializable backend. T-Cache formalizes “cache-serializability,” defined so that for every execution, every partial execution containing all update transactions and all read-only transactions through a single cache server is serializable (Eyal et al., 2014). Its mechanism stores version numbers and bounded dependency lists with cached objects, then checks those dependencies locally during read-only transactions without synchronous database round trips. With unbounded cache size and unbounded dependency lists, T-Cache proves cache-serializability. With bounded resources, the system manager chooses a trade-off between performance and consistency. In workloads based on real-world topologies, T-Cache detects $92.45\%$3–$92.45\%$4 of inconsistencies and increases the rate of consistent transactions by $92.45\%$5–$92.45\%$6.

These two lines of work identify a broad design tension. Asynchronous cache swapping preserves liveness and overlap by removing synchronous coordination, but it also opens questions about what consistency, integrity, or semantic continuity is preserved when internal state changes mid-stream.

5. Distributed, remote-memory, and hierarchical cache systems

Outside attention kernels, asynchronous cache swapping appears as a general systems technique for moving state across memory tiers or aggregation layers. DéjàVu treats the KV cache as a streamable object and uses that abstraction for prompt-token disaggregation, microbatch-level GPU–CPU swapping, and token-level replication for fault tolerance (Strati et al., 2024). The key observation is that in a pipeline of depth $92.45\%$7, only one microbatch’s KV cache is active at a stage at a time, so GPU memory need not permanently hold all $92.45\%$8 in-flight microbatches. DéjàVu stores the KV caches for all in-flight microbatches in CPU memory and transfers a microbatch to GPU only when that microbatch is processed. It reports microbenchmarks with less than $92.45\%$9 overhead for local SSD and remote CPU streaming, a $2.2\times$0 gain from buffered copies compared with naive streaming, throughput improvements up to $2.2\times$1 on OPT-66B and $2.2\times$2 on BLOOM-176B through prompt-token disaggregation, and up to $2.2\times$3 throughput gain from microbatch swapping. For failure recovery, incremental asynchronous replication reduces a microbatch latency increase from $2.2\times$4 to $2.2\times$5 under an injected failure.

LightSwap pushes the same pattern into user-space swapping. It redirects page faults to a user-space handler via eBPF, performs swap I/O through SPDK and user-level RDMA, and co-designs this with lightweight threads so that a fault blocks only the corresponding LWT while other LWTs continue on the same worker thread (Zhong et al., 2021). In production-level evaluation with YCSB workloads on memcached, LightSwap reduces page-fault handling latency by $2.2\times$6–$2.2\times$7 times and improves throughput by more than $2.2\times$8 compared with the state-of-the-art swapping systems. The measurements also show page-fault notification latency remaining around $2.2\times$9–$3.9\times$0 $3.9\times$1s with the eBPF-based scheme even at $3.9\times$2 LWTs, whereas userfaultfd and signal-based alternatives grow sharply under concurrency.

Canvas generalizes asynchronous swap paths to a multi-application remote-memory setting by assigning each cgroup its own swap partition, swap cache, prefetcher, and RDMA bandwidth, then layering adaptive swap entry allocation, semantics-aware prefetching, and two-dimensional RDMA scheduling on top (Wang et al., 2022). The RDMA scheduler uses weighted fair queuing across applications and a timeliness-aware policy within each application that can drop outdated prefetches. The paper reports up to $3.9\times$3 co-run performance improvement, $3.9\times$4 average improvement, $3.9\times$5 higher RDMA read bandwidth utilization, and a weighted min-max ratio of $3.9\times$6. It also reports roughly $3.9\times$7 lower slowdown variability for native applications co-running with managed workloads.

CaBaFL shows that the same vocabulary extends beyond memory pages and KV blocks to model states in federated learning (Xia et al., 2024). Its hierarchical cache-based aggregation keeps up to $3.9\times$8 intermediate models in an L2 cache, promotes sufficiently trained or feature-balanced models to an L1 cache, and asynchronously schedules devices to train them. The reported outcome is up to $3.9\times$9 training acceleration and $1.6\times$0 accuracy improvement versus state-of-the-art FL baselines. This suggests that asynchronous cache swapping need not operate on raw data at all; the cached object may be a learned model checkpoint, and the asynchronous action may be promotion into an aggregation tier.

6. Semantic, plan-level, and policy-driven asynchronous caches

A further extension moves from data movement to policy movement: cached objects are not pages or KV blocks but reusable responses or plans, and the asynchronous action is verification, promotion, or correction. Krites formalizes this for tiered semantic caching in LLM services (Singh et al., 13 Feb 2026). The serving path remains exactly that of a standard static-threshold policy: the nearest static neighbor is accepted if its similarity exceeds $1.6\times$1, otherwise the system falls back to the dynamic tier or backend. The asynchronous component is triggered when the static similarity lies in the grey zone $1.6\times$2. In that case, Krites enqueues a background VerifyAndPromote(q, h_static, v_q) operation; if an LLM judge approves the curated static response for the new prompt, the system upserts $1.6\times$3 into the dynamic cache. On conversational and search-style workloads, this increases the fraction of requests served with curated static answers by up to $1.6\times$4 times relative to tuned baselines, with unchanged critical-path latency.

AgenticCache applies the same asynchronous promotion-and-correction pattern to embodied planning (Kim et al., 27 Apr 2026). Each cache entry is a 2-gram transition $1.6\times$5, scored by

$1.6\times$6

where $1.6\times$7 is local transition count and $1.6\times$8 is global importance based on LLM confirmations. The runtime loop chooses the best feasible cached plan, while a background Cache Updater asynchronously queries the LLM, validates executed transitions, and may immediately replace the active plan if the delayed LLM response disagrees. Across four multi-agent embodied benchmarks and three models, AgenticCache reports $1.6\times$9 average success-rate improvement, $$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$00 simulation-latency reduction, and $$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$01 token reduction across $$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$02 configurations. The cache remains small—$$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$03 to $$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$04 transitions and $$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$05–$$\text{LIS}_p = \alpha_1 \cdot \text{LTS}_p + \alpha_2 \cdot \text{LRS}_p + \beta \cdot \text{MDS}_p^{(Q)},$$06 KB per agent—so replacement is primarily statistical rather than space-driven.

These semantic and agentic systems sharpen a final point. Asynchronous cache swapping is not restricted to moving bytes between physical tiers. It can also mean moving authority between a fast speculative cache and a slow validating oracle. Krites promotes verified static answers into a dynamic tier without changing synchronous serving decisions; AgenticCache speculates with cached plans and later swaps in an LLM-corrected plan if necessary. A plausible implication is that future cache architectures will increasingly mix physical movement, semantic validation, and online control within the same asynchronous substrate.