---
title: 'CacheWise: Optimizing KVCache for Coding Agents'
url: https://www.emergentmind.com/topics/cachewise
type: topic
---

# CacheWise: Optimizing KVCache for Coding Agents

Searching arXiv for the specific paper and closely related work on KV-cache management for coding-agent serving.
arxiv_search(query="CacheWise coding agents KVCache vLLM", max_results=10, sort_by="relevance")
Searching arXiv by the exact title to confirm the primary source.
arxiv_search(query="\"CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents\"", max_results=5, sort_by="relevance")
CacheWise is a KVCache management layer for serving LLM coding agents, introduced to address a workload that differs materially from ordinary chat inference. Coding agents run as long-running closed-loop sessions in which LLM generations alternate with external tool calls, so successive requests repeatedly reuse large prompt prefixes while also placing sustained pressure on accelerator KV memory. CacheWise combines prefix-aware scheduling with reuse-aware eviction guided by lightweight predictions derived from tool call metadata. Implemented in vLLM and evaluated on real coding-assistant traces, it reduces KVCache evictions by up to \(2\)–\(2.6\times\) and improves total agent session completion time by up to \(\sim 3.5\times\) [2606.16824].

## 1. Workload setting and motivation

CacheWise is motivated by the observation that coding agents should be modeled as session-oriented inference workloads rather than as isolated requests. In a typical session, a user provides a high-level instruction, the model generates text or tool calls, external tools execute, tool outputs are appended to context, and the model is invoked again until task completion. This produces repeated revisits to a growing prefix, so KV reuse becomes central to efficient serving [2606.16824].

The workload characterization reported for coding agents is sharply different from chat. Coding-agent sessions have orders of magnitude more turns than other workloads, much larger prefills, and much smaller decodes per request. The paper reports an approximately \(21\times\) higher ratio of prefill to decode tokens compared to chatbot workloads. Tool-triggered requests are also structurally dominant: requests triggered by tool completion are \(20\times\) more frequent than user-initiated requests at the median. This shifts the relevant performance objective away from request-local metrics such as TTFT and TBT toward session completion time.

Sessions are also long-lived. The reported median session duration is 36 minutes, with a tail beyond 2.6 hours, and context length grows monotonically over the session. This combination—large repeated prefixes, many turns, long session lifetimes, and external tool waits—creates sustained KVCache pressure. Conventional serving policies handle this poorly: FCFS scheduling expands the active working set by interleaving many sessions without regard to prefix overlap, while LRU eviction uses only recency and ignores when a blocked session is likely to resume.

## 2. Empirical basis: CATraces and coding-agent behavior

The empirical foundation for CacheWise is CATraces, a dataset collected from consenting participants in the authors’ lab using Claude Code. The traces include user instructions, model outputs, tool calls, tool results, timestamps, token counts, and human interventions. CATraces contains 10M tokens and is described as containing real interactive sessions, real tasks, and real tool usage [2606.16824].

Several findings from CATraces are directly reflected in the CacheWise design. First, coding-agent sessions repeatedly revisit the same large conversational and tool-history prefix. Second, tool execution times are highly heterogeneous and induce irregular inter-request delays. The paper’s examples include `bash ls -la` at 49 ms, `git log ...` at 1143 ms, and `pytest ...` at 83333 ms. Short tools such as `grep` and `ls` coexist with long or highly variable tools such as `bash`, `pytest`, and `WebFetch`. This means that future KV reuse is not well approximated by simple recency.

Third, tool metadata carries predictive signal. The paper argues that tool type and arguments correlate with execution duration, and therefore with time to next reuse. That observation underlies CacheWise’s eviction policy: a session waiting on a short tool should usually retain its KV state, whereas a session waiting on a long-running tool is a better eviction candidate.

## 3. Formal model of KVCache pressure

CacheWise models an inference node with total KVCache block capacity \(M\). At time \(t\), the active sessions are denoted by \(\mathcal{S}_t = \{S_1, S_2, \ldots\}\). For each session \(S_i\), \(d_i\) is the total contiguous, monotonically growing prefix size, and \(k_i(t)\) is the number of blocks of \(S_i\) resident in XPU memory at time \(t\) [2606.16824].

The paper defines the number of additional blocks needed when session \(S_i\) issues request \(r_i\) at time \(t\) as
\[
a_i(t) = d_i - k_i(t).
\]
If sufficient memory is available, the request is admitted without eviction. The paper also defines time to next reuse, \(\tau_i\), as the time until session \(S_i\) issues its next request and its resident blocks are accessed again.

The active working-set notation is slightly inconsistent in the source. One definition gives
\[
W(t) = \sum_{i \in \mathcal{S}_t} d_i,
\]
while a later figure caption describes memory occupancy using
\[
W(t) = \sum_i k_i(t).
\]
Operationally, the distinction is between total per-session demand \(d_i\) and resident blocks \(k_i(t)\). That distinction is important because CacheWise’s scheduler prioritizes requests by marginal additional demand, while its eviction policy operates on already resident blocks.

The ideal eviction target is formulated in Belady-like form: evict the session whose resident KV will be reused farthest in the future,
\[
j^* = \operatorname*{arg\,max}_{j \in \mathcal{S}_t,\; j \ne i} \tau_j.
\]
CacheWise does not compute this oracle directly, but it uses tool-aware prediction as a practical approximation.

## 4. Prefix-aware scheduling

CacheWise’s scheduling policy is prefix-aware. For each queued request, it computes the additional blocks required,
\[
a_i(t) = d_i - k_i(t),
\]
and prioritizes requests in increasing order of \(a_i(t)\) [2606.16824].

This heuristic favors requests that can reuse the most already resident KV state. Equivalently, it schedules the request requiring the fewest new allocations. In coding-agent workloads, that tends to maximize immediate reuse, reduce prefill work, and lower the probability that a queued request triggers additional evictions. The paper also notes that ordering by \(a_i(t)\) approximates shortest-job-first behavior in a regime dominated by large-prefill requests.

The significance of this policy is workload-specific. In ordinary chat serving, prefix overlap is often limited to a system prompt or short conversation history. In coding agents, by contrast, long-running sessions repeatedly revisit a growing session prefix. Prefix-aware scheduling therefore has unusually high leverage: it reduces the active memory footprint exposed by request interleaving and suppresses KVCache thrashing at the scheduler level before eviction policy is even invoked.

## 5. Reuse-aware eviction and tool-aware prediction

When eviction is unavoidable, CacheWise replaces LRU with a reuse-aware policy that estimates which session is least likely to need its resident KV soon. The predictor uses tool metadata
\[
m = (\text{tool\_name}, \text{tool\_args})
\]
together with the tool generation time \(T_i\), and estimates the expected remaining time to reuse conditioned on the tool still running:
\[
\mathbb{E}[\tau_i(t) \mid \tau_i(t) - T_i > t - T_i].
\]
This quantity becomes the priority signal for eviction [2606.16824].

The design assumes that the service does not need a perfect prediction of future reuse; it mainly needs a useful ordering. Sessions predicted to be reused farthest in the future are better eviction candidates. The paper emphasizes that tool name alone is insufficient. For `bash`, for example, execution time depends strongly on arguments: Python one-liners have P50 0.1s and P99 0.4s, whereas mypy has P50 10s and P99 182s, docker builds have P50 22s and P99 152s, and Git operations have P50 0.1s and P99 97s.

To capture this heterogeneity, CacheWise clusters historical tool calls using TF-IDF embeddings of tool arguments, with a maximum vocabulary size of 5000 lexicons and scikit-learn \(\ell_2\)-norm weighting, followed by KMeans clustering. Duration distributions are then maintained per cluster rather than only per tool name. This is a deliberately lightweight predictor: it exploits structured tool metadata already visible at the serving layer and avoids heavy sequence modeling.

Operationally, when a request finishes and its blocks become unreferenced, those blocks retain session metadata and are inserted into an eviction heap. The heap priority is the predicted expected time to next reuse. Because predictions age, CacheWise periodically rebuilds the heap every \(N_{\text{rebuild}}\) engine iterations; the reported evaluation uses
\[
N_{\text{rebuild}} = 3.
\]

## 6. Implementation and evaluation

CacheWise is implemented in vLLM in about 2,500 lines of Python, extending the batch scheduler and KVCache block manager. It associates session-level metadata with blocks, including `tool_name`, `tool_args`, and \(T_i\), and requires no model modification or retraining. The prototype can execute models including grouped-query attention and multi-query attention [2606.16824].

The reported experiments use Qwen2.5-Coder-32B-Instruct on a server with \(2\times\) H200 GPUs, an AMD EPYC 9534 64-core CPU, tensor parallelism \(=2\), and chunked prefill enabled with a maximum of 512 tokens per chunk. CATraces is split 80% for offline predictor training and 20% for evaluation. Baselines are vLLM with FCFS scheduling and block-level LRU eviction, and InferCept with FCFS scheduling and a moving-average tool-duration predictor.

The main results are concentrated under higher load, when KV contention is substantial. At \(N \le 10\) concurrent sessions, all systems perform similarly. For \(N > 10\), CacheWise achieves \(2.7\times\)–\(3.5\times\) lower session completion time than vLLM and InferCept, improves token goodput by \(1.64\times\)–\(2\times\), improves request throughput by \(1.5\times\)–\(2\times\), and reduces KVCache block evictions by \(2\times\)–\(2.6\times\) [2606.16824].

The evaluation also isolates the contributions of the two components. Prefix-aware scheduling alone already improves token goodput by about \(1.38\times\)–\(1.64\times\) at \(N=30\) and \(1.6\times\)–\(1.7\times\) at \(N=40\), while reducing session completion time by about \(1.8\times\)–\(2.35\times\) at \(N=30\) and \(1.85\times\)–\(2.66\times\) at \(N=40\). When all systems use the same prefix-aware scheduler, CacheWise’s predictive eviction still yields \(1.2\times\)–\(1.6\times\) higher token goodput and about \(1.7\times\)–\(2\times\) lower session completion time than the prefix-aware versions of the baselines.

Request latency improves at P50 and P90, with reported \(13\)–\(14\times\) lower P50 request completion times than the baselines, but P99 request latency can worsen. This is a direct consequence of prioritizing requests with large resident prefixes; new or poorly reused requests may wait longer. The paper argues that this tradeoff is acceptable because coding-agent serving is better evaluated by session completion time than by per-request tail latency.

CPU scheduling overhead increases, but the absolute cost remains small relative to saved model execution time. At \(N=40\), reported scheduling overhead rises from 0.33 s in vLLM to 0.99 s in CacheWise, while model execution time drops from 16.6 s to 11.2 s. The paper also evaluates an offloading mode in which evicted KV is transferred between GPU and CPU rather than recomputed. Under \(N=30\), CacheWise still achieves about \(1.19\times\) lower session completion time than vLLM, though the gain is smaller because PCIe transfer is cheaper than recomputing large prefixes.

## 7. Relation to KV-cache research and limitations

CacheWise occupies a specific point in the broader KV-cache literature. It is neither a remote-capacity-tier design nor a generic prefix-cache mechanism. Its main concern is on-node scheduling and eviction for long-running, tool-driven, session-oriented coding workloads. This distinguishes it from systems such as ObjectCache, which studies layerwise retrieval of immutable prefix KV from S3-compatible object storage and reports only 5.6% latency overhead over local DRAM for 64K contexts by overlapping transfer with compute [2605.22850]. CacheWise instead assumes the dominant inefficiency is repeated eviction and recomputation of large session prefixes within a loaded inference node [2606.16824].

The method also has clear limitations. The trace dataset comes from a lab cohort using Claude Code rather than a broad internet-scale deployment. The predictor is intentionally lightweight and uses tool metadata rather than a richer learned model. The paper provides no theoretical guarantee for either scheduling or eviction, no fairness optimization, and no direct comparison against every prefix-aware serving system discussed in the surrounding literature. The reported gains are strongest under high concurrency and tight KV memory; under low load, when evictions are rare, all systems behave similarly.

A further implication is that CacheWise optimizes a workload-specific objective. By design it can worsen P99 request latency while improving session completion time and token goodput. This suggests that its deployment is most appropriate in coding-agent serving stacks where closed-loop session progress, rather than uniform per-request latency, is the primary systems objective.

Source: https://www.emergentmind.com/topics/cachewise