CacheWise: Optimizing KVCache for Coding Agents

• CacheWise is a KVCache management layer designed for long-running, session-oriented coding agents that repeatedly reuse large prompt prefixes.
• It employs prefix-aware scheduling to minimize new memory allocations and improve session completion time and token goodput.
• The system integrates a lightweight, tool-aware prediction model to guide reuse-aware eviction, reducing KVCache evictions by up to 2.6×.

Searching arXiv for the specific paper and closely related work on KV-cache management for coding-agent serving. arxiv_search(query="^^^^^^^^3CacheWise coding agents KVCache vLLM3^^^^^^^^", max_results=^^^^3relevance3CacheWise coding agents KVCache vLLM3^^^^, sort_by="^^^^^^^^3relevance3 Searching arXiv by the exact title to confirm the primary source. arxiv_search(query="^^^^^^^^3\3 Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents^^^^^^^^3\3 max_results=5, sort_by="^^^^^^^^3relevance3 CacheWise is a KVCache management layer for serving LLM coding agents, introduced to address a workload that differs materially from ordinary chat inference. Coding agents run as long-running closed-loop sessions in which LLM generations alternate with external tool calls, so successive requests repeatedly reuse large prompt prefixes while also placing sustained pressure on accelerator KV memory. CacheWise combines prefix-aware scheduling with reuse-aware eviction guided by lightweight predictions derived from tool call metadata. Implemented in vLLM and evaluated on real coding-assistant traces, it reduces KVCache evictions by up to PRESERVED_PLACEHOLDER_^^^^3CacheWise coding agents KVCache vLLM3^^^^–PRESERVED_PLACEHOLDER_^^^^3relevance3^^^^ and improves total agent session completion time by up to PRESERVED_PLACEHOLDER_^^^^3\3^^^^ (&&&^^^^3CacheWise coding agents KVCache vLLM3^^^^&&&).

^^^^3relevance3^^^^. Workload setting and motivation

CacheWise is motivated by the observation that coding agents should be modeled as session-oriented inference workloads rather than as isolated requests. In a typical session, a user provides a high-level instruction, the model generates text or tool calls, external tools execute, tool outputs are appended to context, and the model is invoked again until task completion. This produces repeated revisits to a growing prefix, so KV reuse becomes central to efficient serving (&&&^^^^3CacheWise coding agents KVCache vLLM3^^^^&&&).

The workload characterization reported for coding agents is sharply different from chat. Coding-agent sessions have orders of magnitude more turns than other workloads, much larger prefills, and much smaller decodes per request. The paper reports an approximately $21\times$ higher ratio of prefill to decode tokens compared to chatbot workloads. Tool-triggered requests are also structurally dominant: requests triggered by tool completion are $20\times$ more frequent than user-initiated requests at the median. This shifts the relevant performance objective away from request-local metrics such as TTFT and TBT toward session completion time.

Sessions are also long-lived. The reported median session duration is 36 minutes, with a tail beyond ^^^^3\3^^^^.6 hours, and context length grows monotonically over the session. This combination—large repeated prefixes, many turns, long session lifetimes, and external tool waits—creates sustained KVCache pressure. Conventional serving policies handle this poorly: FCFS scheduling expands the active working set by interleaving many sessions without regard to prefix overlap, while LRU eviction uses only recency and ignores when a blocked session is likely to resume.

^^^^3\3^^^^. Empirical basis: CATraces and coding-agent behavior

The empirical foundation for CacheWise is CATraces, a dataset collected from consenting participants in the authors’ lab using Claude Code. The traces include user instructions, model outputs, tool calls, tool results, timestamps, token counts, and human interventions. CATraces contains ^^^^3relevance3CacheWise coding agents KVCache vLLM3^^^^M tokens and is described as containing real interactive sessions, real tasks, and real tool usage (&&&^^^^3CacheWise coding agents KVCache vLLM3^^^^&&&).

Several findings from CATraces are directly reflected in the CacheWise design. First, coding-agent sessions repeatedly revisit the same large conversational and tool-history prefix. Second, tool execution times are highly heterogeneous and induce irregular inter-request delays. The paper’s examples include bash ls -la at 49 ms, git log ... at ^^^^3relevance3relevance3^^^^43 ms, and pytest ... at 83333 ms. Short tools such as grep and ls coexist with long or highly variable tools such as bash, pytest, and WebFetch. This means that future KV reuse is not well approximated by simple recency.

Third, tool metadata carries predictive signal. The paper argues that tool type and arguments correlate with execution duration, and therefore with time to next reuse. That observation underlies CacheWise’s eviction policy: a session waiting on a short tool should usually retain its KV state, whereas a session waiting on a long-running tool is a better eviction candidate.

3. Formal model of KVCache pressure

CacheWise models an inference node with total KVCache block capacity $M$. At time $t$, the active sessions are denoted by $\mathcal{S}_t = \{S_1, S_2, \ldots\}$. For each session $S_i$, $d_i$ is the total contiguous, monotonically growing prefix size, and PRESERVED_PLACEHOLDER_^^^^3relevance3CacheWise coding agents KVCache vLLM3^^^^ is the number of blocks of PRESERVED_PLACEHOLDER_^^^^3relevance3relevance3^^^^ resident in XPU memory at time PRESERVED_PLACEHOLDER_^^^^3relevance3\3^^^^ (&&&^^^^3CacheWise coding agents KVCache vLLM3^^^^&&&).

The paper defines the number of additional blocks needed when session PRESERVED_PLACEHOLDER_^^^^3relevance33^^^^ issues request PRESERVED_PLACEHOLDER_^^^^3relevance3^^^^4 at time PRESERVED_PLACEHOLDER_^^^^3relevance3^^^^5 as

PRESERVED_PLACEHOLDER_^^^^3relevance3^^^^6

If sufficient memory is available, the request is admitted without eviction. The paper also defines time to next reuse, PRESERVED_PLACEHOLDER_^^^^3relevance3^^^^7, as the time until session PRESERVED_PLACEHOLDER_^^^^3relevance3^^^^8 issues its next request and its resident blocks are accessed again.

The active working-set notation is slightly inconsistent in the source. One definition gives

PRESERVED_PLACEHOLDER_^^^^3relevance3^^^^9

while a later figure caption describes memory occupancy using

PRESERVED_PLACEHOLDER_^^^^3\3CacheWise coding agents KVCache vLLM3^^^^

Operationally, the distinction is between total per-session demand PRESERVED_PLACEHOLDER_^^^^3\3relevance3^^^^ and resident blocks PRESERVED_PLACEHOLDER_^^^^3\3\3^^^^. That distinction is important because CacheWise’s scheduler prioritizes requests by marginal additional demand, while its eviction policy operates on already resident blocks.

The ideal eviction target is formulated in Belady-like form: evict the session whose resident KV will be reused farthest in the future,

PRESERVED_PLACEHOLDER_^^^^3\33^^^^

CacheWise does not compute this oracle directly, but it uses tool-aware prediction as a practical approximation.

4. Prefix-aware scheduling

CacheWise’s scheduling policy is prefix-aware. For each queued request, it computes the additional blocks required,

PRESERVED_PLACEHOLDER_^^^^3\3^^^^4

and prioritizes requests in increasing order of PRESERVED_PLACEHOLDER_^^^^3\3^^^^5 (&&&^^^^3CacheWise coding agents KVCache vLLM3^^^^&&&).

This heuristic favors requests that can reuse the most already resident KV state. Equivalently, it schedules the request requiring the fewest new allocations. In coding-agent workloads, that tends to maximize immediate reuse, reduce prefill work, and lower the probability that a queued request triggers additional evictions. The paper also notes that ordering by PRESERVED_PLACEHOLDER_^^^^3\3^^^^6 approximates shortest-job-first behavior in a regime dominated by large-prefill requests.

The significance of this policy is workload-specific. In ordinary chat serving, prefix overlap is often limited to a system prompt or short conversation history. In coding agents, by contrast, long-running sessions repeatedly revisit a growing session prefix. Prefix-aware scheduling therefore has unusually high leverage: it reduces the active memory footprint exposed by request interleaving and suppresses KVCache thrashing at the scheduler level before eviction policy is even invoked.

5. Reuse-aware eviction and tool-aware prediction

When eviction is unavoidable, CacheWise replaces LRU with a reuse-aware policy that estimates which session is least likely to need its resident KV soon. The predictor uses tool metadata

PRESERVED_PLACEHOLDER_^^^^3\3^^^^7

together with the tool generation time PRESERVED_PLACEHOLDER_^^^^3\3^^^^8, and estimates the expected remaining time to reuse conditioned on the tool still running: PRESERVED_PLACEHOLDER_^^^^3\3^^^^9 This quantity becomes the priority signal for eviction (&&&^^^^3CacheWise coding agents KVCache vLLM3^^^^&&&).

The design assumes that the service does not need a perfect prediction of future reuse; it mainly needs a useful ordering. Sessions predicted to be reused farthest in the future are better eviction candidates. The paper emphasizes that tool name alone is insufficient. For bash, for example, execution time depends strongly on arguments: Python one-liners have P5^^^^3CacheWise coding agents KVCache vLLM3^^^^ ^^^^3CacheWise coding agents KVCache vLLM3^^^^.^^^^3relevance3^^^^s and P99 ^^^^3CacheWise coding agents KVCache vLLM3^^^^.4s, whereas mypy has P5^^^^3CacheWise coding agents KVCache vLLM3^^^^ ^^^^3relevance3CacheWise coding agents KVCache vLLM3^^^^s and P99 ^^^^3relevance3^^^^8^^^^3\3 docker builds have P5^^^^3CacheWise coding agents KVCache vLLM3^^^^ ^^^^3\3\3^^^^s and P99 ^^^^3relevance3^^^^5^^^^3\3 and Git operations have P5^^^^3CacheWise coding agents KVCache vLLM3^^^^ ^^^^3CacheWise coding agents KVCache vLLM3^^^^.^^^^3relevance3^^^^s and P99 97s.

To capture this heterogeneity, CacheWise clusters historical tool calls using TF-IDF embeddings of tool arguments, with a maximum vocabulary size of 5^^^^3CacheWise coding agents KVCache vLLM3CacheWise coding agents KVCache vLLM3CacheWise coding agents KVCache vLLM3^^^^ lexicons and scikit-learn $21\times$^^^^3CacheWise coding agents KVCache vLLM3^^^^-norm weighting, followed by KMeans clustering. Duration distributions are then maintained per cluster rather than only per tool name. This is a deliberately lightweight predictor: it exploits structured tool metadata already visible at the serving layer and avoids heavy sequence modeling.

Operationally, when a request finishes and its blocks become unreferenced, those blocks retain session metadata and are inserted into an eviction heap. The heap priority is the predicted expected time to next reuse. Because predictions age, CacheWise periodically rebuilds the heap every $21\times$^^^^3relevance3^^^^ engine iterations; the reported evaluation uses

$21\times$^^^^3\3^^^^

6. Implementation and evaluation

CacheWise is implemented in vLLM in about ^^^^3\3^^^^,5^^^^3CacheWise coding agents KVCache vLLM3CacheWise coding agents KVCache vLLM3^^^^ lines of Python, extending the batch scheduler and KVCache block manager. It associates session-level metadata with blocks, including tool_name, tool_args, and $21\times$3, and requires no model modification or retraining. The prototype can execute models including grouped-query attention and multi-query attention (&&&^^^^3CacheWise coding agents KVCache vLLM3^^^^&&&).

The reported experiments use Qwen^^^^3\3^^^^.5-Coder-3^^^^3\3 on a server with $21\times$4 H^^^^3\3CacheWise coding agents KVCache vLLM3CacheWise coding agents KVCache vLLM3^^^^ GPUs, an AMD EPYC 9534 64-core CPU, tensor parallelism $21\times$5, and chunked prefill enabled with a maximum of 5^^^^3relevance3\3^^^^ tokens per chunk. CATraces is split 8^^^^3CacheWise coding agents KVCache vLLM3^^^^% for offline predictor training and ^^^^3\3CacheWise coding agents KVCache vLLM3^^^^% for evaluation. Baselines are vLLM with FCFS scheduling and block-level LRU eviction, and InferCept with FCFS scheduling and a moving-average tool-duration predictor.

The main results are concentrated under higher load, when KV contention is substantial. At $21\times$6 concurrent sessions, all systems perform similarly. For $21\times$7, CacheWise achieves $21\times$8–$21\times$9 lower session completion time than vLLM and InferCept, improves token goodput by $20\times$^^^^3CacheWise coding agents KVCache vLLM3^^^^–20×20\times^^^^3relevance3^^^^, improves request throughput by $20\times$^^^^3\3^^^^–20×20\times3, and reduces KVCache block evictions by $20\times$4–$20\times$5 (&&&^^^^3CacheWise coding agents KVCache vLLM3^^^^&&&).

The evaluation also isolates the contributions of the two components. Prefix-aware scheduling alone already improves token goodput by about $20\times$6–$20\times$7 at $20\times$8 and $20\times$9–$M$^^^^3CacheWise coding agents KVCache vLLM3^^^^ at $M$^^^^3relevance3^^^^, while reducing session completion time by about $M$^^^^3\3^^^^–MM3 at $M$4 and $M$5–$M$6 at $M$7. When all systems use the same prefix-aware scheduler, CacheWise’s predictive eviction still yields $M$8–$M$9 higher token goodput and about $t$^^^^3CacheWise coding agents KVCache vLLM3^^^^–tt^^^^3relevance3^^^^ lower session completion time than the prefix-aware versions of the baselines.

Request latency improves at P5^^^^3CacheWise coding agents KVCache vLLM3^^^^ and P9^^^^3CacheWise coding agents KVCache vLLM3^^^^, with reported $t$^^^^3\3^^^^–tt3 lower P5^^^^3CacheWise coding agents KVCache vLLM3^^^^ request completion times than the baselines, but P99 request latency can worsen. This is a direct consequence of prioritizing requests with large resident prefixes; new or poorly reused requests may wait longer. The paper argues that this tradeoff is acceptable because coding-agent serving is better evaluated by session completion time than by per-request tail latency.

CPU scheduling overhead increases, but the absolute cost remains small relative to saved model execution time. At $t$4, reported scheduling overhead rises from ^^^^3CacheWise coding agents KVCache vLLM3^^^^.33 s in vLLM to ^^^^3CacheWise coding agents KVCache vLLM3^^^^.99 s in CacheWise, while model execution time drops from ^^^^3relevance3^^^^6.6 s to ^^^^3relevance3relevance3^^^^.^^^^3\3^^^^ s. The paper also evaluates an offloading mode in which evicted KV is transferred between GPU and CPU rather than recomputed. Under $t$5, CacheWise still achieves about $t$6 lower session completion time than vLLM, though the gain is smaller because PCIe transfer is cheaper than recomputing large prefixes.

7. Relation to KV-cache research and limitations

CacheWise occupies a specific point in the broader KV-cache literature. It is neither a remote-capacity-tier design nor a generic prefix-cache mechanism. Its main concern is on-node scheduling and eviction for long-running, tool-driven, session-oriented coding workloads. This distinguishes it from systems such as ObjectCache, which studies layerwise retrieval of immutable prefix KV from S3-compatible object storage and reports only 5.6% latency overhead over local DRAM for 64K contexts by overlapping transfer with compute (Zhu et al., 16 May 2026). CacheWise instead assumes the dominant inefficiency is repeated eviction and recomputation of large session prefixes within a loaded inference node (&&&^^^^3CacheWise coding agents KVCache vLLM3^^^^&&&).

The method also has clear limitations. The trace dataset comes from a lab cohort using Claude Code rather than a broad internet-scale deployment. The predictor is intentionally lightweight and uses tool metadata rather than a richer learned model. The paper provides no theoretical guarantee for either scheduling or eviction, no fairness optimization, and no direct comparison against every prefix-aware serving system discussed in the surrounding literature. The reported gains are strongest under high concurrency and tight KV memory; under low load, when evictions are rare, all systems behave similarly.

A further implication is that CacheWise optimizes a workload-specific objective. By design it can worsen P99 request latency while improving session completion time and token goodput. This suggests that its deployment is most appropriate in coding-agent serving stacks where closed-loop session progress, rather than uniform per-request latency, is the primary systems objective.