CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents

Abstract: Coding agents are a fast-growing LLM application, executing as long-running closed-loop sessions in which LLM generations alternate with external tool calls. Yet, unlike chat workloads, their serving behavior has not been studied extensively. We address this gap by collecting a dataset of real-world coding assistant traces. Our analysis shows that coding agent sessions repeatedly reuse large prefixes and create sustained KVCache pressure that conventional LLM serving policies handle poorly. Based on our analysis, we present CacheWise, a KVCache management layer that improves KVCache reuse for coding agent workloads. CacheWise combines prefix-aware scheduling with reuse-aware eviction guided by lightweight predictions from tool call metadata. Implemented in vLLM and evaluated on the collected traces, CacheWise reduces KVCache evictions by up to 2-2.6x and improves total agent session completion time by up to 3.5x.

• The paper introduces CacheWise, a KVCache management strategy employing prefix-aware scheduling and predictive eviction tailored for LLM coding agents.
• It quantitatively reduces KVCache evictions by up to 2–2.6×, improves session completion time by approximately 3.5×, and doubles token goodput compared to baselines.
• The study demonstrates that leveraging tool metadata for reuse prediction nearly achieves oracle eviction performance without intrusive modifications to the system.

Efficient KVCache Management for LLM Coding Agents: An Analysis of "CacheWise"

Motivation and Workload Characterization

The proliferation of LLM-based coding agents introduces a new, under-characterized class of serving workload that departs markedly from canonical chatbot or instruction-following interactions. Coding agents execute long, closed-loop sessions, alternating between LLM-driven code generation and external tool execution (Figure 1).

Figure 1: Coding agent workflow highlighting closed-loop execution between LLM requests and tool/environment calls.

Empirical analysis of real-world coding agent traces (CATraces) exposes several unique features: (1) sessions consist of many more turns per task compared to chatbot workloads (Figure 2), (2) tool-initiated requests dominate user-initiated input by over an order of magnitude (Figure 3), and (3) session durations are long-tailed, with median completion time around 36 minutes and tail values exceeding 2.6 hours (Figure 4).

Figure 2: Distribution of number of turns for different workloads, with coding agents exhibiting significantly more turns.

Figure 3: CDF of requests triggered by tool completion versus user input. Tool-initiated requests dominate, indicating a predominantly closed-loop request generation process.

Figure 4: Session duration distribution observed in CATraces.

The resulting workload characteristics raise several design challenges. Each agent session accumulates a growing context, thereby demanding larger and longer-lived KVCache residency on accelerators. Without efficient management, memory pressure and excessive KVCache evictions lead to significant recomputation or offloading overheads, degrading overall token goodput and system efficiency. Figure 5 shows that context length per session expands monotonically over time, exacerbating memory contention.

Figure 5: Context Length growth in number of tokens per session.

Systems Implications of Coding Agents

Unlike traditional, stateless or short-lived LLM serving workloads, coding agents require that accelerators preserve and maximize reuse of large KVCache prefixes. Sessions with highly variable inter-request times (Figure 6), driven by external tool latencies (Figure 7), produce irregular reuse patterns that standard recency-based (LRU) eviction policies fail to capture.

Figure 6: Time between consecutive LLM requests displays large variability due to tool execution latency.

Figure 7: Distribution of tool execution durations reflects the heterogeneity of invoked external tools.

These findings reveal the inadequacy of workload-agnostic FCFS scheduling and LRU cache management. FCFS batch admission leads to KVCache thrashing among co-located sessions, as interleaving increases resident working sets and induces avoidable evictions of soon-to-be-reused prefixes. LRU considers only historical access, missing critical information on imminent future reuse that could be gleaned from tool call metadata.

CacheWise: Prefix-Aware Scheduling and Predictive Eviction

CacheWise introduces an LLM server KVCache management layer specifically designed to address agentic workload requirements. As illustrated in Figure 8, it incorporates two orthogonal policies:

1. Prefix-Aware Request Scheduling: Requests are prioritized based on the degree of overlap between required context and currently resident KVCache state, admitting requests that maximize immediate reuse and minimize additional block admission.
2. Reuse-Aware Predictive Eviction: Instead of recency-based eviction, CacheWise employs predictors trained on historical tool metadata (tool type, arguments) to rank sessions/blocks by anticipated time-to-next-reuse. Sessions with longest expected tool durations (i.e., least likely to generate a new LLM request soon) are evicted first.

   Figure 8: (A) End-to-end LLM serving workflow. (B) Prefix-aware scheduling prioritizes requests for maximal KVCache overlap.

This agent-aware, session-centric approach reduces eviction-induced recomputation and offloading, lowers latency for critical session turns, and maintains high accelerator utilization under memory pressure.

Quantitative Results

Evaluation on realistic multi-turn coding agent traces shows substantial gains over state-of-the-art baselines (vLLM, InferCept). Key numerical results—robust across experimental configurations—include:

• KVCache evictions reduced by 2–2.6× compared to baseline policies (Figure 9).
• Session completion time improved by up to ≈3.5× at high load (Figure 10).
• Token goodput (useful tokens/sec) increased by up to 2× versus baseline (Figures 15, 16).
• The predictive eviction policy approaches the idealized ground-truth (oracle) performance, signifying accurate reuse ranking without requiring invasive changes to agent application code or toolchain instrumentation.

  Figure 10: Session completion time (in seconds) for different KVCache management systems under coding agent workloads.

  

  Figure 9: Serving efficiency of different KVCache management systems, with CacheWise achieving higher throughput and lower latency.

  

Figure 11: Token goodput showing significant improvements due to reduced recomputation and better KVCache retention.

Figure 12: Further breakdown of token goodput across scheduling and eviction strategies.

CacheWise's scheduling and eviction modules incur a modest 2.4–3× increase in CPU scheduler overhead (Figure 13), but this is negligible relative to the order-of-magnitude reduction in GPU-side execution/idle time.

Figure 13: CPU scheduling overheads versus GPU execution time, highlighting the efficiency tradeoff.

Practical and Theoretical Implications

Practically, these results show that general-purpose LLM serving stacks must re-orient toward session-level, agent-aware optimization as closed-loop agents proliferate. This entails deeper coupling of scheduling and memory management policies with workload semantics—specifically, session metadata and tool usage signatures.

Theoretically, the work demonstrates that practical predictors—using lightweight feature extraction and argument clustering—can attain near-Belady eviction performance for agentic LLM serving, a regime that has been elusive in historical caching literature.

Additionally, these mechanisms are complementary to KVCache tiered storage or compression (e.g., Mooncake, LMCache, EvicPress, CacheGen): by minimizing the incidence of unnecessary evictions, CacheWise reduces both recomputation and data transfer volume across all KVCache tiers.

Future Directions

CacheWise’s design philosophy suggests several future research directions:

• Adaptive online learning: Incorporating continual online retraining of tool duration predictors could further improve performance under dynamic workloads or changing code/tool environments.
• Integration with broader serving stacks: Extending session- and tool-aware management principles to disaggregated, multi-node, or cloud-based serving infrastructure may yield global scheduling and cache efficiency gains.
• Generalization to non-coding agents: Investigating the applicability of these methods for other agentic, programmatic, or multi-modal LLM use cases beyond code generation.

Conclusion

"CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents" systematically dissects the unique session dynamics of coding agent workloads and demonstrates that agent-aware, predictive resource management yields strong, quantifiable improvements in serving efficiency. The study establishes that traditional request-level scheduling and recency-based eviction are ill-suited for agent-centric interaction patterns, and that lightweight utilization of tool-level metadata for reuse prediction closes much of the gap to oracle performance. This paradigm is likely to underpin the next generation of high-throughput, low-latency LLM serving infrastructure as agentic applications continue to evolve (2606.16824).