TraceLab: Characterizing Coding Agent Workloads for LLM Serving

Abstract: Coding agents are rapidly becoming a major application of agentic LLMs, but serving them efficiently remains challenging. Progress on this challenge requires understanding real workload patterns, yet the data needed for such analysis is largely absent. Existing public traces and benchmarks do not capture real, day-to-day coding-agent usage across multiple agents and model families for serving-system analysis. To help fill this gap, we collect and release a trace of roughly 4,300 coding-agent sessions, containing about 350,000 LLM steps and 430,000 tool calls from our own day-to-day use of Claude Code and Codex. Our analysis shows that coding-agent workloads feature long autonomous loops, long contexts with short outputs, diverse and heavily-tailed tool calls, and high but imperfect prefix cache hit rates. These findings point to concrete opportunities for optimizing serving, including lower-overhead tool calling, append-length-aware prefill, semantic-aware tool-latency prediction, and improved KV-cache management around human-paced gaps. We release the dataset, trace collection pipeline, and analysis code at https://github.com/uw-syfi/TraceLab.git; the project website is https://tracelab.cs.washington.edu.

• The paper presents TraceLab, a comprehensive cross-provider dataset capturing 4,300 sessions, 350,000 LLM steps, and 430,000 tool calls to evaluate coding agent workloads.
• The study reveals that context replay dominates serving costs with prefix tokens accounting for 59.5% of expenses and highlights heavy-tailed tool latency distributions.
• The analysis identifies opportunities for optimizing LLM serving via improved KV cache management, distinct prefill pathways, and latency-aware tool scheduling.

TraceLab: Systems Analysis of Coding Agent Workloads for LLM Serving

Introduction and Motivation

TraceLab conducts a comprehensive analysis of real-world coding agent workloads, focusing on agentic LLMs such as Claude Code (Anthropic) and Codex (OpenAI). Unlike prior benchmarks and traces, which inadequately capture multi-step tool use and evolving contexts of persistent coding agents, this work presents a cross-provider dataset encompassing approximately 4,300 interactive coding sessions. This dataset records ~350,000 LLM steps and over 430,000 tool invocations, capturing the activity of 43 developers over eight months.

The study fills a critical gap for systems researchers seeking to optimize LLM serving for agentic environments. Key findings highlight distinctive workload properties: sustained context growth, frequent tool invocation with diverse and heavy-tailed latency distributions, high but imperfect reliance on prefix caches, and an autonomous operational pattern dominated by tool-driven loops rather than human-initiated steps.

Dataset and Methodological Framework

TraceLab’s dataset is derived from the production use of Claude Code and Codex coding agents. Sessions are normalized into a unified event schema, supporting detailed per-step dissection of token accounting (prefix, append, output), timing (including human and agentic latencies), and tool interactions. Rigorous anonymization ensures privacy—raw message/text content is omitted, and identifiers are replaced with stable pseudonyms.

At the session level, user requests are resolved by an agentic loop—a sequence of LLM invocations interleaved with tool calls. The persistent session context accumulates user inputs, tool outcomes, and generative outputs, occasionally undergoing compaction when model context limits are approached. The dataset supports fine-grained analysis of context growth, cache reuse, tool call overheads, and system cost structure.

Session Dynamics, Context Evolution, and Cost

Sessions exhibit pronounced autonomy and heavy right-tail distributions in length, requests, and tool calls. On average, each session comprises 8.8 LLM invocations and 10.8 tool calls per request. Human latency (reading, ideation, input) dominates elapsed time, accounting for 92.3% of wall time, whereas LLM generation and tool execution constitute 3.3% and 4.8% respectively.

Context typically grows monotonically, but rare compactions—driven by model context constraints—truncate the history, especially for models with shorter context capacity. Codex exhibits more frequent micro- and compaction-induced reductions compared to Claude.

Cost analysis reveals a counterintuitive dynamic: prefix tokens, despite lower per-token rates, accumulate to comprise 59.5% of serving costs due to being replayed on every step. Append (fresh input) and output tokens represent 29.2% and 11.2%, respectively. The dominant cost driver thus shifts from generation to historical context replay as agentic workflows progress.

LLM Input/Output Patterns and Timing

Input Structure

In agentic steps, the input is overwhelmingly dominated by prefix tokens (replayed context), with median values an order of magnitude higher than append tokens (new input):

Figure 1: Per-step prefix tokens vs. append tokens reveals the dominance of cached context relative to new input at each step.

This bimodal structure reflects two operational regimes: initial or prefix-cache miss steps (large append, small prefix), and steady-state steps with extensive context replay and modest incremental input. Steps with long append lengths are rare but contribute disproportionately to the overall prefill workload.

Figure 2: Most steps append <1k tokens, yet rare long-append steps account for the majority of prefill volume.

Output Attribution

Despite long input contexts, per-step outputs are short (median 214 tokens for Claude, 184 for Codex), a consequence of agentic decomposition into atomic tool-invocation steps. Attribution analysis reveals provider-specific KV cache policies: Claude and GPT-5.5 typically "resend" prior outputs as append tokens for the next step, while GPT-5.4 caches outputs as part of the next prefix.

Figure 3: Merged output-attribution evidence by model shows differing strategies for integrating prior output into subsequent step inputs.

Figure 4: Per-step output-token distribution by provider demonstrates the consistently short output lengths.

Figure 5: Schematic depiction of output caching versus output resend in agentic LLM step sequences.

Latency and Decode Speed

LLM generation speed scales variably with context length. Claude maintains near-constant decode speed across a broader range until very long contexts, while Codex sees a significant drop as context length increases. High variance exists even at fixed context due to scheduler, model, and output-shape heterogeneity.

Figure 6: Trace-observed LLM timing versus total input context length highlights context-dependent decoding speed trends.

Tool Use Pattern and Latency

Tool use is central to coding agents, with a small subset (Bash/exec_command, file reads, and edits) accounting for the vast majority of calls. Tool calls exhibit highly skewed latency distributions: while most complete within a second, a small number of long-running commands dominate total tool-execution time.

Figure 7: Tool call count distribution for Claude and Codex—dominated by command execution and file operations.

Figure 8: Tool-call latency bins reveal that long-latency calls, though rare, dominate cumulative tool-execution time.

Figure 9: Distribution of tool-call latencies for top tools demonstrates significant within-tool-type variability.

Prefix Cache: Behavior and System Implications

High prefix cache hit rates (>95%) are observed, but misses cluster almost entirely in user-initiated steps following extended idle/human thinking periods. Cache eviction is tightly linked to idle duration; gaps over 5 minutes markedly increase miss rates, and almost no cache survives 1-hour gaps.

Figure 10: Example session progression, visualizing prefix/cache evolution and context replays over agentic steps.

Cache misses lead to significant redundant prefill, with actual "fresh" tokens accounting for as little as 12–26% of append work. Amplification factors reach over 8× for Claude and 3.9× for Codex. Storage/retention trade-off curves show that modest increases in timeout can capture large cache efficiency gains, albeit with linear storage cost growth.

Figure 11: Prefix cache hit rate drops sharply with increasing idle gap, especially in user-initiated steps.

Figure 12: Trade-off between prefix cache eviction timeout, achievable hit rate, suspended KV storage ratio, and prefill amplification.

Augmenting cache retention during human pauses (e.g., prefetched refresh) could reduce append prefill cost by up to 46% and overall serving cost by 13%.

Implications, System Opportunities, and Future Directions

Practical and Theoretical Implications:

• Dominance of Context Replay: Unlike stateless or short-turn chat LLM serving, agentic coding workloads are dominated by context replay costs, making KV cache management the primary systems bottleneck.
• Heavy-Tailed Tool Latencies: Serving systems must account for unpredictable, long-running tool calls, motivating semantic- and history-aware eviction/prefetch policies as empirical type-based heuristics are insufficiently predictive.
• Workload-Aware Optimization: The clear bimodal distribution of input structure suggests separate fast-path and bulk-path handling for prefill routing, kernel selection, and parallelism adaptation.

Research and Engineering Opportunities:

• Append-Aware Prefill Pathways: Segregate rare large-prefill steps from dominant short-prefill traffic for throughput and latency gains in batching and scheduling.
• KV Cache Innovation: Pursue advances in cache compression, sparse attention, and cost-effective external storage/refresh to mitigate human-induced cache evictions.
• Fused and Parallel Tool Calls: Reduce round-trip and scheduling overhead by enabling tools to be invoked in denser, possibly parallel, batches.
• Semantic Tool Latency Prediction: Move beyond tool-type-based cache eviction predictors to integrate operation semantics and real-time latency histories (Tiwari et al., 15 Jun 2026).
• Cache Retention During Human Pauses: Employ predictive/refresh-based strategies to minimize cache eviction around human-in-the-loop gaps.

Future Directions:

Extended studies across organizational contexts and tasks beyond coding (e.g., general automation, multi-modal agentics) will further elucidate workload-specific system requirements and generalize the TraceLab methodology. Enhanced provider-level telemetry would enable causal inference regarding back-end scheduling and cache management mechanisms.

Conclusion

TraceLab establishes a benchmark cross-provider trace for coding agent usage, quantifying the agentic LLM serving workload at an unprecedented level of systems-relevant detail. Findings reveal that long-context replay, tool-centric autonomy, and high-variance tool latency, not generation cost, dominate the operational burden. The results delineate concrete system research and engineering targets, particularly in KV cache management, prefill optimization, and human-aware cache retention. TraceLab thus provides a solid foundation for the next generation of LLM serving systems, tuned for agentic, high-context, real-world utility (2606.30560).