KVFlow: Workflow-Aware Key-Value Caching

• KVFlow is a workflow-aware key-value caching framework that leverages an Agent Step Graph to predict fixed prompt reuse in multi-agent LLM workflows.
• It employs a tree-structured eviction policy and proactive CPU-to-GPU prefetching to mitigate recomputation and data transfer overhead.
• Empirical evaluations show speedups up to 2.91× over traditional LRU methods, demonstrating its effectiveness under high memory pressure and long fixed prompts.

Searching arXiv for KVFlow and closely related cache-management papers. KVFlow is a workflow-aware key-value (KV) cache management framework for accelerating LLM-based multi-agent workflows. It targets the specific serving regime in which specialized agents are invoked repeatedly with long fixed prompt prefixes and shorter dynamic suffixes, making prefix KV reuse a dominant systems concern. Rather than relying on generic recency heuristics, KVFlow models agent execution structure explicitly through an Agent Step Graph, derives a steps-to-execution estimate for each agent, applies those estimates to fine-grained eviction in a tree-structured prefix cache, and proactively prefetches CPU-backed KV tensors to GPU in a fully overlapped manner (Pan et al., 10 Jul 2025).

1. Problem setting and motivation

KVFlow is designed for LLM-based agentic workflows in which multiple specialized agents cooperate through a structured execution pattern. In this setting, each LLM call contains a fixed prompt prefix—such as agent identity, role, instructions, behavior, few-shot examples, and task description—and a dynamic suffix containing current user input, intermediate state, or messages from other agents. Because the fixed portion is stable across repeated invocations, its KV tensors are highly reusable (Pan et al., 10 Jul 2025).

The framework addresses a serving bottleneck created by prefix caching under constrained GPU memory. Existing systems can reuse cached KV tensors for matching prefixes, but they typically manage cache residency with a Least Recently Used policy. KVFlow argues that this is a poor fit for multi-agent workflows because recency does not predict future reuse well when execution follows a structured schedule. An agent that has not been invoked recently may be the next agent to run, while a recently executed agent may not be needed again soon. The resulting cache misses cause either redundant recomputation or CPU-to-GPU swapping overhead, and the problem becomes more pronounced when fixed prompts are long (Pan et al., 10 Jul 2025).

A further complication is that modern prefix caches are tree- or radix-structured, so different agents may share common prompt prefixes. In that setting, cache management is not only a question of whether an agent’s prefix should be retained, but also which internal cache nodes should remain resident so that shared subprefixes are preserved efficiently (Pan et al., 10 Jul 2025).

2. Agent Step Graph and the steps-to-execution metric

KVFlow abstracts workflow structure as an Agent Step Graph in which each node corresponds to an agent invocation and edges encode dependency relations. Each node is associated with a step aggregation function that determines its steps-to-execution value from its predecessors. The framework focuses on the earliest possible execution step of each agent, rather than on a fully general workflow semantics (Pan et al., 10 Jul 2025).

The paper illustrates two aggregation patterns. If an agent such as Expresser requires both Executor1 and Executor2, then its step value is computed as

$\text{Expresser} = \max(E1, E2) + 1$

because execution cannot occur until both predecessors have completed. If Expresser can run after either predecessor, then the paper uses

$\text{Expresser} = \min(E1, E2) + 1$

because the earliest valid activation follows the first qualifying branch (Pan et al., 10 Jul 2025).

This metric is the central predictive signal in KVFlow. A smaller steps-to-execution value means that an agent is temporally closer to reuse and its fixed-prompt KV should be retained more aggressively. A larger value indicates that reuse is farther away and the corresponding cache entries are better eviction candidates. The paper’s key claim is that this future-oriented measure is better aligned with actual reuse value than LRU’s backward-looking recency heuristic (Pan et al., 10 Jul 2025).

3. Eviction policy and shared-prefix handling

KVFlow performs eviction at the level of KV nodes in the tree-structured prefix cache rather than at the level of whole-agent cache entries. This matters because different agents may share some prefix nodes but diverge later, and node-level control allows the system to preserve reusable shared structure while discarding less valuable descendants (Pan et al., 10 Jul 2025).

The eviction logic begins by distinguishing fixed prompt KV from varying suffix KV. All varying suffixes are always given the highest eviction priority, so dynamic tails are removed first when memory becomes scarce. For each agent, its steps-to-execution value is assigned to the last node of its fixed prompt, and that value is then propagated upward through the cache tree. When a node aggregates inputs from multiple agents, KVFlow assigns it the minimum priority among its children. The effect is conservative: a shared node is preserved as long as at least one descendant agent is expected to run soon (Pan et al., 10 Jul 2025).

This policy naturally extends to multiple concurrent workflows. If shared nodes participate in several workflows, KVFlow resolves conflicts by choosing the lowest, most conservative priority across workflows. The intent is to preserve entries that are likely to be reused soon anywhere in the active workload, rather than optimizing per agent in isolation (Pan et al., 10 Jul 2025).

The design is therefore best understood not as a token-importance method, but as a workflow-aware prefix-cache residency policy. Its contribution lies in changing how a shared radix-tree cache is prioritized under agentic execution structure.

4. Fully overlapped prefetching and status-aware scheduling

KVFlow complements eviction control with proactive KV prefetching. When fixed-prompt KV has been evicted from GPU but retained in CPU memory, the system prefetches likely next-needed tensors from CPU to GPU while the current agent is still executing. The paper’s motivating example is that while Planner runs, KVFlow can already load Executor1’s fixed-prompt KV in the background if Executor1 is predicted to be next (Pan et al., 10 Jul 2025).

The framework argues that this overlap is feasible because current-agent execution is dominated by GPU model forward and output transfer to CPU, whereas prefix reload is a CPU-to-GPU transfer. Since PCIe supports full-duplex transfer, KV loading can proceed concurrently with ongoing generation. When branching is possible, KVFlow conservatively prefetches all agents that may execute next, subject to a bound on concurrent prefetches (Pan et al., 10 Jul 2025).

Prefetching alone is not sufficient when transfer time exceeds the execution time of the current agent, so KVFlow adds status-aware scheduling. Each cache node is tracked in one of four states: in GPU memory, backup in CPU, loading, or offloading. If a request depends on nodes that are still loading, the scheduler temporarily skips it and dispatches other ready requests instead. Once the background load thread completes, the node state is updated and the request becomes runnable. This mechanism is intended to eliminate cache-miss stalls during generation by ensuring that only ready requests enter execution (Pan et al., 10 Jul 2025).

The result is a pipeline in which cache misses are often converted from blocking events into hidden background transfers, with the scheduler exploiting whatever concurrency is available across workflows.

5. System design and implementation

KVFlow is implemented as a prototype on top of SGLang v0.4.4 and extends SGLang’s radix-tree prefix cache with workflow-aware eviction and fully overlapped KV prefetching (Pan et al., 10 Jul 2025).

The implementation uses a two-tier memory hierarchy: GPU memory is the active KV cache, and CPU memory serves as a secondary backup layer for evicted fixed-prompt KV. Frontend and backend are both modified so that workflow metadata accompanies each request. The implementation assumes that each sgl.function corresponds to an independent agent. During execution, the frontend transmits the current agent identity, the steps-to-execution of all agents in the Agent Step Graph, and information about which agents may be invoked in subsequent steps. The backend uses this metadata to update cache priorities and trigger prefetching when enough evictable GPU memory is available (Pan et al., 10 Jul 2025).

Because KVFlow treats fixed and dynamic prompt regions differently, it must determine where the fixed prefix ends. The paper proposes two mechanisms: an explicit primitive interface through which the user marks the boundary, and a heuristic method that tracks cache-hit history and treats the consistently hit prefix as the fixed portion. To avoid collisions between different applications that use the same agent names, the implementation also adds a unique client ID to each request (Pan et al., 10 Jul 2025).

Although the prototype is built on SGLang, the paper states that the design is not limited to that stack. A plausible implication is that the same control logic can be adapted to other workflow frameworks provided they can transmit the same workflow metadata to the serving backend.

6. Empirical performance

The evaluation compares KVFlow against two SGLang configurations: GPU-only SGLang with radix cache, and SGLang with HiCache, a CPU-backed hierarchical radix cache. Experiments cover a synthetic 10-agent sequential workflow, a high-concurrency setting with many simultaneous workflows, and a more realistic PEER-based workflow with four agents (Pan et al., 10 Jul 2025).

Scenario	Reported comparison	Result
Single workflow, A10G, fixed/dynamic/output = 8192/32/32	KVFlow vs SGLang w/ HiCache	$1.83\times$
Single workflow, A10G, fixed/dynamic/output = 8192/32/32	KVFlow vs GPU-only SGLang	$2.91\times$
Many concurrent workflows, H100	KVFlow vs naive LRU-based HiCache with reactive loading	up to $2.19\times$
PEER-style workflow	KVFlow vs SGLang	up to $1.12\times$
PEER-style workflow	KVFlow vs HiCache	up to $1.08\times$

The reported gains are largest when fixed prompts are long and memory pressure is significant, because cache misses then incur substantial recomputation or transfer cost. The paper also notes that average speedup increases from $1.28\times$ at fixed prompt length 4096 to $1.48\times$ at fixed prompt length 8192, which is consistent with the framework’s emphasis on fixed-prefix reuse (Pan et al., 10 Jul 2025).

Under high concurrency, KVFlow also outperforms both SGLang and HiCache, and the paper highlights that naive CPU-backed caching can even underperform GPU-only SGLang when loading is reactive. As output length grows, however, KVFlow’s relative benefit decreases because autoregressive decode increasingly dominates total latency. Likewise, the gains are more modest for PEER-style prompts because those prompts are shorter and cache-miss overhead is less severe (Pan et al., 10 Jul 2025).

7. Relation to adjacent work, scope, and limitations

KVFlow is specific to workflow-aware prefix caching for LLM-based multi-agent systems and should not be conflated with similarly named methods in other KV-cache subareas. In particular, it is distinct from FlowKV, which addresses multi-turn conversational coherence by isolating previously compressed KV cache segments in dialogue inference rather than managing prefix caching across agents (Liu et al., 21 May 2025).

A later system, PBKV, positions KVFlow as the state-of-the-art workflow-aware baseline for static workflows and characterizes it as assuming a predefined static Agent Step Graph whose steps-to-execution values are known in advance. PBKV argues that this assumption is too strong for realistic dynamic workflows with runtime-dependent loops and branching, where steps-to-execution can become undefined, and reports up to $1.26\times$ speedup over KVFlow on a static workflow while targeting dynamic settings beyond KVFlow’s scope (Zheng et al., 7 May 2026). This does not negate KVFlow’s contribution; rather, it clarifies that KVFlow is optimized for workflows whose near-future structure can be expressed statically.

The KVFlow paper itself also delineates conditions under which its benefits are smaller. Gains diminish when prompts are short, when output generation dominates latency, when cache pressure is weak, or when concurrency is so high that reusable prefix caches can no longer be maintained. The prototype additionally inherits storage-layout limitations from SGLang, and the evaluation notes that fragmented KV layout may prevent full utilization of PCIe bandwidth (Pan et al., 10 Jul 2025).

Taken together, KVFlow established workflow structure as a first-class signal for KV residency decisions in multi-agent LLM serving. Its central technical idea is that future agent activation, not recent access alone, should govern prefix-cache eviction and prefetching. That idea remains the framework’s defining contribution within the broader literature on LLM cache management (Pan et al., 10 Jul 2025).