Stateful Inference for Low-Latency Multi-Agent Tool Calling

Abstract: Multi-agent tool calling is becoming the dominant interaction pattern for LLM-based systems, yet existing inference frameworks treat each tool call as an independent request, re-processing the entire conversation from scratch even though 85-95% of the prompt is unchanged from the previous turn. We present a stateful inference architecture that converts the $O(n_t)$ per-turn cost of conventional serving into an $O(Δ_t)$ delta-only cost: a persistent KV cache lives across turns and advances by ingesting only the new tokens, while a radix prefix cache extends this across interleaved multi-agent traffic and a prompt-lookup speculative decoder accelerates structured output. Against vLLM and SGLang on novel, fully-generated workloads, the reference implementation is $2.1\times$ faster per turn on a 6-turn agentic workflow and $4.2\times$ on the median turn of a 35-turn one, halving end-to-end wall time. The advantage comes from stateful reuse and speculation, not caching.

• The paper reduces per-turn inference cost from O(n_t) to O(Δ_t) by leveraging prefix caching and delta-only computation, achieving up to 4.2× faster performance on deep multi-turn workflows.
• The approach combines nine integrated mechanisms—including a persistent KV cache, radix prefix cache, and prompt-deterministic response cache—to optimize computation reuse in multi-agent sessions.
• The architecture improves hardware utilization and halves wall time for long-running sessions, though it faces challenges like GPU memory overhead and current single-GPU limitations.

Stateful Inference for Low-Latency Multi-Agent Tool Calling

Motivation and Problem Statement

Modern LLM-based systems increasingly orchestrate multi-agent, multi-turn tool calling workflows. Each agent, such as a travel planner or code reviewer, maintains a dynamic conversation history punctuated by structured tool calls and tool results, resulting in prompts composed of a monotonically-growing prefix with small incremental updates per turn. Production inference engines like vLLM and SGLang process each tool call as a fresh request, re-encoding and forwarding the entire conversation history even when most tokens are unchanged, leading to significant wasted computation and GPU underutilization.

The dominant property exploited by this work is that in agentic workloads, the vast majority of each prompt—typically 85–95%—is unchanged from the previous turn. Re-processing this invariant prefix for every tool call results in $\mathcal{O}(n_t)$ per-turn cost, with $n_t$ being the current prompt length and $\Delta_t$ the number of new tokens. The work aims to re-architect inference to align cost with the genuinely new computation, making it $\mathcal{O}(\Delta_t)$ per turn.

Architectural Contributions

This paper proposes a fully stateful inference architecture with nine integrated systems-level mechanisms, designed to persist and share computation across agent turns and concurrent multi-agent sessions, converting full-prefix recomputation to constant-time metadata operations and delta-only computation.

Key architectural elements include:

1. Stateful Sessions via Persistent KV Cache: Each agent maintains a long-lived KV cache, advanced with only the new tokens per turn.
2. Unified Sequence Pool: A pre-allocated pool of sequence slots under a single inference context, partitioned into transient and session regions, enabling lock-free, high-throughput acquisition and release.
3. Radix Prefix Cache: A radix trie over token prefixes, with sequence aliasing enabling constant-time reuse of long prefixes by pointing new requests' state to previously computed cache cells.
4. Prompt-Deterministic Response Cache: Exact prompt repeats (such as retries or polling) trigger cache hits based on FNV-1a prompt hashing, allowing zero-GPU network-latency responses.
5. Prompt-Lookup Speculative Decoding (PLD): Exploits repetitive, structured output by using n-gram suffix matches in prompt or prior output to generate token candidates speculatively, validated in batched passes, massively reducing sequential decode steps on structured tool-call formats.
6. Concurrency-Capped Prefill Scheduler: Cell-budgeted, continuous batching incorporates prefix-aware grouped prefill, collapsing redundant work for requests sharing schema/tool preambles.
7. Streaming Tool-Call Validator: Real-time brace-balanced JSON parsing and tool whitelist filtering enables streaming early-stop and rejection of malformed tool calls for efficient structured output termination.
8. On-Device Greedy Sampling: Device-specific argmax runs directly on accelerator, avoiding host round trips even for short structured outputs.
9. Robust Session Lifecycle Management: Exception-safe RAII guards for sequence acquisition and release, robust under multithreaded orchestration.

Theoretical Analysis and Efficiency

The core theoretical advance is converting per-turn inference cost from

$$\mathcal{O}(n_t) \quad \text{(full prefill per turn)}$$

to

$$\mathcal{O}(\Delta_t) \quad \text{(delta-only prefill, via prefix aliasing)}$$

where typically, $\Delta_t \ll n_t$. The speedup on turn $t$ rapidly approaches $n_t/\Delta_t$, especially as conversations deepen.

The radix prefix cache, in combination with grouped prefill and speculative decoding, ensures that the majority of GPU-side cost is strictly proportional to the true information-theoretic work done—processing new conversational tokens and emitting new tool calls—rather than the redundant recomputation of static prefix context.

Empirical Results

Latency and Wall Time

On realistic multi-agent tool-call workloads, including both shallow multi-turn (6-turn) and deep (35-turn) conversation scenarios, the reference LayerScale implementation is $2.1\times$ faster per turn than vLLM and SGLang in 6-turn workflows, with the advantage widening to $n_t$0 on deep, 35-turn conversations.

Figure 1: Median per-turn latency on novel, fully-generated tool-call workloads (no response-cache hits). LayerScale is $n_t$1 faster than vLLM on the 6-turn agentic workflow and $n_t$2 faster on the median turn of the 35-turn deep-coding workflow; the gap widens as the reusable conversation prefix deepens.

Further, the end-to-end wall time for the 35-turn workflow is halved, with LayerScale completing in $n_t$3 seconds versus $n_t$4 seconds for competitors, reflecting the compounded savings as conversation prefixes grow.

Figure 2: LayerScale speedup over vLLM and SGLang on median per-turn latency, without any kind of caching. The multiplier roughly doubles as workflow depth increases, reflecting the gap between $n_t$5 and $n_t$6 per-turn cost.

Figure 3: Cumulative wall time over the 35-turn novel coding workflow (no response-cache hits). LayerScale's compounding advantage over vLLM and SGLang is due to processing only the per-turn delta versus full prefix each turn.

Additional Observations

• Exact Repeat Traffic: The prompt-deterministic response cache serves idempotent or retry requests at network-roundtrip latency, a capability not present in vLLM or SGLang.
• Burst-Mode Orchestration: In scenarios of concurrent tool calls, the architecture delivers order-of-magnitude better p99 tail latencies by sharing KV state through metadata aliasing, preserving efficiency even under high concurrency.
• Correctness: All engines output correct tool calls under the benchmark; the advantage is strictly in efficiency and latency, not model output quality.

Practical Implications and Limitations

This stateful inference stack is a strong architectural fit for agentic systems, where conversation structure is prefix-extending, and workloads are dominated by structured tool-call output. The reduction in redundant computation translates directly into improved hardware utilization and a decrease in effective cluster FLOPs per agent interaction, addressing a persistent utilization gap in GPU inference clusters.

Limitations include GPU memory overhead scaling linearly with conversation depth and agent concurrency, cache invalidation constraints (the system relies on strictly monotonically extended prefixes), and restriction to single-GPU contexts in the present implementation; distributed, multi-GPU coordination is reserved for future work.

Theoretical and Future Directions

This work complements, rather than replaces, the throughput-oriented advances of systems like vLLM and SGLang, addressing low-latency, multi-agent, real-time structured inferencing scenarios. While the radix prefix cache and speculative decoding directly target redundant per-request computation, integration with cross-GPU distributed KV cache management, as well as more general cache sharing and invalidation strategies for non-monotonic scenarios, remain open.

Given the prevalence of agentic, tool-oriented orchestration in deployed AI applications, further study of such prefix-delta structures and their optimization in distributed inference is likely to be an active systems research direction.

Conclusion

This architecture systematically reduces the per-turn cost of multi-agent tool calling from linear in conversation length to linear in the per-turn delta, driven by a stateful, sequence-pooled, and prefix-cached inference kernel with speculative, cache-aware decoding. On novel, non-repetitive agentic workloads, it empirically demonstrates $n_t$7–$n_t$8 latency improvement over production open-source baselines, and completes long workflows in nearly half the wall time. The approach is immediately practical for high-traffic deployment scenarios and suggests new directions for distributed, cache-centric LLM inference optimization.