ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents

Abstract: Agentic LLMs repeatedly encode tool and skill schemas that recur across requests in different combinations and orders, preventing standard prefix caching from reusing their key--value (KV) states. We introduce \textbf{ReCache}, a framework for independently caching resource representations while reducing their inference-time computational and memory overhead. Resource-wise attention removes cross-resource interactions and assigns resource-local positions, producing composition-invariant KV blocks. ReCache then restricts resource visibility to contribution-selected layer--KV-head-group routes and retains only invocation-critical fields through structural and semantic pruning. We evaluate ReCache on a benchmark assembled from seven public tool- and skill-use datasets, including resource-disjoint tests. Resource-wise attention matches dense invocation performance (82.3\% versus 82.4\% Inv-F1) while providing a 3.655$\times$ time-to-first-token speedup. The complete framework reduces allocated KV-tensor memory by 92.43\% and accelerates attention by 1.423$\times$. These results show that separating reusable schema encoding from selective resource access substantially reduces agentic inference costs with limited effectiveness loss. The code is available at https://github.com/EIT-NLP/ReCache.

• The paper introduces composition-invariant KV blocks with resource-wise attention, structural routing, and field-aware semantic pruning to reuse tool and skill schemas across requests.
• The method matches dense attention within 0.2% on invocation F1 while achieving a 3.655× time-to-first-token speedup, demonstrating that cross-resource attention can be removed after fine-tuning.
• The full framework reduces allocated KV memory by 92.43% and delivers a 1.423× attention speedup, while retaining 97.5% of dense in-distribution and 91.8% out-of-distribution invocation F1.

ReCache addresses a specific inefficiency in tool- and skill-augmented LLM agents: retrieved resource schemas (tools and skills) recur across requests in varying combinations and orders, so standard prefix caching cannot reuse their key–value (KV) states, forcing repeated prefill computation for unchanged content. The paper proposes a framework that encodes each resource as an independently reusable, composition-invariant KV block, then compresses those blocks along structural and semantic dimensions. Evaluation on a unified benchmark built from seven public datasets shows that the reusable encoding matches dense attention within 0.2% on invocation F1 while delivering a 3.655× time-to-first-token (TTFT) speedup, and that the full framework reduces allocated KV-tensor memory by 92.43% with a 1.423× attention speedup (2608.19662).

Motivation and design rationale

The central observation is empirical: attention analysis of Qwen3-4B shows substantially stronger attention within individual resource schemas than across resources, and subsequent conversation tokens concentrate their attention mass on each resource's final suffix token. This supports two hypotheses that ReCache builds on: first, that cross-resource interactions contribute little to resource invocation; second, that global resource ordering carries limited positional information for this task, consistent with prior findings that positional signals are task-dependent (2608.19662). The framework therefore departs from position-independent caching methods such as EPIC, CacheBlend, and KVLink, which preserve or correct original positional encodings and often require selective recomputation; instead, ReCache removes cross-resource attention entirely and re-indexes positions locally, making reusability exact rather than approximate.

Method

Resource-wise attention restricts tokens within each resource $R_i$ to attend only to the shared prefix and their own resource tokens, reducing resource–resource attention from quadratic in total resource length to a sum of per-resource quadratics. Each resource receives identical local positional indices ($pos(t_{i,j})=j$), so its KV block is invariant to retrieval composition and ordering. Lightweight fine-tuning adapts the model to this modified attention and positional layout — a necessary step, since the paper does not claim the representation works out-of-the-box on frozen models.

Structural pruning exploits redundancy across layers and grouped-query-attention (GQA) KV head groups. A routing configuration $\Omega \subseteq [L]\times[G]$ determines which layer–head-group routes expose resource KV states to conversational context; all other routes see only non-resource tokens. Route selection uses leave-one-in contribution scoring: starting from $\Omega=\emptyset$, the reduction in invocation loss from activating one layer or one head group yields normalized contribution weights, and top-$K$ selection over both dimensions produces $\Omega^\star$. The objective is restricted to invocation-span token positions so that unrelated response tokens do not dominate the score. The authors note that leave-one-out scoring produced small loss differences between units and that attention-mass ranking underperformed contribution-based selection, which they attribute to heterogeneous sensitivity across structural dimensions and token positions.

Semantic pruning operates at the field level. Because invocation failures are predominantly caused by invalid names and incorrect arguments, ReCache retains three fields per resource — resource name, argument names, and argument descriptions — plus the final suffix token as a naturally occurring summary state whose causal hidden state aggregates preceding fields. Notably, resource descriptions are dropped: ablations show that adding them changes metrics by less than 0.7% once argument descriptions are preserved. This field-aware criterion distinguishes ReCache from generic text-compression baselines such as LLMLingua-style filtering or learned summary tokens, which are not designed to preserve exact interface identifiers.

Benchmark construction

A substantial methodological contribution is the unified benchmark assembled from ToolACE, APIGEN, ToolMind, ToolRet, Toucan, SkillRouter, and WildToolBench. The authors document severe confounds in direct aggregation: 77.3% of distinct resource names recur across records (reaching 99.96–100% in APIGEN, Toucan, and ToolMind), and 18.8% of candidate records duplicate an exact previously seen candidate subset. Random downsampling would preserve shortcuts based on memorized resource identities, so they apply a two-stage diversity-first sampler that prioritizes unseen called resources and candidate configurations, followed by validity filtering that removes 6.7% of sampled records. The final training set contains 49,424 examples, with matched 1,000-example in-distribution (IND) and resource-disjoint out-of-distribution (OOD) test splits. Effectiveness is measured by turn-averaged invocation F1 (Inv-F1), where a prediction counts as correct only if the resource name and all arguments exactly match gold invocations, alongside resource-ID precision/recall/F1 and hallucination rate.

Structural budget analysis

Leave-one-in profiling reveals highly concentrated layer contributions: retaining 20 layers captures 97.7% cumulative contribution coverage (CC) for Qwen3-4B and 99.8% for Qwen3-1.7B. Head-group behavior differs sharply by scale. For Qwen3-4B, contributions are relatively flat, yet three of eight head groups (covering only 47.3% CC) retain near-full performance, indicating distributed, substitutable resource-related computation. Qwen3-1.7B instead requires seven groups covering 99.3% CC. Appendix experiments add two caveats: head-group sparsification is far more sensitive to limited supervision than layer sparsification (under the 5K subset, six-group retention cost Qwen3-1.7B 18.7% Inv-F1, while 20-layer retention changed it by less than 0.2%), and an auxiliary invocation-oriented loss partially mitigates this degradation. The viable structural budget thus depends jointly on retained route capacity, optimization signal, and model scale — a dependency the paper states explicitly rather than treating budgets as fixed hyperparameters.

Results

Resource-wise attention alone matches dense prefill: Inv-F1 of 82.3% versus 82.4%, ID-F1 identical at 96.0%, with TTFT reduced from 26.319 ms to 7.200 ms (3.655×). This confirms that cross-resource interactions and global ordering can be removed without effectiveness loss when the model is fine-tuned accordingly.

Under matched structural budgets, contribution-based routing consistently outperforms alternatives:

Method	Attn. speedup	Mem. reduction	Inv-F1 (IND)	Inv-F1 (OOD)	Halluc. (OOD)
Dense	–	–	82.4	66.3	0.0
$\Omega_{20,G}$ (contribution)	1.016×	44.71%	82.5	64.3	0.5
$\Omega_{\text{full}}$ + SPEED	1.013×	44.71%	79.3	54.3	13.4
$\Omega_{20,3}$ (contribution)	1.314×	79.27%	82.1	63.2	0.5
Attention-mass $\mathrm{SA}_{20,3}$	1.302×	79.27%	79.1	58.2	5.1
Gist	1.029×	99.22%	39.2	9.7	79.9
Beacon	1.020×	75.42%	78.6	58.5	4.2
Semantic Pruning only	1.021×	63.57%	81.6	62.8	0.3
ReCache (full)	1.423×	92.43%	80.3	60.8	0.6

At equal efficiency, contribution-based selection improves Inv-F1 by 3.2 points over SPEED in-distribution and 10.0 points OOD, and reduces OOD hallucination from 13.4% to 0.5%. Against attention-mass ranking at the same budget, it gains 3.0 and 5.0 points respectively. These gaps widen under distribution shift, indicating that contribution-guided routing generalizes better to unseen resources than depth-asymmetric visibility or attention-mass heuristics.

Semantic baselines expose a sharper contrast. Gist achieves the largest standalone memory reduction but collapses to 39.2% Inv-F1 in-distribution and 9.7% OOD with 79.9% hallucination, showing that a single summary vector cannot carry fine-grained interface information. Beacon performs better but still trails field-aware semantic pruning, particularly OOD, implying that partial coverage of critical fields is insufficient for robust generalization.

The full ReCache configuration retains 97.5% of dense Inv-F1 in-distribution (80.3 vs. 82.4) and 91.8% OOD (60.8 vs. 66.3). Efficiency scaling across resource-length buckets shows Dense TTFT growing to over 5,000 ms for contexts above 10K tokens, while ReCache holds TTFT slightly above 5 ms, TPOT slightly above 6 ms, attention latency below 0.2 ms, and allocated KV memory capped at 0.03 GiB versus roughly 8 GiB for dense inference. The near-flat latency profile across scales is the practically significant outcome: agentic workloads with large retrieved-context windows incur essentially constant serving cost under cache hits.

Limitations and open questions

The paper concedes several constraints directly. End-to-end evaluation covers only Qwen3 backbones (primarily 4B), because full-parameter fine-tuning of larger models exceeded available compute; whether the observed scale-dependent head-group budgets extrapolate to larger models remains open. Because ReCache modifies attention masks and positional layouts, the study is confined to trainable settings, and adaptation to frozen models is unaddressed. The framework also assumes relatively stable system instructions and schemas; environments where cross-resource dependencies or retrieval order carry essential information fall outside its design assumptions, and extending reuse to such settings is left unresolved. Finally, TTFT figures exclude offline cache-construction cost, so end-to-end amortization depends on sufficient cache-hit rates across requests.

Conclusion

ReCache demonstrates that resource-level decomposition of agentic context — composition-invariant KV blocks built by resource-wise attention, pruned by contribution-scored structural routing and field-aware semantic retention — can eliminate repeated schema encoding with minimal invocation degradation. The strongest quantitative claims are the 3.655× TTFT speedup at parity with dense effectiveness, the 92.43% memory reduction and 1.423× attention speedup at 97.5%/91.8% of dense Inv-F1 on IND/OOD splits, and the consistent superiority of contribution-based routing over attention-mass and layer-asymmetric baselines, especially on unseen resources. The accompanying diversity-first benchmark with resource-disjoint evaluation provides a reusable substrate for studying efficiency–effectiveness trade-offs in tool-augmented inference.