Query-State Caching Strategies

• Query-state caching is a strategy that reuses query-dependent state, such as intermediate results and runtime contexts, to eliminate redundant computations.
• It integrates tightly with system optimizers by reusing shared pipeline components and data artifacts, thereby enhancing latency, I/O efficiency, and throughput.
• Implementations range from exact-result caching in SPARQL endpoints to similarity-aware and learned caches, with key challenges in defining state identity and ensuring correctness.

Query-state caching denotes the reuse of query-dependent state that would otherwise be recomputed on later requests. Across the literature, the cached object ranges from final or intermediate query results in data warehouses, to shared pipeline prefixes and transformer outputs in information retrieval, to phase-specific pages in disk-based approximate nearest neighbor search, to keyed operator state in stream processors, to tool-call trajectories with optional sandbox snapshots in LLM-agent post-training, and to successful question-plan-answer episodes in knowledge-graph question answering [0003005, (MacAvaney et al., 14 Apr 2025, Zhou et al., 21 Aug 2025, Zapridou et al., 20 Mar 2026, Kumar et al., 11 Feb 2026, Sun et al., 28 Apr 2026)]. The unifying theme is reduction of redundant computation under repeated, overlapping, or state-correlated workloads; the main technical differences concern what counts as reusable state, how identity is defined, and how correctness is preserved.

1. Scope and relation to result caching

Query-state caching is broader than exact-result caching, but the boundary is important. In the SPARQL endpoint optimizer of “A Cache-based Optimizer for Querying Enhanced Knowledge Bases” (Zhang et al., 2018), the system caches complete (query, result) pairs, predicts likely future full queries from structural similarity, and reuses a cached answer only when a later query exactly matches a cached query; it does not cache query execution states, iterator states, algebra plans, or intermediate subquery/BGP results. In web search, the “Static-Topic-Dynamic cache” keeps exact query results while partitioning cache state by topic, so that non-static queries compete within topic-specific sections rather than only inside one global recency queue (Mele et al., 2020).

Other systems move beyond exact answers. Exchequer caches “final/intermediate results of previous queries” and makes the optimizer responsible for cache decisions rather than separating cache management from optimization [0003005]. In PyTerrier, reusable state includes outputs of shared pipeline prefixes, query rewrites, document rewrites, query-document scores, retriever outputs, and indexing-time transformed streams (MacAvaney et al., 14 Apr 2025). SQLcached pushes the boundary in a different direction by treating cached state as an in-memory relational dataset queried and expired with SQL, with explicit support for per-user, per-page, and per-application scopes (0910.0187).

This suggests that “query state” is best understood as any persistent artifact whose semantics are tied to a query’s logical intent, execution context, or induced data dependency structure, and whose reuse can eliminate later work.

2. State identity, equivalence, and correctness

The central correctness problem is state identity: determining when previously computed state is still valid for a new request. PyTerrier makes this issue explicit. The deprecated unary cache operator ~ was judged too generic because different transformers require different key semantics. For a retriever, caching only on qid may be wrong if query text changes through rewriting, so a safer identity is $\langle qid, query \rangle$. For a cross-encoder reranker, $\langle qid, docno \rangle$ may be insufficient when query or document text changes; the paper recommends the safer key $\langle qid, docno, query, text \rangle$ (MacAvaney et al., 14 Apr 2025). The underlying point is that cache identity must match primary keys and functional dependencies of the operator’s input-output relation.

Stateful agent environments sharpen the same issue. TVCACHE maintains a per-task tool-call graph and declares a hit only when the agent’s full tool history up to and including the current tool call exactly matches a previously executed sequence; longest-prefix matching is then used to resume from a cached sandbox snapshot when the full sequence does not match (Kumar et al., 11 Feb 2026). Its correctness assumptions are explicit: tool output depends only on current sandbox state and tool arguments, and tools marked as state-preserving must not modify sandbox state.

Not all systems require exact identity. QVCache treats approximate nearest-neighbor search as a similarity-aware query-level cache. A hit occurs when the cache-produced $k$-th distance satisfies

$d_{\text{cache}[k]} \le (1 + D)\cdot \theta[k][R],$

where $\theta[k][R]$ is a learned threshold for region $R$ and result size $k$ (Göçer et al., 2 Feb 2026). “Continuous Semantic Caching for Low-Cost LLM Serving” formalizes a related continuous view: queries are embedded in $\mathcal{X}\subset \mathbb{R}^{d_e}$, the cache stores representative query-response pairs, and the expected loss under cache $\mathcal{M}$ is

$\langle qid, docno \rangle$0

(Atalar et al., 21 Apr 2026). These systems replace exact equality by region-wise or semantic equivalence, but only under explicit acceptance criteria.

3. Representative architectural patterns

The major architectural variants differ in the unit of state retained, the moment at which reuse is decided, and the control loop that updates the cache.

System	Cached state	Reuse mechanism
Exchequer [0003005]	Final and intermediate query results	Optimizer-aware cache maintenance with a cache-aware optimizer
PyTerrier / pyterrier-caching (MacAvaney et al., 14 Apr 2025)	Shared pipeline prefixes, row-wise transforms, scorer outputs, retriever outputs, indexing streams	Longest common prefix precomputation plus transformer-specific explicit caches
Distributed raw-array caching (Zhao et al., 2018)	Query-relevant raw-array chunks, chunk bounding boxes, evolving R-tree metadata	Online chunk refinement and placement that collocates dependent cells
GoVector (Zhou et al., 21 Aug 2025)	Static phase-1 nodes and dynamic phase-2 pages	Hybrid cache with phase transition detection and similarity-aware page loading
Keyed Prefetching + TAC (Zapridou et al., 20 Mar 2026)	Previously accessed and prefetched keyed operator state	Upstream hint extraction, asynchronous fetch, timestamp-aware eviction
TVCACHE (Kumar et al., 11 Feb 2026)	Tool result plus optional sandbox snapshot at each tool-call node	Exact longest-prefix matching on tool histories
CacheRAG (Sun et al., 28 Apr 2026)	Successful $\langle qid, docno \rangle$1 tuples	Hierarchical cache retrieval with MMR-selected exemplars
Turbo / PMW-Bypass (Kostopoulou et al., 2023)	Exact answers plus learned histogram state	Bypass-assisted PMW updates and privacy-free histogram answers once accurate

These patterns fall into several classes. One class caches intermediate execution artifacts directly, as in Exchequer and PyTerrier. A second class caches stateful runtime contexts, as in stream processing and tool-using agents. A third class caches learned surrogates of the workload or data distribution, such as PMW-Bypass histograms, QVCache’s regional thresholds and mini-indexes, or CacheRAG’s repository of successful query plans. A fourth class, illustrated by raw-array caching, stores query-relevant data regions together with metadata that makes later access and collocation cheaper.

4. Execution-time integration and adaptive control

A defining property of query-state caching is that it is usually integrated with optimization or execution control rather than bolted on as a passive memo table. Exchequer makes this explicit: “the optimizer takes the decisions for the cache manager,” in contrast to designs where a separate cache manager makes cost-benefit decisions independently of the optimizer [0003005]. This integration is presented as “optimization aware cache maintenance” plus “the use of a cache aware optimizer.”

PyTerrier adopts a similar compile/rewrite perspective for IR experimentation. Before pt.Experiment() executes a set of pipelines, the system can detect the longest common prefix shared by all pipelines, evaluate that prefix once on the query set, and then feed the intermediate result into each suffix pipeline. In formal terms, if $\langle qid, docno \rangle$2 is a set of pipelines, execution proceeds by computing $\langle qid, docno \rangle$3 once and then evaluating each remainder $\langle qid, docno \rangle$4 (MacAvaney et al., 14 Apr 2025). The same framework also offers explicit transformer-level caches, but the experimental optimizer remains responsible for prefix reuse.

In distributed raw-array querying, adaptation occurs at the granularity of data regions rather than operator plans. The framework first identifies cells to be cached locally from each input file by “continuously refining an evolving R-tree index,” then computes a placement that collocates dependent cells so as to minimize overall data transfer (Zhao et al., 2018). Both chunk granularity and placement therefore evolve from the workload.

GoVector introduces a two-phase execution model for disk-based graph ANN search. A static cache serves the rapid-convergence phase, while a dynamic cache is activated after a heuristic phase transition and batch-loads pages near the current expansion node under a similarity-aware disk layout; the experimental static-to-dynamic ratio is $\langle qid, docno \rangle$5, and LFU is the default dynamic-cache replacement policy (Zhou et al., 21 Aug 2025). The cached state is query-dependent because the second phase is driven by the evolving candidate queue and the current node’s position in vector space.

In stateful stream processing, Keyed Prefetching decouples the data path from state access. Upstream lookahead operators extract future access keys and emit hints $\langle qid, docno \rangle$6, while the downstream stateful operator’s Prefetching Manager asynchronously stages the corresponding keyed state before tuples arrive. Timestamp-Aware Caching then orders both previously accessed and prefetched state by event-time timestamps in a single cache (Zapridou et al., 20 Mar 2026). The system thus treats future access state as a first-class online control variable.

A closely related coupling appears in multi-worker LLM serving. “Randomization Boosts KV Caching, Learning Balances Query Load” models the service cost of routing query $\langle qid, docno \rangle$7 to worker $\langle qid, docno \rangle$8 as

$\langle qid, docno \rangle$9

with $\langle qid, docno, query, text \rangle$0 the number of cache-hit tokens (Wu et al., 26 Jan 2026). Routing changes which worker can reuse cached prefix state; eviction changes whether that state survives. The paper’s learning-based greedy routing and randomized leaf-token eviction are therefore designed jointly.

5. Empirical evidence across domains

Reported gains are substantial, but the metrics are domain-specific and not directly comparable. Some systems optimize latency, some optimize I/O, some conserve privacy budget, and some improve answer quality by reusing plan state rather than raw execution artifacts.

System	Representative reported outcome	Context
Exchequer [0003005]	Query response times lower by more than 30% than the best performing competitor; same response time with just one tenth of the cache size	Data warehouse / data mart queries
PyTerrier (MacAvaney et al., 14 Apr 2025)	MSMARCO v1: 3m11s baseline to 1m36s with hot ScorerCache; MSMARCO v2: 5m48s to 2m25s	Comparative IR pipelines
GoVector (Zhou et al., 21 Aug 2025)	At 90% recall, 46% average I/O reduction, 1.73x throughput increase, 42% lower latency	Disk-based graph ANNS
Keyed Prefetching (Zapridou et al., 20 Mar 2026)	$\langle qid, docno, query, text \rangle$1 to $\langle qid, docno, query, text \rangle$2 lower p999 latency than baselines	Stateful stream processing
QVCache (Göçer et al., 2 Feb 2026)	Sub-millisecond cache-hit latency and 40–1000x lower end-to-end latency	Query-level ANN caching
TVCACHE (Kumar et al., 11 Feb 2026)	Cache hit rates of up to 70% and median tool call execution time reduced by up to 6.9X	RL post-training with external tools
CacheRAG (Sun et al., 28 Apr 2026)	+13.2% accuracy and +17.5% truthfulness on CRAG	KGQA with cache-guided planning
Turbo (Kostopoulou et al., 2023)	1.7–15.9x more privacy budget conserved	Differentially private databases

The same pattern appears in adjacent systems. KVRouting reports improvements of up to $\langle qid, docno, query, text \rangle$3 in KV-cache hit rate, $\langle qid, docno, query, text \rangle$4 reduction in latency, $\langle qid, docno, query, text \rangle$5 reduction in TTFT, and 77.4% increase in throughput over state-of-the-art methods (Wu et al., 26 Jan 2026). In web search, topical result caching improves SDC by up to 3% in hit rate and up to 36% of gap reduction with respect to Belady’s optimal policy (Mele et al., 2020). In predictive SPARQL result caching, the proposed system reaches a 76.65% hit rate and 251 ms average query time on DBpedia 3.9, versus 72.63% and 264 ms for ASQC (Zhang et al., 2018). Even when the reused object is not a final result but a learned synopsis, the savings can be decisive: Turbo’s PMW-Bypass preserves future answerability “for free” from a privacy perspective once the learned histogram is accurate enough (Kostopoulou et al., 2023).

6. Limitations, tradeoffs, and common misconceptions

A common misconception is that all systems under the broad caching umbrella perform the same kind of reuse. They do not. Some systems remain exact-result caches even when they use sophisticated prediction or partitioning. The SPARQL optimizer uses structural similarity only to decide what full queries to prefetch; reuse itself is exact lookup over proactively materialized full results (Zhang et al., 2018). STD partitions query-result caching by topic, which preserves topic-conditioned temporal state but still operates on exact query results (Mele et al., 2020). GutenTag materializes popular multi-term posting lists and caches popular keys on gateway nodes, which is query-driven state reuse, but at the level of posting-list materialization rather than general intermediate execution state (Weth et al., 2011).

A second misconception is that richer state is always safely reusable. PyTerrier’s ScorerCache applies only when documents can be scored independently for a query and “does not generally apply to adaptive rerankers, pairwise rerankers, [or] listwise rerankers”; DuoT5 is explicitly “not amenable to caching in these pipelines” (MacAvaney et al., 14 Apr 2025). TVCACHE depends on exact sandbox-state equivalence and on correct will_mutate_state() annotations (Kumar et al., 11 Feb 2026). SQLcached, conversely, shows that richer state semantics can improve whole-system behavior through predicate retrieval and fine-grained expiry, yet it is “suboptimal” as a simple key-value store and does not beat memcached on plain exact-key lookup (0910.0187).

Approximate and learned caches introduce their own tradeoffs. QVCache explicitly does not provide hard worst-case recall guarantees; its hit rule depends on region-specific threshold learning and a user-controlled deviation factor $\langle qid, docno, query, text \rangle$6 (Göçer et al., 2 Feb 2026). Continuous semantic caching assumes an embedding metric, Lipschitz smoothness, and kernel generalization under partial feedback (Atalar et al., 21 Apr 2026). GoVector’s benefits depend on the second search phase dominating latency, on similarity-aware disk reordering, and on a heuristic phase-transition detector (Zhou et al., 21 Aug 2025). Turbo is restricted to linear query workloads, its learned histogram state scales with the domain, and it “does not” handle updates to past data well (Kostopoulou et al., 2023).

A plausible implication is that automation becomes more reliable when systems expose explicit metadata about identity, dependencies, and scope. The surveyed designs repeatedly require some notion of key richness, dependency structure, phase information, semantic region, or logical intent before cached state can be reused safely. Query-state caching, in that sense, is not a single mechanism but a family of execution strategies for preserving and exploiting those structures.