Cache Prior Reranking Techniques

• Cache prior reranking is a technique that leverages cached state—such as precomputed KV blocks, table metadata, or embeddings—to optimize reranking efficiency and reduce redundant computations.
• It spans diverse applications from Text-to-SQL to multimodal systems, where cached objects like visual prefixes and document embeddings guide the processing order and minimize computational overhead.
• Empirical results indicate significant latency reductions and throughput improvements, though benefits depend on cache reuse scenarios and inherent system constraints.

Searching arXiv for papers on cache-aware reranking and cache reuse to ground the article. {"query":"cache reranking KV cache reuse reranker arXiv TableCache miniReranker HyperRAG", "max_results": 10} Cache prior reranking denotes a family of reranking and scheduling techniques that exploit cached state, precomputed intermediate representations, or cache-aware workload structure to reduce redundant computation while preserving ranking or task performance. In current arXiv usage, the term spans several distinct but related settings: batch reordering of Text-to-SQL requests to maximize KV-cache overlap, multimodal reranking with reusable visual prefixes, decoder-only RAG rerankers with document-side KV reuse, IR pipeline execution with cached common prefixes and scorer outputs, post-hoc class reranking with nonparametric caches, and even popularity ranking for coded caching under unknown demand distributions (Su et al., 13 Jan 2026, Fan et al., 9 Jun 2026, An et al., 3 Apr 2025, MacAvaney et al., 14 Apr 2025, Orhan, 2018, Bahadori et al., 7 Mar 2026). Across these settings, the recurring principle is that reranking is not performed in isolation: it is conditioned by a reusable “prior” embodied in cache contents, cache locality, or precomputed summaries.

1. Core concept and scope

The unifying idea is to treat cached computation or cached evidence as a first-class signal in reranking. In TableCache, the “prior” is not probabilistic; it is an implicit cache-aware preference for processing adjacent queries with similar table sets so that GPU-resident KV blocks remain warm (Su et al., 13 Jan 2026). In miniReranker and HyperRAG, the prior is a reusable prefix segment—typically the heavy visual or document side of a query–document pair—whose per-layer K/V tensors can be precomputed and then reused across many pairs (Fan et al., 9 Jun 2026, An et al., 3 Apr 2025). In PyTerrier, the same principle appears at the pipeline level: common prefixes and previously computed scorer outputs are reused so that comparative experiments preserve end-to-end pipeline structure without recomputing shared stages (MacAvaney et al., 14 Apr 2025).

The term also appears in broader memory-augmented or precomputation-based forms. A simple cache model for image recognition mixes the classifier posterior with a similarity-weighted vote over cached training exemplars, thereby reranking class probabilities at test time without retraining (Orhan, 2018). “Reranking with Compressed Document Representation” replaces full document text with a fixed-size precomputed embedding representation, so reranking is conditioned on a stored document prior rather than raw document tokens (Déjean et al., 21 May 2025). In Proximity for RAG, the paper’s core implementation returns cached top-$k$ indices directly, while the detailed discussion presents cache prior reranking as a natural, strictly-compatible enhancement that uses similar prior queries and their retrieved sets as a prior over new candidates (Bergman et al., 7 Mar 2025).

Taken together, these works suggest that cache prior reranking is best understood as a systems-and-inference pattern rather than a single algorithm. The “prior” may be a table set, a prefix KV block, a compressed document representation, a query-neighborhood cache, a class exemplar memory, or a learned partial order over file popularities.

2. Formalizations of the cache prior

A characteristic formalization appears in TableCache. For each query $i$, Table Trie matching yields a set of table IDs $\tau_i \subseteq \{1,\dots,n\}$, represented by a binary indicator vector $I_i \in \{0,1\}^n$. The distance between two queries is the size of the symmetric difference,

$d_{i,j} = \sum_{k=1}^{n} (I_i \oplus I_j)[k],$

and batched reranking seeks a permutation $\pi$ that minimizes total table switches,

$$\min_{\pi} \sum_{t=2}^{N} \big|\tau_{\pi(t-1)} \,\Delta\, \tau_{\pi(t)}\big|,$$

equivalently maximizing adjacent table overlap. The implemented heuristic starts from a random anchor and repeatedly appends the remaining query with minimum $d_{i,j}$, with overall reranking cost $O(\frac{N^2 m}{64})$ under 64-bit bitset optimization (Su et al., 13 Jan 2026).

In multimodal point-wise reranking, miniReranker defines the relevance score from next-token logits: $s(q,d) = \sigma(z_{\text{yes}} - z_{\text{no}}),$ while noting that raw logit differences $i$0 are often used directly for ranking. Its cache prior is embodied in a vision-first prompt ordering that places the visual segment first so its K/V tensors can be reused across pairs. Cross-segment attention is then restricted to an intermediate layer band, and scoring can terminate at a fixed exit layer, making the reusable prefix both a computational prior and an architectural constraint (Fan et al., 9 Jun 2026).

In the image-recognition cache model, the prior is explicitly probabilistic. Cached keys are normalized features from non-final layers and cached values are one-hot class vectors. For a test image $i$1, the cache-induced distribution is

$i$2

with $i$3, and the final class posterior is a convex mixture,

$i$4

Top-$i$5 classification is then obtained by sorting classes by the mixed posterior (Orhan, 2018).

In deep search agents, the cache prior enters through cost modeling rather than the ranking score itself. “Rerank Before You Reason” defines effective token cost as

$i$6

where cached input tokens are discounted by $i$7 and generated tokens are weighted by $i$8. This formalism makes cache reuse analytically visible: reranking inputs are mostly non-cached candidate passages, whereas multi-turn search history contributes heavily to $i$9 (Sharifymoghaddam et al., 20 Jan 2026).

In coded caching, a different formalization appears. TopRank-based delivery optimization tracks pairwise request-count differences within current partitions, uses self-normalized confidence thresholds to infer partial orders over files, and then chooses the union of top partitions as the “popular group” for cache placement. Here reranking is applied to files rather than search results, but the mechanism is still a cache prior in the sense that the ordering controls what content is cached and therefore what future delivery rate is achievable (Bahadori et al., 7 Mar 2026).

3. Algorithmic mechanisms

The main mechanisms can be organized by what is cached and how the reranking or ordering decision uses it.

Setting	Cached object	Reranking mechanism
Text-to-SQL	Per-table KV blocks, Table Trie metadata	Reorder batched queries by minimum table-set symmetric difference
Multimodal reranking	Visual-prefix K/V and optionally FFN outputs	Vision-first prompting, band-limited cross-segment attention, early exit
RAG reranking	Document-side per-layer K/V	Reuse $\tau_i \subseteq \{1,\dots,n\}$0 prefixes across many queries
IR experiments	Common pipeline prefixes, scorer outputs, retrieval results	Reuse shared prefixes or cached scores instead of recomputation
Image recognition	Training-set feature keys and class labels	Mix base posterior with cache posterior and rerank classes
Compressed-document reranking	Fixed-size document embeddings $\tau_i \subseteq \{1,\dots,n\}$1	Score $\tau_i \subseteq \{1,\dots,n\}$2 instead of $\tau_i \subseteq \{1,\dots,n\}$3

In TableCache, the serving path is explicitly split into offline and online phases. Offline, the system builds a primary–foreign key graph, topologically sorts tables, jointly encodes PFK-linked tables, stores KV without positional encoding, and builds a Table Trie over tokenized table content. Online, the query is matched against the trie, the required table caches are loaded from CPU to GPU, and batch-level reranking is executed before inference so that adjacent queries share tables and incur fewer swap-ins and evictions (Su et al., 13 Jan 2026).

miniReranker decomposes the problem differently. It first decides a vision-first ordering: $\tau_i \subseteq \{1,\dots,n\}$4 for $\tau_i \subseteq \{1,\dots,n\}$5, $\tau_i \subseteq \{1,\dots,n\}$6 for $\tau_i \subseteq \{1,\dots,n\}$7, and $\tau_i \subseteq \{1,\dots,n\}$8 for $\tau_i \subseteq \{1,\dots,n\}$9. It then applies three further compression steps: an interaction band that enables cross-segment attention only in specific layers, fixed-depth early exit, and embedder-guided pruning that retains top-$I_i \in \{0,1\}^n$0 visual tokens, with $I_i \in \{0,1\}^n$1 recommended by default (Fan et al., 9 Jun 2026).

HyperRAG uses a decoder-only reranker with the reversed layout $I_i \in \{0,1\}^n$2. The document-side K/V tensors are precomputed once per chunk and stored in a KV store keyed by chunk ID. At inference time, a retriever yields top-$I_i \in \{0,1\}^n$3 chunk IDs, the corresponding document KV blocks are loaded into the reranker’s KV manager, and only query-side projections are recomputed. A tri-mask enforces doc$I_i \in \{0,1\}^n$4doc, query$I_i \in \{0,1\}^n$5doc, and query$I_i \in \{0,1\}^n$6past-query attention while prohibiting doc$I_i \in \{0,1\}^n$7query attention, so two-stage inference remains lossless relative to one-pass scoring (An et al., 3 Apr 2025).

PyTerrier exemplifies a non-neural but conceptually related mechanism. Its implicit caching computes the longest common prefix across pipelines in an experiment and executes that prefix once; its explicit caches include ScorerCache for reranker scores, RetrieverCache for $I_i \in \{0,1\}^n$8 outputs, KeyValueCache for row-wise mappings such as $I_i \in \{0,1\}^n$9 or $d_{i,j} = \sum_{k=1}^{n} (I_i \oplus I_j)[k],$0, and IndexerCache for indexing pipelines. Here the cache prior is operational rather than statistical: reranking experiments become cache-aware because repeated stages and repeated query–document pairs are memoized (MacAvaney et al., 14 Apr 2025).

4. Representative application domains

Text-to-SQL provides the most direct example of cache-aware query ordering. Prompts include large schema metadata, standard prefix caches in SGLang and vLLM require exact token-prefix matches, and varying table orders cause redundant cache copies. TableCache sidesteps entire-prefix matching by caching tables independently and concatenating their KV blocks at inference. The reranking objective directly targets GPU cache locality: “cache-near” queries are processed consecutively so that hot tables remain resident and TTFT is reduced (Su et al., 13 Jan 2026).

Multimodal reranking emphasizes that the most valuable prior is often the heaviest modality. miniReranker argues that query-first and document-first formats are each suboptimal in one of the $d_{i,j} = \sum_{k=1}^{n} (I_i \oplus I_j)[k],$1 or $d_{i,j} = \sum_{k=1}^{n} (I_i \oplus I_j)[k],$2 regimes, whereas vision-first always places the visual segment in the reusable prefix and preserves the model’s native VQA ordering. This converts the visual segment into a reusable cache prior across many query–document pairs and makes subsequent sparsification multiplicative rather than additive (Fan et al., 9 Jun 2026).

RAG reranking introduces a similar asymmetry, but on the text side. HyperRAG exploits the fact that retrieved document chunks are static across many queries, so document-side KV can be precomputed offline and reused at service time. The paper further aligns cache partitions with FAISS IVF centroids so that retrieval probes and KV loads exhibit shared locality (An et al., 3 Apr 2025). Proximity addresses an earlier stage in the same pipeline: instead of reranking documents after retrieval, it caches similar prior queries and their retrieved document sets. The detailed treatment presents cache prior reranking as a compatible extension in which cached query neighborhoods become a prior over candidates before optional reranking and generation (Bergman et al., 7 Mar 2025).

Experimental IR pipelines apply the concept at evaluation time. When several rerankers share the same retriever stage, prefix precomputation avoids repeated retrieval; when multiple retrievers surface overlapping documents, scorer caches avoid rescoring repeated query–document pairs. This preserves the declarative pipeline while avoiding the brittleness of intermediate run files (MacAvaney et al., 14 Apr 2025).

The same structural idea appears outside retrieval. In image recognition, a cache built from training exemplars supplies instance-specific evidence from near-output features that the final linear readout underutilizes, and class probabilities are reranked by a mixture of model and cache posteriors (Orhan, 2018). In coded caching, TopRank-style file ordering determines which files are placed into the “popular group,” directly affecting the coded-multicast delivery rate under non-uniform and initially unknown demands (Bahadori et al., 7 Mar 2026).

5. Empirical behavior and trade-offs

The clearest latency evidence comes from TableCache. Relative to the Transformers baseline, it achieves up to a $d_{i,j} = \sum_{k=1}^{n} (I_i \oplus I_j)[k],$3 reduction in Time to First Token. The reranking ablation isolates a substantial portion of this gain: for OmniSQL-7B on Spider dev, TTFT drops from $d_{i,j} = \sum_{k=1}^{n} (I_i \oplus I_j)[k],$4 s without query reranking to $d_{i,j} = \sum_{k=1}^{n} (I_i \oplus I_j)[k],$5 s with TableCache; on BIRD dev, it drops from $d_{i,j} = \sum_{k=1}^{n} (I_i \oplus I_j)[k],$6 s to $d_{i,j} = \sum_{k=1}^{n} (I_i \oplus I_j)[k],$7 s. The paper also reports that FIFO, LFU, and LRU eviction policies have only minor differences once reranking has concentrated hot tables (Su et al., 13 Jan 2026).

miniReranker reports multiplicative savings from cache reuse, interaction sparsity, token pruning, and early exit. On MMEB-v2, it preserves $d_{i,j} = \sum_{k=1}^{n} (I_i \oplus I_j)[k],$8 of dense performance for the 2B model ($d_{i,j} = \sum_{k=1}^{n} (I_i \oplus I_j)[k],$9 vs $\pi$0) and $\pi$1 for the 4B and 8B models ($\pi$2 vs $\pi$3; $\pi$4 vs $\pi$5). Under high-reuse settings for a single query, reranking runtime falls to $\pi$6 of the dense implementation, and training throughput is approximately $\pi$7 faster. Prompt ordering itself is consequential: vision-first scores $\pi$8 overall, versus $\pi$9 for query-first and $$\min_{\pi} \sum_{t=2}^{N} \big|\tau_{\pi(t-1)} \,\Delta\, \tau_{\pi(t)}\big|,$$0 for document-first (Fan et al., 9 Jun 2026).

HyperRAG reports system-level throughput improvements rather than isolated scoring latency. Passage-level speedups range from approximately $$\min_{\pi} \sum_{t=2}^{N} \big|\tau_{\pi(t-1)} \,\Delta\, \tau_{\pi(t)}\big|,$$1 to $$\min_{\pi} \sum_{t=2}^{N} \big|\tau_{\pi(t-1)} \,\Delta\, \tau_{\pi(t)}\big|,$$2, and document-level speedups from approximately $$\min_{\pi} \sum_{t=2}^{N} \big|\tau_{\pi(t-1)} \,\Delta\, \tau_{\pi(t)}\big|,$$3 to $$\min_{\pi} \sum_{t=2}^{N} \big|\tau_{\pi(t-1)} \,\Delta\, \tau_{\pi(t)}\big|,$$4, with the paper summarizing the overall effect as a $$\min_{\pi} \sum_{t=2}^{N} \big|\tau_{\pi(t-1)} \,\Delta\, \tau_{\pi(t)}\big|,$$5–$$\min_{\pi} \sum_{t=2}^{N} \big|\tau_{\pi(t-1)} \,\Delta\, \tau_{\pi(t)}\big|,$$6 throughput improvement while preserving the decoder reranker’s quality (An et al., 3 Apr 2025). “Reranking with Compressed Document Representation” reaches a different point on the same frontier: by replacing full text with cached fixed-size memory-token representations, RRK achieves approximately $$\min_{\pi} \sum_{t=2}^{N} \big|\tau_{\pi(t-1)} \,\Delta\, \tau_{\pi(t)}\big|,$$7 speed-up versus the same 7B decoder reranker on textual 512-token inputs while remaining within roughly $$\min_{\pi} \sum_{t=2}^{N} \big|\tau_{\pi(t-1)} \,\Delta\, \tau_{\pi(t)}\big|,$$8 nDCG@10 point of strong textual rerankers on average (Déjean et al., 21 May 2025).

Pipeline-level caching yields smaller but still material savings. On MSMARCO v1 and v2, PyTerrier’s precompute-prefix mode reduces total runtime to $$\min_{\pi} \sum_{t=2}^{N} \big|\tau_{\pi(t-1)} \,\Delta\, \tau_{\pi(t)}\big|,$$9 and $d_{i,j}$0 of baseline, while a hot ScorerCache re-run reduces runtime to $d_{i,j}$1 and $d_{i,j}$2 of baseline, respectively (MacAvaney et al., 14 Apr 2025). Proximity reduces retrieval latency by up to $d_{i,j}$3 on MMLU and $d_{i,j}$4 on MedRAG while maintaining accuracy over a broad threshold range, although MedRAG accuracy collapses at an overly permissive threshold of $d_{i,j}$5 (Bergman et al., 7 Mar 2025).

Beyond latency, some papers report broader behavioral effects. The image-recognition cache model improves top-1 accuracy on CIFAR-10, CIFAR-100, and ImageNet, and substantially increases robustness under adversarial attacks, with the mixture model and the cache-only model both outperforming the baseline under FGSM, I-FGSM, single-pixel, and Gaussian-blur attacks (Orhan, 2018). In deep search agents, moderate reranking often yields larger gains than increasing reasoning effort: for oss-120b, medium reasoning with reranking depth $d_{i,j}$6 reaches $d_{i,j}$7 accuracy versus $d_{i,j}$8 for high reasoning with $d_{i,j}$9, but with far lower raw token usage; with $O(\frac{N^2 m}{64})$0, medium reasoning rises to $O(\frac{N^2 m}{64})$1 (Sharifymoghaddam et al., 20 Jan 2026).

6. Limitations, misconceptions, and open directions

A recurrent misconception is that “cache prior” necessarily denotes a probabilistic prior over items. TableCache explicitly rejects that interpretation: its prior is realized as a distance-based greedy heuristic over table sets, not a probabilistic model over tables (Su et al., 13 Jan 2026). More generally, current work uses the term for several different objects—cache locality, reusable prefixes, approximate query neighborhoods, exemplar memories, or popularity partitions—so equivalence between methods should not be assumed.

Another misconception is that cache-aware reranking is universally beneficial. The papers repeatedly delimit the favorable regime. TableCache notes that larger schemas and non-local query distributions reduce cache locality and therefore TTFT gains (Su et al., 13 Jan 2026). miniReranker is best suited to high-reuse scenarios and reports that benefits shrink in low-reuse settings (Fan et al., 9 Jun 2026). HyperRAG identifies diminishing returns when cache hit rate is low, queries are long relative to documents, or storage rather than GPU compute becomes the bottleneck (An et al., 3 Apr 2025). Proximity likewise shows an explicit speed–accuracy trade-off governed by the similarity threshold $O(\frac{N^2 m}{64})$2 (Bergman et al., 7 Mar 2025).

Correctness constraints remain strict. HyperRAG requires identical model weights, tokenizer, prompt templates, chunking, special tokens, masking, and positional offsets between cache build and cache reuse (An et al., 3 Apr 2025). TableCache must preserve primary–foreign key relationships during offline joint encoding to avoid accuracy loss; omitting PFK reconstruction harms Spider and BIRD performance (Su et al., 13 Jan 2026). In IR experimentation, scorer caches are unsuitable for pairwise, listwise, or adaptive rerankers such as DuoT5 because their scores depend on the full candidate set rather than independent query–document pairs (MacAvaney et al., 14 Apr 2025).

Several papers point to more adaptive future formulations. TableCache explicitly proposes learning a better cache prior from historical access frequency or PFK co-occurrence paths and incorporating cache state and micro-batch layout jointly with reranking (Su et al., 13 Jan 2026). Proximity suggests learning thresholding, mixing weights, and eviction policies, while also moving from single-neighbor reuse to richer candidate aggregation (Bergman et al., 7 Mar 2025). TopRank-based coded caching leaves explicit finite-time regret bounds and scalable approximations as open questions (Bahadori et al., 7 Mar 2026). “Rerank Before You Reason” suggests that the next frontier lies in combining reranking with history compression so that both evidence quality and cached context cost improve simultaneously under the ETC model (Sharifymoghaddam et al., 20 Jan 2026).

Taken together, these directions imply that cache prior reranking is evolving from a set of ad hoc optimizations into a broader design principle for inference systems: reorder computation so that reusable state is exposed early, keep expensive content in cache-compatible form, and use reranking not only to improve relevance but also to shape the computational trajectory of the system.