MVR-cache: Multi-Vector LLM Caching

• MVR-cache is a semantic caching system that replaces single-vector cosine similarity with a learned, segment-aware multi-vector retrieval for complex LLM prompts.
• It employs a pointer-network based segmentation policy and reinforcement learning to generate multiple embeddings that capture distinct semantic aspects.
• Empirical results show up to 37% improvement in cache hit rates and reduced latency, maintaining rigorous correctness guarantees.

MVR-cache is a semantic caching system for LLMs that replaces single-vector cosine similarity with a learned, segment-aware multi-vector similarity. In the formulation introduced in “MVR-cache: Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation” (Noshad et al., 24 May 2026), each prompt is segmented by a learned policy, embedded as multiple segment vectors, compared to cached prompts through a symmetric normalized MaxSim score, and then filtered through a vCache-style probabilistic correctness model. The stated objective is to improve cache hit rate without relaxing correctness guarantees, and the system is trained with a reinforcement-learning procedure derived from a theoretical analysis of the similarity–correctness relationship (Noshad et al., 24 May 2026).

1. Definition and problem setting

MVR-cache addresses semantic caching for LLM APIs, where a system must decide whether a new prompt $x$ is sufficiently close to a cached prompt $x_i$ that it can safely reuse the cached response $r(x_i)$ instead of issuing a new LLM call (Noshad et al., 24 May 2026). In this setting, existing systems are described as representing each prompt with a single embedding vector $\phi(x) \in \mathbb{R}^d$, using a scalar similarity such as cosine similarity, and then applying thresholds to decide reuse (Noshad et al., 24 May 2026).

The central claim of MVR-cache is that this single-vector formulation is too coarse for prompts with multiple semantic aspects. The paper states that a single vector cannot capture multiple distinct semantic aspects of a longer or complex prompt, and that cosine similarity often ranks prompts that are topically similar but response-different above prompts that are actually response-equivalent (Noshad et al., 24 May 2026). MVR-cache therefore shifts the design focus from threshold tuning to the similarity layer itself.

Within this framework, semantic caching matters because a cache hit avoids an LLM call, and each LLM call is characterized as orders of magnitude more expensive and slower than embedding plus retrieval. The paper treats correctness as non-negotiable: it keeps the same correctness-guarantee framework as vCache, while attempting to make nearest-neighbor retrieval more faithful to response equivalence (Noshad et al., 24 May 2026).

A plausible implication is that MVR-cache belongs to a broader class of retrieval-augmented cache controllers in which the main bottleneck is not storage but semantic matching fidelity. In that sense, its main contribution is not a new cache eviction policy, but a new similarity representation and training procedure for hit-or-miss decisions.

2. Representation: learned segmentation and multi-vector retrieval

The system begins by identifying candidate split positions $P_x = \{p_1,\dots,p_{|P_x|}\}$ for a prompt $x$, such as token indices of punctuation marks (Noshad et al., 24 May 2026). A learned segmentation model $\mathcal{E}_\theta$ then chooses an ordered subset of these positions,

$$\mathcal{E}_\theta(x)=p=[p_{i_1},\dots,p_{i_{m-1}}],$$

which induces contiguous segments $x^{(1)},\dots,x^{(m)}$ (Noshad et al., 24 May 2026). Each segment is embedded with a shared encoder $\text{Emb}(\cdot)$, yielding a multi-vector representation rather than a single prompt embedding.

The segmentation model is implemented as a pointer network. The prompt tokens are encoded by BERT, projected by an MLP, and processed by a single-layer LSTM decoder. At each decoding step, attention scores

$x_i$0

are computed over candidate split positions, masked to enforce increasing boundary order, and the process continues until a special <stop> token is chosen (Noshad et al., 24 May 2026). The resulting segmentation policy is denoted $x_i$1.

Once segmented, a prompt is represented as

$x_i$2

Similarity between a query prompt and a cached prompt is then computed with ColBERT-style MaxSim. For prompts $x_i$3 and $x_i$4, unidirectional MaxSim is

$x_i$5

where $x_i$6 is a base similarity such as cosine similarity (Noshad et al., 24 May 2026).

For semantic caching, the paper defines a symmetric normalized score,

$x_i$7

and retrieves the nearest neighbor by

$x_i$8

The normalization by segment count is intended to make scores comparable across prompts of different lengths, and the symmetrization is intended to enforce mutual relevance rather than one-sided containment (Noshad et al., 24 May 2026).

This design distinguishes MVR-cache from single-vector semantic caches. The paper’s argument is that partial matching at segment level can preserve response-determining information that would be flattened away by a whole-prompt embedding. A plausible implication is that MVR-cache is best viewed as a retrieval model specialized for cacheability rather than as a general-purpose semantic search model.

3. Correctness model and theoretical objective

MVR-cache preserves the vCache-style correctness model. For a prompt $x_i$9, with nearest neighbor $r(x_i)$0 and similarity score $r(x_i)$1, the probability that reuse is correct is modeled as

$r(x_i)$2

Here $r(x_i)$3 indicates whether the new response equals the cached response, and $r(x_i)$4 are estimated by maximum likelihood from neighbor-specific metadata $r(x_i)$5 (Noshad et al., 24 May 2026).

For each cached prompt $r(x_i)$6, the paper defines

$r(x_i)$7

and fits

$r(x_i)$8

where

$r(x_i)$9

Given these parameters, vCache computes an exploration probability $\phi(x) \in \mathbb{R}^d$0 that guarantees overall error $\phi(x) \in \mathbb{R}^d$1 under its assumptions (Noshad et al., 24 May 2026).

The theoretical analysis in MVR-cache is built on two assumptions. Assumption 3.1 states that, for fixed $\phi(x) \in \mathbb{R}^d$2, similarity scores conditioned on correctness satisfy

$\phi(x) \in \mathbb{R}^d$3

Assumption 3.2 states a balanced prior $\phi(x) \in \mathbb{R}^d$4 (Noshad et al., 24 May 2026). Under these assumptions, the paper gives Theorem 3.3: minimizing the MLE/BCE loss with similarity scores $\phi(x) \in \mathbb{R}^d$5 is equivalent to maximizing the cache hit rate subject to any user-specified error bound $\phi(x) \in \mathbb{R}^d$6 (Noshad et al., 24 May 2026).

The argument relies on the fact that, under the Gaussian assumption, the posterior correctness probability is exactly logistic with

$\phi(x) \in \mathbb{R}^d$7

The population MLE loss depends on the class separation only through $\phi(x) \in \mathbb{R}^d$8, and minimizing the loss is stated to be equivalent to maximizing $\phi(x) \in \mathbb{R}^d$9 (Noshad et al., 24 May 2026). The paper further extends the analysis to imbalanced class priors through a class-rebalanced MLE objective in Lemma 3.4 (Noshad et al., 24 May 2026).

This makes the segmentation model’s training objective unusually explicit. Rather than optimizing a generic ranking loss, MVR-cache optimizes a similarity distribution that is theoretically aligned with the downstream cache-hit objective under the specified correctness constraints. This suggests a direct coupling between retrieval representation learning and cache-control guarantees.

4. Reinforcement-learning training procedure

The segmentation policy induces a combinatorial, non-differentiable optimization problem. The chosen split positions affect segment embeddings, segment counts, $P_x = \{p_1,\dots,p_{|P_x|}\}$0, nearest-neighbor identity, and ultimately the logistic correctness model. Because segmentation decisions are discrete and nearest-neighbor retrieval is itself discrete, the paper formulates training as RL4CO and uses REINFORCE (Noshad et al., 24 May 2026).

For a sampled prompt $P_x = \{p_1,\dots,p_{|P_x|}\}$1, the reward is defined as the negative BCE loss accumulated over prompts whose current nearest neighbor is $P_x = \{p_1,\dots,p_{|P_x|}\}$2: $P_x = \{p_1,\dots,p_{|P_x|}\}$3 The expected objective is

$P_x = \{p_1,\dots,p_{|P_x|}\}$4

and policy gradients are estimated by standard REINFORCE,

$P_x = \{p_1,\dots,p_{|P_x|}\}$5

The training loop samples prompts, samples segmentations, computes $P_x = \{p_1,\dots,p_{|P_x|}\}$6, updates $P_x = \{p_1,\dots,p_{|P_x|}\}$7, and periodically refits $P_x = \{p_1,\dots,p_{|P_x|}\}$8 by MLE (Noshad et al., 24 May 2026).

A practical complication is that the mapping $P_x = \{p_1,\dots,p_{|P_x|}\}$9 also depends on $x$0. Recomputing all nearest neighbors at every update would be expensive, so the paper freezes the nearest-neighbor mapping for $x$1 steps and then recomputes segmentations and nearest neighbors for the full training set every $x$2 steps (Noshad et al., 24 May 2026). This decouples frequent policy updates from expensive global reranking.

The following table summarizes the core training components described for MVR-cache.

Component	Formulation	Role
Policy	$x$3	Samples prompt segmentations
Reward	Negative BCE-based loss	Aligns segmentation with correctness-aware hit rate
Optimizer	REINFORCE	Handles discrete segmentation actions

The training procedure is therefore not a post hoc heuristic over an existing embedding model. It is a learned segmentation policy optimized specifically for cache retrieval under a probabilistic correctness model.

5. Retrieval pipeline and deployment architecture

At inference time, MVR-cache follows a staged retrieval and decision procedure. Given a prompt $x$4, the system computes candidate positions $x$5, segments the prompt with $x$6, embeds the segments, retrieves top candidates using a coarse HNSW index over single-vector full-prompt embeddings, reranks the top-20 candidates with $x$7, and then applies the vCache correctness model to decide exploit versus explore (Noshad et al., 24 May 2026).

The coarse-to-fine retrieval structure is central to the system’s practicality. Each cached prompt stores its segmentation, segment embeddings, response $x$8, and metadata $x$9. Retrieval is approximated by a two-stage pipeline: HNSW on full-prompt embeddings narrows the search space, and symmetric MaxSim reranking recovers the more accurate multi-vector similarity needed for semantic caching (Noshad et al., 24 May 2026).

The paper states that the segmentation model uses BERT, projection MLP, single-layer LSTM decoder, and attention; the embedding model is BGE by default; and the LLM used for generating ground-truth training labels is GPT-4o-mini (Noshad et al., 24 May 2026). Storage overhead consists mainly of multiple vectors per prompt rather than a single vector, but the paper characterizes this overhead as small relative to LLM compute (Noshad et al., 24 May 2026).

The decision rule remains vCache-style. After obtaining the nearest neighbor $\mathcal{E}_\theta$0 and score $\mathcal{E}_\theta$1, the system estimates $\mathcal{E}_\theta$2 from the logistic model associated with $\mathcal{E}_\theta$3, computes an exploration probability $\mathcal{E}_\theta$4 satisfying the user-provided error constraint $\mathcal{E}_\theta$5, and either returns the cached response $\mathcal{E}_\theta$6 with probability $\mathcal{E}_\theta$7 or calls the LLM with probability $\mathcal{E}_\theta$8 (Noshad et al., 24 May 2026).

This deployment structure indicates that MVR-cache is intended as a drop-in semantic-caching layer for LLM APIs rather than a replacement for the cache controller itself. A plausible implication is that its compatibility with existing semantic-caching stacks depends chiefly on whether they can expose per-neighbor correctness metadata in the style of vCache.

6. Empirical results, comparisons, and scope

The experimental evaluation covers four datasets: SemCacheSearchQueries, SemCacheClassification, PromptBench (SQuAD-V2 perturbed), and QNLI-derived QA prompts (Noshad et al., 24 May 2026). Only about 3K prompts per dataset are used for training the segmentation model, which the paper presents as reflecting realistic labeling budgets (Noshad et al., 24 May 2026). Baselines include vCache, vCache with ColBERT-style segmentation, and vCache with POQD (Noshad et al., 24 May 2026).

Across these datasets, the main reported result is that MVR-cache consistently increases cache hit rates by up to 37% while maintaining the same correctness guarantees (Noshad et al., 24 May 2026). More specifically, the paper reports cache-on-miss improvements up to about 25% and always-cache improvements up to 37%, notably on SemCacheSearchQueries (Noshad et al., 24 May 2026). Error rates for all methods converge below $\mathcal{E}_\theta$9, and MVR-cache remains below the same bound while improving hit rate (Noshad et al., 24 May 2026).

Latency results are also notable. Despite the additional segmentation and multi-vector reranking, end-to-end latency is reported to be reduced by up to 6% versus vCache at the same error bound, because the additional cache hits offset the retrieval overhead by reducing expensive LLM calls (Noshad et al., 24 May 2026). The paper characterizes the non-LLM overhead as tiny compared with LLM latency, with segmentation, embedding, and retrieval operating on the scale of tens of milliseconds versus thousands of milliseconds for an LLM call (Noshad et al., 24 May 2026).

Ablations support several design choices. BGE, GTE, and E5 show negligible differences in performance within MVR-cache (Noshad et al., 24 May 2026). Training on 3K prompts performs nearly as well as 6K or 10K (Noshad et al., 24 May 2026). Candidate split positions based on punctuation perform similarly to token-level, keyword, and sentence-level candidates, and are described as slightly best (Noshad et al., 24 May 2026). Symmetric SMaxSim yields a measurable, though modest, improvement over unidirectional MaxSim (Noshad et al., 24 May 2026).

The paper also reports cross-dataset transfer: a segmentation model trained on PromptBench transfers to QNLI and still outperforms vCache and the other baselines on hit rate without rebalance (Noshad et al., 24 May 2026). This suggests that the segmentation policy is not purely dataset-specific, although the paper still frames training as domain- and task-sensitive offline preparation.

The following table condenses the empirical claims emphasized in the paper.

Aspect	Reported outcome	Source
Hit rate	Up to 37% increase	(Noshad et al., 24 May 2026)
Correctness	Same guarantees as vCache; error below $$\mathcal{E}_\theta(x)=p=[p_{i_1},\dots,p_{i_{m-1}}],$$0	(Noshad et al., 24 May 2026)
Latency	Up to 6% reduction vs vCache	(Noshad et al., 24 May 2026)

7. Interpretation, terminology, and limitations

In its primary usage, “MVR-cache” denotes the semantic caching system introduced in (Noshad et al., 24 May 2026). The acronym derives from Multi-Vector Retrieval, and the defining features are learned prompt segmentation, symmetric normalized MaxSim, and correctness-aware RL training. This is distinct from other uses of “MVR” in the literature, such as “Multi-View Video Reward Shaping” in reinforcement learning (Luo et al., 2 Mar 2026), and from separate multimodal KV-cache designs that have been described informally as MVR-style caches in vision or multimodal autoregressive settings (Li et al., 26 May 2025, Ma et al., 22 Jun 2026). The shared acronym should not obscure the fact that (Noshad et al., 24 May 2026) concerns semantic caching for LLM prompts rather than KV-cache reuse or video reward shaping.

Several limitations are explicit in the MVR-cache paper. The method requires labeled pairs or proxy labels for offline training; its formal guarantee relies on the Gaussian class-conditional similarity assumption; RL training is more complex than training-free vCache; and nearest-neighbor recomputation remains expensive even though it is amortized by periodic updates (Noshad et al., 24 May 2026). The current segmentation model is text-only, and the paper identifies multimodal prompts as future work (Noshad et al., 24 May 2026).

These limitations clarify the scope of the contribution. MVR-cache does not replace correctness-aware semantic caching with a looser heuristic; rather, it attempts to improve the retrieval score while keeping the existing correctness framework intact. It also does not directly learn the embedding model jointly with the segmenter. The paper identifies direct multi-vector indexing, richer RL formulations such as actor-critic, and multimodal extensions as future directions (Noshad et al., 24 May 2026).

Taken together, MVR-cache can be situated as a retrieval-centric redesign of semantic caching. Its novelty lies in treating cache hits as a structured retrieval problem over learned prompt segments, and in coupling that retrieval problem to a theorem-backed objective and an RL-trained segmentation policy.