MVR-cache: Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation

Abstract: To reduce LLM costs and latency, semantic caching systems must accurately identify when a new prompt matches a cached one. Current methods often rely on simplistic similarity measures, which limit their effectiveness. We introduce MVR-cache, a novel semantic caching approach that significantly improves retrieval accuracy by integrating Multi-Vector Retrieval (MVR). MVR-cache is built upon a learnable segmentation model that intelligently splits prompts, enabling fine-grained similarity comparisons via MaxSim. We derive the model's training objective from a rigorous theoretical analysis. This can ensure that optimizing this objective directly maximizes cache hits under strict correctness constraints. To solve the resulting non-differentiable combinatorial optimization problem, we leverage a reinforcement learning-based training strategy with the theoretically grounded objectives as the reward. Experimental results on established benchmarks across diverse tasks confirm that in comparison to the state-of-the-art, MVR-cache consistently increases the cache hit rates by up to 37% while maintaining the same correctness guarantees. MVR-cache is available at https://github.com/PKU-SDS-lab/MVR-Cache

• The paper introduces a novel cache optimization method that combines multi-vector retrieval with a learned prompt segmentation policy to boost semantic cache hit rates.
• It employs a symmetric MaxSim similarity measure and RL-based segmentation to ensure correctness while reducing computational latency.
• Experimental results show up to 37% improvement in cache hits and reduced inference latency, demonstrating the method’s efficiency and robustness.

Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation

Introduction

Semantic caching for LLM inference seeks to amortize computational cost and minimize latency by reusing cached responses when incoming prompts are semantically equivalent to previous ones. Conventional methods rely primarily on scalar similarity metrics, such as cosine similarity over embedding vectors, to determine cache hits. However, these representations are insufficiently expressive for complex prompts, especially as semantic variation becomes more subtle yet response-equivalence demands strict caching policy correctness. The paper "MVR-cache: Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation" (2605.24914) introduces a theoretically principled, data-efficient, and empirically validated approach that integrates multi-vector retrieval (MVR) with a learnable, lightweight prompt segmentation model to substantially increase cache hit rates while maintaining correctness guarantees.

Problem Formulation and Motivation

Current semantic caches frequently incur suboptimal hit rates because standard vector similarity fails to detect nuanced but critical semantic differences. For example, two reviews of a crime drama, one positive and one negative, may have high overall embedding cosine similarity but yield divergent LLM responses, thus making naive caching unsound. Furthermore, prior work such as vCache adopts an adaptive but still single-vector similarity, which cannot capture partial match relationships across prompt substructures.

The fundamental challenge is to develop a caching mechanism that captures fine-grained, bidirectional semantic equivalence to maximize cache hits without violating correctness constraints (that is, incorrectly serving a mismatched cached response). Addressing both the representation and policy aspects is necessary for optimal exploitation of semantic caching.

MVR-cache: Architecture and Theoretical Foundations

Multi-Vector Retrieval and Symmetric MaxSim

MVR-cache employs multi-vector retrieval as originally introduced in ColBERT, segmenting both cached and query prompts and embedding each segment independently. Semantic similarity is then measured via a symmetric aggregation of segment-wise maximum similarity (SMaxSim), capturing finer structural correspondence than global prompt embeddings. Bidirectionality is imposed by averaging the normalized MaxSim scores in both directions, which prevents situations where only unidirectional partial overlaps would spuriously trigger cache reuse.

Learned Segmentation Policy

Central to the method is a lightweight, pointer-network-based segmentation module that selects split points (typically at punctuation, but also supporting more granular token-level positions) to decompose each prompt. Unlike prior segmentation-by-heuristics (as in ColBERT or POQD), the policy in MVR-cache is explicitly trained to maximize cache hit rates under correctness guarantees, optimizing the segmentation for task-specific response-equivalence.

The segmentation model uses a compact stack of BERT encoder, MLP, LSTM, and attention, yielding real-time segmentation speeds (approximately 23ms per prompt on tested datasets), and producing a variable number of segments tailored to each prompt.

Training Objective and RL Optimization

Since the cache hit rate is a discrete, non-differentiable metric, the paper introduces a theoretically justified maximum likelihood objective that aligns segmentation-induced SMaxSim with ground-truth response equivalence. Theoretically, under mild assumptions of conditional normality (empirically validated across datasets) and balanced class distributions (addressed via class balancing when not met in practice), minimizing this objective is provably optimal for maximizing hit rates at any fixed error rate $\delta$.

Due to the combinatorial nature of segmentation, the policy is optimized with a reinforcement learning for combinatorial optimization (RL4CO) approach, using REINFORCE, with reward defined as the negative binary cross-entropy loss over batchwise nearest-neighbor pairs. The nearest-neighbor mappings are periodically recomputed to balance training efficiency and up-to-date policy gradients.

Integration with Policy Guarantees

The policy integrates seamlessly with the adaptive-exploration mechanism of vCache. The SMaxSim-derived similarity is used to estimate the probability of correct cache hits, and the caching/exploration threshold is adaptively tuned to ensure the user-specified correctness bound is strictly observed.

Experimental Results

Datasets and Baselines

Extensive experiments were conducted across diverse tasks: short query search (SemCacheSearchQueries), ecommerce text classification (SemCacheClassification), prompt-perturbed QA (PromptBench), and multi-turn QA (QNLI). Strong baselines include vCache, ColBERT, and POQD—the latter two adapted to the vCache policy for comparability.

Cache Hit Rate, Error, and Latency

MVR-cache consistently outperforms all baselines:

• Cache hit rate improvements are up to 37% over vCache (e.g., SemCacheSearchQueries, always-cache protocol).
• Error rates remain tightly bounded under the specified $\delta=0.01$ across all settings.
• End-to-end inference latency is reduced by up to 6% due to higher cache utilization despite the moderate segmentation and retrieval overhead (non-LLM overhead $\sim$54ms vs. LLM call $\sim$1,200–4,200ms).
• Generalization: segmentation models trained on one dataset (e.g., PromptBench) transfer well to others (QNLI), maintaining superior cache hit rates OOD.

Ablation and Robustness Analyses

• Embedding backbone choice is non-critical; BGE, E5-large, and GTE yield similar results.
• Segmentation candidate sets (punctuation, keyword, token, sentence) result in comparable cache hit rates, with punctuation already expressive enough.
• Minimal training set size suffices (3K samples per dataset); larger sets yield negligible further gains.
• Replacing SMaxSim with vanilla MaxSim reduces performance slightly, confirming the value of symmetric similarity.

Long-Context Handling

MVR-cache demonstrates superiority on long-context prompts. Unlike single-vector approaches which dilute local relevance, the learned segmentation preserves distinct semantic units, resulting in stronger retrieval of response-equivalent (not just topically related) cache entries.

Cost-Efficiency

Labeling cost for training (e.g., collecting 3K ground-truth LLM responses) is amortized rapidly—gains in cache hits quickly outweigh this fixed cost. The method is also compatible with weak supervision, and proxy labeling with smaller LLMs can further minimize expenses.

Practical and Theoretical Implications

The integration of learned multi-vector representations with theoretically aligned correctness-driven objectives marks a shift from tuning cache policies to fundamentally improving semantic similarity assessment. Practically, this results in both higher cache efficiency (directly reducing LLM computation cost) and predictable reliability (no tradeoff with correctness).

The modular segmentation component is amenable to further improvements (e.g., multimodal extension, structurally aware segmentation for code or tables) and can be jointly optimized with advanced embedding models or in lower-resource settings via proxy supervision.

Future Directions

Potential research vectors include:

• Multimodal extension: Developing decomposition strategies for non-text inputs.
• Scalable/streaming adaptation: Extending the RL-based segmentation optimization to continual learning in evolving deployment scenarios.
• Structural segmentation: Incorporating syntactic/semantic parsing for hierarchical segment definitions.
• Integration with approximate retrieval: Exploring further acceleration via MVR-specific ANN indexes, conditional on runtime constraints.

Conclusion

MVR-cache establishes a new paradigm for semantic caching by maximizing cache efficacy through learned prompt segmentation and multi-vector retrieval, underpinned by a theoretically optimal training objective and practical RL-based optimization (2605.24914). This approach results in substantial cache hit improvements and reduced LLM utilization while respecting stringent correctness boundaries, and generalizes robustly across diverse datasets and prompt structures. Its design supports further adaptation to new domains and modalities, marking it as a versatile foundation for the next generation of high-efficiency semantic caching systems.