LLM Cache Bandit: Knapsack & Feedback

• LLM cache bandit problems are online decision frameworks where finite caches are managed under uncertain demand and reverse-feedback, optimizing inference cost and stale-answer loss.
• They recast cache selection as a knapsack-constrained bandit problem that requires balancing variable query sizes with value-to-size tradeoffs under limited capacity.
• Proposed methods like VSOCB use lower-confidence estimations, scheduled knapsack oracle calls, and adaptive cache updates to minimize regret and enhance system efficiency.

The LLM cache bandit problem denotes a family of online decision problems in which an LLM-serving system must manage a finite cache under uncertain demand, selective feedback, and resource constraints so as to reduce cumulative inference cost, latency, or stale-answer loss. In the most explicit recent formulation, the cache stores query–response pairs, query arrivals are stochastic, and heterogeneous prompt/response lengths transform cache selection from a fixed-cardinality ranking problem into a knapsack-constrained bandit problem with reverse feedback and restricted cache-update dynamics (Yang et al., 19 Sep 2025). Closely related formulations extend the same bandit lens to semantic retrieval buffers with switching costs, delayed-feedback replacement, and active cache synchronization (Sun et al., 1 Jul 2026, Yusuf et al., 2020, Kolobov et al., 2020).

1. Canonical formulation

A precise formulation considers a finite query universe

$\mathcal{Q}=\{q_1,\dots,q_N\},$

with one query $q_t$ arriving at each round $t\in\{1,\dots,T\}$, sampled i.i.d. from an unknown distribution $P$. Each query $q$ has input length $L(q)$, output length $A(q)$, and cache footprint

$S(q)=L(q)+A(q).$

A feasible cache $\mathcal{M}$ must satisfy the capacity constraint

$$\mathfrak{J}=\left\{\mathcal{M}\middle|\sum_{q\in\mathcal{M}} S(q)\le M\right\}.$$

If $q_t$0, the system returns the stored answer at zero inference cost; otherwise the LLM is invoked and incurs a random cost

$q_t$1

where $q_t$2 is the unknown expected inference cost and $q_t$3 is zero-mean sub-Gaussian noise, with $q_t$4 (Yang et al., 19 Sep 2025).

For a fixed cache $q_t$5, the expected uncached cost is

$q_t$6

Equivalently, the optimal static cache is

$q_t$7

The regret of an online policy relative to this oracle is

$q_t$8

This formulation makes the problem a bandit problem because cache actions determine what is observed: on a hit, no fresh inference cost is revealed; on a miss, the learner observes the realized cost, the answer, and the size. The feedback is therefore “reverse” relative to standard bandits: observation occurs on misses, not on selected cached items (Yang et al., 19 Sep 2025).

A further constraint distinguishes the model from standard combinatorial bandits. Because a response is unavailable once evicted, the next cache can only be formed from the current cache plus the newly arrived query,

$q_t$9

The learner may retain statistics for evicted items, but cannot instantly reinsert them without waiting for their reappearance. This makes the action space history-dependent even when the benchmark is a static cache (Yang et al., 19 Sep 2025).

2. Query heterogeneity and the knapsack turn

The central revision in "LLM Cache Bandit Revisited: Addressing Query Heterogeneity for Cost-Effective LLM Inference" is that variable query sizes qualitatively change the problem (Yang et al., 19 Sep 2025). Under uniform size, cache optimization reduces to selecting the top-$t\in\{1,\dots,T\}$0 queries by $t\in\{1,\dots,T\}$1. Once sizes vary, the offline optimum becomes a $t\in\{1,\dots,T\}$2-$t\in\{1,\dots,T\}$3 knapsack: $t\in\{1,\dots,T\}$4 A large query may force eviction of several small ones, while several small high-value queries may dominate one large expensive query. The optimization difficulty and the statistical difficulty therefore become coupled through a value-to-size tradeoff rather than a simple ranking (Yang et al., 19 Sep 2025).

To capture this heterogeneous-cardinality effect, the paper defines a valid cache set $t\in\{1,\dots,T\}$5 as one that is as full as possible and introduces

$t\in\{1,\dots,T\}$6

These replace the fixed cardinality parameter that exists in the uniform-size case. The resulting complexity term is

$t\in\{1,\dots,T\}$7

which enters both the problem-dependent and problem-independent regret bounds (Yang et al., 19 Sep 2025).

The proposed algorithm, VSOCB (Variable Size Online Cache Bandits), combines lower-confidence estimation, an exact knapsack oracle, an accumulation-based oracle invocation schedule, and a recommend-and-wait mechanism. It tracks total arrivals

$t\in\{1,\dots,T\}$8

total cache misses

$t\in\{1,\dots,T\}$9

and accumulated observed miss cost

$P$0

It then uses lower-confidence estimates for both cost and popularity, deliberately making long-unobserved cached items look less certain and therefore easier to evict, which induces exploration through cache turnover (Yang et al., 19 Sep 2025).

Oracle calls are not made every round. Instead, the algorithm invokes the knapsack oracle only when enough new information has accumulated, namely when either a miss count has grown multiplicatively or enough rounds have elapsed: $P$1 This accumulation schedule is the paper’s main computational–statistical balancing device. It yields only

$P$2

oracle calls while preserving regret guarantees (Yang et al., 19 Sep 2025).

With $P$3, the exact-oracle analysis gives

$P$4

and

$P$5

Since $P$6, this implies

$P$7

improving the earlier $P$8 guarantee cited by the paper for the uniform-size Berkeley line of work. On real data from OpenAssistant with FastChat-T5-3B, using $P$9 distinct queries, $q$0 rounds, and cache size $q$1, the reported total costs are $q$2 for the Berkeley-style baseline and $q$3 for VSOCB, about a $q$4 reduction, summarized in the abstract as “approximately $q$5” (Yang et al., 19 Sep 2025).

3. Alternative feedback models: delayed consequences and stale-state costs

The exact query–answer cache model is only one member of a broader family of cache bandit problems. A conceptually important precursor is "Cache Replacement as a MAB with Delayed Feedback and Decaying Costs" (Yusuf et al., 2020). There the learner does not choose cache entries directly; instead each arm is a replacement policy, and the consequence of an eviction is revealed only if the evicted item is requested again later. If an action chosen at time $q$6 is penalized at time $q$7, the delay is $q$8, and the estimated cost is scaled as

$q$9

with feedback beyond a threshold $L(q)$0 treated as zero. The resulting EXP4-DFDC algorithm satisfies

$L(q)$1

and with

$L(q)$2

achieves $L(q)$3 regret (Yusuf et al., 2020). For LLM systems, the relevance is that the cost of evicting a prompt-result pair, a semantic memory, or a reusable state block is often realized only on later reuse.

A different generalization appears in "Online Learning for Active Cache Synchronization" (Kolobov et al., 2020). Its synchronization bandit model assumes that all arms generate costs at all times, but the learner observes an arm’s instantaneous cost only when it plays that arm. Each arm $L(q)$4 has age state

$L(q)$5

mean instantaneous cost $L(q)$6, and cumulative expected cost

$L(q)$7

Periodic synchronization at rate vector $L(q)$8 induces cost

$L(q)$9

subject to a refresh-budget constraint

$A(q)$0

The MirrorSync algorithm uses random probe plays to build an unbiased gradient estimator and then applies mirror descent; its adversarial regret is

$A(q)$1

(Kolobov et al., 2020). This formulation is directly aligned with stale retrieval/document caches, tool-result caches, and other persistent LLM-serving artifacts whose quality degrades as external state changes.

Together, these models show that “cache bandit” need not mean only popularity learning for exact-match hits. It can also mean learning under delayed eviction feedback, or learning refresh schedules for latent stale costs. The choice of model depends on whether the dominant uncertainty concerns reuse, delayed harm, or exogenous drift.

4. Semantic retrieval buffers and predictive cache population

The semantic-retrieval variant is formulated most clearly in "When Classic Cache Policies Fail: Learning-Augmented Replacement for Semantic Retrieval Buffers" (Sun et al., 1 Jul 2026). There the cache state is

$A(q)$2

retrieval is by embedding similarity, and the loss is continuous rather than binary: $A(q)$3 The total objective includes both miss cost and switching cost,

$A(q)$4

The paper’s main empirical finding is that on semantic workloads, LRU and LFU consistently underperform the naive FIFO baseline because temporal locality and frequency concentration are weak. The proposed SOLAR framework separates modification timing from eviction content: modification is triggered by regret accumulation, while eviction uses Bayesian online learning with Beta posteriors and Thompson-style sampling. The paper reports about a 17\% modification rate, proves a constant competitive ratio

$A(q)$5

independent of cache size and horizon under bounded stale advantage, and gives an eviction regret bound

$A(q)$6

matching the $A(q)$7 lower bound up to logarithmic factors (Sun et al., 1 Jul 2026). This is a genuine LLM cache bandit formulation when the cache stores semantic memories rather than exact query–answer pairs.

A complementary but non-bandit direction is "InstCache: A Predictive Cache for LLM Serving" (Zou et al., 2024). InstCache is not an online learning method; it is an offline predictive cache populated by likely future instructions generated from an instruction-aligned LM. The key selection rule is

$A(q)$8

and the hit-rate estimate is based on the NLL CDF. On LMSys, InstCache reports 51.34\% hit rate, about 2x speedup, and approximately 4.5GB memory for 4,253,981 instruction-answer pairs (Zou et al., 2024). Within the LLM cache bandit literature, its role is best understood as an offline predictive prior: it supplies a model-based notion of future request likelihood, but it does not perform online exploration, regret minimization, or dynamic replacement.

The contrast between SOLAR and InstCache is revealing. SOLAR assumes online semantic retrieval with partial implicit feedback and switching costs. InstCache assumes that a strong predictive model can pre-populate an exact-match cache offline. The former is a bandit problem; the latter is supervised predictive caching. Both are relevant because practical LLM systems often need both an offline prior over likely requests and an online controller for nonstationarity and capacity pressure.

5. Boundary with neighboring non-bandit cache-control problems

A recurring misconception is to treat every LLM cache-management problem as a bandit. Several influential papers instead formulate optimization or scheduling problems with known structure and no exploration component. "XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference" assigns per-layer KV budgets by solving a combinatorial allocation problem over retained-importance ratios; it reports 61.6\% average KV memory reduction, 2.1x computational efficiency improvement, and up to 5.2x throughput increase, but it is an offline profiling-and-allocation method rather than a bandit (Li et al., 2024). "Reformulating KV Cache Eviction Problem for Long-Context LLM Inference" introduces LaProx, an output-aware token-importance score

$A(q)$9

and shows that model performance can be maintained with only 5\% of the KV cache while achieving up to 2\times accuracy-loss reduction under extreme compression; again, the method is a deterministic scoring rule, not an online learner (Mai et al., 8 May 2026).

Serving-time scheduling problems make the distinction even sharper. "Apt-Serve" formulates per-iteration scheduling with a hybrid KV/hidden cache as an NP-hard knapsack-like optimization and reports up to 8.8x improvement in effective throughput over baseline serving systems (Gao et al., 10 Apr 2025). "CacheFlow" formulates KV restoration as a 3D parallel scheduling problem across tokens, layers, and GPUs, and reports 10\%–62\% TTFT reduction (Nian et al., 28 Apr 2026). "Low-Latency Edge LLM Handover via Joint KV Cache Transfer and Token Prefill" optimizes a split between KV transfer and target-side prefill, proving that the optimal split equalizes prefill and transfer delay whenever possible; it reports up to 3.1\times and 3.7\times improvement over pure transfer and pure prefill baselines (Lee et al., 30 Mar 2026). These are online systems problems with constrained resources, but they are not bandits unless uncertainty and exploration are made explicit.

The practical implication is that the LLM cache bandit problem occupies only one region of a larger design space. When the main difficulty is unknown popularity, delayed reuse, semantic hit quality, or unobserved staleness, bandit formulations are natural. When the main difficulty is precedence-constrained restoration, per-layer allocation under profiled costs, or deterministic batch scheduling, direct optimization may be the more faithful abstraction.

6. Limitations, misconceptions, and open directions

The current literature makes several limitations explicit. In the heterogeneous query-answer formulation, the analysis assumes a fixed query distribution $S(q)=L(q)+A(q).$0, known sizes once a query has been observed, and regret against the best static cache rather than a dynamic benchmark; experiments are also modest in scale (Yang et al., 19 Sep 2025). In semantic retrieval buffers, the strongest guarantees rely on bounded stale advantage and stylized stochastic utility models, while the empirical regime is nonstationary and feedback is only implicit (Sun et al., 1 Jul 2026). Predictive caches such as InstCache depend on distributional stability and exact-match coverage; drift is handled by periodic rebuilding rather than continual adaptation (Zou et al., 2024).

Another recurring misconception is that classical heuristics remain reliable baselines in all LLM settings. The evidence does not support that claim. In semantic retrieval buffers, LRU and LFU can underperform FIFO because semantic workloads may lack temporal locality and frequency concentration (Sun et al., 1 Jul 2026). In long-context KV management, head-local or layer-uniform heuristics can underperform output-aware or personalized allocations (Mai et al., 8 May 2026, Li et al., 2024). In heterogeneous query-answer caching, local one-for-one replacement can be suboptimal because variable-size entries induce nonlocal knapsack interactions (Yang et al., 19 Sep 2025).

A broader synthesis is that the term “LLM cache bandit problem” now refers less to one fixed model than to a technical family. Exact query–answer reuse leads to reverse-feedback knapsack bandits (Yang et al., 19 Sep 2025). Semantic memory leads to switching-cost online learning with local Bayesian eviction (Sun et al., 1 Jul 2026). Stale knowledge artifacts lead to synchronization bandits with probe-based gradient estimation (Kolobov et al., 2020). Delayed eviction consequences motivate delayed-feedback and decaying-cost expert aggregation (Yusuf et al., 2020). This suggests that future work will likely center on contextual combinatorial bandits that incorporate variable object size, delayed and partial feedback, switching costs, nonstationarity, and cross-request resource coupling rather than on a single universal cache-bandit formalism.