Ideal Cache Model: Theory & Applications

• Ideal Cache Model is a theoretical abstraction that simplifies analyzing data transfers between fast and slower memory using an omniscient off-line optimal replacement policy.
• It underlies cache-oblivious algorithm design, enabling asymptotically optimal cache miss bounds in sequential and parallel settings through recursive divide-and-conquer strategies.
• Extensions such as distributed cache models and KV caching in LLM serving demonstrate its practical applications in workload optimization and capacity planning.

Searching arXiv for the cited papers and closely related material. The ideal cache model is a theoretical abstraction for analyzing data movement between a fast memory and a conceptually unbounded slower memory. In its canonical sequential form, it consists of a cache of size $Z$, a cache-line size $L$, full associativity, and an omniscient off-line optimal replacement policy usually described as Belady’s rule; the principal cost measure is the number of cache misses, equivalently the number of block transfers. Within cache-oblivious algorithmics, this abstraction underlies bounds such as $Q_1(n; Z, L)$, and it supports the classic claim that a cache-oblivious algorithm can achieve asymptotically optimal $Q_1$ simultaneously at all levels of a multilevel memory hierarchy without being told $Z$ or $L$. More recent systems work has reused the phrase in a different but related way: for LLM KV caching, an “ideal cache” can denote an offline upper bound on reuse under infinite capacity, or the minimum capacity needed to realize that upper bound under a clairvoyant “never-evict-reused” policy (Tang et al., 2020, Wang et al., 3 Jun 2025).

1. Sequential ideal-cache abstraction

In the formulation adopted by Frigo et al. and used in later cache-oblivious work, the sequential ideal cache has four defining properties: capacity $Z$, cache-line size $L$, full associativity, and off-line optimal replacement. The cache-miss complexity of an algorithm on input size $n$ is denoted $Q_1(n; Z, L)$, or simply $L$0. The idealization is deliberate: replacement is omniscient, main memory is conceptually infinite, and the abstraction counts only transfers of cache lines between the two levels (Tang et al., 2020).

The significance of the model lies in its separation of algorithmic locality from machine-specific tuning. A cache-oblivious algorithm, in the sense used by the literature summarized in the PACO paper, does not receive $L$1 or $L$2 as parameters, yet it can attain asymptotically optimal $L$3 across all hierarchy levels. This makes the model foundational for recursive divide-and-conquer designs in dynamic programming, matrix multiplication, Strassen’s algorithm, and sorting.

The abstraction is also intentionally stronger than typical hardware. Full associativity and Belady-style replacement are not intended as literal hardware descriptions; they provide an analytic lower-bound-like environment in which one can expose whether an algorithm has the right locality structure. This suggests that the ideal cache model is best understood as a communication complexity model rather than as a microarchitectural simulator.

2. Ideal distributed cache model

For parallel algorithms, the relevant extension is the ideal distributed cache model of Frigo & Strumpen. In that setting there are $L$4 processors $L$5, each with a private ideal cache of size $L$6 and line size $L$7, backed by an unbounded shared memory. A processor can access only data in its private cache; on a miss it fetches an entire line from shared memory. Replacement in each private cache remains off-line optimal, and the caches are assumed to be non-interfering, so one processor’s cache misses can be analyzed independently of the others. The analysis further assumes DAG-consistent models such as the Backer protocol and the HSMS model, no data races, and ignored false sharing (Tang et al., 2020).

Under this model, standard processor-oblivious quantities include $L$8, $L$9, $Q_1(n; Z, L)$0, $Q_1(n; Z, L)$1, and $Q_1(n; Z, L)$2. The PACO framework additionally tracks overall work over $Q_1(n; Z, L)$3 processors, critical-path work, overall cache complexity $Q_1(n; Z, L)$4, and critical-path cache complexity $Q_1(n; Z, L)$5. The memory parameters $Q_1(n; Z, L)$6 and $Q_1(n; Z, L)$7 remain explicit in the bounds. For example, for matrix multiplication the sequential cache complexity is stated as

$Q_1(n; Z, L)$8

The non-interference assumption is central. It makes the distributed model analytically tractable, but it also marks the boundary between ideal-cache theory and concrete shared-memory implementations with coherence traffic, false sharing, and associativity effects. The model is therefore strongest when used to reason about asymptotic communication structure and less informative when used to predict low-level hardware behavior.

3. Cache-oblivious algorithms and locality bounds

The PACO study applies the ideal-cache framework to several classic cache-oblivious algorithms and then re-partitions their recursion trees for arbitrary processor counts. The sequential cache bounds and parallel strong-scaling ranges are algorithm-specific, but the analytic pattern is consistent: a cache-oblivious sequential kernel supplies the $Q_1(n; Z, L)$9 bound, and the partitioning aims to recover $Q_1$0-type scaling in both work and cache complexity over a suitable range of $Q_1$1 (Tang et al., 2020).

Problem	Sequential ideal-cache complexity	Stated perfect strong scaling range
LCS	$Q_1$2	$Q_1$3
1D	$Q_1$4	$Q_1$5
GAP	$Q_1$6	$Q_1$7
MM	$Q_1$8	$Q_1$9
Strassen	$Z$0	$Z$1
Sorting	$Z$2	$Z$3

For LCS, 1D, and GAP, the geometry of the dynamic-programming dependency structure determines the cache bound. For classical matrix multiplication, the analysis is expressed through a rectangular cuboid $Z$4, whose surface area controls fit in cache and whose volume controls communication when it does not. For Strassen, the relevant recursion tree is 7-ary, with $Z$5, and the cache bound depends on whether the input-plus-output footprint of a subproblem fits within $Z$6. Sorting is treated differently: after balanced sample splitting, each processor executes a local sequential sort on a subset of size $Z$7, so the overall $Z$8 can be smaller than the best purely sequential $Z$9.

These results show how the ideal cache model functions as a unifying language for locality across very different recursive structures. The common feature is not a particular data structure but the existence of a recursion geometry in which sufficiently small subproblems become surface-area dominated rather than volume-dominated.

4. Processor-aware but cache-oblivious partitioning

The PACO framework classifies parallel methods by what they know. Processor-oblivious algorithms know neither $L$0 nor memory details and typically rely on schedulers such as randomized work-stealing. Processor-aware algorithms explicitly use $L$1, and sometimes memory or network details, to engineer a particular decomposition. PACO occupies the intermediate position: it uses $L$2 to partition the recursion tree, but it does not use $L$3 or $L$4, and it retains the same cache-oblivious sequential kernels as the underlying Frigo-style algorithms (Tang et al., 2020).

The generic partitioning scheme is a pruned breadth-first traversal of the divide-and-conquer tree. Nodes are expanded level by level until a level has at least $L$5 ready and independent nodes. Exactly $L$6 of those nodes are then assigned to the $L$7 processors in round-robin fashion, and each assigned node is executed sequentially using the cache-oblivious kernel. If no such level appears, base cases are eventually assigned in round-robin fashion. The key invariant is that the sequence of nodes assigned to any processor is geometrically decreasing in size, so the largest assigned node dominates both work and cache misses.

This is the basis for the paper’s notion of perfect strong scaling. Computationally, each processor’s work $L$8 satisfies

$L$9

Analogously, each processor’s cache misses satisfy

$Z$0

The emphasis on additive lower-order imbalance, rather than a constant-factor overhead, is stronger than many conventional parallel formulations.

Within this framework, the paper presents PACO variants for LCS, 1D, GAP, matrix multiplication, Strassen, and sorting, and argues that the approach provides a new perspective on Ballard et al.’s open problem of extending recursive cache-oblivious techniques to arbitrary architectures. For Strassen in particular, the paper states that the PACO variants work for any $Z$1, match the computational lower bound exactly, match Ballard et al.’s memory-dependent communication lower bounds in form, and achieve $Z$2 latency when interpreted in distributed-memory terms. This suggests that the ideal cache model, originally a sequential abstraction, can support processor-count-aware parallel design without abandoning cache-oblivious sequential structure.

5. KV-cache reinterpretation in LLM serving

In LLM serving, the phrase “ideal cache” is used in a more operational way. The Alibaba Cloud study introduces two distinct offline baselines tailored to KV caching. The first is an ideal hit ratio under infinite capacity: assume a cache with infinite capacity and a perfectly persistent KV store that never evicts anything, and count a hit whenever a KV block appears in the trace after a previous appearance. If the trace contains $Z$3 KV blocks and $Z$4 is the identity of block $Z$5, then

$Z$6

The second is the ideal capacity for a given workload: the minimum capacity required to achieve that ideal hit ratio assuming an ideal eviction policy, defined as never evicting a KV that will be reused (Wang et al., 3 Jun 2025).

For the capacity notion, if $Z$7 is the set of KV blocks that will be reused later and are currently alive at time $Z$8, and $Z$9 is the size of block $L$0, then

$L$1

Operationally, the trace is scanned to determine whether a block is reused and when it dies; replay then adds a block’s size when it is first generated and will be reused, subtracts its size when replay passes its last use, and tracks the maximum running total.

Ideal notion	Definition	Reported role
Ideal hit ratio	Infinite-capacity reuse upper bound	Fraction of all KV blocks ever reused
Ideal capacity	Offline “never-evict-reused” capacity	Minimum space to attain the ideal hit ratio

The empirical results are specific. On one representative day from two production traces, the ideal infinite-capacity hit ratio is 62% for Trace A and 54% for Trace B. Trace A is a to-C workload with Text, File, Multimodal, and Search requests; Trace B is a to-B OpenAI-compatible API workload. In Trace B, the multi-turn ratio is less than 0.1%, yet 97% of all cache hits come from single-turn API requests because of consistent system prompts hard-coded in client code. Cross-user sharing is almost negligible, most hits come from the same user’s prior requests, and reuse is extremely skewed: in Trace A, 19% of users contribute more than 90% of KV hits, and in Trace B, 4% of users do so. The study also reports that 10% of KV blocks contribute 77% of reuses.

Temporal locality is described in terms of reuse time and lifespan. In Trace A, 80% of reuse times are below 10 minutes; in Trace B, 80% are below 10 seconds. The reuse time distribution for a given request category is stable across hours during similar traffic conditions and is well approximated by an exponential distribution. Lifespan is defined as the time from first caching until the time after which the KV is never reused. In Trace A, 90% of KV blocks die within 612 seconds; in Trace B, 90% die within 0.3 seconds, although some system-prompt KV have lifespans up to 1,200 minutes, and the P99 lifespan of KV in to-B workloads is 97 seconds.

Capacity results then connect the ideal baseline to actual system sizing. For GQA models such as Llama3-70B on Trace A, approximately 4× the HBM space reserved for KV is sufficient to reach the ideal infinite-capacity hit ratio. On Trace B, the required KV capacity is smaller than the reserved HBM, so GPU-only caching suffices to reach the ideal hit ratio. MHA models require much larger capacity and can exceed local DRAM. The paper therefore concludes that “the overall cache size required for an ideal cache hit ratio is moderate,” but only for the studied workloads and especially for modern GQA models.

6. Limits, misconceptions, and scope

A persistent misconception is that all uses of “ideal cache” refer to the same object. In the classical cache-oblivious literature, the ideal cache model is a memory abstraction with full associativity, off-line optimal replacement, and a miss-count objective. In the KVCache study, the “ideal cache hit ratio” is not Bélády’s MIN under finite capacity; it is an infinite-capacity reuse statistic. The “ideal capacity” is closer to a Bélády-like offline optimum, but it is specialized to the rule “never evict a KV that will be reused” and is computed from exact death times rather than next-use distance (Wang et al., 3 Jun 2025).

Another misconception is that frequency alone should drive eviction. The KVCache study explicitly argues against this. Because the lifespan of KV is ephemeral, a block can be frequent early and then die, which makes LFU harmful in this setting. The proposed workload-aware policy replaces GDFS’s frequency-cost-size priority with a lexicographic tuple

$L$2

where $L$3 is the workload category, $L$4 is time since last access, and lower offset corresponds to head KV. The policy removes frequency from the priority, uses category-specific exponential reuse distributions and average lifespan, and reduces naive $L$5 eviction cost to $L$6 by maintaining one priority queue per category. The reported overhead is approximately 79 $L$7s per eviction and approximately 1.2% of total scheduling overhead in vLLM, while the hit rate improves by 8.1–23.9% versus standard policies and by 1.5–3.9% versus the best other baseline at each capacity; QTTFT reductions reach 28.3–41.9% in some configurations.

The assumptions and scope boundaries are explicit in both lines of work. Classical ideal-cache analyses assume fully associative, optimal replacement, and, in the distributed setting, non-interfering private caches with no false sharing and no data races. The PACO results guarantee perfect strong scaling only within parameter ranges stated in terms of $L$8, $L$9, $n$0, and $n$1. The KVCache study covers one week of traces from one large provider, reports one representative day per trace, studies mainstream chatbots and OpenAI-compatible APIs, and uses hashed 4-token blocks in analysis even though deployed systems such as vLLM typically use 16-token blocks. It also does not provide an explicit global online optimum or a competitive analysis against Bélády’s MIN.

These limits matter for interpretation. Theoretical ideal-cache results are strongest as asymptotic communication analyses, not as exact models of real set-associative hardware. The KV-specific ideal baselines are strongest as offline benchmarks for workload-aware eviction, not as universal properties of all LLM traffic. The combined picture nevertheless identifies a common theme: ideal-cache reasoning is most informative when locality has stable structure. In classic cache-oblivious algorithms that structure is recursive geometry; in KV caching it is category-specific temporal behavior, short lifespans, and prefix-dominated reuse.