Cache-Resident Execution Model

• Cache-resident execution is a method that keeps critical data and state in on-chip cache to reduce latency and enhance throughput.
• It organizes execution by defining resident objects—such as working sets, graph partitions, or model weights—and setting policies for eviction and failure handling.
• Implementations demonstrate significant improvements, including up to 1.88× speedups and reduced cache misses in applications like CPU processing, graph workloads, and LLM inference.

Cache-resident execution model denotes an execution organization in which useful work is structured around keeping a chosen working set, execution state, or protection domain resident in cache or other on-chip memory, and around making the consequences of residency failure explicit. In the cited literature, the term is not singular: it appears as a shared-cache throughput model for multithreaded processors, as a partition-centric scheduling discipline for irregular graph workloads, as a security and predictability mechanism for cache isolation, as a formal abstraction for cache-aware analysis, and as a runtime model for LLM serving in which KV state, complete execution state, or model weights become first-class resident objects (Desai, 2023, Lu et al., 2021, Dessouky et al., 2021, Touzeau et al., 2017, Stepanek, 22 May 2026, Su, 18 Jun 2026, Zhang et al., 24 Jun 2026).

1. Core meanings and recurring structure

Across these uses, the resident object varies, but the structural pattern is consistent: a subset of state is designated as worth keeping on chip; execution is reorganized so that this state is reused before eviction; and arbitration, isolation, or analysis is centered on that state rather than on undifferentiated cache lines. In processor throughput studies, the resident object is the combined instruction and data working set of concurrent threads. In graph systems, it is an LLC-sized partition plus buffered per-query work. In security architectures, it is an isolated domain’s private cache region or a shared-cache-only view of selected data. In recent LLM systems, it becomes either a future-reuse claim over KV, a graph-bound execution-state capsule, or a weight shard intended to remain in a GB-scale last-level cache (Desai, 2023, Lu et al., 2021, Dessouky et al., 2021, Bansal et al., 2019, Stepanek, 22 May 2026, Su, 18 Jun 2026, Zhang et al., 24 Jun 2026).

Setting	Resident object	Primary objective
Shared-cache CPU throughput	Combined instruction/data working sets	Keep shared cache from becoming the bottleneck
Graph FPP execution	LLC-sized graph partition and buffered operations	Reduce LLC misses and memory stalls
Resident KV serving	Future-reuse KV object with lifecycle and predicate	Make active/resident arbitration explicit
Execution-state capsules	Complete graph-bound restorable state	Low-latency snapshot, restore, fork, rollback
CPU LLM inference	Weight shards in GB-scale LLC	Turn DRAM-bound decode into LLC-bound decode
Secure/predictable caches	Isolated or shared-cache-only data/domain	Bound interference and suppress leakage

A common misconception is to equate cache-resident execution with simple cache priority or with “everything fits in cache.” Several of the papers reject that interpretation explicitly. Resident KV claims are distinguished from ordinary priority because they introduce lifecycle, materialization, and harm semantics rather than merely ranking victims for eviction (Stepanek, 22 May 2026). Chunked-Cache states that “cache-resident” secure execution does not mean that no misses occur; it means that private accesses are confined to an exclusive LLC chunk, so any misses are self-induced rather than cross-domain (Dessouky et al., 2021).

2. Performance-oriented residency in shared caches

In classical microarchitecture, a cache-resident execution model is a performance regime. The dual-hardware-threaded AJIT processor duplicates a single-issue in-order pipeline while keeping a single shared memory unit consisting of shared I-cache, shared D-cache, and shared MMU. The evaluation uses a 32 KB I$and 32 KB D$, both 4-way, with 1-cycle hit latency and about a 30-cycle miss penalty in the FPGA setup. The design achieves speedups of 1.6X to 1.88X on parallelizable applications when the combined working sets of the two hardware threads stay effectively cache-resident and shared-cache bandwidth is not saturated; performance degrades when cache misses increase, and mem-copy shows a 0.87× slowdown because the D-cache miss rate reaches 87% (Desai, 2023). This establishes one baseline meaning of the term: near-ideal speedup is available only while the execution remains largely resident in the shared on-chip cache.

Analytical shared-cache modeling for OpenMP programs makes the same idea explicit through reuse distance. The model derives a concurrent reuse-distance profile from a sequential small-input trace, then maps that profile to shared-cache hit probability using a probabilistic set-associative cache model. For a shared L3, the overall hit rate is computed from the reuse-distance distribution $\Pr(D)$ and the conditional hit probability $P(h \mid D)$, allowing cache residency to be treated as the condition that most reuses have distance below effective cache capacity. On the reported Intel Core i7 and Xeon systems, the predicted shared-cache hit rates remain close to Valgrind-derived values, with BFS at about $96.77\%$ predicted versus $98.49\%$ observed on the 6 MB L3, and MatMul at about $95.31\%$ predicted versus $97.83\%$ observed (Barai et al., 2019). In this line of work, cache-resident execution is a quantitative criterion for deciding whether a parallel phase remains inside the shared cache rather than spilling into memory-bound behavior.

ForkGraph turns that criterion into an execution strategy for the fork-processing pattern on large graphs. The graph is divided into partitions each sized of LLC capacity; each partition owns a buffer of operations targeting vertices in that partition; and execution becomes partition-centric rather than query-centric. Within a partition, queries are executed with sequential algorithms, and atomic-free query processing is obtained by consolidating contending operations to the cache-resident graph partition. Inter-partition execution adds yielding and priority-based scheduling to reduce redundant work. On real-world graphs, the system reports up to two orders of magnitude speedups over state-of-the-art graph processing systems, with LLC misses reduced by up to 100× in the reported comparisons (Lu et al., 2021). Here residency is not merely an observed property; it is the organizing principle of the scheduler.

3. LLM runtimes: resident state as an explicit runtime object

Recent LLM systems recast cache residency from an emergent performance regime into an explicit runtime contract. “Resident KV Claims” defines a resident claim as “an application-visible future-reuse object submitted to the runtime,” with a formal ResidentClaimInput containing claim_id, owner_scope, cache_identity, object_id, materialization_predicate, footprint_blocks, protection_mode, and optional duration_steps. The runtime separates input, decision, lifecycle state, and telemetry, and introduces the feasibility boundary

$$\text{protected\_resident\_kv} + \text{active\_live\_kv} \le \text{usable\_kv}.$$

When this boundary fails, the runtime must choose and report an explicit action. In the canonical 60/70/80 scenario, a 60-block resident claim and a 70-block active prefill exceed an 80-block usable KV pool; native eviction and write no-admit both allow resident value to be lost, whereas hard protected resident claims convert the failure mode into scheduler-visible active refusal with direct blocking-claim attribution. The paper states that the result is not a production speedup or a new cache-replacement algorithm, but a runtime contract that turns unreported resident loss into reconstructable active/resident arbitration (Stepanek, 22 May 2026).

A second LLM strand broadens the resident object from KV fragments to complete graph-bound execution state. “Execution-State Capsules” defines the boundary state as

$$S(P)=\{S_{\text{KV}}, S_{\text{rec}}, S_{\text{conv}}, S_{\text{MTP}}, S_{\text{meta}}\},$$

and packages it as a graph-bound checkpoint and restore unit. FlashRT captures the forward pass as graph plans over contiguous static buffers with no block-table indirection, so a capsule can snapshot, restore, fork, or roll back the whole execution boundary. On an RTX 5090, GPU-resident snapshot and restore are sub-millisecond; TTFT speedup over cold prefill grows from 3.9x at 2k tokens to 27x at 16k tokens; and a KV-only ablation diverges, showing that recurrent state is load-bearing (Su, 18 Jun 2026). The paper is explicit that capsules are not a replacement for high-throughput KV-cache serving; they define a complementary latency-first serving point.

A third strand places the resident object at the level of model weights in GB-scale last-level caches. “Cache-Resident LLM Inference in GB-Scale Last-Level Caches” separates weight-centric operators from attention and KV-cache management into dedicated resource domains, keeping reusable weights cache-resident while scaling KV capacity independently of pipeline depth. The same execution model relaxes synchronization from operator boundaries to true sub-operator dependencies, because in the cache-resident regime operator-boundary synchronization becomes visible overhead. On deployed Llama-3.2-3B and Llama-2-7B configurations, the prototype reports 2.04x-11.51x speedup on TPOT relative to equally provisioned llama.cpp, and the validated analytical model reaches up to 13.9x TPOT speedup across model sizes, context lengths, and batch sizes (Zhang et al., 24 Jun 2026). This use of the term emphasizes resident weights rather than resident per-request state, but the logic is the same: reorganize execution so the reusable object remains on chip and the nonresident state is decoupled.

4. Privatization and state manipulation in cache-resident form

Cache-resident execution can also be a model of privatized state manipulation. CCache implements on-demand privatization for commutative updates without allocating extra copies in main memory. On first commutative access to a cache line, the line is fetched into L1, a source copy is stored in a small fully associative source buffer, and the L1 line becomes the updated copy. During the update phase, commutative operations act only on the L1 copy, CData lines do not participate in coherence, and memory is updated only through an explicit merge that combines the memory copy, the source copy, and the updated copy through a programmer-supplied merge function. The paper characterizes this as keeping all intermediate state in the core’s cache and source buffer while memory acts as the final reconciliation point, and reports speedups up to 3.2X on applications including a random access key-value store, clustering, breadth first search, and graph ranking (Balaji et al., 2017).

This model differs from both conventional privatization and transactional speculation. Conventional duplication allocates per-thread copies in memory and increases memory footprint and shared-cache occupancy; CCache avoids that by making the cache itself the location of the duplicated state. Transactional memory speculates and may roll back; CCache resolves conflicts by merge, provided the updates are commutative (Balaji et al., 2017). A plausible implication is that cache-resident execution need not imply passively “keeping data hot”; it can instead elevate the cache into the primary locus of state evolution.

5. Isolation, predictability, and secure residency

In security and real-time systems, the principal concern is not throughput but the semantics of who may occupy, evict, or observe cache lines. Chunked-Cache provides on-demand, domain-private LLC regions for TEE architectures. An isolated domain in EXCLUSIVE-CACHE mode maps all non-shared accesses only into its chunk’s sets; other domains cannot access or evict lines in that chunk; and the NI-Domain continues to use a principal chunk plus any unallocated mainstream sets. The design adds a per-line DID and SHARED bit, a Cache Set Status Table, and a Domain Cache Allocation Table, while keeping the LLC physically set-associative. Evaluated against conventional way-based partitioning, Chunked-Cache outperforms it by 43% and scales significantly better to support a larger number of protection domains (Dessouky et al., 2021). The paper also clarifies that exclusive cache residency is an isolation property, not a guarantee that the working set fully fits: if the domain’s footprint exceeds the chunk capacity, capacity and conflict misses still occur, but only from the domain’s own accesses.

“Cache Where you Want!” introduces a different predictable-residency mechanism through the ARMv8-A memory type INC-OC, or Inner Non-Cacheable, Outer Cacheable. INC-OC data bypasses private caches and may be cached only in the shared cache, so the latency of shared accesses becomes independent of coherence state. In the trace-mode experiment with four cores writing the same line, the worst-case execution time for a single memory write request is reduced by 52%, and program-aware use on benchmark workloads has near-identical performance to the normal cacheable baseline (Bansal et al., 2019). This model does not lock data in cache; rather, it constrains where the data may become resident, removing coherence-induced variability at private levels.

HybCache and RaS pursue side-channel resilience by controlling both residency and observability. HybCache selectively assigns isolated domains to a subcache formed from designated ways, with domain-tagged lines, fully-associative lookup for isolated execution, and random replacement in the isolated subcache. It reports a performance overhead of 3.5–5% for isolated execution while incurring no performance overhead for the remaining execution workload (Dessouky et al., 2019). The remaining leakage is largely reduced to occupancy rather than address-specific access patterns. RaS, by contrast, decorrelates cache state changes from memory requests themselves. It uses NoFill to prevent demand-fetched security-sensitive lines from being filled, populates a Safe History Buffer only with authorized addresses, issues constant-rate randomized prefetches from safe windows, and uses random replacement. RaS-Spec reports a 3.8% average performance overhead against speculative cache attacks, while RaS+ variants span 7.9% to 45.2% average overhead as the security-performance trade-off is tightened (Hu et al., 2023). These systems show that secure cache-resident execution is often defined less by maximizing hits than by controlling which accesses are allowed to create resident state.

6. Formal analysis, telemetry, and unresolved tensions

Several papers make cache residency analyzable by introducing explicit state models. For WCET refinement under LRU caches, the cache-resident state of a tracked block $a$ is represented either as $\Pr(D)$0, meaning that $\Pr(D)$1 is absent from the cache, or as $\Pr(D)$2, meaning that $\Pr(D)$3 is present and the set $\Pr(D)$4 contains precisely the blocks younger than $\Pr(D)$5. A model checker then refines classical may/must cache analysis by exploring this block-centric state transition system along CFG paths, resolving up to about 60% of previously unknown accesses in some benchmarks (Touzeau et al., 2017). In this tradition, a cache-resident execution model is literally an execution semantics annotated with per-block residency.

Speculative symbolic execution tools extend the same idea to secret-dependent transient residency. KLEESPECTRE models whether speculatively loaded secret-dependent addresses remain cache-resident at an observation point by encoding set, tag, and associativity constraints over speculative and non-speculative traces (Wang et al., 2019). SpecuSym similarly records memory traces, models speculative states whose architectural effects roll back but whose cache effects survive, and constructs leak predicates by comparing speculative and non-speculative cache behavior; in its evaluation it detects from 2 to 61 leaks in 6 programs under 3 different cache settings and identifies false positives in 2 programs reported by recent work (Guo et al., 2019). These analyses reinforce a broad interpretation: cache-resident execution is not only a runtime organization but also a formal object of reasoning.

The main tension across the literature is that making residency explicit reveals competing obligations. Protected residency can block active work, as in resident KV claims; full-state capsules improve TTFT but are explicitly not a replacement for throughput-oriented KV-cache serving; secure chunking and subcaches suppress cross-domain interference but may leave occupancy channels or reduce effective capacity; and strong decorrelation mechanisms such as RaS+ impose substantial overhead as windows grow (Stepanek, 22 May 2026, Su, 18 Jun 2026, Dessouky et al., 2021, Dessouky et al., 2019, Hu et al., 2023). This suggests a unifying interpretation: a cache-resident execution model is most useful when the runtime, architecture, or analysis no longer treats cache residency as incidental, but instead defines a resident object, a policy for maintaining it, and an explicit account of what happens when that policy collides with capacity, concurrency, or security constraints.