InstCache: Caching Strategies in Computing

• InstCache is a multi-faceted caching framework that employs predictive instruction-level and test-time memory techniques to enhance throughput, accuracy, and energy efficiency in deep learning and microarchitecture.
• Its methodologies span offline NLL-based pre-population in LLM serving, key-value blending in image classifiers, and two-level temporal reuse predictors in processor front-ends.
• Empirical results show up to a 51% hit rate in LLM workloads, significant error reductions in image classification, and 18% miss reduction in hardware caches, underscoring its practical impact.

InstCache is a term encompassing multiple research lines on caching strategies in advanced computing contexts, most prominently in deep learning inference and processor instruction delivery. The InstCache family includes (1) a predictive instruction-level cache for LLM serving, (2) a “simple cache model” for augmenting image classifier test-time performance, and (3) variants in processor frontends, notably the Admission-Controlled Instruction Cache (ACIC). These mechanisms exploit the predictability and temporal locality in input distributions to improve throughput, accuracy, energy efficiency, and adversarial robustness.

1. InstCache in LLM Serving

InstCache for LLM serving refers to a predictive, instruction-level cache positioned before the LLM inference stack (e.g., vLLM, SGLang), designed to answer frequently repeated (“hot”) instructions directly from CPU memory, drastically reducing GPU calls and latency (Zou et al., 2024).

The key architectural insight is that most user instructions submitted to public LLM services are short, highly repetitive, and can be predicted with high likelihood by an “instruction-aligned” (fine-tuned) LLM. The InstCache workflow is divided into two phases: (a) an offline pre-population phase, and (b) a deployment phase for live serving integration.

Pre-population Phase

An instruction-aligned LLM, fine-tuned on the dominant input distribution, enumerates all instructions whose negative log-likelihood (NLL) falls below a threshold $\sigma$. The process uses a breadth-first tree search over token sequences; branches whose partial NLL ever exceeds $\sigma$ are pruned, controlling coverage and memory footprint. When an <eos> token is reached and the cumulative NLL is acceptable, the instruction is materialized along with its LLM-generated answer (typically via greedy or top-K decode).

Mathematically, letting instruction $s = (t_1, t_2, ..., t_n)$, the cumulative NLL is:

$$L(s) = -\sum_{i=1}^n \log P(t_i\,|\,t_1 \ldots t_{i-1})$$

Instructions with $L(s) \leq \sigma$ are selected for caching.

Deployment Phase

The resulting (instruction, answer) pairs are stored in a compact hash table:

• Keys: 64- or 128-bit fingerprints (e.g., CityHash) of the UTF-8 instruction string.
• Values: Pointers to answer token buffers.
• Open addressing (linear or Robin Hood hashing) with load factor $\alpha \approx 0.7$ is used for predictable probe counts and low memory overhead.

At inference:

• On a cache hit, the answer is immediately returned from RAM.
• On a miss, the request is forwarded to the standard LLM serving process.

Empirical results show InstCache achieves up to 51.34% hit rate on the LMSys dataset with only 4.5 GB of CPU RAM and enables $2 \times$ throughput by bypassing up to half of GPU invocations (Zou et al., 2024). This is markedly more memory- and latency-efficient than semantic or key-value caches at token or embedding level.

Cache hit rate and population size scale predictably via the empirical CDF $F_\mathcal{N}(\sigma)$ of NLL:

• $\text{HitRate}(\sigma) = F_\mathcal{N}(\sigma)$.
• Number of instructions: $N(\sigma) \approx a (e^{b\sigma} - 1)$ for empirical parameters $a, b$.

Limitations include degraded hit rate if the user instruction distribution shifts or if the predictor LLM is misaligned. Very novel or long instructions yield low cache coverage but comprise a small query fraction.

2. InstCache in Neural Image Recognition

In image recognition, InstCache refers to a test-time cache memory mechanism extending any pre-trained classifier without gradient updates or data augmentation (Orhan, 2018). The design leverages the observation that layers immediately preceding the output softmax in deep networks contain extractable, class-discriminative information not present at the logit or output layer itself.

Model and Workflow

At test time:

1. The input $x$ is forward-propagated to obtain both:
   • the final network posterior $$p_\text{net}(y|x) = \text{softmax}(f_{\mathrm{out}}(x))$$
   • intermediate feature activations $\phi(x)$ from selected non-output layers.
2. The normalized feature $\phi(x)$ is compared (dot product) to a cache of stored training-set keys via a sharpness kernel:

$$\sigma_i(x) = \exp\big[ \beta\, k_\text{test}^\top k_i \big]$$

where $\beta$ is a tunable bandwidth.

1. The cache-distribution is

$$p_\text{mem}(y|x) = \frac{\sum_{i=1}^{K} \sigma_i(x) v_i}{\sum_{i=1}^{K} \sigma_i(x)}$$

with $v_i$ the one-hot value for the class label.

2. The final output is an affine blend:

$$p_\text{final}(y|x) = (1 - \lambda) p_\text{net}(y|x) + \lambda p_\text{mem}(y|x)$$

Only $\beta, \lambda$ require tuning on a validation set.

Construction and Query Algorithms

Cache construction involves storing $(k_i, v_i)$ pairs for each (optionally subsampled) training example, where $k_i$ is the concatenated, normalized activation from selected high-level layers, and $v_i$ the one-hot label.

Query/prediction for a test input follows the computation above, using layer activations, dot products, and normalization. For large $K$, approximate nearest neighbor acceleration (e.g., FAISS, Annoy, HNSW), subsampling, or key-dimension reduction are recommended (Orhan, 2018).

Empirical Outcomes

• Significant test error reductions: e.g., ResNet32 on CIFAR-100: 32.94% baseline $\rightarrow$ 29.36% with cache ($\sim3.6\%$ absolute gain).
• Improved adversarial robustness: e.g., FGSM-attacked ResNet32 baseline 5.2% $\rightarrow$ Cache3 72.8%.
• Regularization evidenced by decreased Jacobian norms and smoother predictive landscapes.

Keys from layers near—but not including—the output layer, or concatenations of several high-level layers, yield the greatest performance gains. The cache is a “light-touch” add-on, requiring no retraining or fine-tuning (Orhan, 2018).

3. Instruction Cache Design: Admission-Controlled Instruction Cache (ACIC)

In microarchitectural contexts, InstCache may colloquially refer to instruction caches optimized for front-end delivery of executable code, with ACIC as a key example (Wang et al., 2022).

Motivation

Datacenter applications commonly exhibit bursty instruction block accesses, including short-term spatial and temporal locality intermixed, which conventional LRU replacement fails to optimize. The ideal OPT policy is unattainable in hardware, but ACIC aims to bridge over half of the LRU-OPT gap.

ACIC Architecture

ACIC combines:

• A small, fully-associative “i-Filter” (16 entries, ~1.123 KB) that isolates spatial bursts from subsequent temporal reuses.
• A two-level temporal locality predictor: Comparison Status Holding Registers (CSHR), History Register Table (HRT), and Pattern Table (PT) predict whether a recently evicted i-Filter victim (V) will be reused sooner than the L1i replacement candidate (C).

On i-Filter eviction: The predictor hashes tags and consults the HRT/PT for pattern-based admission vs. bypassing into the main L1i. On subsequent fetches, outcome feedback loops reinforce or weaken future predictions. Hardware overhead is ~2.67KB per 32KB L1i (about 8%).

Performance Findings

• ACIC achieves 1.0223$\times$ geomean speedup over LRU+FDP and reduces i-cache misses by 18.14%, bridging 55.85% of the theoretical gap to OPT.
• Outperforms GHRP, SHiP, Hawkeye/Harmony, victim caches, and simple L1i size increases.
• Per-application, MPKI reductions of 24–28% translate to single-digit percent speedups, substantial for datacenter workloads.
• Energy analysis shows net savings due to reduced memory system activity.

The predictor is most effective in “useful” regions of short reuse, and design sensitivity to i-Filter and predictor table sizes is empirically characterized (Wang et al., 2022).

4. Comparative Summary of InstCache Paradigms

Domain	Design Principle	Core Mechanism	Noted Efficiency Gains
LLM Serving (Zou et al., 2024)	Predictive instruction cache	NLL-guided pre-population, hash table	51.34% hit-rate at 4.5 GB RAM, 2$×$ speedup
Image Classification (Orhan, 2018)	Test-time key/value augment	High-level feature–label memory, blend	$0.7$–$3.6\%$ absolute error reduction, strong adversarial robustness
CPU Frontend (Wang et al., 2022)	Spatio-temporal access separation	i-Filter, two-level reuse predictor	18.14% miss reduction, bridges 56% of LRU-OPT gap

5. Practical Considerations and Limitations

• Scalability: LLM InstCache’s memory cost is precisely tunable via $\sigma$; 4.25M entries require 4.5GB RAM.
• Hardware constraints: ACIC’s area/energy impact is minimal compared to performance benefits in server processors.
• Adaptivity: Both LLM and hardware InstCache mechanisms require periodic re-alignment or parameter tuning to cope with distributional shift or evolving workloads.
• Miss/novelty handling: Rare or long-tail inputs, especially in LLM serving, bypass the cache gracefully at the expense of GPU invocation and are logged to enable subsequent model adaptation.
• Deployment: LLM InstCache can be sharded via Redis or similar, is easily integrated in cloud scaling, and supports rapid incremental re-population.
• Robustness: The cache-augmented classifier approach inherits benefits of smooth predictive behavior and resistance to adversarial attacks without sacrificing accuracy (Orhan, 2018).

6. Research Trajectories and Ongoing Work

Open directions include adaptive online re-population for InstCache in LLM serving to address rapid distributional drift, hardware prefetch-awareness for ACIC, multicore/hierarchical cache coordination, and automated selection/concatenation of feature layers for neural test-time caches. The quantifiable trade-off between cache size, hit-rate, and system throughput defined by the InstCache framework has motivated cross-domain adoption, from deep learning to microarchitecture, and remains a focus for future design of efficient computational inference infrastructure.