LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer Inference (2512.16843v1)

Abstract: Transformer-based LLMs have achieved remarkable performance across a wide range of tasks, yet their high inference latency poses a significant challenge for real-timeand large-scale deployment. While existing caching mechanisms,such as token-level key-value caches, offer speedups in autore-gressive decoding, they are limited in scope and applicability. In this paper, we present LLMCache, a novel layer-wise caching framework that accelerates transformer inference by reusing intermediate activations based on semantic similarity of input sequences. Unlike prior work, LLMCache is model-agnostic,operates across both encoder and decoder architectures, and supports caching at arbitrary transformer layers. We introduce a lightweight fingerprinting mechanism for matching seman-tically similar inputs and propose adaptive eviction strategies to manage cache staleness. Experiments on BERT and GPT-2 across SQuAD, WikiText-103, and OpenBookQA show up to 3.1 X speedup in inference time with <0.5% accuracy degradation. Our results highlight LLMCache as a practical and general-purpose solution for optimizing transformer inference in real-world applications

• The paper introduces a layer-wise caching mechanism that reuses intermediate transformer representations to reduce inference latency by up to 3.1× with less than 0.5% accuracy drop.
• It outlines a modular architecture employing input fingerprinting, cache lookup, and dynamic eviction strategies to balance speed gains with memory overhead.
• Experimental results on BERT-base, DistilBERT, and GPT-2 demonstrate significant latency reductions on real-world NLP tasks while maintaining robust task performance.

LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer Inference

Introduction

The paper "LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer Inference" (2512.16843) presents a framework for optimizing transformer inference latency by introducing a model-agnostic, layer-wise caching mechanism. Conventional transformer-based LLMs such as BERT and GPT-2, although performant across diverse NLP tasks, suffer from significant computational overhead during inference, particularly in scenarios with highly similar or repetitive inputs. Existing methods including quantization, pruning, and token-level key-value (KV) caching either introduce non-trivial trade-offs or are overly restrictive in architectural scope. LLMCache addresses these limitations by enabling the caching and reuse of intermediate hidden representations at arbitrary transformer layers based on semantic similarity metrics, yielding substantial speedups with negligible accuracy degradation.

System Architecture

LLMCache is modular and compatible with both encoder and decoder transformer architectures. At its core, each layer is augmented with its own cache bank and lookup logic to facilitate the storage and retrieval of intermediate activations. The system comprises several dedicated modules: a lightweight input fingerprint generator, layer-wise cache banks, a cache matching and lookup engine, a layer execution manager, and a refresh/replacement controller. The workflow begins by generating a semantic fingerprint for each input sequence, followed by a cache lookup at every transformer layer. On cache hit, stored activations are reused, bypassing full layer computation; otherwise, computation proceeds as normal and the result is cached for potential future reuse.

Figure 1: High-level LLMCache system architecture illustrating the addition of layer-local cache banks and decision logic for caching.

The inference flow explicitly involves fingerprinting, cache lookup, cache reuse decision logic, and fallback to computation when necessary.

Figure 2: Inference flow in LLMCache, showcasing the interaction between fingerprint generation, lookup, reuse, and fallback.

Methodology

LLMCache’s methodology formalizes layer-wise caching based on semantic similarity. Each input sequence is mapped to a fixed-length fingerprint using aggregation and dimensionality reduction techniques (e.g., SimHash, MinHash, or PCA projections). Each transformer layer maintains an independent cache bank mapping fingerprints to hidden states. During inference, layer execution is conditioned on cache lookup: if a similar fingerprint is present (according to a configurable threshold $\tau$), the cached hidden state is reused; otherwise, the output is freshly computed and stored.

To manage cache staleness and memory usage, LLMCache implements LRU, staleness-aware, and divergence-based eviction policies—ensuring cache freshness and bounded memory requirements. The similarity threshold $\tau$ and cache size are system hyperparameters that balance reuse frequency, memory overhead, and inference fidelity.

Experimental Results

LLMCache was extensively evaluated using BERT-base, DistilBERT, and GPT-2 on WikiText-103, SQuAD v2, and OpenBookQA. The primary metrics include inference latency, cache hit rate, accuracy, and memory overhead.

Inference Latency Reduction

When compared to vanilla inference (NoCache) and traditional KV-caching, LLMCache delivers consistent and marked speedups. For instance, on BERT-base, a 2.4× reduction is observed, and on GPT-2, LLMCache surpasses even KV-caching due to deeper granularity in reuse.

Cache Hit Rate by Layer

Analysis shows cache hit rates are higher in lower and mid transformer layers, hitting up to 92%, with reduced rates in upper layers attributable to greater sensitivity to semantic drift.

Figure 3: Cache hit rates across transformer layers in GPT-2 on WikiText, indicating high reuse in lower layers.

Accuracy and Task Robustness

Across all tested datasets, LLMCache achieves $<$0.5% relative reduction in accuracy, demonstrating that layer-wise reuse does not substantively degrade task performance. LLMCache outperforms document-level caching methods by virtue of its finer granularity and online cache management.

Memory-Performance Trade-off

As cache size increases, cache hit rates improve sub-linearly, resulting in modest memory overhead increases for significant speedup benefits. The design ensures system-level tunability to control the balance between latency savings and memory footprint.

Figure 4: Relationship between memory overhead and cache hit rate in BERT-base, highlighting scalable efficiency.

Sensitivity to Similarity Thresholds

Lowering $\tau$ increases cache hits and speedup at the cost of potential output drift; optimal choices of $\tau$ are model- and domain-dependent but typically fall in the range [0.82, 0.88].

Figure 5: Impact of varying cache similarity thresholds on hit rate and downstream performance.

Discussion

LLMCache is particularly effective in deployment scenarios characterized by high template reuse or semantic overlap across requests, including conversational agents, document pipelines, and RAG environments. Its model-agnostic nature ensures applicability across both encoder and decoder stacks without retraining or architectural changes. However, cache efficacy diminishes with unpredictable, high-entropy inputs and in very deep architectures where memory may become a bottleneck.

In contrast to token-level KV-caching—limited to autoregressive models—and document-level caching—constrained by passage granularity—LLMCache offers a unified, flexible solution for layer-wise inference optimization.

Critical system trade-offs involve setting the similarity threshold, cache size, and selectively choosing which layers to cache, allowing for fine-tuning of the deployment according to workload and hardware constraints. Potential extensions include dynamic threshold adaptation, distributed cache sharing for multi-node systems, and learned fingerprinting for improved semantically-aware matching.

Conclusion

LLMCache introduces a system-level paradigm for accelerating transformer inference via semantically-matched, layer-wise activation reuse. The framework achieves up to 3.1× latency reduction with minor ($<$0.5%) accuracy impact, without requiring model retraining or architectural modification. Its design enables broad applicability and underlines the utility of input-aware caching as a complement to traditional model compression and pruning. LLMCache provides a foundation for future research in scalable, high-throughput inference in real-world LLM deployments.