RegCache: Diverse Mechanisms for Constrained Caching
- RegCache is a family of caching mechanisms that manage constrained state across diverse computing applications by tailoring the cached object and optimization target to specific runtime budgets.
- It encompasses techniques such as regression-based KV compression in long-context LLMs, prefix-based cache strategies for vision encoder quantization, agent-level RAG caching, no-regret online optimization, KV-fusion for RAG, and GPU register-file caching.
- These mechanisms enhance performance by reducing memory and energy overhead while improving inference speed, accuracy, and robustness across varied systems with minimal computational cost.
RegCache is a label that has been used for multiple technically distinct mechanisms in recent arXiv literature. Depending on context, it may denote regression-based KV-cache compression for long-context LLMs, a prefix-based outlier-mitigation method for post-training quantization of vision encoders, an agent-level RAG cache policy, an Online Linear Optimization formulation of cache control, a KV-fusion framework for position-invariant RAG, or a register-file cache in GPU microarchitecture. The common thread is constrained-state management under a fixed budget, but the object being cached ranges from Transformer keys and values to passages, fractional file allocations, and warp-local registers (Peng et al., 29 May 2026, Kim et al., 6 Oct 2025, Lin et al., 4 Nov 2025, Paschos et al., 2019, Oh et al., 13 Jan 2025, Shoushtary et al., 2023).
1. Terminological scope and recurring structure
In recent usage, “RegCache” does not designate a single canonical algorithm. Instead, it names a set of domain-specific mechanisms whose technical content depends on the underlying memory object, optimization target, and runtime constraints. This suggests that the term functions as a family label rather than a uniquely standardized abstraction (Peng et al., 29 May 2026, Kim et al., 6 Oct 2025, Lin et al., 4 Nov 2025, Paschos et al., 2019, Oh et al., 13 Jan 2025, Shoushtary et al., 2023).
| Usage of “RegCache” | Cached object | Core mechanism |
|---|---|---|
| Regression-based KV compression | Retained KV cache in long-context LLM inference | Ridge-regression-based merge steps minimizing discrepancy between compressed-cache and full-cache attention outputs |
| Vision-encoder PTQ | Prefix KV entries and intermediate tokens | Middle-layer prefixing and token deletion |
| Agent RAG cache | Small, high-value subset of passages | DRF, hubness, and memory-penalized priority |
| OLO-based cache control | Fractional file placements | OGA and BSA with no-regret guarantees |
| Position-invariant RAG | Per-passage KV caches | Parallel Key-Value Cache Fusion |
| GPU register-file cache | Warp-local register operands | Compiler-assisted CCU-based RF caching |
A recurrent misconception is that “cache” has a uniform systems meaning across these works. The literature shows otherwise. In long-context inference it refers to Transformer attention state; in vision PTQ it refers to precomputed KV entries used to reshape activation distributions; in agent RAG it denotes a compact retrieved corpus; in OLO it is a decision variable over file placements; in KV-fusion it is a fused memory built from per-passage caches; and in GPUs it is an RF-adjacent microarchitectural buffer.
2. Regression-based KV-cache compression in long-context LLMs
In long-context LLM inference, RegCache denotes regression-based KV-cache compression, and GRKV is its concrete training-free instantiation. The setting is standard autoregressive attention with per-head keys and values and , and attention output
The motivating problem is that KV caches grow linearly with context length and head count, becoming the dominant memory bottleneck in LLM inference. Modern eviction methods often use span-based retention because preserving contiguous spans is empirically effective and better preserves semantic coherence. However, when span-based retention is followed by local merging, evicted tokens are funneled predominantly into span-boundary carriers, which concentrates merge load, amplifies over-merging, and increases information loss. GRKV replaces local merge heuristics with a global regression objective that directly minimizes the discrepancy between compressed-cache and full-cache attention outputs over a surrogate query window (Peng et al., 29 May 2026).
The core optimization uses the retained set and evicted set , with updated retained values . For a query window , GRKV defines
and minimizes
With keys fixed, the value update becomes a ridge regression:
where 0 and 1. Because 2, the paper emphasizes the dual form
3
This “RegCache” view treats all retained tokens as a global carrier set, rather than assigning evicted tokens to a small number of local carriers.
GRKV also permits a limited key correction. With values fixed, it linearizes 4 around the current keys and solves a local ridge problem using Jacobian-vector products. In practice, it keeps a subset of retained tokens fixed: sink tokens, surrogate-window queries, and the top 5 retained tokens by attention, with 6 by default. Hyperparameters are explicitly specified: 7 on Llama-3.1-8B, 8 on Mistral-7B and Qwen3-14B, 9, and 0 alternating step.
Empirically, GRKV is reported as the only merging method that consistently improves overall performance with span-based retention while keeping overhead minimal. On LongBench at 10% budget, Llama-3.1-8B improves from 33.96 to 34.58 over SnapKV and from 36.00 to 36.58 over CriticalKV; Mistral-7B improves from 33.12 to 33.75 over SnapKV and from 33.69 to 34.30 over CriticalKV. On RULER at 10% budget and 16K context, Llama-3.1-8B improves from 27.44 to 29.09 over SnapKV and from 40.83 to 41.51 over CriticalKV. At 64K, decoding latency is reduced to approximately 37.75 ms/token versus approximately 64.84 ms/token for full cache, and at 96K SnapKV+GRKV TTFT is approximately 49.71 s, close to D2O at approximately 48.20 s and far below AsymKV at approximately 788.49 s. Limitations remain: extremely high compression reduces the feasible reconstruction space, the key step relies on local linearization of softmax, and tasks with highly non-stationary query distributions may require adaptive windows or multi-window aggregation.
3. Prefix-register RegCache for post-training quantization of vision encoders
A separate usage of RegCache refers to a training-free, prefix-based outlier mitigation method for post-training activation quantization of transformer vision encoders such as CLIP, SigLIP, and DINOv2. The motivating observation is that ViT encoders exhibit heavy-tailed activation distributions, with a small subset of tokens or channels in specific layers reaching extremely large magnitudes. These outliers expand the dynamic range required for per-tensor uniform quantizers, increasing rounding and clipping error and causing large accuracy drops even at W8A8. RegCache addresses this by introducing semantically meaningless yet outlier-prone prefix tokens, called registers, at selected intermediate blocks, and by deleting internally emerging sink tokens before the most quantization-sensitive layer (Kim et al., 6 Oct 2025).
The method is built around two empirical findings. First, outliers in ViTs emerge primarily in middle blocks rather than from the input. Second, token deletion is only beneficial when external registers are present to stabilize computation. The paper therefore uses middle-layer prefixing instead of input-level prefixing. Registers are inserted as KV-cache entries in attention blocks so that they attract attention and accumulate magnitude themselves, protecting image tokens from becoming outliers. The candidate set is constructed from reference images by selecting tokens with largest 1 norm at the sensitive layer:
2
Candidate selection then searches over token choice and repetition count,
3
and token deletion at inference removes the top-4 image tokens by 5 norm at the input to 6.
The algorithmic pipeline is explicit. It first identifies the quantization-sensitive layer 7 by quantizing each candidate layer in isolation to W8A8 and measuring zero-shot accuracy on ImageNet-1k. It then curates candidates from 50,000 random training images from ImageNet-1k, selecting top-8 by 9 norm with 0 per block, optionally from the sensitive block and up to three preceding blocks. For each candidate token and repetition count 1, it precomputes KV caches from a few blocks before 2 to the end using the unquantized model, inserts those caches into the quantized model, evaluates reference accuracy, and chooses 3. At deployment, CLIP, SigLIP, and SigLIP2 prefix from the searched middle block through the final block, whereas DINOv2 prefixes only at the searched block. The final step deletes top-4 sinks at 5.
The reported gains are large. For ImageNet-1k zero-shot classification under naive W8A8 quantization, CLIP-B/16 improves from 34.01 to 61.44, OpenCLIP-B/16 from 46.12 to 67.14, SigLIP-B/16 from 69.71 to 74.42, SigLIP2-B/16 from 26.04 to 68.65, and DINOv2-B/14 from 19.20 to 22.07. With PTQ4ViT, RepQ-ViT, and NoisyQuant at W8A8 and W6A6, RegCache consistently improves accuracy across models. On MS-COCO zero-shot retrieval, CLIP-B/16 naive 6 R@1 rises from 22.76 to 47.78 and 7 R@1 from 14.08 to 29.47. Outlier reduction is targeted: the max token norm at 8 under W8A8 drops for CLIP from 61.17 to 15.30, OpenCLIP from 122.99 to 12.38, SigLIP from 78.09 to 12.15, and SigLIP2 from 244.78 to 30.45, while the average norm of other tokens remains essentially unchanged. The paper also states that the “Cache” in RegCache refers to reusing precomputed KV entries across inputs, not expensive dynamic computation. Overhead is negligible in GFLOPs and runtime memory, but the method does require a search over candidate blocks, repetition counts, and deletion budgets, and self-supervised encoders such as DINOv2 show weaker outlier reduction.
4. Agent-level RegCache for RAG systems
In agent RAG, the relevant usage is ARC, the Agent RAG Cache Mechanism, which is explicitly described as a regulated, agent-centric cache layer. The cache stores a small, high-value subset of passages tailored to an individual agent’s query distribution. ARC differs from naive query or result caching and from corpus-level pruning by combining a query-driven statistic with embedding-space geometry. Specifically, it synthesizes a rank- and distance-weighted frequency, DRF, learned online from queries, with a query-agnostic hubness centrality computed in the embedding space, and then penalizes large items by weight. The goal is to maximize expected has-answer rate over future queries subject to a memory budget (Lin et al., 4 Nov 2025).
The formal setup uses an external corpus 9, an agent-specific cache 0, and a capacity constraint 1. For query 2 and item 3, ARC defines
4
and aggregates over historical queries to obtain
5
Hubness is
6
The eviction priority is then
7
Items with lower Priority are more likely to be evicted. ARC also includes a drift-aware refresh rule: it first retrieves from the cache, and if the average distance across the cache’s top-8 exceeds a threshold 9, it escalates to full-corpus retrieval and updates the cache.
The implementation uses FAISS IndexFlatIP, with embeddings from bge-small-en and LLM-embedder, top-0, and a cache capacity of 3.0 MB. The reported corpus is English 2023 Wikipedia, with more than 6.4M documents chunked into more than 14M passages. ARC achieves 0.015% of the original storage by bytes. On bge-small-en, has-answer rate reaches 62.63 on MMLU, 71.18 on AdversarialQA, and 79.80 on SQuAD, outperforming LFU, FIFO, GPTCache, Proximity, and an ARC variant without hubness. AMAT is 0.556 s on MMLU, 0.377 s on AdversarialQA, and 0.269 s on SQuAD, corresponding to approximately 80% reduction versus 1.313 s full-index retrieval on SQuAD. The ablation without hubness remains strong, indicating that DRF alone is already informative, while hubness yields up to approximately 3% further improvement in has-answer rate and additional AMAT reductions. The design is annotation-free and handles cold start and moderate concept drift through the 1-based escalation rule, though extreme drift may require periodic hubness recomputation and retuning of 2.
5. No-regret online caching and the OLO interpretation
A different line of work uses RegCache to denote a regret-minimizing view of caching based on Online Linear Optimization. In the single-cache model, files are indexed by 3, the cache has capacity 4, the request at slot 5 is a one-hot vector 6, and the cache configuration is a fractional allocation 7 satisfying 8. The per-round utility is linear:
9
Regret is defined relative to the best static configuration in hindsight, with no modeling assumptions on the request sequence. The principal result is that Online Gradient Ascent matches a problem-specific lower bound, yielding a universally optimal no-regret caching policy up to constants (Paschos et al., 2019).
The paper proves that classical heuristics incur linear regret in the worst case:
0
By contrast, OGA updates the fractional cache configuration through projected ascent,
1
where 2 is Euclidean projection onto the capped simplex. With step-size
3
OGA satisfies
4
For the common case 5 with uniform hit weights, the regret scales as 6, and the paper states that OGA is universally optimal up to constants.
The same framework extends to a network of caches arranged as a bipartite graph. There, the feasible set is a product of capped simplexes, the per-slot utility is a concave function induced by optimal routing, and the Bipartite Subgradient Algorithm updates the fractional placements via projected subgradient ascent. For equal capacities 7, BSA satisfies
8
The practical significance is model-free robustness: the policies adapt to popularity shifts and adversarial request sequences without assuming stationarity. In experiments with a single cache, OGA consistently tracks or outperforms the better of LRU and LFU and beats the weaker heuristic by up to 16% and 20% in different scenarios. In a networked scenario with 3 caches and 4 user locations, BSA outperforms lazy-LRU by 45.8% in average utility. Limitations remain around fractional relaxation, 9 projection cost, single-request-per-slot analysis, and extending the guarantees to richer topologies or explicit expiry and invalidation.
6. RegCache as parallel KV-fusion for position-invariant RAG
In position-invariant RAG, RegCache refers to the cache-fusion framework introduced as Key-Value Fusion. The motivation is the Lost-in-the-Middle phenomenon in decoder-only LLMs, where relevant information placed in the middle of a long concatenated context is often under-attended because of a U-shaped positional attention bias. KV-Fusion removes this dependence on global passage order by processing retrieved passages independently, assigning each passage the same local positional indices, extracting per-passage KV caches in parallel, and then concatenating those caches into a single fused memory for a trainable answer decoder (Oh et al., 13 Jan 2025).
The architecture uses two decoders. A frozen prefill decoder 0 runs on each passage 1 independently, using identical local positions 2 within every passage, and extracts per-layer caches
3
These are fused by concatenation along the token axis:
4
A second decoder, initialized from the same base model but trained for the task, consumes short target tokens at positions 5 and attends over the fused cache:
6
Because passages are encoded with identical local positional embeddings and 7 and 8 are permuted jointly under passage reordering, the weighted sum is unchanged. This is the basis of the order-invariance claim.
The reported results are strong on Natural Questions, TriviaQA, and POPQA. Token-level consistency between a “gold-first” input and a shuffled input rises from 17.6, 14.4, and 8.2 for Llama3 to 99.6, 99.2, and 99.4 for KV-Llama3, and from 14.6, 10.6, and 4.6 for Llama3.1 to 99.6, 99.8, and 99.6 for KV-Llama3.1. Accuracy remains stable across top-9 retrieved passages: for KV-Llama3 on NQ, top-5/10/20/40 accuracy is 51.6/51.7/51.4/49.8; on TQA it is 67.5/68.8/69.3/69.3. The framework also outperforms PAM QA and RePlug across most dataset and top-0 settings, and the paper notes that KV-Fusion on the original top-40 without reranking can exceed reranked top-20 baselines. A common misconception is that simple concatenation alone yields invariance; the paper is explicit that identical local positional indices per passage are critical, and reintroducing global offsets would restore positional bias. Unlike several other RegCache usages, this method is not training-free: the answer decoder must be fine-tuned.
7. Register-file RegCache in GPU microarchitecture
In GPU architecture, RegCache refers to a lightweight, compiler-assisted register-file cache implemented by repurposing Operand Collector Units into Caching Collector Units. The design, introduced as Malekeh, targets modern sub-core GPUs with single-ported RF banks and tensor cores. Its objective is to reduce reads to large RF banks, alleviate bank and port conflicts, lower RF dynamic energy, and improve IPC without substantial area overhead (Shoushtary et al., 2023).
The microarchitecture is explicit. Each sub-core contains 2 CCUs, and each CCU has 8 fully associative entries. Every entry stores a 128B vector register, a 1-byte tag, a lock bit, a 1-bit reuse-distance hint, and 3 LRU priority bits. A 6-slot Operand Collector Table supports tensor-core instructions. The baseline per-SM RF is 256KB, while the added cache storage is 2KB per SM, approximately 0.78% overhead. Reads from RF banks arrive via a source port, destination values enter through a write-back port, and a status port communicates CCU metadata with the scheduler.
Operation is guided by compiler-derived reuse-distance metadata. During CCU allocation, source operands hit in the cache if their tags are already present; on misses, the replacement policy excludes locked entries, prefers replacing “far” reuse-distance entries chosen randomly, and falls back to LRU among “near” entries. Destination writes always update the authoritative RF banks, but only near-reuse writes are selectively allocated into the CCU. Scheduling is GTO-inspired and reuse-aware: if the previously issued warp is ready, it is selected; otherwise, ready warps with data resident in any CCU are preferred over ready warps without CCU data. A waiting mechanism, regulated by a global threshold STHLD and adapted online through a 10,000-cycle IPC-driven FSM, delays flushing CCUs containing near values so that reuse can be harvested before eviction.
Quantitatively, Malekeh reduces RF bank reads by 46.4% on average, RF dynamic energy by 28.3% on average, and improves IPC by 6.1%, while adding only 2KB of storage per core to the baseline RF of 256KB. RF dynamic energy reduction reaches up to 47.3% for rnn_bench_t2, and IPC improvement reaches up to 28.4% for rnn_bench_i2. The reported RF cache hit ratio averages near 46% across benchmarks. Compared with RFC, Software RFC, LTRF, and BOW, the design emphasizes minimal integration cost by reusing existing OCUs and avoiding two-level scheduler stalls or large private per-warp operand buffers. Its limitations are also stated explicitly: memory-bound phases may not realize IPC gains, 1-bit reuse-distance classification is approximate, 8-entry CT sizing is a tuned sweet spot rather than a universal optimum, and a single write port can serialize rare multi-destination cases.