Smarter and Cheaper at Once: Byte-Exact KV-Cache Grafting Turns a Frozen Small Model into a Verified-Knowledge Flywheel

Abstract: We report a way to make a frozen small LLM both more capable and dramatically cheaper at once, without changing any weights. Verified knowledge is deposited once as a byte-exact key-value (KV) state artifact and later restored, by graft, into a fresh inference context. The restore is bit-exact: under a pinned deterministic configuration, the grafted logits are byte-for-byte identical to a fresh computation (SHA-256 equality), with zero KL divergence and 100% argmax agreement over fifty samples. We show that own-position graft is the unique numerically exact operating point on a model with floating-point rotary encoding, and we verify byte-exactness on two model scales (12B, 31B) and two GPU targets, one through a pre-registered replay. On AIME 2025, a frozen Gemma-4-12B moves from 80.0% to 93.3% once a verified solution library is grafted, above its own 77.5% and its 31B sibling's 89.2% published anchors. On the recurring case, eight problems the base model never solves within a 401,026-token budget are answered from cached verified solutions in 61 total decode tokens, a factor of 6,574 fewer tokens and about 8,700x less energy; the capability claim proper rests on held-out transfer (7 of 7 at 31B). The same byte-exact store widens usable context from 32,768 to 2,854,766 tokens at zero extra accelerator memory, and moves byte-identical between machines of the same architecture. We describe the system at the behavior level; the engine is proprietary, and every reported number is backed by committed input and output hashes so the scoring can be re-checked without it.

• The paper introduces byte-exact KV cache grafting that enables inference-time learning in frozen LLMs, improving benchmark performance from 80.0% to 93.3%.
• It employs the Taliesin engine and Galahad protocol to deterministically cache and restore KV states without altering model weights or incurring extra memory costs.
• The approach yields dramatic token and energy savings, extends the context window, and verifies exactness via SHA-256 hash matching and zero KL divergence.

Byte-Exact KV-Cache Grafting for Inference-Time Learning in Frozen LLMs

Introduction

This work investigates a paradigm-shifting mechanism for enhancing both the intelligence and inference efficiency of frozen, small-scale LLMs. Eschewing model re-training or architectural modifications, the paper introduces byte-exact key-value (KV) cache grafting as a deterministic, verifiable means to transfer and persist knowledge across inference contexts. The approach deposits verified solutions to challenging tasks as KV states—captured precisely at the byte level—and restores them during subsequent inference by grafting. The mechanism neither modifies model weights nor requires additional accelerator memory, and the process upholds the strongest possible determinism guarantees, with empirical confirmation via SHA-256 hash equality and zero KL divergence between fresh and grafted computations.

Mechanism Design and Exactness

Central to the method are two system components: Taliesin (the byte-exact KV grafting engine) and Galahad (the verify-then-cache flywheel learning protocol). When the model solves a problem (via inference-time search plus external verification), its resultant KV state is serialized and stored. Upon recurrence or closely related query (transfer), the relevant state is restored in a way that is bit-identical to a fresh computation. The system mandates a rigorous notion of exactness: byte-wise logit outputs (via SHA-256), $D_{\mathrm{KL}} = 0$ between fresh and grafted softmax distributions, and 100% argmax match. Determinism is pinned at the compute stack (compiler, cuBLAS, environmental controls), ensuring reproducible cache restoration.

A significant theoretical and empirical observation is that byte-exactness is achievable exclusively through own-position grafting—restoring the cached KV state at precisely the sequence positions at which it was deposited. Positional encodings (specifically, rotary embeddings) induce floating-point position-sensitivity; absolute positional re-alignment incurs inherent, irreducible discrepancies at the $10^{-3}$--$10^{-2}$ KL regime due to non-associativity and rounding in IEEE 754 arithmetic.

Empirical Results

Capability and Cost Benefits

Evaluation on the AIME 2025 benchmark (post-pretraining for Gemma-4-12B) demonstrates substantial gains. The base frozen 12B model scored 80.0% in its best local configuration. Grafting a library of externally-verified KV states elevates performance to 93.3%—exceeding both the vendor’s official 12B (77.5%) and 31B (89.2%) model-card scores. The setup targets only the subset of problems unsolved by the base model, leveraging retrieval of case-specific solution blocks.

For recurrence (repeat queries), cost reduction is dramatic: tasks that were previously unsolvable by the base model (even with an aggregate 401,026 decode-token sampling budget) are answered perfectly from disk-cached state in 61 decode tokens. This reflects a 6,574-fold reduction in token budget and 3,000–8,700$\times$ lower energy consumption for those cases, with zero additional accelerator memory or re-computation overhead.

The system also enables a massive effective context window extension. The archived KV store can hold $>2.8$ million tokens—an 87$\times$ expansion of the active memory window—with access latency that does not increase with depth due to blockwise, targeted restoration.

Generalization and Transfer

Held-out transfer experiments (new problems matched structurally but distinct in content) show that the grafted method generalizes if the cached solution is parameteric; brittle, non-parametric solutions (with hard-coded constants) fail to transfer, thus delineating the effective regime of the approach. In the scale-up, a frozen Gemma-4-31B achieves 100% (7/7) transfer generalization on AIME variants, outperforming the base model's vendor-reported anchor.

Systems and Portability

Distributed operation is facilitated by the architecture: KV blocks are portable, byte-exact assets within a compute architecture. Knowledge deposited on one server is a zero-cost import on another machine with matching compute stack. However, byte-exact guarantees are architecture- and configuration-specific due to floating-point accumulation idiosyncrasies.

Key systemic questions—misrouting and disk-paging impact—are addressed through direct measurement. Block misses route to abstention rather than incorrect answers, and restore-from-disk cost grows sublinearly with block size, preserving the overall speedup even for large block stores.

Negative Results

The method brings no benefit when the model already solves the task class at minimal cost. Additionally, forcibly installing a method the model already executes well can degrade both efficiency and performance. LiveBench recurrence is weaker for complex multi-task blocks, attributed to retrieval policy limitations rather than cache mechanism defects.

Theoretical and Practical Implications

This work concretely demonstrates that verified, byte-exact KV cache grafting enables inference-time iterative learning in LLMs, without modifying weights or increasing marginal serving cost. The result is a decoupling of capability gains from compute scale-up: new knowledge is integrated once, operationalized at negligible cost, and transferred across servers. This paradigm threatens the dominant model-scaling narrative by enabling capability expansion and inference cost reduction in small, frozen models.

Theoretical implications include a clarified boundary for exact state reuse in positional-encoded transformer architectures: byte-exact restoration is only possible at original capture positions; all nontrivial re-basing is inherently lossy due to floating-point artifacts. Retrieval and composition regimes are formally characterized and empirically delimited.

Practically, this method offers a compelling path to economic and ecological efficiency in LLM deployment. The measured energy and token savings are significant, suggesting that inference-time learning via external, persistent, and byte-exact knowledge transfer is a viable alternative to retraining or scaling model parameters.

Future Directions

Areas warranting further investigation include:

• Automated routing with robust abstention and confidence-gating,
• Transfer to related tasks beyond parametric code adaptation,
• Comprehensive block composition and overlap resolution,
• Architecture-agnostic cache serialization, and
• High-concurrency, distributed serving regimes.

Further study is needed for multi-block and fine-grained retrieval strategies, as well as cross-architecture and heterogeneous deployment scenarios.

Conclusion

Byte-exact KV-cache grafting provides a deterministic, auditable, and compositionally robust substrate for inference-time knowledge injection in LLMs. By externally verifying and caching solutions into persistent KV artifacts, models can permanently encode new skills without retraining or increasing accelerator consumption. The approach demonstrably lifts the performance ceiling and slashes recurring inference costs, reframing the economics of LLM advancement. Given the environmental and operational cost constraints in state-of-the-art deployment, byte-exact cache grafting constitutes a strong empirical alternative to relentless upscaling and retraining.