Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute

Abstract: A LLM can only use the text that fits in its context window, and it recomputes its internal key-value (KV) state for a prompt every time the prompt is sent. We test a memory layer, the public package galahad-kv, that saves the KV state of each block of about 16,000 tokens to encrypted local NVMe disk and loads it back later, byte-exact, without recomputing it. We ran it on 50,000,000 tokens of real public text, served through vLLM on one NVIDIA H100, with Gemma 4 12B and Gemma 4 31B. Every block we probed was loaded back from the encrypted store with no recompute (100 of 100, at depths from 0 to 50M tokens) on both models. Loading a block was 2.8x to 4.3x faster than recomputing it and used 8.8x to 12.3x less GPU energy, and GPU memory stayed flat over the whole 50M-token stream. Asked about facts planted millions of tokens earlier, the 12B model gave the right answer 82 times out of 100 and the 31B model 98 times out of 100. Neither model made up an answer. The limits are as follows. This is reuse of stored state, not a wider attention window: one block is loaded at a time, and how well a question is answered depends on the model. Writing the memory is a one-time cost, and the store takes terabytes of local NVMe disk. We describe the test protocol, which is built to resist common ways of gaming long-context benchmarks, and give a single-GPU reproduction that uses public software and a free licence for the package.

• The paper presents `galahad-kv`, a KV-state layer that extends the context window of AI models to 50 million tokens using encrypted NVMe storage, while maintaining efficient GPU memory usage.
• The system successfully restored 100/100 blocks with no recomputation for both Gemma 4 12B and 31B models, demonstrating durability and reliability.
• The smaller 12B model recovered 82 of 100 planted facts, while the 31B model recovered 98 of 100, showcasing the dependence of memory retrieval effectiveness on model complexity.
• Galahad-KV avoids the session-based limitations of conventional page attention mechanisms: for text spanning 50 million syllables, users can sequentially retrieve desired sections of the encased corpus, leveraging persistent, disk-backed memory.

The paper presents galahad-kv, a persistent KV-state layer for served transformer models. Its central claim is that a model with a fixed native context window can support durable, arbitrarily extended access to previously processed material by storing each approximately 16,000-token KV block on encrypted local NVMe and restoring the block on demand. The mechanism does not enlarge the model’s attention window, modify its weights, or perform semantic retrieval. Instead, it eliminates repeated prefill for previously processed blocks and permits the model to reuse state that lies far outside its native context.

The evaluation uses Gemma 4 12B and Gemma 4 31B under vLLM on a single NVIDIA H100 80 GB. The authors report a 50-million-token corpus, comprising 3,125 blocks, with constant GPU memory during ingestion and subsequent block restoration. The strongest result is an engine-level guarantee within the tested protocol: all 100 probed blocks, including blocks at depths approaching 50 million tokens, were restored from encrypted storage with no recomputation. The model-level result is distinct: the 12B model recovered 82 of 100 planted facts, whereas the 31B model recovered 98 of 100. Neither model fabricated a code in the negative-control tests.

System design and scope

The system integrates with vLLM 0.31.0 through a GalahadConnector KV-transfer plugin. During deposit, the connector obtains the per-block KV state produced during prefill, serializes it into an encrypted MRLNCRY1 store using AES-256-GCM, and releases the corresponding GPU memory. During restoration, it grafts a stored block’s KV state back into the running request so that subsequent computation proceeds as though the block had just been computed.

This design creates an operational distinction between processed material, GPU-resident material, and material that can later be restored. Only one block is resident at a time; older blocks are not simultaneously available to attention. Consequently, the system should not be interpreted as providing global attention over 50 million tokens. It provides persistent, indexed access to separately restored blocks. The distinction is important because long-range usability here depends on selecting the relevant block and reinstating its state, not on allowing one forward pass to attend jointly to the entire corpus.

The storage footprint is determined by the model’s KV representation. For Gemma 4 12B, the measured footprint is 36.4 KiB per token, producing a 1.86 TB store for 50 million tokens. For Gemma 4 31B in FP8, the footprint is 134 KiB per token, producing a 6.25 TB store. The relatively low 12B footprint is attributed to Gemma 4’s sliding-window attention, rather than to compression introduced by Galahad. The storage requirement therefore remains a substantial systems cost, even though GPU memory remains effectively constant.

The approach differs from conventional prefix caching and KV offloading primarily in persistence and scope. Existing systems such as vLLM’s PagedAttention (Kwon et al., 2023), Prompt Cache (Gim et al., 2023), LMCache (Cheng et al., 8 Oct 2025), CacheGen (Liu et al., 2023), CachedAttention (Gao et al., 2024), and InfiniGen (Lee et al., 2024) address reuse or movement of KV state, but generally operate as session- or request-oriented caching mechanisms. The present work emphasizes durable encrypted storage, restart persistence, and restoration of content far outside the native context window. It also differs from RAG (Lewis et al., 2020) and cache-augmented generation (Chan et al., 2024), where the retrieved or cached source text is placed back into the model’s context and incurs another reading or prefill operation.

Evaluation methodology

The evaluation is designed to separate the correctness of KV restoration from the model’s ability to answer from restored content. This separation is methodologically important. A successful restore demonstrates that the serving layer supplied the stored state without recomputation; it does not by itself establish that the model correctly interprets the restored state.

The corpus contains public text from WildChat, source code, arXiv papers, and educational web text. Before ingestion, a runtime mutator replaces names, dates, amounts, and e-mail addresses with random nonces. The authors then insert 100 high-entropy needle records, one per 500,000 tokens, together with same-format decoys. Ingestion is blind to the later queries, and queries are issued only after the complete 50-million-token deposit. These controls target several possible sources of invalid performance: pretraining contamination, query lookahead, block dropping, lexical shortcutting, and fabricated telemetry.

The protocol also audits the physical store. Each deposited block must contain the MRLNCRY1 header, the number of stored blocks must match the number deposited, and the observed disk footprint must agree with the expected KV size. This is intended to prevent a benchmark implementation from claiming persistent storage while silently pruning or reconstructing blocks. The answer key is withheld from the serving engine, and energy is measured using an idle-subtracted NVML GPU counter.

The benchmark’s controls are useful but narrowly targeted. They establish that the tested blocks were physically persisted and later restored, and that the planted answers were not simply available as unchanged source text or obvious lexical matches. They do not establish semantic retrieval over arbitrary unstructured corpora, because the probe locations are known and the experimental mechanism is block restoration rather than learned selection.

Restoration correctness and long-range access

The paper reports 100/100 zero-recompute restores for both models over 100 probed depths between the beginning of the stream and 49,488,000 tokens. This is the clearest evidence for the paper’s primary systems claim: restoration success does not degrade with token depth when the block is addressed through the persistent store. GPU memory also remains flat during the 50-million-token deposit, supporting the claimed $O(1)$ moving-window behavior with respect to the total amount of processed material.

The model-level recall results are lower and model-dependent. Gemma 4 12B returns the exact planted code in 82 of 100 probes, while Gemma 4 31B succeeds in 98 of 100. The misses are described as selections of other values genuinely present in the same block, including decoys, rather than evidence that the correct block was unavailable. Both models produce zero fabricated codes in the positive and negative controls: the reported negative-control rate is 0/20 for each model. Thus, in this experiment, restoration is more reliable than factual extraction. The memory mechanism can make the relevant state available without guaranteeing that a particular model will identify the intended record within that state.

This distinction has a direct operational implication. A smaller model can use the same persistent memory with lower inference cost, while a larger model can be invoked when accurate discrimination among multiple in-block candidates is required. The memory layer does not itself improve the model’s reading or disambiguation capability; the difference between 82% and 98% is attributed to the model performing the read.

The authors appropriately avoid claiming byte-exact logit equality in this paper. They report zero-recompute restoration and rely on a companion work for the logit-level correctness gate. Therefore, the present evaluation directly establishes that the connector restores stored KV state without reprocessing the block, but the stronger claim that grafted state is computationally identical to freshly computed state is not independently re-established here.

Latency and energy results

The central performance comparison is between restoring a stored 16,000-token block and recomputing an unseen block of the same size. The reported medians are:

Model	Restore	Recompute	Speedup	GPU-energy reduction
Gemma 4 12B	0.266 s, 59 J	0.760 s, 521 J	2.8x	8.8x
Gemma 4 31B	0.347 s, 80 J	1.475 s, 976 J	4.25x	12.3x

Restoration is therefore reported as 2.8–4.3 times faster and 8.8–12.3 times less GPU-energy-intensive than recomputation. The larger model benefits more because its block recomputation is more expensive, while the restoration path is dominated by state transfer and connector overhead rather than by the full prefill computation. This result supports the paper’s claim that persistent KV reuse becomes increasingly advantageous as model-side prefill cost rises.

The latency is also largely independent of historical depth. For Gemma 4 12B, low-third to high-third restore TTFT changes from 0.193 to 0.274 seconds; for Gemma 4 31B, it changes from 0.343 to 0.375 seconds. These measurements indicate that accessing a block near the end of the 50-million-token stream does not impose a latency proportional to its distance from the current request. The relevant cost is storage access and restoration, not traversal or recomputation of all preceding blocks.

The energy result should be interpreted more cautiously than the speed result. Each restore takes approximately 0.3 seconds, close to the resolution limits of the NVML measurement procedure. The authors regard the restore-versus-recompute ratio as reliable but acknowledge greater uncertainty in the absolute per-probe joule values. In addition, the comparison excludes the one-time deposit cost. The claimed savings apply when a stored block is reused sufficiently often to amortize its initial prefill and serialization cost.

One-time deposit cost and deployment economics

The deposit phase processes the corpus once at 20,839 tokens per second for the 12B model and 10,700 tokens per second for the 31B model. Peak VRAM remains flat at approximately 70.9 GB and 75.1 GB, respectively. The one-time cost is consequently shifted from repeated GPU prefill to persistent disk capacity and initial ingestion.

This trade-off is favorable only under a reuse workload. A 50-million-token store requires 1.86 TB for the 12B model and 6.25 TB for the 31B model, and the reported latency and energy measurements depend on local NVMe. The paper explicitly notes that a network volume could invalidate those figures through additional latency and energy overhead. The system is therefore not a storage-free extension of context; it is a disk-backed state-management architecture whose economics depend on reuse frequency, NVMe capacity, and the cost of maintaining model-specific KV representations.

Encryption at rest provides a privacy property absent from many KV offloading configurations. The use of AES-256-GCM with an organization-specific key protects stored state against direct disk disclosure, although the paper does not report an independent security analysis of key management, access control, metadata leakage, or compromise of the serving process. The persistence mechanism should consequently be understood as encrypted storage rather than as a complete end-to-end memory-security architecture.

Relationship to retrieval and long-context evaluation

The paper’s contribution is deliberately narrower than a general long-context model. Long-context evaluations such as “Lost in the Middle” (Liu et al., 2023) and RULER (Hsieh et al., 2024) test whether a model can use information positioned at varying locations within a single input context. Galahad instead partitions history into blocks and restores one block when requested. Its success condition is thus a combination of block addressing, KV restoration, and model-level extraction.

This architecture avoids the prefill cost associated with RAG because the restored block’s source tokens are not read again. However, it does not solve the selection problem that RAG systems solve. The paper excludes Blaise, the authors’ learned retrieval component, and does not measure how an arbitrary query maps to the correct stored block. The experiment probes known block depths, with one needle per designated interval. As a result, the reported 100/100 restoration statistic should not be interpreted as 100% retrieval accuracy for an unconstrained memory system.

The distinction also affects the claim that the model has “memory.” The stored state is durable, encrypted, model-specific, and reusable after restart, but it is not a compressed semantic memory independent of the model architecture. It is a persistent representation of intermediate transformer computation. A change in model weights, tokenizer, attention implementation, quantization configuration, or relevant serving kernels may invalidate the stored KV state or require a new store. The claimed model-agnosticism is therefore best understood as model-agnostic operation across the two tested Gemma configurations and the connector interface, not as architecture-independent portability of stored states.

Limitations and open questions

The system does not provide simultaneous attention over the full 50-million-token corpus. Only one block is resident, and the method restores depth-addressed content rather than retaining all historical content in GPU memory. The paper also does not evaluate multiple-block composition, cross-block reasoning, or a query whose answer requires integrating evidence distributed across many separately restored blocks.

Semantic memory retrieval remains outside the evaluation. The benchmark establishes that known blocks can be restored and queried, but it does not measure recall, precision, or latency for selecting relevant blocks from arbitrary user queries. Combining learned retrieval with persistent KV restoration at 50-million-token scale is explicitly left open.

The storage overhead is substantial: 36.4 KiB per token for the 12B model and 134 KiB per token for the 31B model in the reported configurations. The architecture therefore exchanges repeated computation for durable storage, and the paper does not provide a break-even analysis across reuse frequency, NVMe cost, storage bandwidth, concurrency, or multi-tenant workloads. Nor does it evaluate failure recovery, partial corruption, key rotation, concurrent writes, or consistency under model upgrades.

Finally, the performance study uses one H100, local NVMe, two models, a fixed block size, temperature zero, and a controlled probe protocol. The results demonstrate feasibility under these conditions but do not establish scaling behavior under concurrent serving, heterogeneous hardware, remote storage, varying block sizes, or models with different attention and KV layouts. The absent logit-equivalence experiment is also material: the paper directly measures zero-recompute restoration, while exact equivalence to fresh computation is delegated to companion work.

Conclusion

The paper demonstrates a persistent KV-state layer that restores previously processed 16,000-token blocks from encrypted NVMe storage at depths approaching 50 million tokens while maintaining flat GPU memory usage. In the reported single-GPU experiments, restoration is 2.8–4.3 times faster and uses 8.8–12.3 times less GPU energy than recomputation, with 100/100 zero-recompute restores for both tested Gemma models. The results establish durable state reuse beyond the native context window, but not wider attention or semantic retrieval. The principal unresolved questions concern block selection, multi-block reasoning, storage economics, and correctness under broader model and serving configurations.