Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute
This presentation explains Galahad-KV, a persistent memory layer that enables language models to access previously processed material at depths approaching 50 million tokens without recomputation. By storing KV state blocks on encrypted local storage and restoring them on demand, the system achieves 2.8–4.3× faster processing and 8.8–12.3× lower GPU energy use compared to recomputation, while maintaining flat GPU memory during ingestion. The evaluation demonstrates 100% successful zero-recompute restoration across all tested depths, though model-level factual recall varies by model size and the system does not provide simultaneous attention over the entire corpus.Script
A language model today processes millions of tokens during a conversation, then forgets everything when the session ends. Galahad-KV changes that by storing each processed block on encrypted local storage and retrieving it on demand, turning ephemeral computation into persistent memory that survives restarts and scales to 50 million tokens.
The system divides processed material into 16,000-token blocks, stores each block's KV state on NVMe, and keeps only one block resident in GPU memory at a time. This design eliminates repeated prefill for previously seen content and keeps memory footprint flat even as total processed material grows without bound.
Restoring a stored block is 2.8 to 4.3 times faster than recomputing it from scratch, and uses 8.8 to 12.3 times less GPU energy. The larger the model, the greater the advantage, because restoration cost is dominated by state transfer while recomputation scales with model size.
The evaluation planted 100 high-entropy codes across 50 million tokens and queried them after full ingestion. Every single block was restored from encrypted storage without recomputation, including blocks at depths approaching 50 million tokens, and the Gemma 31B model recovered 98 of 100 planted facts.
The trade-off is storage capacity: the 12B model requires 1.86 terabytes for 50 million tokens, and the 31B model requires 6.25 terabytes. The reported speed and energy results also depend on local NVMe; network storage could invalidate those figures through added latency and overhead.
The system restores blocks without recomputation, but it does not guarantee the model will find the right answer within that block. Smaller models may miss the target among decoys, and the architecture does not yet address how to select the relevant block from an arbitrary query. To explore this work further and create your own research videos, visit EmergentMind.com.
