Effective one-shot KV context compaction

Determine efficient single-pass key–value cache compaction procedures for Transformer-based autoregressive language models that reduce the size of the KV cache while preserving downstream model behavior when the compacted prefix is later concatenated with uncompacted and future tokens.

Background

The paper addresses the growing memory bottleneck in Transformer-based LLMs caused by the KV cache, especially in long-context settings. Common practices such as summarization or token dropping are lossy and can harm downstream performance, motivating more principled compaction methods.

Cartridges provide high-quality latent-space compaction but require expensive gradient-based optimization. The authors propose Attention Matching as a faster alternative that approximates attention outputs and mass, aiming to preserve model behavior after compaction. Despite these advances, the general problem of achieving effective compaction in a single pass without degrading behavior is explicitly stated as open.

References

Effective context compaction—reducing KV cache size in a single pass while preserving downstream model behavior—remains an important open problem.

— Fast KV Compaction via Attention Matching  (2602.16284 - Zweiger et al., 18 Feb 2026) in Section 1 (Introduction)

This approach seems restricted to KV cache updates during generation, it is not clear what is done when a large prompt needs to be processed.

— Learning how to Forget: Fine-tuning for Long-Context Sparse Attention  (2608.19920 - Seeger et al., 20 Aug 2026) in Section 2, Related Work

For multi-turn conversations with cumulative context approaching the model's window, periodic compaction (physically trimming the cache to remove all masked positions) would be needed; masked positions are compacted away when cache state is serialized at end of generation (\S\ref{sec:prefix_cache}), but in-generation compaction remains unimplemented and unmeasured.

— Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets  (2608.19147 - Berenbaum et al., 19 Aug 2026) in Section 4.2, “Mask-Based Rewind”

The accuracy of an evictor is decided by what it protects, not by how it ranks the rest, which moves the open questions to where protection still matters: how to budget long prompts, especially in code tasks where protecting the entire prompt consumes a substantial fraction of the cache budget, and how to recover rare once-stated facts that only a content-dependent signal can preserve (\S\ref{sec:mech_boundary}).

— Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning  (2609.03430 - Wang et al., 3 Sep 2026) in Section 1, Introduction