Online adaptation of KV-cache compression to non-stationary marginal utility

Develop online updates to the compression basis of the Key–Value cache so that the cache budget can be adaptively reallocated as new tokens are processed and the eigenvalue spectrum of the projected covariance operator changes during generation, particularly for borehole logs whose information-dense tokens are distributed throughout the sequence.

Background

The paper observes that the spectrum of the operator W V Wᵀ is dynamic during autoregressive generation because newly processed tokens alter the information structure of the sequence. Compression based only on the initial spectrum may therefore become suboptimal for long-context inputs.

The proposed direction is to update the compression basis online, analogously to incremental principal-component analysis, so that the finite KV-cache budget follows changes in the information value of tokens. The application motivation is especially strong for borehole logs, where depth measurements and mineralogical observations may occur throughout the document rather than near its beginning.

References

Four research directions, identified in the original formulation, remain open and are sharpened by the application context.

\paragraph{Non-stationary marginal utility.} The eigenvalue spectrum of $W V W{\top}$ changes during a generation sequence as new tokens are processed. Static compression schemes based on the initial spectrum will be suboptimal for long-context generation. In the geo-mining context, this is particularly acute for borehole logs, where the most information-dense tokens (depth measurements, mineralogical observations) are distributed throughout the sequence rather than concentrated at the beginning. Online updates to the compression basis, analogous to online PCA (principal component analysis performed incrementally on a data stream: the eigenbasis is updated with each new observation via a rank-one correction, avoiding the need to store or reprocess past data \citep{oja1982}), could adaptively reallocate the KV cache budget as new tokens are processed.

— Marginal utility, matrix factorization, and the Key-Value (KV) cache: a unified information-economic framework for sovereign geo-mining inference  (2609.20068 - Combe, 17 Sep 2026) in Section 14, “Open problems,” paragraph “Non-stationary marginal utility”