Learned Structure in CARTRIDGES: Keys as Shareable Routers in Self-Studied Representations

Abstract: A bottleneck for long-context LLM inference is the linearly growing KV cache. Recent work has proposed CARTRIDGES, an approach which leverages offline compute to train a much smaller KV cache than is typically required for a full document (up to 40x less memory usage at inference time). In this paper, we present the first mechanistic exploration of the learned CARTRIDGE key-value cache structure. In particular, we propose that (1) CARTRIDGE keys act as stable, shareable retrieval routers for the compressed corpora and (2) most of the learned compression occurs within the CARTRIDGE value vectors. We present empirical evidence of our routing theory across tasks, model families, and model sizes; for example, we can ablate the learned CARTRIDGE key vectors between tasks with little performance loss. Finally, we propose a slight improvement in initialization called Sampled Chunk Initialization (SCI). We suggest that SCI can lead to faster CARTRIDGE convergence than previously demonstrated in the literature. Our findings lay the groundwork for broader empirical study of CARTRIDGE training optimization which may be crucial for further scaling.