Cartridges at Scale: Training Modular KV Caches over Large Document Collections

Abstract: LLMs can reason over long contexts, yet prefilling millions of tokens is wasteful as much of the content remains static across queries. Cartridges address this by distilling document collections into reusable key-value (KV) caches that eliminate prefilling while preserving accuracy. A critical limitation of this approach is that cartridges are monolithic and non-compositional: encoding an entire collection into a single KV block does not scale, and naively mixing cartridges trained in isolation collapses performance to near chance. We introduce Cartridges at Scale (CAS), a training framework for scalable multi-cartridge learning with dynamic distractor mixing and a memory-efficient budget manager that rotates hundreds of per-document cartridges between GPU and persistent storage. Our approach scales to collections exceeding a million tokens, improving over a monolithic cartridge by 10-31 points at comparable token budgets. Oracle cartridge accuracy falls within 2-6 points of full in-context learning even at high compression. When paired with retrieval for cartridge selection, CAS matches or exceeds conventional RAG accuracy while consuming 3-4x fewer prompt tokens.

• The paper demonstrates that modular, joint training of per-document KV caches with dynamic distractor mixing mitigates catastrophic interference and achieves robust compression.
• It details the CAS pipeline where memory-efficient joint learning, dynamic cartridge rotation, and optimized self-study data synthesis enable nearly oracle-level QA performance at high compression rates.
• The study indicates practical gains in retrieval-augmented generation by reducing prompt token usage by 3–4×, promising scalable long-context reasoning in LLMs.

Cartridges at Scale: Modular KV Cache Training for Large Document Collections

Introduction

The "Cartridges at Scale" (CAS) framework (2606.04557) addresses fundamental efficiency and scalability issues of long-context reasoning in LLMs, focusing on modular, reusable key-value (KV) cache representations—called cartridges — distilled from large document collections. While prior cartridge methods demonstrated favorable compression and fast inference by storing context as KV caches, these approaches were hampered by monolithic, non-compositional representations. Naïvely aggregating independently trained per-document cartridges leads to catastrophic interference due to the frozen LLM’s inability to selectively leverage the correct document context in the presence of distractors.

CAS introduces a multi-cartridge training and serving regime, dynamic cartridge rotation, memory-efficient joint learning, and improved self-study data generation. The result is a pipeline that scales to hundreds of documents and over a million tokens, achieving robust compression (up to 100× for some domains), modularity, and accuracy close to full in-context learning, while enabling practical retrieval-augmented generation (RAG) with much lower prompt token requirements.

The CAS Framework: Modular, Scalable Cartridges

CAS begins by associating each document $d_i$ in a collection with a dedicated cartridge $Z_i$, trained as a frozen LLM KV cache to answer queries about its content. Unlike monolithic approaches, CAS enables simultaneous learning and joint inference over hundreds of cartridges.

Dynamic distractor mixing during training exposes the LLM to both relevant and irrelevant document cartridges in the KV prefix, directly optimizing selective attention over distractors. The CAS pipeline is underpinned by a memory-efficient budget manager that rotates active cartridges between GPU and persistent storage, making large-scale composition feasible within hardware constraints.

Figure 1: Overview of the CAS pipeline, showing self-study data generation, cartridge training with active pool rotation, and inference via retrieval-based cartridge selection.

Self-study data synthesis is improved with proportional-to-length document sampling and batched, multi-question generation. These optimizations favor coverage of dense, fact-heavy content (e.g., tables in FinQA), reduce synthesis cost ($\sim 20\times$), and increase accuracy. Cartridge-specific KV cache initialization (using the first $p$ tokens of each $d_i$) further accelerates convergence and lowers the final loss.

Multi-Cartridge Joint Training and Robustness

Isolated per-document cartridge training leads to severe interference at test time: in LongHealth, oracle accuracy drops from $\sim 74\%$ (oracle context) to chance when all distractors are present. Mixed-visibility training, incorporating variable numbers of active distractors in each step, resolves collapse by teaching the attention mechanism to select the correct cartridge.

Figure 2: Per-document cartridge performance (LongHealth), highlighting catastrophic interference in the “all-cartridges” setting with isolated training versus robust performance with joint training.

Experimentally, joint training with mixed cartridge pools yields position-invariant attention and strong selectivity: accuracy on present documents is nearly unchanged whether cartridges are loaded in order or shuffled, and presence/absence of relevant cartridges is sharply distinguished.

Figure 3: Accuracy as a function of active cartridge count in LongHealth. Model performance on questions for absent cartridges drops to chance, confirming effective knowledge partitioning.

Compression, Efficiency, and Retrieval-Enhanced Inference

CAS-trained cartridges consistently outperform monolithic approaches at fixed token budgets. On LongHealth and QuALITY, per-doc cartridges achieve up to a 30-point gain at 10–20× compression, while the monolithic variant suffers destructive gradient interference.

Document complexity modulates compression tolerance: table-rich or fact-dense tasks (FinQA, LongHealth) exhibit loss at high compression, with optimal rates below 10×; technical or extractive tasks (TechQA) permit 100× with negligible performance drop. Notably, cartridges approach within 2–6 points of the uncompressed oracle baseline at high compression.

The integration of CAS with existing dense retrieval enables Cartridge RAG: instead of retrieving chunk text, relevant cartridges (often entire documents) are selected and loaded as KV caches. This results in 3–4× fewer prompt tokens than conventional Text RAG to achieve comparable or higher accuracy on LongHealth and QuALITY.

Figure 4: Accuracy vs. additional prompt tokens comparing Text RAG and Cartridge RAG. CAS consistently achieves higher efficiency, approaching oracle accuracy with substantially lower prompt overhead.

Optimization and Data Generation Strategies

CAS also introduces several key changes in optimization, including slow-decay learning rate schedules and per-cartridge warmup, which are critical under pool rotation and large-scale joint learning.

Figure 5: Learning rate scheduling ablation on FinQA demonstrating the benefit of slow linear decay and per-cartridge warmup in improving final accuracy.

Improvements in synthetic data generation (using a strong, decoupled question model and batched sample generation) yield measurable gains in coverage and final QA scores on dense, complex documents, closing much of the residual gap to the oracle setting.

Practical and Theoretical Implications

CAS demonstrates that modular, query-independent context compression can be trained and deployed at the granularity of individual documents, yielding scalable, composable, and computationally efficient context representations. The framework is extensible: grouped-document cartridges can further improve context granularity and RAG recall, and direct cartridge-level retrieval architectures are a promising future direction. Potential limitations remain in cartridge retrieval calibration, multilingual generality, and optimal grouping strategies.

From a theoretical standpoint, CAS highlights that catastrophic interference is not an inherent property of attention over compressed KV caches, but rather a failure of training regimes to present compositional context during distillation. Proper training with distractors restores selective attention without parameter adaptation, supporting further modularization and system scaling.

Future Directions

Key avenues for extension include:

• Dedicated cartridge retrievers: Training models that encode queries in the same space as cartridges will allow greater retrieval quality and efficiency, especially in domains where text-based retrieval is suboptimal.
• Flexible cartridge manipulation: Dynamically grouping or splitting document cartridges based on retrieval or content structure.
• Multilingual and cross-institutional deployment: Investigating cartridge distillation and transfer in multilingual or data-governed environments.
• Downstream utility and privacy: Auditing and redacting cartridge content, especially for sensitive domains.

Conclusion

CAS establishes that modular, multi-cartridge approaches can scale document context compression and retrieval for LLMs well beyond the limitations of monolithic or naïve per-document schemes. By resolving catastrophic interference through joint training with dynamic distractor mixing and advanced data synthesis strategies, CAS delivers efficient, compositional, and nearly oracle-level QA at large scale, with practical speed and token efficiency gains over current RAG pipelines. The resulting framework marks a significant advance in the persistent, efficient, and modular use of context in LLM-based retrieval.

Reference: "Cartridges at Scale: Training Modular KV Caches over Large Document Collections" (2606.04557)