Cartridges at Scale: Modular KV Caches

• Cartridges at Scale (CAS) is a framework that scales long-context KV compression by converting extensive document collections into modular, per-document cartridges.
• CAS employs dynamic distractor mixing and joint training to mitigate prefix growth and interference issues when loading multiple cartridges during inference.
• Empirical results demonstrate that CAS improves performance by 10–31 points over monolithic approaches while using 3–4× fewer prompt tokens in retrieval scenarios.

Searching arXiv for the named CAS and related Cartridges papers to ground the article in the cited literature. arxiv.search_query({"query":"ti:\"Cartridges at Scale\" OR ti:\"Cartridges\" long context KV cache self-study", "max_results": 10, "sort_by": "submittedDate", "sort_order": "descending"}) arXiv search results include:

1. "Cartridges at Scale: Training Modular KV Caches over Large Document Collections" (Hardalov et al., 3 Jun 2026)
2. "Learned Structure in CARTRIDGES: Keys as Shareable Routers in Self-Studied Representations" (Diaz, 23 Aug 2025)
3. "Cartridges: Lightweight and general-purpose long context representations via self-study" (Eyuboglu et al., 6 Jun 2025) Cartridges at Scale (CAS) is a training framework for scalable multi-cartridge learning over large document collections. It extends the original Cartridges formulation, in which a long corpus is distilled into a reusable key-value (KV) cache prefix, by addressing the failure modes that emerge when many document-specific cartridges must coexist in a single inference system. In CAS, LLMs avoid repeated prefilling of static corpora by loading compact, document-level KV artifacts selected for a query, while training explicitly teaches the frozen base model to attend to relevant cartridges and ignore distractors. The framework scales to collections exceeding a million tokens, improves over a monolithic cartridge by 10–31 points at comparable token budgets, places oracle cartridge accuracy within 2–6 points of full in-context learning even at high compression, and when paired with retrieval matches or exceeds conventional RAG accuracy while consuming 3–4x fewer prompt tokens (Hardalov et al., 3 Jun 2026).

1. Origins in long-context KV compression

CAS is rooted in the earlier Cartridges formulation, which reinterprets long-context inference as an offline compression problem. Instead of placing an entire corpus in the prompt for every query, the model trains a small corpus-specific KV cache offline and reuses it across future queries about that corpus. The original motivation is serving efficiency: the KV cache grows linearly with context length, so full in-context learning (ICL) over long corpora is memory-hungry and throughput-limited. A concrete example given for this regime is that LLaMA 70B needs about 84 GB of KV-cache memory for a single 128k-token context, and on a single H100, LLaMA-8B throughput drops by about $77\times$ when moving from 1k to 120k context length. Cartridges therefore target repeated-query settings such as codebases, financial filings, medical records, legal documents, and chat histories, where an offline training cost can be amortized over many future requests (Eyuboglu et al., 6 Jun 2025).

Formally, for a corpus $C$, a Cartridge is a set of parameters $Z$ that augments a frozen LLM $F$. In the main parameterization, the learned state is a simplified prefix-tuning object with trainable key and value vectors,

$$\mathbf{z}_\text{k}, \mathbf{z}_\text{v} \in \mathbb{R}^{p\times d},$$

with total size

$Z\in \mathbb{R}^{L\times p\times d\times 2},$

where $p$ is the number of virtual tokens, $d$ is the model hidden size, and the final dimension corresponds to the KV pair. At training time, the corpus KV pairs are replaced by $Z$, gradients are backpropagated only into those key/value vectors, and all model weights remain frozen. Inference then consists of loading the Cartridge as a cached prefix, appending a query $q$, and decoding normally.

The original paper also established that naive next-token prediction on raw corpus text is not sufficient. Although such training can achieve near-perfect perplexity while using about $C$0 less memory than full ICL, it does not recover general-purpose question answering over the corpus. To address this, Cartridges introduced Self-Study: synthetic conversation generation over corpus chunks followed by context distillation from a teacher with the document in context to a student using the Cartridge alone. On challenging long-context benchmarks, Self-Study Cartridges match ICL performance while using $C$1 less memory and enabling $C$2 higher throughput, and they can extend effective context length, for example from 128k to 484k tokens on MTOB (Eyuboglu et al., 6 Jun 2025).

2. Why collection scale changes the problem

CAS begins from the observation that the single-cartridge setting does not directly generalize to large document collections. A monolithic cartridge that attempts to encode an entire collection into one KV block does not scale, and a system that simply concatenates independently trained cartridges is not reliably compositional at collection scale. The paper characterizes the first failure mode as prefix growth: if all cartridges are loaded at once, the prefix cost scales like

$C$3

where $C$4 is the number of documents or cartridges and $C$5 is cartridge length. The second failure mode is interference: if cartridges are trained in isolation, loading many of them jointly at inference can collapse performance to near chance because the model was never trained to choose among multiple KV prefixes (Hardalov et al., 3 Jun 2026).

This scaling diagnosis qualifies, rather than negates, earlier evidence that Cartridges can sometimes be concatenated without retraining. The original Cartridges work reported that separately trained cartridges for different 10-K filings could be concatenated at inference time without joint optimization and still outperform both a single-Cartridge baseline and truncated ICL on multi-document questions (Eyuboglu et al., 6 Jun 2025). CAS, by contrast, shows that for larger collections this behavior does not persist automatically. On LongHealth, isolated per-document cartridges perform reasonably in an oracle setting, but when all are loaded together accuracy can collapse to 26.0, whereas joint training reaches 77.8. This suggests that limited composition and large-scale modular deployment are distinct regimes with different optimization requirements (Eyuboglu et al., 6 Jun 2025, Hardalov et al., 3 Jun 2026).

3. Multi-cartridge learning in CAS

The central contribution of CAS is a training framework for modular multi-cartridge learning with dynamic distractor mixing. The framework retains the Cartridge representation

$C$6

but shifts the unit of compression from a single corpus to many per-document cartridges that must function jointly. Training still uses context distillation, but the visible cartridge set is randomized. For a target cartridge $C$7, CAS defines the active set as

$C$8

with $C$9. Thus, with probability $Z$0 the model trains on the correct cartridge alone, and otherwise it trains with the correct cartridge plus random distractors. The practical purpose is to teach the frozen LLM to attend to the correct cartridge and ignore unrelated ones under inference-time conditions where multiple cartridges may be present (Hardalov et al., 3 Jun 2026).

The training objective remains a KL divergence between teacher and student distributions. In the original single-cartridge setting, the distillation objective is

$Z$1

CAS preserves this teacher-student logic while changing the student’s visible memory from one cartridge to a mixed set of target and distractor cartridges. Packing multiple cartridges during training requires all active cartridges to be in the prompt, with attention masks distinguishing the sample-specific visible subset. Because cartridges occupy prefix positions, RoPE position IDs for query tokens must be offset by the total number of active cartridge tokens (Eyuboglu et al., 6 Jun 2025, Hardalov et al., 3 Jun 2026).

A second core component is the budget manager, which makes large-scale training feasible under fixed GPU memory. CAS keeps only a subset of cartridges on GPU, with GPU pool size $Z$2, while the rest reside in CPU memory or disk. Every $Z$3 optimizer steps, the system rotates the pool: it evicts a fraction $Z$4 of active cartridges and loads new ones from persistent storage, while prioritizing cartridges with the fewest optimizer steps so that training coverage remains uniform. The need for this mechanism is illustrated quantitatively: a 1K-token Qwen3-8B cartridge takes about 551 MiB during backprop, 500K tokens would require about 274 GiB, and gradients and Adam moments increase the footprint by $Z$5 (Hardalov et al., 3 Jun 2026).

4. Synthetic supervision, initialization, and optimization

CAS also revises the Self-Study data pipeline. The paper identifies three deficiencies in the earlier synthetic-data regime: uneven fact coverage, because questions overfocus on prose rather than dense facts and tables; unbalanced document sampling, because long documents are underrepresented if chunks are sampled uniformly; and low throughput, because one question per API call is expensive and slow. CAS addresses these issues with proportional-to-length sampling, where document $Z$6 receives weight

$Z$7

multi-question generation with $Z$8 questions per call, and a decoupled question/answer setup. In the reported system, $Z$9 is GPT-OSS 120B and $F$0 is Qwen3-8B. Generating 20 questions per call increases throughput, lowers synthesis cost up to $F$1, and improves factual coverage (Hardalov et al., 3 Jun 2026).

These changes refine the original Self-Study principle rather than replacing it. In the original formulation, the corpus is chunked into a subcorpus that fits in context, a generic seed prompt biases the conversation type, and two roles are simulated by the same base model to create synthetic conversations. The seed prompts come from five generic categories—structuring, summarization, question, use cases, and creative—and multiple seed types improve performance over a single seed prompt by up to 7.9 chrF on MTOB and 4.8 accuracy points on LongHealth. CAS inherits the view that diverse synthetic conversations, not raw next-token prediction on the corpus alone, are needed to learn ICL-like functionality (Eyuboglu et al., 6 Jun 2025).

Initialization and optimization are also specialized to the rotating multi-cartridge setting. CAS uses cart-specific initialization, in which each cartridge is initialized from the KV cache of its own document rather than unrelated text; this yields about 50% lower initial loss and about 10% lower final loss. Optimization uses Adam in FP32, BF16 weights, slow linear learning-rate decay, and per-cartridge warmup: $F$2 where $F$3 is the current global learning rate, $F$4 is the number of optimizer steps received by the cartridge, and $F$5 is the per-cartridge warmup length. To avoid too many optimizer parameter groups, the warmup scale is bucketed into 4 discrete levels (Hardalov et al., 3 Jun 2026).

5. Empirical performance across training regimes and retrieval settings

The clearest empirical demonstration of CAS is the contrast between isolated and joint training on LongHealth. Isolated training performs adequately when the correct document is known in advance, but fails when all cartridges are present; joint multi-cartridge training largely closes that gap.

Setting	Oracle (1 doc)	Full (20 docs)
Oracle Context	87.4	65.5
Train in Isolation	73.6	26.0
Train Jointly (25% ex., 10 active)	78.9	56.4
Train Jointly (25% ex., 20 active)	79.0	77.8

The same study shows that the principal advantage of CAS is not merely additional capacity but modular specialization under controlled interference. At large token budgets, a monolithic cartridge reaches 72.6 on LongHealth, 65.2 on QuALITY, and 14.1 on FinQA, whereas per-document cartridges achieve 80.1, 78.6, and 62.7 respectively. Oracle single-document cartridges remain competitive across substantial compression ratios: on LongHealth, 81.1 at $F$6, 80.1 at $F$7, and 77.3 at $F$8; on QuALITY, 78.6, 77.6, and 76.4; on QASPER, around 54–55 across compressions; on FinQA, 62.7, 50.7, and 23.0; and on TechQA, around 75 even at $F$9. The reported pattern is document-type dependent: dense numeric or table-heavy content is harder to distill than verbose or extractive text (Hardalov et al., 3 Jun 2026).

CAS is also evaluated as a retrieval-conditioned system, termed Cartridge RAG. At inference time, a dense retriever selects relevant text chunks, these are mapped to source document identifiers, and the corresponding cartridges are loaded instead of the raw text chunks themselves. In this setting, CAS matches or exceeds conventional Text RAG while using 3–4x fewer prompt tokens. On LongHealth, Cartridge RAG at $$\mathbf{z}_\text{k}, \mathbf{z}_\text{v} \in \mathbb{R}^{p\times d},$$0, $$\mathbf{z}_\text{k}, \mathbf{z}_\text{v} \in \mathbb{R}^{p\times d},$$1 reaches 75.4 with 2,673 tokens versus Text RAG with 9,860 tokens; at $$\mathbf{z}_\text{k}, \mathbf{z}_\text{v} \in \mathbb{R}^{p\times d},$$2, $$\mathbf{z}_\text{k}, \mathbf{z}_\text{v} \in \mathbb{R}^{p\times d},$$3, it reaches 76.5 with only 566 tokens. On QuALITY, Cartridge RAG at $$\mathbf{z}_\text{k}, \mathbf{z}_\text{v} \in \mathbb{R}^{p\times d},$$4, $$\mathbf{z}_\text{k}, \mathbf{z}_\text{v} \in \mathbb{R}^{p\times d},$$5 achieves 75.3, exceeding Text RAG at 73.9 while using 2,635 versus 4,754 tokens. FinQA peaks around $$\mathbf{z}_\text{k}, \mathbf{z}_\text{v} \in \mathbb{R}^{p\times d},$$6, $$\mathbf{z}_\text{k}, \mathbf{z}_\text{v} \in \mathbb{R}^{p\times d},$$7 with 52.8, and increasing $$\mathbf{z}_\text{k}, \mathbf{z}_\text{v} \in \mathbb{R}^{p\times d},$$8 beyond that hurts. TechQA is the main counterexample: Text RAG slightly beats Cartridge RAG, 73.2 versus 71.0. The paper also notes that at $$\mathbf{z}_\text{k}, \mathbf{z}_\text{v} \in \mathbb{R}^{p\times d},$$9 compression, cartridge inference uses about 1,200 tokens versus about 12,000 for text oracle and about 9,860 for Text RAG at $Z\in \mathbb{R}^{L\times p\times d\times 2},$0; since prefill FLOPs scale like $Z\in \mathbb{R}^{L\times p\times d\times 2},$1, a $Z\in \mathbb{R}^{L\times p\times d\times 2},$2 token reduction yields roughly $Z\in \mathbb{R}^{L\times p\times d\times 2},$3 fewer prefill FLOPs (Hardalov et al., 3 Jun 2026).

6. Mechanistic interpretation, limitations, and terminological ambiguity

A mechanistic study of CARTRIDGES provides additional insight into why CAS-style systems may be scalable. That work argues for a division of labor in the learned KV cache: keys act as stable, shareable retrieval routers, while values carry most of the task- or corpus-specific compression payload. Empirically, value rotations during training are often an order of magnitude larger than key rotations, values continue changing late into training, and keys barely move after early training. Key-swapping ablations across tasks and model families show only modest performance loss—about 4–5% for Llama models and about 7% for Qwen3—while the ablated systems remain better than the no-Cartridge and random-choice baselines. The same study proposes Sampled Chunk Initialization (SCI), which samples multiple random chunks across the corpus instead of using only the first $Z\in \mathbb{R}^{L\times p\times d\times 2},$4 tokens, and reports faster convergence than First-$Z\in \mathbb{R}^{L\times p\times d\times 2},$5 Token Initialization under a paired $Z\in \mathbb{R}^{L\times p\times d\times 2},$6-test with $Z\in \mathbb{R}^{L\times p\times d\times 2},$7. A plausible implication is that future CAS variants may be able to freeze or simplify keys and concentrate optimization on values (Diaz, 23 Aug 2025).

CAS also states several limitations explicitly. Prepending cartridges is awkward for multi-turn dialogue, because inserting a cartridge mid-conversation would invalidate prior KV states and require recomputing the dialogue history. Cartridge selection remains text-based, since Cartridge RAG uses a standard dense retriever over raw text rather than a cartridge-native retrieval space. The reported evaluation is English-only, and all experiments use Qwen3-8B as the base model. Task dependence is significant: TechQA benefits from verbatim span extraction, where raw text is advantageous, and FinQA remains difficult because dense tables and numeric relations are hard to compress (Hardalov et al., 3 Jun 2026).

Finally, the acronym “CAS” is not unique in the arXiv literature. It is also used for the Sloan Digital Sky Survey’s Catalog Archive Server in “Batch is back: CasJobs, serving multi-TB data on the Web” and for compare-and-swap in “Efficient Multi-word Compare and Swap.” In the present long-context LLM setting, however, CAS denotes “Cartridges at Scale,” a framework for training modular KV caches over large document collections 0502072.