Context-Resident Compression for LLMs

• Context-resident compression is a technique that replaces extensive token sequences with smaller, actionable representations integrated directly into the LLM’s inference pathway.
• It includes both hard compression (selective token retention) and soft compression (latent embedding techniques) to balance accuracy, latency, and memory efficiency.
• Adaptive methods, such as dynamic prefix selection and recurrent memory updates, enable optimized performance while significantly reducing token counts.

Context-resident compression denotes a family of compression strategies in which long context is replaced by a smaller representation that remains directly usable inside the model’s inference pathway, rather than being preserved only as external retrieval data or as permanent parameter updates. In contemporary LLM systems, that resident form may be a retained top-$k$ document prefix, a shorter plain-text prompt, a sequence of continuous compression tokens, a compact KV-cache state, a recurrent compressed memory, a portable latent buffer, or a typed semantic state. A useful historical antecedent appears in compressed context modeling for PPM, where the conditioning context is itself represented in compressed form rather than as literal symbol suffixes (Kulekci, 2012, Ye et al., 3 Feb 2026, Kim et al., 2023).

1. Conceptual scope and boundaries

A basic distinction in this area is between hard and soft context compression. Hard compression keeps some original tokens or spans and discards the rest; soft compression condenses a long context into a small number of continuous representations. ComprExIT states this distinction explicitly and treats soft compression as replacing a long token sequence with a compact latent memory that can be directly consumed during inference (Ye et al., 3 Feb 2026). QUITO, by contrast, remains on the hard side: it filters the context into shorter ordinary text and passes that text to the downstream model unchanged, which preserves black-box compatibility with standard LLM APIs (Wang et al., 2024).

A second distinction is between context compression and model compression. The paper on Contextual Compression Encoding is explicit that it compresses model parameters and cross-layer redundancy, not runtime prompt state, KV caches, or long-context memory. It is therefore adjacent to, but not itself an example of, context-resident compression in the prompt- or memory-state sense (Schmitt et al., 12 Feb 2025).

A third distinction concerns whether the compressed artifact remains inside the model’s normal context interface. Several recent LLM papers emphasize precisely this property. QGC replaces retrieved documents with sequences of continuous embeddings inserted into the LLM context (Cao et al., 2024). UniGist keeps gist-token KV states in cache after old raw tokens are removed (Deng et al., 19 Sep 2025). LCC compiles a long context into buffer tokens and then discards the temporary LoRA compiler, leaving a portable memory artifact compatible with the frozen base model (Li et al., 31 Jan 2026). This suggests that “resident” should be understood operationally: the compressed state continues to participate in ordinary decoding as if it were context.

2. Main representational forms

The literature spans several representational regimes rather than a single method family.

Resident form	Representative methods	Core mechanism
Ranked document prefix	AdaComp	Keep the predicted minimal top-$k$ retrieved documents
Filtered plain text	QUITO	Query-guided phrase/sentence filtering under a budget
Continuous compression tokens or slots	ComprExIT, IC-Former, QGC, CompLLM	Replace long text with short latent embeddings
KV-resident memory	CCM, UniGist, ATACompressor, SAC	Keep compact KV states instead of full token histories
Portable latent buffer	LCC	Compile context into reusable buffer tokens / KV cache
Natural-language or symbolic state	TaC-C, Context Codec	Use reasoning traces or typed semantic atoms as compressed context

At the extractive document level, AdaComp compresses by retaining a top-$k$ prefix of the ranked retrieval list rather than by pruning within documents. Its central definition is the minimal sufficient prefix

$$\mathcal{D'}=\arg\min_k \{k \mid \textnormal{Correct}(q,\mathcal{D}_k)=1\},$$

so compression is measured at the document-prefix level rather than by token or sentence reduction (Zhang et al., 2024).

At the soft latent level, several methods replace text with learned continuous vectors. QGC compresses each retrieved document into a query-conditioned sequence of continuous embeddings and feeds those embeddings to the frozen target LLM in place of the original document tokens (Cao et al., 2024). IC-Former uses learnable digest tokens and cross-attention to condense a long context into soft prompt vectors, explicitly separating the compressor from the target LLM and reducing compression complexity to $O(kn+k^2)\sim O(kn)$ (Wang et al., 2024). ComprExIT turns a long context into a small number of continuous slots through depth-wise and width-wise information transmission over frozen hidden states, then aligns those slots to the decoder-facing space (Ye et al., 3 Feb 2026). CompLLM makes the same broad move but at segment level: each segment is compressed independently into concept embeddings, and the concatenation of those segment embeddings serves as reusable compressed context (Berton et al., 23 Sep 2025).

A KV-native branch of the literature keeps compressed information directly in attention-compatible memory. CCM compresses online interaction history into compact key/value memory and updates that memory recurrently across time steps (Kim et al., 2023). UniGist replaces old raw-token KV entries with gist-token KV entries that remain in cache and continue to be attendable by future tokens (Deng et al., 19 Sep 2025). ATACompressor outputs compressed tokens whose KV representations are passed to the downstream LLM (Li et al., 3 Feb 2026). SAC goes further by selecting original context tokens as semantic anchors and using their layerwise KV pairs as the compressed representation, rather than learning context-agnostic compression tokens through autoencoding (Liu et al., 10 Oct 2025).

A separate textual branch treats compressed context as human-readable text. TaC and TaC-C use > traces as compressed context, and those traces are then reinserted into the prompt as the only context for downstream answering (Ma et al., 27 May 2026). Context Codec represents dialogue state as typed semantic atoms and renders those atoms in CCL, an ASCII-first notation intended to keep durable commitments explicit and auditable (Trukhina et al., 17 May 2026).

3. Adaptive and query/task-aware compression

A recurring conclusion is that one compression level is not appropriate for all inputs. AdaComp makes this explicit in retrieval-augmented QA: the minimum top-$k$ documents needed is not constant, but depends on both query complexity and retrieval quality. Complex multi-hop queries may need more retrieved documents; weak retrieval may also force the system to retain more of the ranked list (Zhang et al., 2024).

Recent methods operationalize this adaptivity in different ways. ACC-RAG uses a fixed offline hierarchical compressor but an online selector that stops at the earliest compressed prefix judged sufficient. Its selection process tests increasingly larger compressed prefixes and stops when the selector predicts sufficiency, so the effective compression rate is determined by the consumed prefix length rather than by changing the underlying compressor (Guo et al., 24 Jul 2025). ATACompressor similarly separates content selection from budget allocation: a selective encoder compresses only query-relevant parts of the context, while an Adaptive Allocation Controller predicts relevant-content length $\hat{L}_{Rel}=\zeta(\mathbf{H}_\varphi)$ and sets the compressed token budget by $k=\eta(\hat{L}_{Rel})$ (Li et al., 3 Feb 2026).

Query conditioning is central in several other systems. QGC is built around the claim that high-ratio compression fails primarily because key query-relevant information is lost, so it injects the query at multiple stages: joint query-document encoding, query-guided pooling, a reviewing layer, and document-level dynamic compression. Its reported aggregate result is an average improvement of $+5.03$ points over LongLLMLingua with LongChat-13B and $+12.87$ points with LLaMA-2-7B, together with average $k$0 higher compression ratio and $k$1 higher throughput (Cao et al., 2024). QUITO also uses query-conditioned signals, but via trigger-token attention over the full prompt, then budgets retained words or sentences under $k$2 or $k$3 compression (Wang et al., 2024). In tool-use, selective compression is task-aware in a narrower sense: tool names and parameter names are designated as key information and deliberately preserved as raw text while the remaining documentation is compressed into soft tokens (Xu et al., 2024).

This broad pattern suggests that adaptivity in context-resident compression has at least three distinct meanings: adapting the amount of retained context, adapting which context is preserved, and adapting the form of representation to the current task.

4. Online, recurrent, and reusable resident memory

Some of the strongest context-resident methods are explicitly designed for long-lived or reusable memory. CCM is the clearest recurrent formulation. It defines an online state

$k$4

and then conditions future prediction on $k$5 rather than on the full accumulated context (Kim et al., 2023). The paper studies both a concatenative memory and a fixed-size merge memory, the latter becoming a true recurrent compressed state.

UniGist implements a different online memory discipline. Old spans of raw tokens are replaced by gist tokens, and once raw tokens fall outside the local window they become permanently invisible, so their full KV states can be physically removed from cache. The resident cache is then a mixture of sink tokens, gist tokens, and a bounded local window of recent raw tokens (Deng et al., 19 Sep 2025). RCC scales long contexts by recurrent chunkwise compression rather than explicit cache eviction: each encoder window is compressed into a shorter latent sequence, and those latent vectors are concatenated across chunks, yielding a stored context that grows far more slowly than the original document (Huang et al., 2024).

Reusability is the defining property of CompLLM. Because each segment is compressed independently, compressed segments can be cached and reused across multiple queries with overlapping contexts. The paper highlights this as a practical advantage over holistic compressors, since only changed segments need recompression and long contexts can be handled by repeated local compression rather than a single global pass (Berton et al., 23 Sep 2025). Tool-document compression adopts the same reuse logic at application level: stable documentation is compressed once and then reused whenever the tool must be invoked (Xu et al., 2024).

LCC pushes this reuse principle furthest by distinguishing memory artifacts from model adaptation. A disposable LoRA module acts only as a compiler; after optimization, the long context has been distilled into $k$6, the LoRA is discarded, and the retained memory is a portable latent buffer or its standard KV cache (Li et al., 31 Jan 2026). TaC-C provides a different form of portability: because the compressed artifact is natural-language reasoning text, it can be consumed by different downstream answer models without architectural changes (Ma et al., 27 May 2026).

5. Empirical quality–efficiency trade-offs

The empirical literature consistently shows that context-resident compression is not only about shrinking prompts but about finding a favorable accuracy–latency–memory point.

AdaComp illustrates the document-prefix regime clearly. On Natural Questions with Llama2-7B, Top-5 retrieval used 802 tokens and achieved EM/F1 $k$7, while AdaComp used 441 tokens and achieved $k$8; on HotpotQA, Top-5 used 819 tokens and scored $k$9, whereas AdaComp used 527 tokens and scored $k$0 (Zhang et al., 2024). The resulting picture is not extreme compression, but substantial token reduction with performance near or above the uncompressed baseline.

Soft latent methods often push harder on compression ratio. On six in-domain MRQA benchmarks at $k$1 compression, ComprExIT reports average performance of $k$2 EM / $k$3 F1 on Llama-3.2-1B and $k$4 on Llama-3.2-3B while adding only about $k$5 additional parameters (Ye et al., 3 Feb 2026). SAC reports stronger performance than ICAE, 500xCompressor, and EPL across $k$6, $k$7, and $k$8 compression, with advantages widening as compression becomes more aggressive; on out-of-domain MRQA at $k$9, SAC reaches average $$\mathcal{D'}=\arg\min_k \{k \mid \textnormal{Correct}(q,\mathcal{D}_k)=1\},$$0 versus EPL’s $$\mathcal{D'}=\arg\min_k \{k \mid \textnormal{Correct}(q,\mathcal{D}_k)=1\},$$1 (Liu et al., 10 Oct 2025).

For deployment-oriented compressors, the throughput story is often as important as quality. IC-Former reports theoretical FLOPs about $$\mathcal{D'}=\arg\min_k \{k \mid \textnormal{Correct}(q,\mathcal{D}_k)=1\},$$2 of ICAE in its studied setting, compression speedups of $$\mathcal{D'}=\arg\min_k \{k \mid \textnormal{Correct}(q,\mathcal{D}_k)=1\},$$3 to $$\mathcal{D'}=\arg\min_k \{k \mid \textnormal{Correct}(q,\mathcal{D}_k)=1\},$$4 times, and over $$\mathcal{D'}=\arg\min_k \{k \mid \textnormal{Correct}(q,\mathcal{D}_k)=1\},$$5 of baseline evaluation performance (Wang et al., 2024). CompLLM, at $$\mathcal{D'}=\arg\min_k \{k \mid \textnormal{Correct}(q,\mathcal{D}_k)=1\},$$6 compression, reports TTFT speedups up to $$\mathcal{D'}=\arg\min_k \{k \mid \textnormal{Correct}(q,\mathcal{D}_k)=1\},$$7 at high context lengths and a 50% reduction in KV cache size, while remaining comparable to uncompressed context and sometimes surpassing it on very long sequences (Berton et al., 23 Sep 2025). UniGist shows that hardware-aligned sequence-level compression can translate into real kernel gains: at 128K sequence length, forward attention time drops from 863.9 ms for causal full attention to 271.8 ms at ratio 4 and 130.9 ms at ratio 8 (Deng et al., 19 Sep 2025).

Textual resident compression can also be competitive. TaC-C reports that at $$\mathcal{D'}=\arg\min_k \{k \mid \textnormal{Correct}(q,\mathcal{D}_k)=1\},$$8 and $$\mathcal{D'}=\arg\min_k \{k \mid \textnormal{Correct}(q,\mathcal{D}_k)=1\},$$9 compression it surpasses the strongest competitor by $O(kn+k^2)\sim O(kn)$0 and $O(kn+k^2)\sim O(kn)$1 in average F1, and by $O(kn+k^2)\sim O(kn)$2 and $O(kn+k^2)\sim O(kn)$3 in average EM, respectively (Ma et al., 27 May 2026). QUITO shows that plain-text query-guided filtering can still outperform token-importance baselines: on NaturalQuestions under LongChat-13B-16k, it reaches 58.9 accuracy at $O(kn+k^2)\sim O(kn)$4 compression and 53.1 at $O(kn+k^2)\sim O(kn)$5, both above the reported baselines (Wang et al., 2024). In tool use, selective plus block compression achieves performance comparable to the no-compression upper bound under up to $O(kn+k^2)\sim O(kn)$6 compression on API-Bank and APIBench (Xu et al., 2024).

RCC demonstrates that recurrent latent compression can scale context length dramatically: it reports BLEU-4 close to 0.95 at $O(kn+k^2)\sim O(kn)$7 compression on text reconstruction and nearly 100% passkey retrieval at sequence length 1M (Huang et al., 2024). These results are unusually strong evidence that resident latent context can preserve targeted long-range information even under severe storage constraints.

6. Verification, limitations, and persistent misconceptions

A central limitation across the literature is that compression is often evaluated only by downstream task accuracy, not by what semantic state was preserved. Context Codec is the clearest attempt to close that gap. It introduces Critical Atom Recall

$O(kn+k^2)\sim O(kn)$8

Weighted Atom Recall

$O(kn+k^2)\sim O(kn)$9

and Commitment Density

$k$0

together with a round-trip recoverability criterion over normalized semantic atoms (Trukhina et al., 17 May 2026). Its taxonomy of omission, weakening, mutation, polarity flip, scope error, temporal/decision error, hallucinated commitment, and safety-boundary erasure is a direct statement that not all compression failures are equally harmful.

Other limitations are more architectural. AdaComp depends on retriever ranking quality and cannot rescue cases where crucial evidence lies lower in the ranking but the intervening documents are poor (Zhang et al., 2024). SAC is sensitive to anchor choice: random anchor selection degrades performance sharply, showing that anchor quality remains a real bottleneck (Liu et al., 10 Oct 2025). ATACompressor’s adaptive allocation can under-allocate on short but subtle inputs, because relevant-length prediction is only an approximation of needed latent capacity (Li et al., 3 Feb 2026). CompLLM’s segment independence enables reuse and scalability, but it also means cross-segment interactions are deferred to the downstream LLM rather than modeled during compression itself (Berton et al., 23 Sep 2025).

Interpretability and faithfulness remain open issues for latent resident states. ComprExIT notes that continuous compressed representations are less interpretable and that important information may be omitted (Ye et al., 3 Feb 2026). LCC explicitly trades interpretability for portable latent memory and incurs substantial one-time compilation cost, making it most attractive in write-once, read-many settings rather than single-query use (Li et al., 31 Jan 2026). TaC-C shows a distinct failure mode for textual latent compression: without anti-hacking constraints, the thinking trace degenerates into an answer channel rather than a reusable compressed context (Ma et al., 27 May 2026).

A final misconception is terminological. Not every method with “contextual compression” in its name is about compressing runtime context state. CCE is explicit that it compresses model parameters, not prompt state, KV-cache state, or long-context memory (Schmitt et al., 12 Feb 2025). The modern context-resident literature is instead defined by where the compressed information lives after compression: inside retained documents, text spans, soft tokens, KV caches, recurrent memories, compiled latent buffers, or explicit semantic state.

Context-resident compression has therefore evolved into a broad design space rather than a single technique. Its unifying concern is durable access to the right information under tight token, compute, and memory budgets. The strongest current systems differ sharply in representation—document prefixes, plain text, latent slots, KV states, reasoning traces, or commitment atoms—but converge on the same systems objective: keep the information that must remain live, and make that live state cheaper to store, transmit, and decode.