Adaptive Context Compression for RAG (ACC-RAG)

• ACC-RAG is a framework that replaces fixed context retrieval with query-dependent compression, selecting minimal yet sufficient evidence for accurate responses.
• It employs various strategies—including subset selection, abstractive summarization, dense embedding, and graph-based methods—to optimize context usage under strict budgets.
• Empirical evaluations show that ACC-RAG can significantly reduce token counts and latency while maintaining or improving answer accuracy across diverse QA tasks.

Adaptive Context Compression for Retrieval-Augmented Generation (ACC-RAG) denotes a class of retrieval-time and generation-time strategies that replace fixed context construction with query-conditional compression, selection, or expansion under explicit token, memory, or latency budgets. In current literature, the term appears both as the name of a specific framework that “dynamically adjusts compression rates based on input complexity” and as a broader description of methods that seek “minimal sufficient information” rather than fixed top-$k$ context, fixed compression ratios, or unconditional window expansion (Guo et al., 24 Jul 2025, Zhang et al., 2024, Peng et al., 31 Dec 2025, Langlois, 15 Jun 2026). Across these works, ACC-RAG is best understood not as a single algorithm but as an organizing principle: preserve answer-critical evidence while suppressing redundancy, boundary noise, misleading material, or structurally irrelevant context.

1. Conceptual scope and problem formulation

ACC-RAG is motivated by a recurrent failure mode of standard RAG: the retrieved context is usually much longer than the query, and the additional tokens are not uniformly useful. Several papers articulate this in distinct but compatible forms. Fixed-rate embedding compression can over-compress hard queries and under-compress easy ones; top-$k$ retrieval can waste budget on near-duplicates; fixed-length chunking can split a minimal evidence set across adjacent chunks; and abstractive compressors can omit answer-bearing spans when retrieved documents contain irrelevant or factually wrong material (Guo et al., 24 Jul 2025, Peng et al., 31 Dec 2025, Langlois, 15 Jun 2026, Kim, 19 Nov 2025).

A common formalization is to define compression as selecting the smallest context that still supports a correct answer. AdaComp labels the “compression rate” by the minimal top-$k$ prefix $\mathcal{D}_k$ such that the deployed RAG system answers correctly, effectively learning $k^*(q)$ from system behavior rather than from heuristic budgets (Zhang et al., 2024). AdaGReS makes the same intuition set-theoretic by maximizing a relevance term minus a redundancy term under a token budget,

$F(q, C) = \alpha S_{qC}(q, C) - \beta S_{CC}(C),$

where $S_{qC}$ sums query–chunk similarity and $S_{CC}$ sums pairwise chunk–chunk similarity inside the selected set (Peng et al., 31 Dec 2025). In SCAR, the problem is phrased as boundary fragmentation: the evidence required to answer a query spans adjacent chunks, but the retriever returns only a subset, so the LLM sees an incomplete logical unit (Langlois, 15 Jun 2026).

Taken together, these formulations show that ACC-RAG is not merely “compression after retrieval.” It is a budgeted inference problem in which the system must decide, per query, whether to keep raw passages, summarized representations, structured concepts, local neighbors, graph neighborhoods, or dense context embeddings. This suggests that the central object of optimization is not raw recall or raw compression ratio alone, but the utility of context under constraints.

2. Major compression primitives

The literature instantiates ACC-RAG through several distinct primitives. Some methods compress by selection, some by abstraction, some by representation learning, and some by structured expansion followed by pruning. The following families recur across the current corpus.

Family	Core mechanism	Representative systems
Adaptive subset selection	Choose the smallest useful subset under relevance, redundancy, or local-continuity constraints	AdaComp (Zhang et al., 2024), AdaGReS (Peng et al., 31 Dec 2025), SCAR (Langlois, 15 Jun 2026)
Abstractive or semantic compression	Rewrite noisy evidence into shorter answer-focused or concept-focused forms	ACoRN (Kim, 19 Nov 2025), AMR concept distillation (Shi et al., 2024)
Dense embedding compression	Replace long token sequences with learned context embeddings or hierarchical compressed prefixes	COCOM (Rau et al., 2024), FlexRAG (Liu et al., 2024), ACC-RAG (Guo et al., 24 Jul 2025)
Multi-scale and structural compression	Retrieve on compressed fine-grained units, then reconstruct only relevant neighborhoods	MacRAG (Lim et al., 10 May 2025), REFRAG (Lin et al., 1 Sep 2025)
Structured memory or graph evidence	Compress evidence into higher-order graph, hypergraph, or evidence-subgraph states	HGMem (Zhou et al., 30 Dec 2025), MAGE-RAG (Zuo et al., 14 Jun 2026)
Budget recovery in agentic systems	Compress non-document context that competes with retrieval budget	Tool-schema compression (Sakizli, 24 May 2026)

Compression by selection is the most direct ACC-RAG variant. AdaComp predicts how many of the top retrieved documents to keep; AdaGReS greedily adds chunks while the marginal gain remains positive under a relevance–redundancy objective; SCAR starts from dense retrieval seeds and adds neighboring chunks only when a continuity-aware relevance condition is satisfied (Zhang et al., 2024, Peng et al., 31 Dec 2025, Langlois, 15 Jun 2026).

Compression by abstraction operates on content rather than indices. ACoRN inserts a query-focused compressor between retrieval and generation and trains it against evidential-only teacher summaries while exposing it to irrelevant and factual-error documents, thereby making summarization robust to retrieval noise and positional bias (Kim, 19 Nov 2025). The AMR-based concept distillation line compresses long retrieved documents into concept lists derived from informative AMR nodes, especially named entities, wiki anchors, and dates, then supplies those concepts as “facts” to the generator (Shi et al., 2024).

Dense embedding compression replaces long raw text with compact learned vectors. COCOM maps a long context $\mathcal{C}$ to a handful of context embeddings $\mathcal{E}$, yielding large reductions in decoding time and FLOPs while preserving generation quality relative to earlier embedding compressors (Rau et al., 2024). FlexRAG similarly compresses retrieved contexts into compact embeddings before the LLM encodes them, but adds selective preservation of important contexts and training across multiple compression ratios, making the compressor itself flexible with respect to budget (Liu et al., 2024). The framework explicitly titled ACC-RAG combines a hierarchical compressor with a selector that decides how many compressed embeddings are sufficient for a given query (Guo et al., 24 Jul 2025).

Multi-scale and structural methods compress at one level and expand at another. MacRAG compresses chunks into summaries, slices those summaries for fine-grained retrieval, then scales back up to original chunks, documents, and $k$0-hop neighbors, creating a bounded long context from a coarse-to-fine index (Lim et al., 10 May 2025). REFRAG observes that RAG contexts often induce block-diagonal attention because retrieved passages are semantically sparse with respect to one another; it therefore compresses context into chunk embeddings, senses usefulness, and selectively expands chunks, reducing unnecessary prefill computation during decoding (Lin et al., 1 Sep 2025).

Graph and memory methods generalize ACC-RAG beyond flat passage sets. HGMem treats memory as a hypergraph whose hyperedges are higher-order memory points; update, insertion, and especially merging compress primitive facts into more expressive propositions for subsequent multi-step reasoning (Zhou et al., 30 Dec 2025). MAGE-RAG builds a multigranular evidence graph over pages and elements in long multimodal PDFs, then uses an online evidence controller to activate, open, search, and prune evidence under explicit budgets (Zuo et al., 14 Jun 2026). Tool-schema compression extends the same logic to agentic RAG, where tool definitions occupy the same context budget as retrieved evidence; compressing schemas can restore a non-zero RAG budget under tight windows (Sakizli, 24 May 2026).

3. Canonical decision rules and optimization patterns

A distinctive feature of ACC-RAG is that compression is usually expressed as a policy, not a fixed heuristic. Three optimization patterns dominate: relative local expansion, set-level budgeted selection, and learned stopping over hierarchical representations.

SCAR provides a canonical local policy. For a retrieved chunk $k$1 and its neighbor $k$2, it defines a boundary penalty

$k$3

an expansion score

$k$4

and an adaptive expansion condition

$k$5

With $k$6 and $k$7, the policy keeps a neighbor only if it is sufficiently query-relevant relative to the base chunk after applying a continuity penalty. Because the threshold is relative to the base chunk’s own relevance, the paper argues that the rule is approximately scale-robust across embedding models (Langlois, 15 Jun 2026).

AdaGReS provides the canonical set-level formulation. Its objective

$k$8

is optimized greedily under a token budget using the marginal gain

$k$9

Its distinctive contribution is an instance-adaptive $k$0, calibrated from candidate-pool relevance, candidate-pool redundancy, and expected set size induced by $k$1. The paper further shows that the objective is $k$2-approximately submodular under bounded similarity assumptions, providing a near-optimality argument for the greedy selector (Peng et al., 31 Dec 2025).

The hierarchical ACC-RAG framework makes compression a sequential stopping problem over prefixes of compressed embeddings. Each retrieved document $k$3 is compressed to

$k$4

the retrieved sequence is concatenated, and a selector progressively exposes prefixes $k$5 to the decoder. The stopping policy is trained with a trajectory reward

$k$6

so that the model learns to stop as soon as a compressed prefix is sufficient for a correct answer (Guo et al., 24 Jul 2025).

Other systems fit into the same pattern with different state spaces. ACoRN trains a compressor $k$7 on noisy inputs $k$8 but supervises it with teacher summaries $k$9 generated only from evidential documents, using the standard sequence loss

$\mathcal{D}_k$0

which makes answer-focused compression emerge from supervision design rather than from an explicit compression controller (Kim, 19 Nov 2025). REFRAG, by contrast, adds an RL policy over chunk embeddings that decides which chunks to expand from compressed embeddings back into full tokens, using reward based on decoder perplexity over the next segment (Lin et al., 1 Sep 2025).

These formulations indicate that ACC-RAG spans both discrete and continuous control. Some methods choose integers such as $\mathcal{D}_k$1 or $\mathcal{D}_k$2; some choose graph actions such as ActivateNode or OpenNode; some choose local binary expansion decisions; and some choose dense embedding counts. The shared property is that context size is treated as an optimizable variable coupled to downstream utility.

4. Empirical behavior and operating regimes

Across the literature, ACC-RAG methods repeatedly show that adaptive compression can recover most or all of the benefit of long retrieved context while materially reducing token count, latency, or memory. The exact trade-off depends on task structure.

The ACC-RAG framework built around hierarchical compression reports an average Match of 35.23 with a first-token inference time of 697 ms, compared with 38.17 and 3371 ms for standard RAG on five QA datasets; on some datasets it matches or exceeds vanilla RAG while remaining more than $\mathcal{D}_k$3 faster in first-token latency (Guo et al., 24 Jul 2025). AdaComp reports roughly 45–55% fewer tokens than Top-5 retrieval while remaining near Top-5 performance on Natural Questions, TriviaQA, and HotpotQA, and it improves more clearly on open-ended conversational questions, where fixed top-$\mathcal{D}_k$4 is especially misaligned with context need (Zhang et al., 2024).

SCAR isolates the boundary-fragmentation regime and shows that adaptive local expansion can reduce context without collapsing recall. On boundary-fragmented queries, SCAR achieves 92.8% recall with 7.84 chunks, versus 96.7% recall with 10.16 chunks for static windowing; on downstream RAGAS evaluation over the 10-K corpus, it preserves faithfulness at 4.99/5 while reducing context tokens from 1,281 to 934, a 27.1% reduction (Langlois, 15 Jun 2026). AdaGReS addresses a different regime—redundancy under token budgets—and reports consistently higher IoU than similarity-only baselines on Natural Questions and a high-redundancy biomedical corpus, with dynamic $\mathcal{D}_k$5 improving robustness over fixed-$\mathcal{D}_k$6 variants (Peng et al., 31 Dec 2025).

Abstractive compression can be highly effective when retrieval noise is the primary bottleneck. ACoRN trains T5-large and Flan-T5-large compressors that operate at compression ratios near $\mathcal{D}_k$7–$\mathcal{D}_k$8 while improving or preserving EM and F1 on NQ, TriviaQA, and PopQA relative to uncompressed Top-5 or other compression baselines; its preserving-answer-string ratio remains competitive even under much stronger compression than token-pruning methods (Kim, 19 Nov 2025). Semantic compression via AMR concept distillation shows a different pattern: it improves robustness as the number of supporting documents rises, particularly for larger backbone LLMs, while reducing word count by more than 60% relative to feeding full retrieved documents (Shi et al., 2024).

Multi-scale retrieval and graph-based evidence construction broaden the empirical picture. MacRAG improves LongBench variants of HotpotQA, 2WikiMultihopQA, and Musique, and its efficient generation modes can outperform LongRAG’s best Full_EF strategy while using 8.19% less context on 2WikiMultihopQA and 45.87% less context on Musique (Lim et al., 10 May 2025). MAGE-RAG reports 52.75 overall accuracy on LongDocURL and 53.26 accuracy with 51.19 F1 on MMLongBench-Doc, with budget-performance curves showing that evidence subgraph construction can improve dispersed-evidence coverage without simply inflating page count (Zuo et al., 14 Jun 2026). In agentic RAG, tool-schema compression demonstrates a sharply budget-dependent effect: under 8K context, verbose JSON tool definitions can consume the entire window, yielding near-zero EM, whereas compressed schemas restore RAG functionality and large exact-match gains; at 32K, where both schema formats fit, the differences largely disappear, confirming that the effect is budget-driven rather than intrinsic to the compressed representation (Sakizli, 24 May 2026).

A consistent empirical theme therefore emerges: adaptive compression helps most when the baseline sits in a pathological regime—boundary fragmentation, heavy redundancy, retrieval noise, multimodal overbreadth, agentic tool overflow, or long-context decoding overhead. When raw context already fits comfortably and is reasonably clean, the gains become smaller and may even flatten.

5. Limitations, failure modes, and recurrent misconceptions

ACC-RAG does not remove the dependence of RAG on retrieval quality. Several methods are explicit about this. SCAR can only expand to contiguous neighbors and cannot recover evidence that is non-adjacent to the initial Top-$\mathcal{D}_k$9 seed; it also remains dependent on the base retriever’s neighborhood (Langlois, 15 Jun 2026). AdaComp cannot compensate for documents that were never retrieved into the top-$k^*(q)$0 list because its action space is adaptive truncation of retrieved prefixes rather than retrieval correction (Zhang et al., 2024). AdaGReS assumes a reasonably well-behaved embedding space and can miss ideal combinations under highly skewed redundancy structure, despite its approximate submodularity analysis (Peng et al., 31 Dec 2025).

Abstractive methods carry a different set of liabilities. ACoRN defines evidential documents by answer-string inclusion, which can misclassify paraphrased evidence or answer-bearing but misleading passages; it also relies on teacher summaries from GPT-3.5 and retains residual vulnerability when factual-error documents are present (Kim, 19 Nov 2025). The AMR concept-distillation approach is query-independent at the node-selection stage and depends on parser quality; it can therefore preserve concepts that are structurally salient but not query-salient, and smaller LMs may struggle to reconstruct full semantics from sparse concept lists (Shi et al., 2024).

A common misconception is that more context is always better. Multiple papers argue against this. AdaComp’s performance curves peak and then decline as additional documents introduce noise (Zhang et al., 2024). Tool-schema compression shows that recovering more available chunk slots is essential when the system is in the $k^*(q)$1 or near-overflow regime, but the same paper also observes distractor dilution at 16K when extra retrieved chunks do not improve—and can slightly hurt—accuracy for smaller models (Sakizli, 24 May 2026). MAGE-RAG’s budget sweeps likewise show non-monotonic effects of increasing initial page budget $k^*(q)$2, actions per round, or iterations; larger candidate pools can reduce unanswerable calibration and increase noise (Zuo et al., 14 Jun 2026).

Another misconception is that ACC-RAG is synonymous with aggressive summarization. The literature shows at least four incompatible but equally legitimate meanings of “compression”: removing redundant retrieved items, replacing text with embeddings, summarizing evidence, and compressing non-document context such as tool schemas. A plausible implication is that ACC-RAG is best treated as a systems-level resource-allocation problem rather than as a single summarization technique.

6. Extensions and research directions

The present literature suggests several converging directions. Relative decision rules, such as SCAR’s comparison of neighbor utility to the relevance of the base chunk, appear unusually robust across corpora and embedding models and may generalize to broader chunk- or graph-expansion settings (Langlois, 15 Jun 2026). Set-level budgeted optimization, as in AdaGReS, suggests that ACC-RAG can benefit from explicit redundancy terms, approximate-submodular objectives, and instance-adaptive trade-off parameters rather than fixed compression schedules (Peng et al., 31 Dec 2025).

Learned dense compression remains an open frontier. The hierarchical ACC-RAG framework, COCOM, and FlexRAG all show that a compressor can expose a monotone, multigranular representation from which a selector or budget controller can decide how much context to reveal; this suggests future systems in which retrieval, compression, and stopping are trained jointly rather than as separate modules (Guo et al., 24 Jul 2025, Rau et al., 2024, Liu et al., 2024). REFRAG extends this to the decoding path by exploiting the sparsity structure of RAG attention and selectively expanding chunks, indicating that ACC-RAG need not end at prompt construction but can continue into runtime representation management (Lin et al., 1 Sep 2025).

Structured evidence spaces provide another extension route. MacRAG’s coarse-to-fine retrieval, HGMem’s hypergraph memory, and MAGE-RAG’s evidence-subgraph controller all suggest that ACC-RAG can be reframed as budgeted graph construction, where the compressor decides not only how much evidence to keep but also what structural relations must remain explicit for subsequent reasoning (Lim et al., 10 May 2025, Zhou et al., 30 Dec 2025, Zuo et al., 14 Jun 2026). In multimodal and agentic settings, this perspective becomes especially natural because pages, elements, tool schemas, and working memory all compete for the same context budget.

Finally, several papers explicitly point toward learned policy improvement. SCAR proposes that future work could learn $k^*(q)$3, $k^*(q)$4, or a more complex expansion decision; AdaGReS suggests RL or meta-learning for $k^*(q)$5; ACoRN notes the absence of an explicit adaptive compression controller; MAGE-RAG’s action traces invite imitation or reinforcement learning; and tool-schema compression suggests a unified budget allocator spanning schemas, history, retrieval, and output (Langlois, 15 Jun 2026, Peng et al., 31 Dec 2025, Kim, 19 Nov 2025, Zuo et al., 14 Jun 2026, Sakizli, 24 May 2026). This suggests that the next phase of ACC-RAG research may move from handcrafted but effective policies toward end-to-end or semi-end-to-end controllers that optimize answer quality, latency, and memory jointly across heterogeneous context types.