CompactRAG: Efficient Retrieval and Generation

Updated 9 July 2026

CompactRAG is a family of systems that minimizes retrieved evidence to supply only the necessary context for accurate answers.
It integrates techniques such as adaptive document truncation, query-aware summarization, and embedding-space compression to reduce token usage and inference latency.
In multi-hop QA frameworks, CompactRAG uses a two-stage process with offline atomic QA extraction and minimal online LLM calls to enhance efficiency.

CompactRAG denotes a family of retrieval-augmented generation systems that deliberately reduce the amount of retrieved evidence, prompt context, retrieval work, index footprint, or inference orchestration required for a query, while attempting to preserve answer quality and factual grounding. In the recent literature, the term is used both as a general design principle for making RAG shorter, denser, and more query-relevant under tight resource budgets, and as the name of a specific multi-hop question-answering framework that replaces iterative passage reading with an offline atomic QA knowledge base (Li et al., 4 Apr 2025, Jiang, 23 Oct 2025, Yang et al., 5 Feb 2026). Across these usages, the central objective is to supply only the minimal sufficient information for a given query, reader, and deployment regime.

1. Conceptual scope and formal problem setting

CompactRAG is motivated by a common failure mode of standard RAG: noisy or overly long retrieved contexts degrade generation quality and raise inference cost. The literature frames compactness through several jointly optimized quantities, including prompt token count, retrieval depth $k$ , index memory, compute budget, and end-to-end latency (Jiang, 23 Oct 2025). In that sense, CompactRAG is not a single algorithm but a systems objective spanning retrieval, compression, ranking, prompting, caching, and hardware placement.

A recurring formalization treats retrieved documents as a ranked set $D = \{d_1, d_2, \dots, d_N\}$ and asks for the smallest prefix that still permits a correct answer. AdaComp defines the minimum top- $k$ $k^*$ as the smallest $k$ such that the RAG system generates the correct answer $y$ with $D_k = \{d_1,\dots,d_k\}$ , and defines the compression rate as $r = k^*/K_{\max}$ (Zhang et al., 2024). This formulation makes compactness query-dependent: simple queries and high-quality retrieval may require only a small prefix, whereas multi-hop questions or low-quality retrieval may require substantially more evidence.

The literature also distinguishes two compression regimes. ArcAligner describes hard compression in text space, such as token pruning, sentence selection, and summarization, and soft compression in embedding space, where passages are encoded into a small number of dense vectors that are injected into the generator (Li et al., 8 Jan 2026). Hard compression directly shortens the prompt, while soft compression reduces sequence length more aggressively but introduces an alignment problem between compressed embeddings and the LLM’s hidden space.

2. Major architectural families

The term “CompactRAG” covers several recurring architectural patterns rather than a single canonical pipeline.

Family	Representative systems	Characteristic mechanism
Adaptive document truncation	AdaComp (Zhang et al., 2024)	Predict the number of top-ranked documents to keep
Query-aware summarization and clustering	EDC $^{2}$ -RAG (Li et al., 4 Apr 2025), MacRAG (Lim et al., 10 May 2025)	Cluster, summarize, slice, and merge only relevant evidence
Embedding-space compression	FlexRAG (Liu et al., 2024), ACC-RAG (Guo et al., 24 Jul 2025), ArcAligner (Li et al., 8 Jan 2026)	Replace raw text with compressed embeddings or slots
Retrieval gating and fallback	L-RAG (Voloshyn, 10 Jan 2026), Tiny-Critic RAG (Wu et al., 1 Mar 2026)	Defer or trigger retrieval only when uncertainty or a binary gate fires
Post-retrieval relevance control	Lightweight Relevance Grader (Jeong, 17 Jun 2025)	Filter or rerank retrieved candidates with a small classifier
Compact indexing and storage	MiniRAG (Fan et al., 12 Jan 2025), ARC (Lin et al., 4 Nov 2025), CUBO (Astrino, 3 Feb 2026), DIRC-RAG (Shao et al., 29 Oct 2025)	Reduce storage, cache size, or retrieval hardware overhead

These families are complementary. A single deployment can, for example, use a compact index, retrieve a modest candidate pool, apply a lightweight relevance grader, compress the surviving context, and then invoke a small or medium reader with a bounded prompt budget. This suggests that CompactRAG is best understood as a layered optimization stack rather than a single retrieval heuristic.

3. Adaptive context compression and context construction

Document-level adaptive compression is exemplified by AdaComp, which retains the top- $\hat{k}$ documents predicted by a fine-tuned Llama2-7B + LoRA classifier. The predictor takes the query and the full top- $D = \{d_1, d_2, \dots, d_N\}$ 0 retrieved documents as text-only input and outputs a discrete class representing how many documents to keep. On open-domain QA with a Llama2-7B generator, AdaComp reduced tokens from 802 to 441 on Natural Questions while moving from EM 40.64 / F1 71.09 to EM 40.13 / F1 70.96; on TriviaQA it reduced tokens from 808 to 468 with EM 48.58 / F1 80.20 versus EM 47.15 / F1 79.40; and on HotpotQA it reduced tokens from 819 to 527 while improving from EM 25.09 / F1 59.56 to EM 26.36 / F1 60.46. The predictor achieved approximately 65% accuracy, with most mispredictions within $D = \{d_1, d_2, \dots, d_N\}$ 1 classes (Zhang et al., 2024).

A second line of work compresses retrieved evidence through query-aware summarization or multi-scale reorganization. EDC $D = \{d_1, d_2, \dots, d_N\}$ 2-RAG groups retrieved documents into dynamic, query-rooted clusters, then compresses each cluster into query-aware summaries that remove irrelevant spans and redundancy. On WebQ, it reported average F1 89.23 versus 88.75 for RALM and 88.25 for Raw Compression; on FELM it achieved Top-10 Balanced Acc 64.03 and average accuracy 62.26 versus 55.65 for RALM (Li et al., 4 Apr 2025). MacRAG similarly compresses documents into summaries, slices them into overlapping fine-grained units, retrieves slices, reranks parent chunks, and then expands to neighboring chunks and distinct documents. On LongRAG Full_EF with GPT-4o, MacRAG improved average F1 from 58.64 to 63.93, including 2WikiMultiHopQA from 65.89 to 73.19 and MuSiQue from 43.83 to 50.09 (Lim et al., 10 May 2025).

Embedding-space compression shifts compactness from text selection to representation learning. FlexRAG compresses retrieved contexts into compact embeddings that are optimized for the downstream frozen LLM, with optional selective preservation of important context. At 8× compression, it improved ODQA average EM from 37.25 to 45.37 while reducing CUDA time from 7.78 s to 2.48 s, and improved LMQA average F1 from 22.34 to 29.93 (Liu et al., 2024). ACC-RAG pushes this further by precompressing documents into hierarchical embeddings offline and using a reinforcement-learned selector to stop at the minimal sufficient prefix online; its average Match was 35.23 with FTIT 697, versus Vanilla RAG at Match 38.17 with FTIT 3371, yielding more than 4× faster inference on average (Guo et al., 24 Jul 2025). ArcAligner addresses the alignment problem introduced by aggressive embedding compression by inserting slot-only LoRA adapters and gate-controlled recursion into transformer layers; at ×24 compression it reported HotpotQA EM 26.60, F1 34.92, Acc 33.40, outperforming COCOM ×16 on F1 and Acc at comparable compression regimes (Li et al., 8 Jan 2026).

4. Retrieval minimization, gating, and lightweight control

CompactRAG does not always begin by compressing already retrieved text; some systems first ask whether expensive retrieval should occur at all. L-RAG implements a two-tier architecture in which the model first sees a compact document summary and only retrieves detailed chunks when early-token predictive entropy exceeds a threshold. On SQuAD 2.0 with Phi-2, $D = \{d_1, d_2, \dots, d_N\}$ 3 yielded 78.2% accuracy versus 77.8% for Standard RAG while reducing retrieval by 8%, and $D = \{d_1, d_2, \dots, d_N\}$ 4 yielded 76.0% accuracy with 26% retrieval reduction. When retrieval latency exceeded 500 ms, L-RAG saved about 80–210 ms per query, and correct versus incorrect predictions showed statistically significant entropy separation with $D = \{d_1, d_2, \dots, d_N\}$ 5 (Voloshyn, 10 Jan 2026).

A closely related strategy is to separate retrieval evaluation from answer generation with a very small gate. Tiny-Critic RAG uses a LoRA-adapted Qwen3-1.7B gate that emits only two masked tokens, $D = \{d_1, d_2, \dots, d_N\}$ 6 or $D = \{d_1, d_2, \dots, d_N\}$ 7, and triggers a fallback tool call only when evidence is judged deficient. Under 45% adversarial noise, it achieved Routing F1 0.912 and Faithfulness 0.86, compared with Heavy-CRAG at Routing F1 0.934 and Faithfulness 0.88, while reducing TTFT from 1235 ms to 492 ms and explicit routing cost from $D = \{d_1, d_2, \dots, d_N\}$80.06 per 10k queries (Wu et al., 1 Mar 2026).

Between retrieval and generation, some CompactRAG systems insert a relevance-verification stage. “Lightweight Relevance Grader in RAG” fine-tuned Llama-3.2-1B as a binary relevance classifier over concatenated query-document pairs. In its best configuration, precision rose from approximately 0.1301–0.1312 to 0.7750, with Accuracy 0.9353, Recall 0.6670, and F1 0.7170, approaching the reported precision of Llama-3.1-70B while using a 1B-class model (Jeong, 17 Jun 2025). This line of work treats compactness not only as shorter prompts, but as stricter control over which retrieved evidence is allowed into the prompt at all.

5. Compact indexes, caches, and edge deployment

At the retrieval-system layer, CompactRAG often means storing only a compact, high-value working set. ARC constructs an agent-specific cache $D = \{d_1, d_2, \dots, d_N\}$9 under a strict memory budget, using a priority score that combines Distance–Rank Frequency, hubness centrality, and a memory penalty. On a 3.0 MB cache, ARC reduced storage to 0.015% of the original corpus, reached up to 79.8% has-answer rate, and reduced average retrieval latency by about 80% (Lin et al., 4 Nov 2025). MiniRAG addresses the same constraint from the indexing side: it builds a semantic-aware heterogeneous graph over chunks and named entities, then performs topology-enhanced retrieval suitable for SLMs. It reported comparable performance to LLM-based methods while requiring only 25% of the storage space (Fan et al., 12 Jan 2025).

Laptop-scale and privacy-preserving deployments translate compactness into strict systems budgets. CUBO presents a self-contained RAG platform for consumer laptops with 16 GB shared memory, using streaming ingestion, a tiered hybrid index, and a 4-bit local LLM. It reported competitive Recall@10 ranging from 0.48 to 0.97 across BEIR domains, p50 retrieval latency of 185 ms on C1,300 laptops, and operation within a hard 15.5 GB RAM ceiling (Astrino, 3 Feb 2026). The system-level lesson is that CompactRAG can be realized without cloud vector stores or large-memory servers when retrieval, indexing, and orchestration are co-designed for bounded memory.

At the hardware extreme, DIRC-RAG pushes CompactRAG into compute-in-memory retrieval. It stores document embeddings directly in a Digital In-ReRAM Computation macro, supports query-stationary dataflow, and reports an on-chip non-volatile memory density of 5.178 Mb/mm $k$ 0, throughput of 131 TOPS, retrieval latency of 5.6 $k$ 1s/query for a 4 MB store, and energy consumption of 0.956 $k$ 2J/query while maintaining retrieval precision (Shao et al., 29 Oct 2025). This suggests that compactness can be pursued not only through algorithmic compression but also through physical colocation of storage and similarity computation.

6. “CompactRAG” as a specific multi-hop QA framework

A distinct use of the term appears in “CompactRAG: Reducing LLM Calls and Token Overhead in Multi-Hop Question Answering,” which defines a two-stage framework for multi-hop QA (Yang et al., 5 Feb 2026). In the offline stage, an LLM reads the corpus once and converts each document into an atomic QA knowledge base consisting of minimal, fine-grained question-answer pairs. Questions must be short wh-questions with explicit entity names, and answers must be exact substrings of the passage. Each pair is concatenated as $k$ 3 and embedded with Contriever for dense retrieval.

In the online stage, the LLM is invoked exactly twice, regardless of hop count: once for sub-question decomposition and once for final answer synthesis. The intermediate loop uses only local modules. For each sub-question, CompactRAG retrieves top- $k$ 4 atomic QA pairs from the knowledge base, extracts the answer with a RoBERTa-base span predictor, and rewrites the next sub-question with Flan-T5-small so that entity mentions remain explicit rather than pronominal. The retrieval similarity is cosine similarity over the shared encoder space. This architecture directly targets the inefficiencies of iterative multi-hop RAG, where LLM calls and token usage scale with the number of reasoning hops (Yang et al., 5 Feb 2026).

On sampled dev subsets of 250 questions per dataset, CompactRAG reported average tokens per query of 1.9K, versus 2.7K for Vanilla RAG, 6.9K for Self-Ask, 10.2K for IRCoT, and 4.7K for Iter-RetGen. On HotpotQA, the LLaMA Reader version achieved EM 45.20, F1 66.21, Acc 70.40, while the GPT-4 Reader version achieved EM 49.60, F1 69.54, Acc 77.20. On MuSiQue, the LLaMA Reader version achieved EM 26.80, F1 37.63, Acc 41.20, surpassing Iter-RetGen on all three reported metrics (Yang et al., 5 Feb 2026).

The ablations clarify the role of the local modules. On HotpotQA / 2WikiMultiHopQA / MuSiQue, the full system achieved Acc 70.4 / 53.2 / 41.2; removing the rewriter reduced these to 63.2 / 48.8 / 35.8; removing both extractor and rewriter reduced them further to 58.4 / 44.2 / 32.6 (Yang et al., 5 Feb 2026). In this framework, compactness is achieved less by shortening passages than by changing the corpus representation itself: the online reader no longer consumes multi-hop raw documents, but a compact atomic QA substrate with stable entity grounding.

7. Limitations, evaluation controversies, and reader-aware design

Compactness is not evaluation-neutral. “Fixed RAG Compression Collapses Measured Reader Scaling” shows that a fixed compressed evidence layer can raise average accuracy while obscuring the advantages of stronger readers. Across 20 readers and ten domain-method settings, compression gain decreased with reader baseline in nine of ten settings with $k$ 5. Generic summarization flipped 31% of pairwise model rankings on LongMemEval-S, and a fixed HotpotQA compressor hid 80% of the raw upgrade from Qwen 7B to GPT-4.1-mini. The paper models the net compression effect as $k$ 6, where noise reduction helps weak readers and information loss harms strong readers (Panthi et al., 20 Jun 2026). This result establishes an important methodological constraint: CompactRAG should be evaluated across multiple readers, not treated as a universally beneficial preprocessing layer.

This controversy connects directly to common failure modes reported elsewhere. AdaComp’s predictor has approximately 65% accuracy, so underestimation can cause over-compression and overestimation can pass more tokens than necessary (Zhang et al., 2024). L-RAG notes that entropy is a reliable but imperfect uncertainty signal: some confident errors have low entropy, and some correct answers have high entropy, which can trigger unnecessary retrieval (Voloshyn, 10 Jan 2026). ARC warns that performance may degrade under domain shift or rapid concept drift, because its demand model is derived from historical query distributions (Lin et al., 4 Nov 2025). At the text-compression level, EDC $k$ 7-RAG explicitly includes a fallback to the original retrieved documents because compression can omit critical details (Li et al., 4 Apr 2025).

A plausible implication is that the future of CompactRAG lies in adaptive rather than fixed compaction policies: reader-aware compression, query-type-aware retrieval, structure-preserving evidence transformation, and evaluation protocols that report both raw and compressed performance. The emerging literature already points in that direction through adaptive top- $k$ 8 selection, entropy gating, gate-controlled fallback, and multi-reader audits (Zhang et al., 2024, Voloshyn, 10 Jan 2026, Panthi et al., 20 Jun 2026). CompactRAG is therefore best understood not as “smaller context” in the abstract, but as a family of mechanisms for finding the smallest context, retrieval path, or memory footprint that remains sufficient for a particular reasoning task and a particular reader.