AttnComp: Adaptive Compression for RAG

• AttnComp is an attention-guided, extractive context compressor for RAG that adaptively selects document subsets by scoring retrieved content with query-to-context attention.
• It uses an adaptive Top-p thresholding mechanism to retain only the most relevant evidence, thereby reducing latency while maintaining or improving answer quality.
• LongAttnComp extends these principles to long-context settings by employing token-level chunking and confidence signals to enhance multi-hop reasoning.

AttnComp most directly denotes an attention-guided, extractive context compression framework for retrieval-augmented generation (RAG) that scores retrieved documents with query-to-context attention and retains the minimal subset whose cumulative attention mass exceeds a Top-$p$ threshold (Luo et al., 22 Sep 2025). In this formulation, compression is adaptive rather than fixed-rate, is inserted between retrieval and generation, and also yields a confidence signal derived from instruction attention. A later system, LongAttnComp, generalizes the same scoring mechanism to 100k$+$-token settings by replacing document-level selection with token-level chunking and token-budget Top-$p$ selection (Ji et al., 31 May 2026). The label has also appeared in other settings, notably “Attention for Inference Compilation,” where attention is used over latent-variable traces in probabilistic programs rather than retrieved documents (Harvey et al., 2019).

1. Definition, scope, and terminology

In its RAG-specific sense, AttnComp addresses the setting in which a retriever returns documents

$D=\{d_1,d_2,\dots,d_k\},$

and a LLM generates an answer conditioned on retrieved context and query,

$M(y \mid D, q).$

The framework compresses $D$ into a reduced subset $D' \subseteq D$ so that irrelevant material is removed before answer generation. The paper’s stated goal is to minimize the size of $D'$ while maintaining or improving the quality of $M(y \mid D', q)$ (Luo et al., 22 Sep 2025).

The name is not entirely univocal in the literature. In the RAG paper, AttnComp is an extractive compressor operating over retrieved documents and their attention-derived relevance. In “Attention for Inference Compilation,” the same shorthand is associated with attention over previously sampled latent variables in universal probabilistic programs, added to inference compilation to recover long-range latent dependencies (Harvey et al., 2019). These are distinct frameworks that share an attention-centered rationale but differ in task, object of compression, and deployment point.

A common misconception is to treat AttnComp as a generic synonym for any attention compression method. The surrounding literature is broader. TriAttention compresses the KV cache for long-context reasoning by exploiting pre-RoPE Q/K concentration and a trigonometric distance-preference score; CompRank compresses the document tokens exposed to attention-based reranking; and Attention-State Memory replaces direct prefix attention with retrieval of precomputed attention states (Mao et al., 6 Apr 2026, Lu et al., 10 Jun 2026, Okoshi et al., 18 May 2026). AttnComp, by contrast, is specifically a pre-generation context compressor for RAG and, in its long-context extension, a trainable query-context compressor.

2. Core architecture and attention scoring

AttnComp constructs an input by prefixing a predefined instruction to the concatenated retrieved documents, followed by the query. Conceptually, the sequence is

$[\text{instruction}; d_1; d_2; \dots; d_k; q].$

The compressor consists of the first $+$0 transformer layers of the original LLM, followed by an added cross-attention layer. After the first $+$1 layers, the model obtains context hidden states

$+$2

and query hidden states

$+$3

The added cross-attention computes

$+$4

and

$+$5

where $+$6 is the number of heads, $+$7 are head-specific projections, and $+$8 is the aggregated query-to-context attention matrix (Luo et al., 22 Sep 2025).

Relevance is defined by aggregating query attention over context spans. For a segment with token indices $+$9, and query token indices $p$0, the segment score is

$p$1

The same aggregation yields the instruction score $p$2 and document scores $p$3: $p$4 The default unit is the whole retrieved document, although the appendix also evaluates a sentence-level variant.

The architecture is motivated by the empirical observation that middle-layer attention heads focus most consistently on supporting evidence. In the main implementation with Llama-3.1-8B-Instruct, AttnComp keeps $p$5 transformer layers, uses $p$6 heads in the cross-attention layer, and initializes that layer from the top 16 attention heads from the 14th layer (Luo et al., 22 Sep 2025).

3. Adaptive Top-$p$7 compression and confidence estimation

AttnComp’s selection rule is the mechanism that makes compression adaptive. Documents are first sorted by descending relevance,

$p$8

and cumulative mass is initialized with the instruction score,

$p$9

Documents are then added in ranked order until either the cumulative score exceeds a threshold $D=\{d_1,d_2,\dots,d_k\},$0, or the next document score falls below a minimum threshold $D=\{d_1,d_2,\dots,d_k\},$1. Equivalently, with

$D=\{d_1,d_2,\dots,d_k\},$2

AttnComp returns the smallest ranked prefix whose cumulative relevance satisfies

$D=\{d_1,d_2,\dots,d_k\},$3

subject to the minimum-score cutoff $D=\{d_1,d_2,\dots,d_k\},$4 for retained documents (Luo et al., 22 Sep 2025).

This rule behaves differently depending on how relevance is distributed. If only a few documents absorb most attention, compression is aggressive. If evidence is spread across many documents, more are retained. If all documents are irrelevant, the instruction score can dominate, and the method can return $D=\{d_1,d_2,\dots,d_k\},$5. The paper presents this as a response to fixed-ratio compressors, which may over-compress distributed evidence or under-compress noisy retrieval sets.

AttnComp also defines a confidence signal from the same scores. The confidence score is

$D=\{d_1,d_2,\dots,d_k\},$6

Here higher instruction attention corresponds to lower confidence in the usefulness of the retrieved context. The paper is explicit that this is a proxy for retrieved-context relevance rather than a fully calibrated probability of answer correctness. A plausible implication is that the confidence score is best interpreted as a retrieval-quality signal that may be useful for fallback or iterative RAG policies rather than as a standalone verifier.

4. Supervision, annotation, and implementation

The strongest AttnComp results do not rely on raw attention alone. The framework fine-tunes only the added cross-attention layer, while freezing the first $D=\{d_1,d_2,\dots,d_k\},$7 LLM layers; the paper states that this updates about $D=\{d_1,d_2,\dots,d_k\},$8 of total parameters. Supervision uses binary document relevance labels $D=\{d_1,d_2,\dots,d_k\},$9, with a document-level loss

$M(y \mid D, q).$0

To handle cases in which none of the retrieved documents are relevant, AttnComp adds an instruction-level objective

$M(y \mid D, q).$1

and the total loss is

$M(y \mid D, q).$2

The instruction term is intended to move attention toward the instruction region when the retrieval set is wholly irrelevant (Luo et al., 22 Sep 2025).

Because standard QA datasets often have incomplete document-level relevance annotations, the paper constructs training labels with a dedicated pipeline. Retrieved documents are shuffled multiple times, an untrained compressor is run repeatedly with Top-$M(y \mid D, q).$3 compression, consistently retained documents are treated as relevant, and the result is verified by asking an LLM to answer using the retained subset. The main training set contains 8,000 examples from HotpotQA, of which 2,000 are negative instances with all documents irrelevant. Optimization uses Adam with learning rate $M(y \mid D, q).$4, batch size 8, and 8 epochs; the appendix reports training on 4 NVIDIA RTX 4090 GPUs for about 4 hours (Luo et al., 22 Sep 2025).

The implementation is not black-box. It requires access to hidden states from the first $M(y \mid D, q).$5 layers and insertion of the cross-attention scoring layer. This makes the method naturally compatible with open-weight or modifiable reader models, but the paper does not present a black-box API-only version.

5. Empirical behavior, ablations, and observed trade-offs

AttnComp is evaluated on HotpotQA, 2WikiMultiHopQA, MuSiQue, Natural Questions, and PopQA, using E5-base-v2 retrieval over top-100 Wikipedia segments of 100 words each. With Llama-3.1-8B-Instruct as both compressor backbone and reader, the main result is that AttnComp achieves average F1 $M(y \mid D, q).$6, average accuracy $M(y \mid D, q).$7, and average compression $M(y \mid D, q).$8, compared with $M(y \mid D, q).$9, $D$0, and $D$1 for the uncompressed “All Documents” baseline (Luo et al., 22 Sep 2025).

The gains are concentrated especially in multi-hop settings. On average, AttnComp improves accuracy by 3.3 points over the uncompressed baseline on the multi-hop tasks. On 2WikiMultiHopQA, the paper reports accuracy $D$2 for AttnComp, compared with $D$3 for “All Documents” and $D$4 for Provence. On HotpotQA, accuracy rises from $D$5 to $D$6; on MuSiQue, from $D$7 to $D$8. On Natural Questions, AttnComp is slightly below the uncompressed baseline, $D$9 versus $D' \subseteq D$0, while on PopQA it is slightly above, $D' \subseteq D$1 versus $D' \subseteq D$2. This pattern suggests that the method is particularly well matched to settings in which evidence integration across several retrieved items matters.

Efficiency is part of the claim rather than a side effect. The uncompressed baseline has total latency $D' \subseteq D$3 s, while AttnComp’s compression latency is $D' \subseteq D$4 s and its generation latency is $D' \subseteq D$5 s; the paper states that AttnComp’s total latency is $D' \subseteq D$6 of the uncompressed baseline. Unlike abstractive compressors such as CompAct, which achieve stronger raw compression but very high end-to-end latency, AttnComp is designed so that compression cost does not erase the gains from shorter generation.

The ablations refine the picture. The number of retained documents varies from $D' \subseteq D$7 to $D' \subseteq D$8, with average retained documents $D' \subseteq D$9 on HotpotQA and $D'$0 on PopQA, which is the paper’s direct evidence that the Top-$D'$1 rule is adaptive. Performance remains stable for $D'$2. A sentence-level variant pushes average compression from $D'$3 to $D'$4, but average accuracy declines from $D'$5 to $D'$6. Layer-depth ablations show that middle-layer settings are best without fine-tuning, and fine-tuning improves both compression and accuracy across all layer settings. For confidence estimation on HotpotQA, examples with confidence below $D'$7 have average F1 $D'$8, while examples above $D'$9 have average F1 $M(y \mid D', q)$0; Pearson correlation between confidence and F1 is $M(y \mid D', q)$1, compared with $M(y \mid D', q)$2 for the uncompressed baseline.

6. LongAttnComp and the broader attention-compression landscape

LongAttnComp is presented as a long-context adaptation of AttnComp for 100k$M(y \mid D', q)$3-token prompts. It retains AttnComp’s core mechanism and loss formulation, but replaces document-level scoring with fixed-size token-level chunking, replaces minimum-score stopping with a token-budget Top-$M(y \mid D', q)$4 rule, restores selected chunks to original order before passing them to the target model, and introduces a format-agnostic query parser. The compressor uses the first 13 layers of Llama-3.1-8B-Instruct as a frozen backbone with one trainable cross-attention scoring layer on top, and is trained with a two-stage recipe in which Stage 1 builds a retrieval foundation from SQuAD and HotpotQA-style data and Stage 2 adds MuSiQue and 2WikiMultiHopQA for broader reasoning coverage (Ji et al., 31 May 2026).

The long-context results sharpen what AttnComp is and is not. On InfiniteBench Code-Debug with DeepSeek-R1-0528 as target, full context at 120k tokens yields $M(y \mid D', q)$5, Speculative Prefill yields $M(y \mid D', q)$6, LongAttnComp Stage 1 yields $M(y \mid D', q)$7, and Stage 2 subq yields $M(y \mid D', q)$8. On LongBench v2 with the same target, full context yields $M(y \mid D', q)$9, Speculative Prefill $[\text{instruction}; d_1; d_2; \dots; d_k; q].$0, Stage 1 $[\text{instruction}; d_1; d_2; \dots; d_k; q].$1, and Stage 2 nosubq $[\text{instruction}; d_1; d_2; \dots; d_k; q].$2. This indicates that AttnComp-style learned compression transfers well to long code reasoning, while naturalistic multi-document reasoning remains harder and more sensitive to training data composition.

Relative to adjacent methods, AttnComp occupies a specific niche. TriAttention is a training-free KV-cache compression method for long-context reasoning that scores cached keys from pre-RoPE geometry and a trigonometric distance-preference curve rather than from retrieved-document relevance (Mao et al., 6 Apr 2026). CompRank is a reranking framework that compresses the document tokens exposed to query-side attention and scores candidates by aggregated attention mass over those retained tokens (Lu et al., 10 Jun 2026). Attention-State Memory externalizes the effect of a reusable long prefix into a lookup-based memory of precomputed attention states, replacing direct prefix attention during decoding (Okoshi et al., 18 May 2026). This suggests that AttnComp is best situated as a pre-generation, query-conditioned context selector for retrieval-augmented inference, distinct from KV eviction, reranking-time token subsampling, or prefix-attention replacement.

The principal limitations stated for AttnComp and its long-context extension are also revealing. The RAG paper evaluates dense models up to 8B parameters and does not study MoE architectures; its confidence score measures retrieval quality more directly than full answer correctness. LongAttnComp shows that the original AttnComp setting assumed relatively short-context retrieval-augmented QA with clean document structure, and that scaling the method requires modifications in compression unit, selection rule, positional reconstruction, and training distribution. A plausible implication is that AttnComp’s central idea is stable—query-conditioned relevance estimation from internal attention—but its effective deployment depends strongly on what is being compressed: documents in RAG, chunks in long-context reasoning, or, in other lines of work, KV states, reranking views, or reusable prefix effects.