CompLLM: Compression for Long Context Q&A (2509.19228v1)

Abstract: LLMs face significant computational challenges when processing long contexts due to the quadratic complexity of self-attention. While soft context compression methods, which map input text to smaller latent representations, have shown promise, their real-world adoption is limited. Existing techniques typically compress the context as a single unit, which leads to quadratic compression complexity and an inability to reuse computations across queries with overlapping contexts. In this work, we introduce CompLLM, a soft compression technique designed for practical deployment. Instead of processing the context holistically, CompLLM divides it into segments and compresses each one independently. This simple design choice yields three critical properties: efficiency, as the compression step scales linearly with the context length; scalability, enabling models trained on short sequences (e.g., 1k tokens) to generalize to contexts of 100k tokens; and reusability, allowing compressed segments to be cached and reused across different queries. Our experiments show that with a 2x compression rate, at high context lengths CompLLM speeds up Time To First Token (TTFT) by up to 4x and reduces the KV cache size by 50%. Furthermore, CompLLM achieves performance comparable to that obtained with the uncompressed context, and even surpasses it on very long sequences, demonstrating its effectiveness and practical utility.

• The paper introduces a segment-wise soft compression framework that reduces the quadratic cost of long-context processing to linear complexity.
• It employs independent compression of context segments into concept embeddings using a LoRA-adapted transformer, enabling reusable and efficient inference.
• Empirical results demonstrate up to 4x latency reduction and maintained or improved accuracy on long-context QA benchmarks.

CompLLM: Segment-Wise Soft Compression for Efficient Long-Context LLM Inference

Introduction

CompLLM introduces a segment-wise soft compression framework for long-context question answering (QA) with LLMs. The method addresses the quadratic computational and memory complexity of self-attention in transformers when processing extended contexts. Unlike prior soft compression approaches that compress the entire context holistically—resulting in quadratic compression cost and non-reusable representations—CompLLM divides the context into independent segments, compresses each segment separately, and feeds the resulting concept embeddings (CEs) to the LLM. This design yields linear compression complexity, enables reusability of compressed segments, and allows models trained on short contexts to generalize to much longer ones.

Methodology

Segment-Wise Compression and Concept Embeddings

CompLLM operates by splitting the input context into segments (e.g., sentences or fixed-length token blocks), each of which is independently compressed into a set of CEs. These CEs reside in the same embedding space as the LLM's token embeddings (TEs) and can be directly input to the LLM without any modification or fine-tuning of the base model. The compression rate $C$ determines the reduction in sequence length (e.g., $C=2$ halves the number of embeddings).

Figure 1: Token Embeddings (TEs) versus Concept Embeddings (CEs); CEs provide a more compact representation while preserving output fidelity.

The architecture for the compressor is a LoRA-adapted version of the same LLM used for generation, with an appended linear layer. For each segment, the compressor takes $S$ TEs and outputs $S/C$ CEs. The CEs are concatenated across all segments to form the compressed context.

Training Protocol

The compressor is trained via hidden state distillation: for each context-question pair, the compressor is optimized to produce CEs such that, when fed to the frozen LLM (along with the uncompressed question), the hidden activations on the answer tokens match those produced by the original, uncompressed context. The loss is a Smooth-$L_1$ distance between teacher and student activations, normalized per layer.

Figure 2: CompLLM training protocol: segments are compressed independently, and the loss is computed only on answer token activations.

This approach does not require ground-truth labels; the answer segment is generated by the LLM itself. Only the context is compressed; the question remains uncompressed, reflecting real-world usage where contexts are long and often static, while questions are short and provided online.

Computational Complexity

The segment-wise design ensures that compression cost is $O(NS)$ (linear in context length $N$ and segment size $S$), as opposed to $O(N^2)$ for holistic compression. The LLM's inference cost is reduced from $O(N^2)$ (KV cache prefill) and $O(NT)$ (next-token prediction for $T$ tokens) to $O(N^2/C^2)$ and $O(NT/C)$, respectively. For large $N$ and small $T$, CompLLM achieves up to $C^2$ speedup in latency; for large $T$, the speedup approaches $C$.

Figure 3: CompLLM achieves significant TTFT speedup at high context lengths by reducing the number of embeddings fed to the LLM.

Figure 4: Inference speed with and without CompLLM for various context and output lengths; compression time becomes negligible for large contexts.

Experimental Results

Datasets and Evaluation

CompLLM is evaluated on open-ended (NarrativeQA, SQuAD) and multiple-choice (RACE, QuAIL) QA datasets, as well as the LOFT RAG benchmark, which concatenates multiple contexts to stress-test long-context capabilities. The compressor is trained on NarrativeQA and RACE, and tested on all datasets to assess generalization.

Main Findings

• Latency and Memory: With $C=2$, CompLLM reduces TTFT by up to 4x and KV cache size by 2x for long contexts.
• Accuracy: CompLLM matches or exceeds baseline LLM performance at high context lengths, with negligible or slightly reduced performance at short lengths. At very long contexts, CompLLM often outperforms the baseline, likely due to reduced attention dilution.
• Reusability: Compressed segments can be cached and reused across queries, enabling efficient multi-query scenarios (e.g., RAG, coding assistants).

  Figure 5: CompLLM maintains or improves accuracy across increasing context lengths for multiple datasets and LLMs.

• LOFT Benchmark: CompLLM improves long-context QA accuracy for smaller open-source LLMs, closing the gap with frontier models on 128k-token contexts.

Comparison with Prior Methods

Most soft compression methods (e.g., ICAE, InContextFormer) compress the context as a whole, resulting in non-reusable, quadratic-cost representations. LLMLingua-2 is an exception, using a BERT-like encoder for sentence-wise compression. CompLLM outperforms LLMLingua-2 at moderate context lengths and is competitive at extreme lengths.

Figure 6: CompLLM versus LLMLingua-2 and no compression; CompLLM is superior at moderate context lengths and competitive at extreme lengths.

Practical Implications

CompLLM is directly applicable to any LLM supporting external embeddings, without requiring model modification or fine-tuning. It is particularly suited for:

• Retrieval-Augmented Generation (RAG): Enables offline compression and caching of document embeddings, reducing online inference cost.
• Coding Assistants: Allows efficient handling of large codebases, with segment-level re-compression upon file changes.
• Web Agents and Document QA: Facilitates scalable, low-latency QA over large, partially overlapping contexts.

The method is orthogonal to other inference-time optimizations (e.g., chain-of-thought, paged attention) and can be integrated into existing LLM pipelines.

Limitations and Future Directions

CompLLM is optimized for semantic compression; it does not preserve fine-grained structural information, making it unsuitable for tasks requiring exact token-level fidelity (e.g., typo detection, character counting). However, the compressor can be bypassed for such cases, as the base LLM remains unmodified.

Future research directions include:

• Dynamic, input-adaptive compression rates.
• Scaling compression rates with model size and embedding dimensionality.
• Exploring alternative compressor architectures (e.g., encoder-only, fully-tuned LLMs).
• Leveraging unsupervised or plain-text data for compressor training.
• Extending to code and multimodal domains.

Conclusion

CompLLM provides a practical, segment-wise soft compression framework for long-context LLM inference, achieving substantial reductions in latency and memory while maintaining or improving QA accuracy at scale. Its linear compression complexity, reusability of compressed segments, and compatibility with unmodified LLMs make it a strong candidate for integration into production LLM systems handling large, repetitive, or overlapping contexts. The approach opens new avenues for efficient, scalable deployment of LLMs in real-world long-context applications.