Task-Agnostic Prompt Compression (TPC)

• Task-agnostic Prompt Compression (TPC) is a set of techniques that reduce LLM prompt lengths by maximizing information retention via methods like token extraction, attention fusion, and RL-based pruning.
• TPC integrates strategies such as hard extractive selection, soft representational bottlenecks, and attribution-based pruning, achieving competitive metrics while minimizing performance degradation.
• TPC improves computational efficiency by reducing memory and latency costs, making it vital for scaling LLM applications across diverse tasks and datasets.

Task-agnostic Prompt Compression (TPC) refers to a collection of model-agnostic strategies and algorithms for automatically shortening LLM prompts without leveraging downstream-task labels, queries, or handcrafted templates. The objective is to substantially reduce the number of input tokens—thereby decreasing memory and latency—and at the same time retain as much essential information as possible, so that LLM-generated outputs from compressed prompts remain accurate and faithful to the original context. TPC is grounded in both information theory and modern neural sequence modeling, and has led to a rich taxonomy of approaches, encompassing hard extractive selection, soft representational bottlenecks, attribution-based pruning, and structured compression policies suitable for diverse, unseen tasks.

1. Theoretical Foundation and Formalization

Task-agnostic prompt compression systems are formulated as algorithms operating on raw prompt sequences with the aim of maximizing information retention subject to aggressive length reduction. In the canonical formalism, for an initial prompt $x$ of length $L$, the aim is to derive a compressed prompt $\tilde{x}$ of length $\tilde{L} \ll L$, such that $$\mathrm{Dist}(\mathrm{LLM}(x), \mathrm{LLM}(\tilde{x}))$$ is minimized for a pre-specified distance metric (e.g., task EM, BLEU, ROUGE, KL divergence of LLM output distributions) (Nagle et al., 2024, Pan et al., 2024). In contrast to query-aware or task-conditioned compression (where a specific downstream label or question is given a priori), TPC explicitly forbids any such side information at training or inference time, reflecting its goal of universal applicability.

The information-theoretic core is provided by a rate-distortion framework: the minimum achievable distortion $D^*(R)$ as a function of the average compression rate $$R = \mathbb{E}[\ell(\tilde{x})] / \mathbb{E}[\ell(x)]$$ is derived via convex optimization, and empirical compressors are benchmarked against this fundamental limit (Nagle et al., 2024). This analysis demonstrates that all current TPC algorithms fall well above the theoretical optimum, due to their reliance on myopic, local heuristics rather than globally optimized, mixture or variable-rate policies.

2. Algorithmic Taxonomy and Representative Approaches

There are five principal classes of TPC algorithms, differentiated by their selection granularity, scoring strategy, and degree of supervision:

• Token-level extractive selection/classification: A model assigns each token a "keep/discard" score via either entropy [Selective-Context, LLMLingua], pseudo-labeled data distilled from teacher LLMs [LLMLingua-2], or self-supervised next-token prediction [Selection-p]. The highest scoring $\tau L$ tokens (for compression rate $\tau$) are preserved in their original order. Bidirectional encoders (e.g., XLM-RoBERTa, mBERT) are superior to uni-directional/casual models for judging token salience (Pan et al., 2024, Chung et al., 2024).
• Attention and dynamic-entropy fusion: Algorithms such as DAC combine tokenwise conditional entropy and cross-layer attention scores to capture both information content and algorithmic importance to the underlying self-attention mechanism. DAC prunes over multiple dynamic rounds, recalculates entropy after each round, and enforces no-consecutive-drop constraints to prevent catastrophic accumulation of local entropy artifacts. The additive fusion $M_t^a = (1-\alpha)I_t + \alpha\bar s_t$ (typically with $L$0) empirically outperforms multiplicative variants (Zhao et al., 16 Jul 2025).
• Reinforcement learning over sequential pruning: LLM-DCP models TPC as a finite-horizon Markov Decision Process (MDP), where the action space is selection vectors over the prompt, the reward function balances compression ratio, information fidelity (BERTScore), and preservation of the output distribution (KL divergence with a student LLM). A staged curriculum (Hierarchical Prompt Compression) is used for progressive-hardness training, and the agent is updated using PPO (Hu et al., 15 Apr 2025).
• Global importance via cross-attention pooling: R2C leverages Fusion-in-Decoder (FiD) architectures to compute full-prompt cross-attention scores for each chunk and sentence, then hierarchically prunes least important units to meet a budget. No explicit pseudo-labeled training data is required: the pretrained FiD model's attention naturally encodes salience for arbitrary prompts (Choi et al., 2024).
• Attribution and segment-level utility analysis: ProCut segments prompts into semantic units (e.g., blocks, sentences, or LLM-proposed units), computes segment-wise Shapley values, leave-one-out (LOO) scores, or LASSO attributions over the marginal drop in task performance, and ranks/prunes segments accordingly. An LLM-driven fast attribution variant (mask/rank loop) achieves production-scale performance (Xu et al., 4 Aug 2025).

3. Key Models, Datasets, and Empirical Performance

TPC algorithms have been systematically evaluated across a suite of task-diverse datasets: LongBench (long-context QA, summarization, coding), GSM8K (math word problems), BBH (BigBench Hard, few-shot reasoning), MeetingBank (real-world long transcripts), and ZeroSCROLLS (multitask long-text benchmarks). Results consistently indicate:

Model/Method	Typical Compression	Main Evaluated Metric	Retention Gap vs. Full Prompt	Outlier Findings
LLMLingua-2	2–5×	F1, EM, ROUGE, BERTScore	0–3% (QA/Summ); ≤0.8% (Class)	Strong out-of-domain transfer (Pan et al., 2024)
LLM-DCP	6–12×	BLEU, ROUGE-2, EM	–0.2% to +4% (ShareGPT/Arxiv)	>20×: degraded in multi-step reasoning (Hu et al., 15 Apr 2025)
DAC	2–5×	F1, EM (LongBench/GSM8K/BBH)	+1–4 F1 vs. baselines	Attention-critical tokens decisive (Zhao et al., 16 Jul 2025)
R2C	5–10×	SpanEM, F1 (QA), latency	+6.9% (LLaMA2, QA), −0.2% GPT-3.5	Outperforms uncompressed in denoising (Choi et al., 2024)
ProCut	2–4× (template)	Exact Match, Pass@1, Accuracy	Neutral to +2% (prod)	Up to 84% token reduction (Xu et al., 4 Aug 2025)
Selection-p	10×	Classification accuracy	−0.8%	Transferable across LLMs/APIs (Chung et al., 2024)
Prompt-SAW	1.5× (CoT)	EM (GSM8K-aug), SpanAcc (NQ)	+8.7–13.7% over best baseline	Graph-based redundancy (Ali et al., 2024)

Performance under high compression rates reflects trade-offs between token-level extractiveness, semantic coverage, and the fidelity objective. For instance, LLMLingua-2’s bidirectional encoder, distilled on GPT-4 compressions, consistently generalizes well both in-domain and out-of-domain, frequently matching or outperforming long/deep RL or entropy-pruning systems (Pan et al., 2024, Hu et al., 15 Apr 2025).

4. Practical Integration and Computational Impact

TPC is attractive for deployment due to substantial reductions in quadratic attention, GPU/TPU memory savings, end-to-end latency, and API billing. For example, lossless sequence compression via meta-tokens (LTSC) yields a 27% input reduction and 47% self-attention computation reduction in parsing tasks, with strictly zero semantic loss (Harvill et al., 30 May 2025). In contrast, lossy methods (LLMLingua, DAC, R2C) typically achieve 3–12× compression rates, with only a 0–2% drop in F1 on QA/summarization and comparable acceleration in wall-clock time.

Hybrid approaches (EFPC) combine task-aware and task-agnostic settings, using instruction-prepending at inference to switch modes (Cao et al., 11 Mar 2025). Compression models are typically lightweight: where LLMLingua-2 is 110–355M parameters and R2C uses T5-base (223M), these can be deployed as pre-processing modules with single-pass throughput that is 3–30× faster than entropy-only baselines (Pan et al., 2024, Choi et al., 2024).

Energy consumption is nuanced. While a 17.4% token reduction led to only a 4.8% real joule reduction for TinyLlama on RTX4090, suggesting that naive token-count proxies may overestimate true energy savings. Direct GPU power telemetry is recommended over purely computational estimates (Johnson, 6 Mar 2026).

5. Design Considerations, Limitations, and Extensions

Key considerations for TPC system design include:

• Instruction/segment survival: Structural analysis using instruction-survival probability $L$1 shows that careless truncation (especially "first-N tokens") risks catastrophic loss of task-critical segments, leading to output explosion (e.g., 56× token growth at $L$2) on benchmarks like MBPP (Johnson, 6 Mar 2026). Structure- and semantic-aware scoring or adaptive-ratio policies should be used to always maintain a minimum $L$3 and preserve critical segments.
• Compression-granularity: Sentence- or chunk-level selection generally retains more information and coherence (higher BERTScore, entity recall) than token-only models. Multi-granular or hierarchical policies, as in LLMLingua-2 and R2C, outperform flat classifiers (Choi et al., 2024, Hu et al., 15 Apr 2025).
• Faithfulness and hallucination risks: Most loss in performance and accuracy is via information-loss hallucinations (ILH), where omitted tokens lead to plausible but spurious LLM outputs. Aggressive pruning heightens ILH, especially with function words and coreference tokens in long contexts (Zhang et al., 24 Apr 2025).
• Transferability and robustness: Selection-p and LLMLingua-2 demonstrate strong transfer to larger LLMs, and across closed/open APIs, affirming the value of universal, non-model-specific compression heads (Chung et al., 2024, Pan et al., 2024). Cross-benchmark robustness should be measured by quality-efficiency metrics such as the Compression Robustness Index (CRI) (Johnson, 6 Mar 2026).

Limitations affect all current TPC systems:

• At extreme compression rates ($L$4), multi-step reasoning chains are often irrecoverably lost (Hu et al., 15 Apr 2025).
• Lossy compressors fail dramatically on syntax- or structure-sensitive tasks (e.g., parsing, code completion), where lossless meta-token methods like LTSC are required (Harvill et al., 30 May 2025).
• Most existing models, especially those requiring attention scores (e.g., DAC), are constrained by efficient-attention implementations or the opacity of black-box APIs (Zhao et al., 16 Jul 2025).
• Over-compression or underestimation of segment utility leads to instruction dropping or grammar-breaking outputs.

Future work includes multi-granular token/sentence/segment compressors, online bandit feedback for reward refinement, adaptive compression scheduling based on input structure or model response, and integration with context retrieval or prompt optimization frameworks (Hu et al., 15 Apr 2025, Xu et al., 4 Aug 2025).

6. Integration into LLM Workflows and Recommendations

TPC systems are now routinely deployed as modular components for long-context inference, QA, summarization, multi-document synthesis, and continual prompt tuning in LLMs. Best practices include:

• Pre-processing input prompts with a semantic- or structure-aware compressor (e.g., LLMLingua-2, DAC) to filter and rank tokens, then set an adaptive (per-prompt) compression ratio via $L$5-based analysis to avoid critical instruction loss (Johnson, 6 Mar 2026).
• For batch settings (retrieval, ICL), combine segment-level scoring (ProCut, R2C, Prompt-SAW) with hierarchical pruning to ensure both input diversity and semantic coverage (Xu et al., 4 Aug 2025, Choi et al., 2024, Ali et al., 2024).
• Always profile impact on both output quality and downstream inference cost (latency, memory, energy), and validate under multiple benchmarks and scenarios before production integration.
• Monitor hallucination rates and regularize for factual consistency or integrate with post-filtering modules as needed (Zhang et al., 24 Apr 2025).

TPC is increasingly foundational for scalable, cost-efficient, and reliable LLM-based systems, as large-context transformers and multimodal models become pervasive. The continuing development of universally robust, adaptive, and information-theoretically informed compression algorithms remains a core research frontier.