Task-Agnostic Prompt Compression

• Task-agnostic prompt compression is the process of reducing the input length for large language models while retaining essential information for diverse downstream tasks.
• It employs a variety of techniques such as reinforcement learning, attribution analysis, and distillation to decouple compression from task-specific signals.
• Empirical evaluations indicate that these methods can achieve 3× to 12× prompt reduction while maintaining 80–90% of original task performance, enabling efficient LLM deployment.

Task-agnostic prompt compression refers to algorithmic strategies for reducing the length of prompts provided to LLMs while preserving the essential information necessary for effective downstream inference—irrespective of any specific task, user query, or context. This paradigm addresses the computational, latency, and financial costs associated with lengthy prompts, particularly in long-context scenarios, and is designed for maximal generality: a single compressed prompt serves across diverse tasks without per-query recompression. Early work in prompt compression focused primarily on task-aware or query-driven extractive methods, but the field now encompasses a range of architectures, learning frameworks, and evaluation protocols that explicitly decouple compression from task-specific signals or query knowledge. Task-agnostic prompt compression therefore underpins scalable, efficient LLM deployment in production applications where prompt re-use, latency, and cost are critical concerns.

1. Formal Problem Statement and Objectives

Let $x = [x_1, x_2, \dots, x_L]$ denote an input prompt of $L$ tokens. Task-agnostic prompt compression seeks a compressed subsequence $\widetilde{x} \subseteq x$ of length $|\widetilde{x}| = \tau L$ (for compression ratio $\tau \in (0,1]$), such that

$$\min_{\widetilde{x}} D\bigl( P(y|x), \, P(y|\widetilde{x}) \bigr),$$

where $P(y|x)$ is the output distribution of the target LLM, $D(\cdot,\cdot)$ measures distributional or information loss, and the selection of $\widetilde{x}$ is performed independent of any downstream task or query-specific signal. The goal is to maximize information retention and maintain output quality, while minimizing prompt length and, consequently, computational resource usage. This provides a universal, reusable compressed representation across potentially unknown or varying downstream tasks (Hu et al., 15 Apr 2025, Choi et al., 2024, Cao et al., 11 Mar 2025).

Key objectives include:

• Generality (one compressed prompt for all tasks)
• Fidelity (preserve LLM utility: QA, summarization, reasoning)
• Efficiency (reduce input length for inference speed/cost)
• Safety (avoid information loss hallucination or output explosion).

2. Taxonomy of Methodologies

Task-agnostic prompt compression has catalyzed a range of methodologies, including reinforcement-learning, attribution analysis, distillation-based classification, attention-entropy fusion, and information-theoretic or graph-structural approaches. Representative methods include:

Approach	Core Mechanism	Notable Example(s)
RL-based sequential	MDP/token-picking agent (PPO, curriculum)	LLM-DCP (Hu et al., 15 Apr 2025)
Segment attribution	Prune units by importance to output	ProCut (Xu et al., 4 Aug 2025)
Token classification	Distilled binary classifier from LLM	LLMLingua-2 (Pan et al., 2024), EFPC (Cao et al., 11 Mar 2025)
Information + attention	Reweight tokens by entropy + attention	DAC (Zhao et al., 16 Jul 2025)
Cross-attention mining	Exploit multi-document reader attentions	R2C (Choi et al., 2024)
Parse-tree pruning	Dynamic-program over syntactic tree	PartPrompt (Mao et al., 2024)
Graph-based subgraph	Prune fact KG for diversity	Prompt-SAW (Ali et al., 2024)
Self-supervised scoring	LoRA + classifier on frozen LLM	Selection-p (Chung et al., 2024)
Abstractive/Verbalization	Gist tokens + decoder mapping	Gist-COCO (Li et al., 2024), Cmprsr (Zakazov et al., 15 Nov 2025)
Lossless sequence coding	Meta-token (LZ77)	LTSC (Harvill et al., 30 May 2025)
Continual prompt pool compression	Gradient-based selection	GRID (Tiwari et al., 19 Jul 2025)

Most methods proceed by:

• Segmenting the prompt (tokens, sentences, chunks, or graph nodes)
• Scoring units by information density, importance (learned or analytical)
• Pruning to a desired length under explicit or implicit constraints
• Forming the compressed prompt by concatenation or abstraction.

Compression can be purely extractive (selecting original tokens/units) or abstractive (paraphrasing/generating gist prompts). Crucially, all selection must be independent of future task inputs.

3. Model Architectures and Algorithmic Strategies

RL-based Sequential Compression

LLM-DCP models prompt compression as a finite-horizon Markov Decision Process, where states encode the current compressed prompt as a sequence; actions are binary token keep/drop decisions; and transitions apply a masking operator. The agent’s policy is a transformer encoder, optimized by PPO to maximize a reward blending compression rate, key information retention (measured by BERTScore), and output distribution similarity (measured by KL divergence to a small, output-aligned reference model). Hierarchical Prompt Compression (HPC) uses curriculum learning to gradually tighten compression constraints—beginning with mild reductions and progressing to aggressive regimes. This RL-based sequential agent adapts to evolving prompt contexts and learns robust token elision policies (Hu et al., 15 Apr 2025).

Attribution-Driven Segment Pruning

ProCut segments input prompts into logical units (manually or automatically), then estimates segment-level attributions for downstream task utility. Classic and LLM-driven approaches can be used (Shapley, LOO, LASSO, greedy, or LLM-elicited). Top-scoring segments are retained; the rest are pruned. This approach yields interpretable, batched compression, and is training-free except for optional LLM-driven attribution acceleration (Xu et al., 4 Aug 2025).

Distillation-Based Token Classifiers

LLMLingua-2, EFPC, and Selection-p formulate compression as a binary token classification problem. A small transformer classifier is trained to predict keep/discard labels for each token, using labels derived from distillation of strong LLMs (e.g. GPT-4 via extractive compression) (Pan et al., 2024, Cao et al., 11 Mar 2025). Selection-p additionally explores self-supervised training, introducing only minimal parameters to a frozen backbone, and optimizes over next-token prediction with compressed contexts (Chung et al., 2024).

Attention-Aware and Dynamic Reweighting

DAC integrates token-wise information entropy (from a small LM) with per-token attention scores (from the LLM) to produce a fusion metric that more reliably preserves attention-critical tokens. A dynamic, iterative pruning process is employed, recalculating entropy after each removal stage and enforcing a “no two adjacent drops” constraint to limit cascading information loss. Empirically, the attention-entropy fusion improves downstream robustness, especially under high compression (Zhao et al., 16 Jul 2025).

Parse, Graph, and Cross-Attention-based Approaches

• Parse-tree pruning organizes tokens into hierarchical trees (sentence, paragraph, section) and recursively prunes using adjusted entropy scores, yielding outputs that better preserve linguistic structure and coherence (Mao et al., 2024).
• Relation-aware graphs (Prompt-SAW) transform the prompt into an entity-relation triplet graph, then prune for semantic diversity by thresholding vector distance, with the remaining facts reassembled into fluent, readable compressed prompts (Ali et al., 2024).
• Cross-attention mining (R2C) leverages the attention maps of a Fusion-in-Decoder QA model to identify globally salient units; hierarchical two-pass pruning (chunk then sentence) enables high-fidelity, pseudo-label-free, task-agnostic compression (Choi et al., 2024).

4. Empirical Performance and Evaluation

State-of-the-art task-agnostic compressors achieve substantial and robust prompt length reductions, commonly retaining at least 80–90% of downstream task performance even at compression ratios ranging from 3× to 12×. Empirical highlights include:

• LLM-DCP: Attains up to 12.9× reduction on Arxiv paper summaries with superior ROUGE-2 and competitive BLEU compared to LLMLingua-2 (Hu et al., 15 Apr 2025).
• R2C: Reduces prompt tokens by factors of 5–6× while delivering 3–11% higher accuracy on out-of-domain tasks, including LongBench and multi-document QA (Choi et al., 2024).
• ProCut: Achieves 78% average token reduction in production settings with <2% accuracy loss, outperforming prior extractive and random pruning policies (Xu et al., 4 Aug 2025).
• LLMLingua-2: At 3–5× compression, mitigates latency by 1.6–2.9×, outperforms standard entropy-pruning, and transfers across domain/language (Pan et al., 2024).
• Selection-p: 10× compression for in-context learning with only 0.8% average accuracy drop, state-of-the-art transferability across models (Chung et al., 2024).
• Gist-COCO, Cmprsr: Abstractive/gist-style compressors retain high utility even at >99% removal of raw context, surpassing prior extractive baselines on open- and closed-domain QA (Li et al., 2024, Zakazov et al., 15 Nov 2025).

Practical analysis reveals several trends:

• Long-context QA and summarization often benefit from moderate compression (up to 3–5×), which can improve performance by reducing noise and redundancy, effectively denoising the prompt (Zhang et al., 24 Apr 2025).
• For very aggressive compression (>10×), faithfulness and factual retention degrade unless inductive biases (e.g., linguistic structure, graph abstraction) are exploited.
• Model-agnostic/self-supervised approaches yield strong zero-shot and transfer robustness across models, tasks, and domains (Chung et al., 2024, Tiwari et al., 19 Jul 2025).
• Careful preservation of function words and instruction segments is necessary to avoid output explosion or semantic drift (Johnson, 6 Mar 2026).

5. Limitations, Failure Modes, and Evaluation Protocols

Despite substantial progress, several limitations and open challenges persist:

• Lossless Compression: Lossless token sequence compression (e.g., LZ77-style with meta-tokens) is possible and guarantees 100% semantic retention, but typically only achieves ~18–27% prompt length reduction and requires vocabulary extension; lossy strategies yield higher compression rates but may sacrifice minor details or global consistency (Harvill et al., 30 May 2025).
• Hallucination and Output Explosion: Excessive or poorly-structured compression can increase hallucination rates (information-loss or semantic-alteration), or cause “output explosion” where LLMs produce excessively long off-target completions. Structural metrics (e.g., instruction survival probability Ψ) and the Compression Robustness Index (CRI) have been proposed to quantify and mitigate these risks (Johnson, 6 Mar 2026).
• Generalization: Some methods depend on LLMs trained with limited task diversity (e.g., QA only for R2C), possibly limiting cross-domain application (Choi et al., 2024). Others require explicit output-alignment or access to model internals.
• Latency and Overhead: While inference-time efficiency is dramatically improved, some compressors introduce nontrivial pre-processing or attention extraction overhead, especially under deep or API-restricted settings (Zhao et al., 16 Jul 2025).

Best practice is to benchmark compressors across structurally diverse datasets, report Ψ and CRI, and monitor hallucination and energy metrics alongside traditional accuracy and F1/BERTScore, ensuring robustness not only to compression level but also to prompt structure (Johnson, 6 Mar 2026, Zhang et al., 24 Apr 2025).

6. Guidelines, Applications, and Future Directions

Guidelines for deployment and research include:

• Compression Rate Selection: For short contexts (<512 tokens), keep compression modest ($\rho \leq 0.2$). For long contexts (≥2K tokens), moderate rates ($L$0) often enhance performance; above $L$1 fidelity risks escalate (Zhang et al., 24 Apr 2025).
• Compressor Choice: For maximal extraction quality at high compression, favor data-distilled classifiers (LLMLingua-2, EFPC) or dynamic sequential agents (LLM-DCP). For real-time applications, lightweight segment- or entropy-based schemes (SCRL, Selective Context) may suffice.
• Structure-Aware Policies: Implement or simulate instruction-preservation constraints to ensure critical semantic units survive truncation (Johnson, 6 Mar 2026).
• Model Generalization: Whenever possible, use self-supervised or instruction-distilled compressors with model-agnostic token/segment selection (e.g., Selection-p, ProCut) (Chung et al., 2024, Xu et al., 4 Aug 2025).
• Hybrid Paradigms: Combining extractive, attention-guided, and abstractive compression or multitask/hybrid training signals holds promise for universal robust performance (Hu et al., 15 Apr 2025, Ali et al., 2024, Zakazov et al., 15 Nov 2025).

Emerging areas include lossless sequence transforms for strict syntactic/semantic fidelity (Harvill et al., 30 May 2025), transfer-safe model-agnostic scoring (Chung et al., 2024), and fully differentiable, structure-aware, or meta-learned compression agents (Xu et al., 4 Aug 2025, Hu et al., 15 Apr 2025).

Task-agnostic prompt compression thus constitutes a critical infrastructural layer for scalable LLM deployment, providing a toolbox of proven strategies to balance cost, latency, and information retention across domains, models, and prompt paradigms.