CompactPrompt: Efficient LLM Prompt Compression

• CompactPrompt is a prompt compression technique that systematically reduces token count while preserving essential task fidelity for LLMs.
• It employs hard pruning, self-information scoring, dependency-based grouping, and reinforcement learning to optimize token selection.
• Experiments show up to 58% token reduction with minimal performance loss, enabling cost-effective and model-agnostic deployments.

A compact prompt, sometimes referred to as a "CompactPrompt" (Editor's term), denotes a systematically compressed input prompt for LLMs that preserves essential task fidelity while reducing token count and, by extension, inference cost. This paradigm encompasses both hard, interpretable prompt pruning and explicit data-level compression mechanisms, leveraging self-information, dependency-based grouping, quantization for numerical data, and occasionally hierarchical or reinforcement learning-based selection. CompactPrompt strategies aim to address practical constraints, notably the limited context window and full- or partial-token API billing in LLM deployments, while maintaining or improving model utility across a range of downstream tasks (Choi et al., 20 Oct 2025, Jung et al., 2023, Ma et al., 2022, Li et al., 2024).

1. Motivations and Problem Definition

Contemporary LLM-based workflows frequently exceed manageable context window sizes, especially in agentic or retrieval-augmented generation pipelines, resulting in high computational and monetary costs. This motivates formal prompt compression, where the goal is to map an original prompt or context $C$ of length $L_O$ to a compressed version $C_c$ of length $L_c$ such that task-relevant output is preserved:

$\rho = 1 - \frac{L_c}{L_O}$

The compression process is typically framed as:

$$\min_{C_c \subseteq C} |C_c| \quad \text{s.t.} \quad \mathrm{Sim}(f(C_c), f(C)) \geq \tau$$

where $f(\cdot)$ is the (typically frozen) LLM’s function and $\mathrm{Sim}(\cdot, \cdot)$ a semantic or output similarity metric (Li et al., 2024). The field distinguishes between compressions performed over original, human-interpretable tokens (hard/discrete pruning) and those manipulating soft embeddings ("soft prompts"). CompactPrompt pursues the former for maximal utility and cross-system compatibility (Choi et al., 20 Oct 2025, Jung et al., 2023).

2. Hard Prompt Pruning and Self-Information Strategies

A core principle in the CompactPrompt methodology is hard prompt pruning using data-driven, model-agnostic saliency functions:

• Self-Information Scoring: Tokens are assigned informativeness scores based on their static corpus frequency ($s_{\stat}(t) = -\log_2 p_{\stat}(t)$) and/or model-driven context probability ($s_{\dyn}(t \mid c) = -\log_2 p_{\mathrm{model}}(t \mid c)$). Scores can be dynamically combined to yield robust token rankings for pruning (Choi et al., 20 Oct 2025).
• Dependency-Based Phrase Grouping: To preserve grammaticality and avoid syntactic artifacts from token-level filtering, dependency parsing groups tokens into minimal heads/subtree phrases. Pruning proceeds at the phrase level by aggregating informativeness (Choi et al., 20 Oct 2025).
• Selective Context (SC) Methods: These remove tokens with the lowest aggregate self-information until a user-specified compression ratio $L_O$0 is reached, maximizing semantic content remaining under a token budget (Li et al., 2024).

This approach consistently targets and removes function words, high-frequency stopwords, and redundant phraseology, while retaining task-informative and content-dense expressions, as shown in output heat-map analyses (Jung et al., 2023, Choi et al., 20 Oct 2025).

3. Structured Soft Prompt Compression and Pruning

While hard pruning operates over textual tokens, prompt compression can also target soft prompts—trainable continuous embeddings prepended to the input sequence:

• Hierarchical Structured Pruning: As introduced in XPrompt, trained soft prompt tokens $L_O$1 are subjected to token- and piece-wise (embedding-dimension-wise) pruning based on gradient-based importance measures. Masks $L_O$2 (tokens) and $L_O$3 (embedding pieces) screen for the so-called "winning tickets" under the lottery ticket hypothesis (Ma et al., 2022).
• Rewinding: After aggressive pruning, the non-zero prompt parameters are rewound to their values before pruning and re-trained, recovering or exceeding the pre-pruning performance—especially beneficial for small–medium PLMs (Ma et al., 2022).

Compact soft prompt strategies can yield up to 90% sparsity with no performance loss, and often close the traditional gap between pure prompt- and full model fine-tuning for tasks such as those in SuperGLUE (Ma et al., 2022).

4. Reinforcement Learning Approaches to Discrete Compression

Reinforcement learning (RL), particularly policy-gradient methods, enables data-driven, unsupervised exploration of optimal discrete token subsets:

• One-Step Markov Decision Process (Contextual Bandit): PCRL frames prompt editing as a single-step MDP, where a token-wise binary decision vector $L_O$4 defines the compressed prompt. The reward balances a faithfulness metric (ROUGE-L between outputs on original and compressed input) and possible penalties for destructive edits (Jung et al., 2023).
• Policy Network Architecture: A frozen transformer encoder (e.g., DistilRoberta) produces token embeddings, passed to a two-layer MLP classification head for per-token inclusion probabilities. Action masking preserves necessary structural tokens (Jung et al., 2023).
• Gradient-Free, Black-box LM Compatibility: RL-based compression trains policies using black-box access only to LM generations, comparing outputs via similarity metrics without requiring LM gradients or gold summaries (Jung et al., 2023).

PCRL and its derivatives demonstrate 22–26% prompt reduction with >90% retention of output fidelity using ROUGE-L and preference-based metrics, outperforming comparable baseline trimming methods (Jung et al., 2023).

5. File-Level Data Compression and Mixed-Content Payloads

Beyond prompt text, the CompactPrompt pipeline accommodates compression of document attachments and tabular/numerical data:

• N-gram Abbreviation: Reversible LZW-style dictionary substitution identifies high-frequency n-grams (n=2–5), replacing them with short placeholder tokens, reducing redundant sequence length with minimal semantic loss (Choi et al., 20 Oct 2025).
• Uniform Quantization for Numerical Data: Numerical columns are quantized to a specified bit-width, storing min/max/range metadata for later reconstruction. The quantization error is explicitly controlled (Choi et al., 20 Oct 2025).

These mechanisms facilitate end-to-end payload minimization in practical, document-rich LLM pipelines, further decreasing token-induced computational or API load.

Compression Type	Method	Scope/Modality
Hard prompt pruning	Self-info scoring, SC	Token/phrase-level text
Structured soft prompt	XPrompt, HL pruning	Embedding-level
RL-based discrete	PCRL	Token selection (hard)
File-level	N-gram abbreviation, quant	Documents, numerical data

6. Experimental Outcomes and Best Practices

CompactPrompt pipelines and related toolkits such as PCToolkit have been benchmarked across diverse tasks:

• Token Reduction and Cost Savings: On datasets such as TAT-QA and FinQA, CompactPrompt yields up to 2.35× token reduction (≈58% fewer tokens), and end-to-end inference cost reductions close to 60% (Choi et al., 20 Oct 2025).
• Accuracy Impacts: Conservative settings preserve output quality within 5 percentage points of baseline. In some cases (e.g., FinQA with Claude-3.5-Sonnet), accuracy increased with compression, attributed to noise removal and representation regularization (Choi et al., 20 Oct 2025).
• Transferability: Discrete compressed prompts generalize to diverse LMs (LLaMa-2, Falcon-7B, FLAN-T5-XXL, GPT-3.5-turbo) without significant degradation, supporting model-agnostic deployment (Jung et al., 2023).
• Benchmarked Compressors: PCToolkit reports highest reconstruction/summarization fidelity for SC and LLMLingua, with moderate ratios (30–50%) recommended for minimal downstream degradation (Li et al., 2024).
• Best Practices: Empirically, aggressive pruning beyond 60% compression impairs performance; moderate levels preserve task fidelity. Phrase-level grouping and real-world benchmarks help identify suitable parameterizations. Rigorous evaluation using BLEU, ROUGE, BERTScore, and domain-specific accuracy metrics is essential (Choi et al., 20 Oct 2025, Li et al., 2024).

7. Limitations and Future Directions

Documented limitations include:

• Model-Specific Sensitivities: Compression performance and optimal hyperparameter settings vary by LM and task, especially for numerical data quantization and file-level abbreviation (Choi et al., 20 Oct 2025).
• Proprietary Tooling: Some pipeline components (e.g., CompactPrompt GUI) require licensing (Choi et al., 20 Oct 2025).
• Adaptive and Privacy-Aware Compression: Future work is directed towards adaptive policies that optimize compression given a token budget, privacy-aware filters that remove sensitive terms, and multimodal payloads encompassing figures, charts, or time-series (Choi et al., 20 Oct 2025).
• Reduced Reliance on Retrieval-Augmented Generation: CompactPrompt approaches can facilitate embedding larger document corpora directly, possibly reducing the complexity or need for external retrieval steps (Choi et al., 20 Oct 2025).

A plausible implication is that, as foundation LMs scale and deployment scenarios diversify, prompt and payload compression will be central in ensuring feasible, efficient, and robust use, motivating continual development in both methodological innovations and evaluation standardization.