---
title: Input Compression in Machine Learning
url: https://www.emergentmind.com/topics/input-compression
type: topic
---

# Input Compression in Machine Learning

Input compression is the reduction of the representation presented to a model before or during processing, typically by shortening a token sequence, lowering spatial or spectral resolution, pruning visual tokens, collapsing redundant temporal states, or quantizing features. Recent work treats it as a distinct optimization axis rather than a mere preprocessing trick: in large language models it is evaluated against realized per-item API cost and semantic fidelity [2606.24083]; in multimodal systems it is used to reduce FLOPs, prefilling latency, inference time, and token counts [2509.12159; 2511.02650]; in speech, source separation, and edge inference it is used to trade sequence length, memory, and bandwidth against downstream quality [2102.01578; 2602.08671; 2603.05172]. The literature shows that input compression is not intrinsically beneficial: its effect depends on what is compressed, how the compressed representation preserves task-critical structure, and which deployment metric is actually measured [2606.24083; 2603.23527].

## 1. Formalizations and problem structure

A common formalization views input compression as an operator that maps an original input sequence to a shorter one. In ICPC, an input \(x=(x_1,x_2,\dots,x_n)\) is mapped by a compression operator \(C_l(\cdot)\) at compression level \(l\) to a compressed sequence \(x' = C_l(x)\) with \(m = |x'| < n\), and the compression ratio is \(r = m/n\). In Transformer settings, this changes self-attention cost from \(O(n^2)\) to \(O(m^2)=O(r^2 n^2)\) [2312.12385].

In prompt compression for LLMs, CAVEWOMAN defines a user question as a token sequence \(x = (w_1,\dots,w_n)\in V^n\), with deterministic POS tags \(g(w_i)\) from spaCy, and introduces reduction functions \(\phi_0,\dots,\phi_4\). These range from the identity \(\phi_0(x)=x\), to deletion of function-word categories, to noun/verb filtering, to noun-only truncation with \(\mathrm{trunc}_{15}(\cdot)\). Under Condition A, the model receives \(\phi_\ell(x)\) in place of \(x\) under a neutral system prompt [2606.24083].

Other domains use different state spaces but analogous compression ratios. UniPruneBench defines visual-token retention \(\rho = N_{\rm ret}/N_{\rm orig}\) and pruning ratio \(r = 1-\rho\) for multimodal LLMs [2511.02650]. Spectral Feature Compression (SFC) compresses a frequency axis \(F\) to \(K\) bands, giving compression ratio \(K/F\), for example \(64/1025 \approx 1/16\) [2602.08671]. STAR defines the compression ratio for streaming sequence transduction as \(r = T_x/K\), where \(T_x\) is the number of low-level frames and \(K\) the number of anchor representations [2402.01172]. Trainable Bitwise Soft Quantization compresses each \(32\)-bit feature to a user-defined \(B\)-bit code, with compression approximately \(32/B\) [2603.05172].

These formulations differ in modality and objective, but they share a structural pattern: an original representation is mapped to a smaller one before the main compute bottleneck. This suggests that input compression is best understood as a front-end budget-allocation mechanism rather than a single algorithmic family.

## 2. Prompt-side input compression in language models

CAVEWOMAN makes the strongest recent case that prompt shortening and cost reduction should be treated as separate questions. Its two-channel protocol evaluates the same items under Condition A, where the prompt is filtered via \(\phi_\ell\) and the output is unconstrained, and Condition B, where the prompt is unfiltered but the response is constrained to a matching register. Every generation is scored on task accuracy, realized per-item cost
\[
C = n_{\rm in} p_{\rm in} + n_{\rm out} p_{\rm out},
\]
and reference-text agreement using bidirectional NLI against the model’s own \(L0\) generation, plus \(11\) complementary measures [2606.24083].

The central empirical finding is that input compression is a strict lose-lose under realistic API pricing. Across all eight models and five benchmarks, input compression raises net API cost by approximately \(1.15\times\) at \(L1\), reaches up to \(1.8\times\) on the worst per-dataset cell, and grows to approximately \(2.7\times\) under stronger compression as accuracy collapses. The mechanism is compensatory verbosity: input token savings are small, while output tokens increase, and \(p_{\rm out} \gg p_{\rm in}\), so output expansion dominates the bill [2606.24083].

At \(L1\), the mean relative cost changes over the four API models make this asymmetry explicit. GPT-4o saves \(15.3\) input tokens, adds \(0.1\) output tokens, and changes cost by \(-2.6\%\); GPT-5.4 saves \(15.3\) input tokens, adds \(18.0\) output tokens, and increases cost by \(15.4\%\); Claude Haiku 4.5 saves \(15.7\) input tokens, adds \(9.9\) output tokens, and increases cost by \(3.1\%\); Claude Sonnet 4.6 saves \(15.7\) input tokens, adds \(12.9\) output tokens, and increases cost by \(5.1\%\) [2606.24083].

Quality degrades in parallel. Task accuracy and same-channel entailment both worsen monotonically with compression level \(\ell\). Wilcoxon tests against \(L0\) show significant accuracy drops already at \(L1\) in \(20/20\) \((\text{model},\text{dataset})\) cells with \(p<0.01\), and mean bidirectional NLI in Condition A at \(L1\) falls to approximately \(50\%\) from \(100\%\). For input compression specifically, there is no dissociation phenomenon: correct answers almost always still entail the \(L0\) reference [2606.24083].

A related prompt-compression study refines the structural explanation. “Compression Method Matters” introduces instruction survival probability
\[
\Psi(\mathbf{x},r)=\sum_{j=1}^k w_j\,\psi(I_j,r),
\]
where \(\psi(I_j,r)\) indicates whether a task-critical segment survives first-\(N\)-words truncation. Under \(r=0.3\), DeepSeek shows a \(56.4\times\) output expansion on MBPP with \(\Psi \approx 0.15\), but only \(5.2\times\) on HumanEval with \(\Psi \approx 0.72\); GPT-4o-mini is comparatively stable across benchmarks. The same work introduces the Compression Robustness Index (CRI) and argues that prompt structure, not provider identity alone, is the primary moderator of output explosion [2603.23527]. This suggests that destructive prompt compression is less about compression per se than about whether task-defining segments survive.

A contrasting result comes from AdaCoder, which compresses long programmatic VQA preprompts by preserving API scaffolding and specializing code snippets by question type. In ViperGPT, the preprompt length falls from \(3{,}434\) tokens to \(993\) tokens, a \(71.1\%\) reduction, while answer accuracy improves on GQA, VQAv2, and NLVR2. The ablations show that removing question-type specialization collapses accuracy to \(28.9\%\), whereas oracle types reach \(44.5\%\) [2407.19410]. The plausible implication is that prompt compression can succeed when it is structure-aware and task-conditional rather than telegraphic.

## 3. Vision and multimodal input compression

In vision and multimodal systems, input compression is usually motivated by prefilling cost, quadratic attention, or latency inside large image-conditioned backbones. The methods differ sharply in whether they are training-free heuristics, architecture-preserving rescalings, or task-specific token selectors.

| Setting | Compression mechanism | Reported effect |
|---|---|---|
| Programmatic VQA | Type-conditioned compressed preprompts in AdaCoder | \(3{,}434 \rightarrow 993\) tokens; \(71.1\%\) reduction [2407.19410] |
| UI code generation | ELTC + RTR in EfficientUICoder | \(55\%-60\%\) compression ratio; FLOPs \(\downarrow 44.9\%\); prefill \(\downarrow 46.6\%\); inference \(\downarrow 48.8\%\) [2509.12159] |
| Image editing | HiLo-Token high/low-frequency token allocation | DiT speedups \(3.13\times\), \(2.59\times\), \(1.67\times\) for small, medium, large masks [2606.13898] |
| Visual-token pruning benchmark | UniPruneBench standardized retention \(\rho\) | Random pruning is strong; OCR is most vulnerable; pruning ratio dominates degradation [2511.02650] |
| Vision backbones | Input resolution downsizing | ViT-S \(224\times224 \rightarrow 192\times192\): Top-1 \(81.1\%\rightarrow 80.2\%\), FLOPs \(4.6 \rightarrow 3.3\) GFLOPs [2504.03749] |

EfficientUICoder is representative of task-structured multimodal compression. Its Element and Layout-aware Token Compression detects UI elements, builds a graph over bounding boxes with edge weights \(w_{ij}\) defined by minimum Euclidean perimeter distance, extracts a minimum spanning tree, and retains patch tokens whose receptive fields overlap the selected regions. Region-aware Token Refinement then swaps low-attention retained tokens for high-attention discarded ones using the final CLIP vision-layer attention. Under \(34\)B-level MLLMs, this produces a \(55\%-60\%\) compression ratio without compromising webpage quality, reduces generated tokens by \(41.4\%\), and lowers redundant-sample count from \(136\) to \(16\) [2509.12159].

HiLo-Token addresses a different regime: DiT-based image editing under user masks. It retains all latent tokens inside a dilated mask, selects additional high-frequency tokens outside the mask using Sobel-derived spatial frequency, and represents the rest with tokens from a \(16\times\) downsampled image. On A100-80GB, it reduces the DiT token count by up to \(85\%\) for small masks and yields end-to-end inference speedups of \(1.77\times\), \(1.66\times\), and \(1.33\times\) across small, medium, and large mask categories, with no measurable regression in key perceptual or structural-fidelity metrics [2606.13898].

A more architecture-agnostic approach is input resolution downsizing. For convnets, per-layer FLOPs scale as \(H\cdot W\cdot C_{\rm in}\cdot C_{\rm out}\cdot K^2\), so uniform downsampling by factor \(r\) gives an \(O(r^2)\) reduction; for ViTs, reducing token-grid resolution by \(r\) reduces FLOPs roughly by \(r^4\) and memory by \(r^2\). On ImageNet with ViT-S, moving from \(224\times224\) to \(192\times192\) changes Top-1 from \(81.1\%\) to \(80.2\%\) while lowering FLOPs from \(4.6\) to \(3.3\) GFLOPs and activation memory from \(40\) to \(32\) MB [2504.03749].

UniPruneBench systematizes evaluation of visual-token compression across ten algorithms, ten datasets, and three LMM families. Its main findings are that random pruning is a surprisingly strong baseline, no single method consistently dominates, OCR tasks degrade drastically under aggressive pruning, and the pruning ratio is the dominant factor governing performance degradation. On InternVL3-8B at \(\rho=11.1\%\), DivPrune reduces total time from \(761\,\mathrm{s}\) to \(469\,\mathrm{s}\) and prefill time from \(320\,\mathrm{s}\) to \(185\,\mathrm{s}\), while method time remains below \(0.5\,\mathrm{s}\) for all methods [2511.02650].

## 4. Speech, audio, vector, and edge settings

In speech and audio, input compression often arises as sequence shortening inside encoder-decoder pipelines. CTC-based compression for direct speech translation branches from encoder layer \(N_{\rm CTC}=8\), predicts phone or transcript labels, groups consecutive frames with the same label, and replaces each group with a single vector using AVG, WEIGHTED, or SOFTMAX collapse. On MuST-C English\(\rightarrow\)Italian, adding phone-informed CTC compression with AVG at layer \(8\) improves BLEU from \(22.1\) to \(23.2\) and reduces RAM from \(9624\) MB to \(8554\) MB; on English\(\rightarrow\)German, BLEU rises from \(20.4\) to \(21.9\) while RAM drops from \(9166\) MB to \(7348\) MB [2102.01578].

STAR compresses streaming inputs by dynamically segmenting a frame sequence with a sigmoid read-score \(\alpha_t\), emitting an anchor whenever the running accumulator reaches \(\beta=1\), and using the last hidden state of the segment as its anchor representation. In ASR, STAR reports nearly lossless \(12\times\) compression, with WER changing from \(15.0\) to \(15.9\), and shows \(30\%-40\%\) memory savings already at \(r=10\) for long inputs. In simultaneous ASR and EN\(\rightarrow\)DE speech translation it obtains more favorable latency-quality trade-offs than CIF [2402.01172].

Spectral Feature Compression moves from temporal compression to frequency compression. Instead of the fixed band-split (BS) encoder, which uses \(K\) separate subband modules, SFC uses a single sequence-modeling block to compress \(F\rightarrow K\) and symmetrically decode \(K\rightarrow F\). The cross-attention variant, SFC-CA, injects BS-inspired positional bias while remaining input-adaptive. On MUSDB18-HQ with a small separator, BS reaches \(uSDR=8.26\) dB and \(cSDR=8.72\) dB, whereas SFC-CA reaches \(uSDR=8.95\) and \(cSDR=9.27\); the encoder-decoder parameter count drops from about \(29.6\) M for BS to about \(0.80\) M for SFC-CA [2602.08671].

At the client-server boundary, Trainable Bitwise Soft Quantization compresses each scalar input feature with \(B\) thresholded bits using differentiable sigmoids during training and hard bit-packing at inference. Across six regression datasets, it reports compression factors of \(5\times\) to \(16\times\) relative to \(32\)-bit input without significant performance loss, with a mean compression factor of \(11.1\times\) while maintaining statistical parity with full precision [2603.05172].

RQ-MoE addresses input-dependent vector compression rather than raw sensory input. It combines a hyper-dimensional codebook, a second-level MoE that deforms base centroids into an input-specific codebook, and dual-stream quantization that decouples instruction from quantization. It matches or exceeds QINCo in reconstruction and Recall@1 while providing \(6\times\) to \(14\times\) faster decoding; for example, at \(16\) bytes on BigANN1M it improves Recall@1 from \(71.6\%\) to \(72.3\%\) and MSE from \(0.33\) to \(0.30\) [2605.14359].

## 5. Evaluation criteria, robustness, and failure modes

A central lesson of recent work is that token reduction alone is an inadequate evaluation target. CAVEWOMAN therefore measures realized cost, not nominal prompt length, and compares compressed outputs to the model’s own unconstrained baseline with bidirectional NLI [2606.24083]. “Compression Method Matters” adds instruction survival probability \(\Psi\), CRI, Welch’s \(t\), bootstrap confidence intervals, Tobit regression for censoring, and companion NVML measurements to show that token savings can overstate joule savings [2603.23527]. UniPruneBench adds total time, prefill time, and compression-method overhead to task accuracy [2511.02650].

These richer metrics expose recurrent failure modes. In prompt compression, stripping function words can preserve syntactic remnants while deleting task-critical segments, causing output expansion and accuracy collapse [2606.24083; 2603.23527]. In visual-token pruning, OCR is much more fragile than general VQA, and no compression method dominates across benchmarks [2511.02650]. In image editing, fixed heuristics such as threshold \(T\) and dilation radius \(r\) can miss smooth but semantically important regions [2606.13898]. In speech translation, applying CTC compression too early saves more memory but hurts translation quality more severely [2102.01578].

Some work turns these fragilities into adaptive inference policies. ICPC varies compression level across epochs as data augmentation and at test time uses progressive re-evaluation: samples are first inferred at heavy compression, then reprocessed at medium or full size if confidence remains below a threshold. On average, more than \(75\%\) of samples terminate at heavy compression and fewer than \(15\%\) require full size, yielding net inference speed-ups of approximately \(2\times\) with no accuracy loss; across nine tasks it reports training and inference speed-ups of up to \(2.9\times\) and \(2.6\times\), with accuracy gains of up to \(1\%\) [2312.12385].

Taken together, these results indicate that robustness under input compression is benchmark-dependent and modality-dependent. A plausible implication is that compression policies should be evaluated as conditional control policies over structure, difficulty, and downstream billing, not as single scalar reductions in token count.

## 6. Compression theory, information-theoretic views, and broader significance

Although current ML papers use input compression primarily to control compute and latency, related work connects it to classical compression objectives and to generalization theory. In bit-optimal LZ77 parsing, the objective is to minimize
\[
\sum_{i=1}^k \bigl(|f(d_i)| + |g(\ell_i)|\bigr)
\]
over all parsings of a string, equivalently the shortest path in a DAG whose edges represent phrases. The resulting algorithm computes a bit-optimal parsing in \(O(n\cdot(Q(f,n)+Q(g,n)) + T_{\rm sort}(n,\sigma))\) time and \(O(n)\) space under increasing-cost codeword encodings [0802.0835]. In sequential-access compression, one-pass algorithms face nearly tight memory-quality trade-offs, and one stream is insufficient for grammar-based compression or entropy-only bounds under polylogarithmic resources [0902.0133].

An information-theoretic perspective appears in work on generalization bounds. “Bounding generalization error with input compression” uses the Xu-Raginsky bound
\[
R_{\mathcal P}(\theta)\le R_S(\theta)+\sqrt{\frac{2\,[I(X;Z)+\ln(2/\delta)]}{n}},
\]
and, in the infinite-width limit with additive Gaussian noise, computes
\[
I(X;\widetilde Z)=\tfrac12 \log\det\!\bigl(I_n + K(X,X)/\beta^2\bigr).
\]
Empirically, the bound is often tight for best-performing models, detects randomized labels, reflects robustness to perturbations, and works with few training samples [2207.09408].

These results do not collapse into a single doctrine. In classical coding, compression is about minimizing output bits of a source representation [0802.0835; 0902.0133]. In modern ML systems, it is often about reducing the effective size of the representation *consumed* by a model, or the information retained about the original input, while preserving task utility [2606.24083; 2312.12385; 2207.09408]. This suggests that the contemporary significance of input compression lies in its role as a unifying systems-and-representation problem: it links efficiency, fidelity, and inductive bias, but the correct objective depends on whether the bottleneck is billing, latency, memory, bandwidth, or generalization.

Source: https://www.emergentmind.com/topics/input-compression