Compression Robustness Index (CRI)

• CRI is a metric that evaluates prompt compression by jointly measuring quality retention and output-length penalties across diverse benchmarks.
• Its formulation combines a quality-retention factor and an output-explosion penalty, accounting for deployment costs like energy use and token efficiency.
• Empirical results differentiate models such as GPT-4o-mini, Mistral-Large, and DeepSeek-Chat, highlighting CRI's role in assessing practical compression robustness.

Searching arXiv for papers on “Compression Robustness Index” and closely related compression–robustness evaluation work. Compression Robustness Index (CRI) is a benchmark-aggregated metric for evaluating whether compression remains useful and controlled under deployment-relevant conditions. In the literature summarized here, the term is defined explicitly for large-language-model prompt compression, where it combines quality retention with a penalty for output-length explosion across a diverse benchmark set (Johnson, 6 Mar 2026). Closely related work in certified neural network compression, adversarially robust model compression, corruption robustness, BERT compression, index compression, and robust distillation does not always use the name CRI, but it repeatedly adopts the same underlying principle: compression should be assessed jointly with robustness, fidelity, and deployment efficiency rather than by size reduction alone (Kaur et al., 2022).

1. Definition and formal structure

The explicit definition of CRI is given for prompt compression as

$$\mathrm{CRI}(M,r)=\frac{1}{|\mathcal{B}|}\sum_{b\in\mathcal{B}} \frac{Q_r^{(b)}}{Q_0^{(b)}} \left(1-\frac{\max\!\bigl(0,\;T_r^{(b)}-T_0^{(b)}\bigr)}{T_{\max}}\right),$$

where $M$ is the model, $r$ is the compression ratio, $\mathcal{B}$ is a diverse benchmark set, $Q_0^{(b)}$ and $Q_r^{(b)}$ are baseline and compressed quality on benchmark $b$, $T_0^{(b)}$ and $T_r^{(b)}$ are baseline and compressed mean output lengths, and $T_{\max}=1024$ is the generation ceiling (Johnson, 6 Mar 2026). In the reported experiments, quality is measured using pass@1 functional correctness (Johnson, 6 Mar 2026).

The formula is an average over benchmarks of a product of two terms. The first term,

$M$0

is a quality-retention factor. The second term,

$M$1

is an output-explosion penalty. The penalty is one-sided: if compression leaves output length unchanged or reduces it, the $M$2 term is zero and the benchmark incurs no penalty (Johnson, 6 Mar 2026).

The paper states directly that higher CRI indicates better compression robustness (Johnson, 6 Mar 2026). It also states that CRI is not a pure safety metric, not a pure quality metric, and not a pure cost metric; it is a joint robustness indicator intended to capture whether compression remains beneficial and controlled across benchmarks (Johnson, 6 Mar 2026). This makes CRI structurally different from single-axis measures such as input-token reduction, clean accuracy, or compression ratio alone.

2. Motivation: benchmark dependence and structural survival

The motivation for CRI is the observation that prompt compression cannot be judged by input-token reduction alone. A compressed prompt may save input tokens yet trigger much longer outputs, thereby reducing or reversing savings in cost and energy (Johnson, 6 Mar 2026). The paper describes this as a compression paradox and ties it to an energy model

$M$3

with

$M$4

so output tokens are approximately $M$5 more expensive than input processing (Johnson, 6 Mar 2026). CRI does not include energy directly, but its output-length penalty is motivated by this asymmetry.

A second motivation is benchmark dependence. The paper states that single-benchmark assessments can produce misleading conclusions about compression safety and efficiency (Johnson, 6 Mar 2026). The central explanatory variable is instruction survival probability $M$6, defined at segment level by

$M$7

and at prompt level by

$M$8

where $M$9 is an instruction segment and the weights satisfy $r$0 (Johnson, 6 Mar 2026). For deterministic first-$r$1-words compression,

$r$2

The reported benchmark-level values at $r$3 are $r$4 for MBPP, $r$5 for HumanEval, and $r$6 for GSM8K (Johnson, 6 Mar 2026). The paper further hypothesizes a thresholded relation between $r$7 and output length, with empirical threshold $r$8 (Johnson, 6 Mar 2026). Low $r$9 is associated with severe output expansion and quality loss; high $\mathcal{B}$0 is associated with more stable generation. The paper does not define CRI directly as a function of $\mathcal{B}$1, and it distinguishes the two roles clearly: $\mathcal{B}$2 is explanatory, whereas CRI is evaluative (Johnson, 6 Mar 2026).

3. Experimental methodology and empirical profile

The CRI study evaluates three models—DeepSeek-Chat, GPT-4o-mini, and Mistral-Large—on three structurally distinct benchmarks: MBPP ($\mathcal{B}$3), HumanEval ($\mathcal{B}$4), and GSM8K ($\mathcal{B}$5) (Johnson, 6 Mar 2026). Compression is deterministic first-$\mathcal{B}$6-words truncation with

$\mathcal{B}$7

and CRI is reported at $\mathcal{B}$8 (Johnson, 6 Mar 2026). The study comprises 5,400 API calls; each $\mathcal{B}$9 cell uses $Q_0^{(b)}$0 independent prompts with 3 replicates each, giving 150 observations per cell, and temperature is fixed at $Q_0^{(b)}$1 (Johnson, 6 Mar 2026). Statistical analysis uses Welch’s heteroscedastic $Q_0^{(b)}$2-test, bootstrap confidence intervals with 10,000 BCa resamples, and Tobit regression for right-censored outputs at $Q_0^{(b)}$3 (Johnson, 6 Mar 2026).

The main CRI results at $Q_0^{(b)}$4 are as follows:

Model	CRI	Interpretation
GPT-4o-mini	0.848	Highly robust
Mistral-Large	0.424	Moderately robust
DeepSeek-Chat	0.090	Compression-sensitive

These values are reported with 95% confidence intervals of $Q_0^{(b)}$5, $Q_0^{(b)}$6, and $Q_0^{(b)}$7, respectively (Johnson, 6 Mar 2026). The implied ranking is GPT-4o-mini first, Mistral-Large second, and DeepSeek-Chat third (Johnson, 6 Mar 2026).

The benchmark-specific behavior explains why aggregation is necessary. For DeepSeek at $Q_0^{(b)}$8, MBPP shows baseline output length $Q_0^{(b)}$9, compressed output length $Q_r^{(b)}$0, output ratio $Q_r^{(b)}$1, and pass@1 drop from $Q_r^{(b)}$2 to $Q_r^{(b)}$3; HumanEval shows $Q_r^{(b)}$4 to $Q_r^{(b)}$5, ratio $Q_r^{(b)}$6, and pass@1 drop from $Q_r^{(b)}$7 to $Q_r^{(b)}$8; GSM8K shows $Q_r^{(b)}$9 to $b$0, ratio $b$1, and pass@1 drop from $b$2 to $b$3 (Johnson, 6 Mar 2026). The paper emphasizes that MBPP alone would overestimate explosion risk, whereas HumanEval alone would underestimate it (Johnson, 6 Mar 2026).

An additional interpretive caveat is ceiling censoring. The paper states that 74% of DeepSeek MBPP trials at $b$4 hit the $b$5-token ceiling, so the observed mean output length may be a lower bound; Tobit correction suggests the uncensored mean could exceed 2000 tokens, implying explosion factors above $b$6 (Johnson, 6 Mar 2026). This matters because the CRI penalty is bounded by $b$7, whereas the underlying instability may be more severe.

4. CRI as a general compression–robustness design pattern

Outside prompt compression, several papers do not define a single scalar CRI, yet they supply the ingredients for one by treating compression and robustness as coupled objectives rather than separate reporting axes. In verified model synthesis, VeriCompress seeks a sparse subnetwork $b$8 with $b$9 while preserving what the paper operationally calls “generalizability,” defined as simultaneously high Standard Accuracy and Verified Accuracy, alongside deployability under architecture and size limits (Kaur et al., 2022). Standard Accuracy is “percentage of benign samples classified correctly,” and Verified Accuracy is “percentage of benign samples which are certified to be robust using verified robustness mechanism IBP” (Kaur et al., 2022). Parameters, relative model size, memory footprint, inference latency, peak RAM, peak CPU, and training time are all reported, which directly supports deployment-aware CRI constructions (Kaur et al., 2022).

Compression-aware certified training extends the same logic to pruning and quantization. The CRANE framework trains over a set $T_0^{(b)}$0 of compressed networks and optimizes an average robust objective over that set rather than over a single model (Xu et al., 13 Jun 2025). It reports standard accuracy, certified accuracy, pruning level, and quantization format, and the technical synthesis identifies compression-averaged, worst-case, retention-ratio, and area-under-curve constructions as natural CRI analogues (Xu et al., 13 Jun 2025). A plausible implication is that, in certified settings, CRI is best viewed as a set-valued robustness-retention summary rather than a one-point score.

Adversarially robust model compression gives a related but attack-based formulation. ATMC minimizes adversarial loss under joint sparsity and quantization constraints and evaluates robust accuracy under PGD as a function of compressed model size in bits (Gui et al., 2019). That paper’s framing is explicitly tri-objective: model compactness, clean accuracy, and adversarial robustness (Gui et al., 2019). This suggests a CRI for attack-based compression should normalize robust accuracy by a dense adversarially trained reference and use actual compressed size in bits rather than sparsity alone.

The same pattern appears in language-model compression and corruption robustness. For BERT compression, preserved accuracy is supplemented by label loyalty, probability loyalty, after-attack accuracy, and query number, showing that models with similar clean accuracy can diverge sharply in teacher fidelity and attack resistance (Xu et al., 2021). For compressed CNNs under natural corruptions, robustness is measured by baseline-relative mean Corruption Error (mCE), alongside clean accuracy and compression ratio, and model selection is organized through Pareto fronts that maximize accuracy, minimize mCE, and maximize compression ratio (Silva et al., 31 Dec 2025). These papers do not define CRI explicitly, but the technical syntheses present the same design principle: robustness under compression is a joint property of utility preservation and deployment efficiency.

5. Alternative robustness objects associated with compression

The phrase compression robustness is not confined to prompt compression or neural network pruning. In distributed information retrieval, robustness can refer to the stability of compressed inverted-index size under random local index partitioning. The paper on random index-partitioning shows that aggregated postings size can decrease logarithmically with the number of servers under random ordering, yet increase logarithmically under URL sorting or clustering/TSP orderings (Feldman et al., 2011). The technical synthesis proposes relative degradation

$T_0^{(b)}$1

and log-sensitivity

$T_0^{(b)}$2

as natural partition-robustness indices, while also emphasizing that the direction of the effect depends on the ordering scheme (Feldman et al., 2011). Here, “robustness” concerns compression stability under sharding, not predictive reliability.

In robust model compression via deep hypotheses, robustness is defined through predicate depth and breakdown point rather than adversarial perturbations. The paper defines predicate depth

$T_0^{(b)}$3

and a compression-specific breakdown point

$T_0^{(b)}$4

for the compression algorithm $T_0^{(b)}$5 relative to oracle perturbations (Armstrong et al., 2021). The paper’s empirical robustness metric is prediction agreement across compressed models obtained from perturbed training sets (Armstrong et al., 2021). This suggests a different CRI lineage in which robustness denotes stability of the compression mapping itself, rather than resistance to input attacks.

A third variant appears in compression-in-the-loop adversarial inference. There, compression is inserted before inference, attacks are applied in the compressed representation, and robustness degradation is attributed to decision space reduction (Evans et al., 8 Apr 2026). The paper defines candidate accuracy-drop and accuracy-ratio CRIs in the technical synthesis, such as

$T_0^{(b)}$6

while also introducing geometric quantities such as true-class area fraction, mean margin, and boundary intrusion as mechanistic sub-indices (Evans et al., 8 Apr 2026). In this setting, compression robustness concerns how much robustness survives when the deployed representation itself is compressed.

6. Limitations, interpretation, and scope

CRI is not a universal constant of a model. The prompt-compression paper states that CRI depends on the chosen benchmark set $T_0^{(b)}$7, and if benchmark diversity is poor, the score can still mislead (Johnson, 6 Mar 2026). It also evaluates only three providers/models and uses naive first-$T_0^{(b)}$8-words truncation as the compression method, explicitly noting that the main experiments rely on closed APIs and that energy is estimated rather than directly measured in those runs (Johnson, 6 Mar 2026). The same paper therefore supports using CRI as an evaluation framework, not as a benchmark-independent property.

Across adjacent domains, the caveats multiply. Verified-compression papers are pruning-centric or quantization-centric and often rely on verifier-specific quantities such as IBP or $T_0^{(b)}$9-CROWN, so certified robustness scores are method-dependent (Kaur et al., 2022); (Xu et al., 13 Jun 2025). Natural-corruption studies normalize robustness relative to the uncompressed baseline of the same architecture, which is well suited to within-architecture comparisons but not to absolute cross-architecture ranking (Silva et al., 31 Dec 2025). Attack-based compression papers use empirical robust accuracy under fixed attack suites rather than certification, so any CRI derived from them is attack-dependent (Gui et al., 2019); (Evans et al., 8 Apr 2026). BERT-compression robustness is operationalized through TextFooler after-attack accuracy and query count, not through a norm-bounded worst-case guarantee (Xu et al., 2021).

Several common misconceptions are explicitly rejected by the source material. CRI is not equivalent to input-token savings, because output explosion can erase nominal savings (Johnson, 6 Mar 2026). It is not equivalent to energy savings, because token reductions can overstate joule reductions (Johnson, 6 Mar 2026). It is not equivalent to clean accuracy, because compression methods with similar accuracy can differ substantially in loyalty, verified robustness, or attacked accuracy (Xu et al., 2021); (Kaur et al., 2022). Nor is compression robustness universally monotone in compression severity: some architectures retain robustness well under moderate compression, whereas others exhibit cliffs or benchmark-specific failure modes (Kaur et al., 2022); (Silva et al., 31 Dec 2025).

Taken together, the literature supports a precise but domain-sensitive understanding of CRI. In its explicit form, CRI is a cross-benchmark prompt-compression metric that averages quality retention and an output-explosion penalty (Johnson, 6 Mar 2026). In broader usage, this suggests an umbrella concept for scalar or Pareto-faithful summaries that quantify how much utility, robustness, or fidelity survives compression under realistic deployment constraints. The consistent methodological lesson is that compression must be evaluated jointly with the failure modes it can induce.