The Perplexity Paradox: Why Code Compresses Better Than Math in LLM Prompts

Abstract: In "Compress or Route?" (Johnson, 2026), we found that code generation tolerates aggressive prompt compression (r >= 0.6) while chain-of-thought reasoning degrades gradually. That study was limited to HumanEval (164 problems), left the "perplexity paradox" mechanism unvalidated, and provided no adaptive algorithm. This paper addresses all three gaps. First, we validate across six code benchmarks (HumanEval, MBPP, HumanEval+, MultiPL-E) and four reasoning benchmarks (GSM8K, MATH, ARC-Challenge, MMLU-STEM), confirming the compression threshold generalizes across languages and difficulties. Second, we conduct the first per-token perplexity analysis (n=723 tokens), revealing a "perplexity paradox": code syntax tokens are preserved (high perplexity) while numerical values in math problems are pruned despite being task-critical (low perplexity). Signature injection recovers +34 percentage points in pass rate (5.3% to 39.3%; Cohen's h=0.890). Third, we propose TAAC (Task-Aware Adaptive Compression), achieving 22% cost reduction with 96% quality preservation, outperforming fixed-ratio compression by 7%. MBPP validation (n=1,800 trials) confirms systematic variation: 3.6% at r=0.3 to 54.6% at r=1.0.

• The paper validates the perplexity paradox, confirming that code prompts tolerate aggressive compression (up to 65%) better than math due to high perplexity in coding syntax.
• The study introduces TAAC, an adaptive quality-gated algorithm, improving code preservation (+6.5% additional quality) while beating fixed compression ratios for cost-efficient quality savings.
• The authors experimentally revealed significant cause of math or CoT prompts failure in LLM tasks which lie in the pruning of critical numbers data.

This paper extends a prior study on task-dependent prompt compression by validating the code-versus-reasoning dichotomy across larger benchmarks, providing the first per-token empirical evidence for the hypothesized "perplexity paradox," and introducing TAAC, an adaptive, quality-gated compression algorithm (2602.15843).

Background and motivation

The predecessor study, "Compress or Route?", observed that code generation tolerates aggressive compression (ratios $r \geq 0.6$) while chain-of-thought (CoT) reasoning degrades gradually, but it relied solely on HumanEval (164 problems), left its proposed mechanism unvalidated, and offered no adaptive algorithm. The present work addresses three questions: whether the threshold behavior generalizes beyond HumanEval; why code and math respond differently to perplexity-based compression; and whether these patterns can be exploited algorithmically.

Ruling out prompt length as a confound

Because HumanEval prompts are shorter than GSM8K prompts, compression tolerance could in principle correlate with length rather than task structure. The paper tests this with two complementary designs. An ANCOVA with prompt length as covariate yields a highly significant Task × Compression interaction: $F(5, 2019) = 57.84$, $p = .000108$, $\eta^2 = .081$. Bin-matched sampling over the overlapping length range (67–134 tokens), verified by a Kolmogorov–Smirnov test ($D = 0.089$, $p = .312$), produces an even larger interaction effect ($\eta^2 = .102$). Notably, length differences were attenuating rather than creating the dichotomy.

Effect sizes reveal a crossover interaction: Cohen's $d = +2.14$ favoring code at $r = 0.3$, negligible at $r = 0.6$, and $F(5, 2019) = 57.84$0 favoring CoT at $F(5, 2019) = 57.84$1. This pattern is inconsistent with any monotone length-based explanation and establishes task structure as the operative variable.

Benchmark generalization

Validation on MBPP ($F(5, 2019) = 57.84$2 trials across six ratios, three models) shows monotone degradation: pass rates of 3.7% at $F(5, 2019) = 57.84$3, 11.3% at $F(5, 2019) = 57.84$4, 23.3% at $F(5, 2019) = 57.84$5, 32.3% at $F(5, 2019) = 57.84$6, 42.7% at $F(5, 2019) = 57.84$7, versus a 54.7% uncompressed baseline (Cochran–Armitage trend $F(5, 2019) = 57.84$8). One departure from the original threshold narrative deserves emphasis: MBPP exhibits continuous, approximately linear degradation rather than a sharp cliff—quality retention is only 59% at $F(5, 2019) = 57.84$9. The authors' own H1 anticipated this shift toward more conservative thresholds for shorter, information-denser prompts, and the data support it. The practical implication is that the $p = .000108$0 "safe zone" established on HumanEval does not transfer cleanly to natural-language-style code prompts.

The perplexity paradox

Perplexity-based compressors (LLMLingua, Selective Context) prune low-perplexity tokens as redundant, implicitly assuming that linguistic predictability tracks task importance. The paper's per-token analysis of 723 tokens falsifies this assumption for structured tasks:

Token category	Mean PPL	Count
Python syntax	928,636	46
Brackets/delimiters	68,593	39
Content words	11,697	105
Variable names	10,227	89
Numbers	9,195	54
Stopwords	1,652	36

Python syntax tokens show 79× higher perplexity than content words and are therefore preserved; numerical values in math problems show lower perplexity than surrounding content words despite being task-critical, and are preferentially pruned. Kept tokens average perplexity 143,768 versus 2.03 for removed tokens—a 71,000× disparity—quantifying how extreme the compressor's perplexity bias is.

The causal validation is the strongest result in the paper. In a controlled signature-preservation experiment ($p = .000108$1 pooled trials), injecting function signatures after aggressive compression recovers +34 percentage points in pass rate (5.3% → 39.3%; Cohen's $p = .000108$2), with NameError rates falling from 86.1% to 6.1%. Failures shift to AssertionError (logic errors), indicating the model can now generate syntactically valid code. This identifies Function Identity Collapse—the pruning of signatures—as the dominant failure mode under aggressive code compression, and it implies that cheap, targeted preservation of high-perplexity structural tokens can substitute for expensive global quality recovery.

The paper proposes Semantic Necessity Scoring (SNS), multiplying per-token perplexity by task-category weights (e.g., numbers weighted 3.0 for CoT tasks) to correct the misalignment. However, the weight matrix appears hand-specified, and no ablation validating the specific weight values is reported—an assumption the reader should note.

Multi-algorithm validation

To rule out LLMLingua-2-specific artifacts, the paper designs a phased comparison of LLMLingua-2, LLMLingua-1, Selective Context, and random compression (13,200 planned trials). A Threshold Homogeneity hypothesis posits thresholds within 0.05 across algorithms, to be tested via TOST equivalence testing. Importantly, the results section reports expected findings and predictions rather than completed multi-algorithm results, so cross-algorithm generality remains asserted by design rather than demonstrated empirically—a significant open item.

Task-Aware Adaptive Compression (TAAC)

TAAC operates in three stages: a DistilBERT task classifier (<10 ms); information density estimation via the coefficient of variation of per-token perplexity; and iterative quality-gated compression using task-specific targets ($p = .000108$3, $p = .000108$4, hybrid 0.72) with a 2-layer MLP quality predictor trained on ~50K samples. On a synthetic validation set (220 prompts), TAAC achieves 95.6% quality preservation at 21.8% cost savings, dominating fixed $p = .000108$5 (89.1% quality, 41.2% savings) by +6.5 points of quality and outperforming task-based fixed thresholds by 7% on cost-quality tradeoff. Component ablation attributes most quality benefit to gating. The trade-off is explicit: TAAC sacrifices roughly half the savings of aggressive fixed-ratio compression to obtain its quality guarantee, and its quality predictor is trained on the authors' own experimental distribution, raising distribution-shift concerns for deployment.

Limitations and open questions

The paper concedes several constraints. Code benchmarks cover function completion only; longer files may behave differently. Perplexity measurements use a single pilot model, and different model families may yield different category-level patterns. TAAC has not been evaluated in agentic or multi-turn settings. Additionally, the SNS weights are unvalidated, the multi-algorithm comparison is incomplete, and the MBPP results suggest the celebrated $p = .000108$6 cliff may be benchmark-dependent. Whether the perplexity paradox persists for models heavily pretrained on code—where syntax tokens would presumably have low perplexity—is a concrete question the paper leaves unanswered.

Conclusion

The paper converts a previously hypothesized mechanism into measured evidence: code survives compression because its syntax is high-perplexity and retained, while math fails because numerals are low-perplexity and pruned despite being essential. The signature-preservation intervention (+34 pp, $p = .000108$7) provides causal support, and TAAC demonstrates that task-aware, quality-gated compression yields Pareto-improving cost-quality tradeoffs. The main caveats—single-pilot-model perplexity analysis, hand-set SNS weights, incomplete multi-algorithm validation, and benchmark-dependent threshold sharpness—define the boundaries within which these conclusions currently hold.