Perplexity-Guided Pruning in Language Models

• Perplexity-guided pruning is a strategy that leverages token-level negative log-likelihood scores to rank and remove redundant pretraining data, network components, or reasoning steps.
• The method adapts metrics such as cross-entropy and KL divergence when direct perplexity search is impractical, addressing diverse pruning needs from efficiency to reasoning fidelity.
• Empirical findings reveal that targeting an optimal perplexity band—rather than solely low perplexity—can improve model performance and reduce computational demands.

Perplexity-guided pruning is a family of pruning strategies in which perplexity, or the token-level negative log-likelihood from which perplexity is derived, is used to decide what to remove from a language-model pipeline. In recent work, the pruned object varies widely: fixed-length pretraining sequences can be filtered before model training, attention heads or MLP channels can be removed from pretrained transformers, chain-of-thought steps can be deleted or merged, and unlabeled pools in active learning can be reduced before acquisition. The literature also contains many closely related methods that do not use perplexity explicitly, but replace it with cross-entropy, decode-only negative log-likelihood, KL divergence, activation geometry, or representation-similarity proxies because direct perplexity search is either too expensive or too poorly aligned with the target behavior (Marion et al., 2023, Wang et al., 20 Oct 2025, Cui et al., 18 Feb 2025).

1. Mathematical basis and methodological scope

In the direct formulation used for chain-of-thought refinement, perplexity is defined over a generated sequence conditioned on a prompt $x$ as

$$\text{PPL}(x,\{w_k\}_{k=1}^{N}) = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log p(w_i \mid x, w_1,\dots, w_{i-1})\right),$$

so lower perplexity means the sequence is easier for the model to predict under its own autoregressive distribution (Cui et al., 18 Feb 2025). In pretraining-data pruning, the same idea is instantiated at the sample level: each example $x^{(i)}$ is scored by a length-normalized negative log-likelihood,

$$\mathrm{NLL}_{x^{(i)}} = \frac{1}{|x^{(i)}|} \sum_{t_j \in x^{(i)}} -\log P(t_j \mid t_{<j}; \theta_{\mathrm{ref}}),$$

and then converted into sequence perplexity,

$$\mathrm{PPLX}_{x^{(i)}} = 2^{\mathrm{NLL}_{x^{(i)}}}.$$

The resulting scalar can then be used for percentile-based retention or rejection (Ankner et al., 2024).

When perplexity guides model pruning directly, the operational quantity is often the language-model cross-entropy itself rather than its exponential. "GISP-Global Iterative Structured Pruning" defines the language-modeling loss as

$L = -\frac{1}{N}\sum_{i=1}^{N}\log p(x_i|x_{<i}),$

and computes first-order importance by the loss-gradient-weight product, so the pruning signal is explicitly model-level and loss-based rather than layer-local reconstruction error (Wang et al., 20 Oct 2025). This establishes an important distinction within the field: some methods are directly perplexity-guided, while others are only perplexity-evaluated.

The recent literature therefore spans at least three direct uses of perplexity. First, perplexity can rank training data before pretraining begins (Marion et al., 2023, Ankner et al., 2024). Second, it can define the loss whose gradients induce structural saliency in post-training model pruning (Wang et al., 20 Oct 2025), or serve as a leave-one-structure-out criterion for depth pruning (Sandri et al., 29 Jan 2025). Third, it can score the importance of individual reasoning steps in chain-of-thought compression by measuring the perplexity increase caused by removing them (Cui et al., 18 Feb 2025). Several adjacent papers then argue that these direct uses must often be modified or replaced when the target is reasoning fidelity, code logic, safety, or other non-generic capabilities (Wang et al., 1 Dec 2025, Zeng et al., 8 Aug 2025, Xu et al., 2024).

2. Perplexity-guided pruning of pretraining data

The clearest direct formulation appears in large-scale data pruning for pretraining. "When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale" scores fixed-length $2048$-token sequences by reference-model perplexity and then keeps different percentile regions of the distribution. Its main result is that the strongest subsets are not the lowest-perplexity ones: keeping the middle $50\%$ of the perplexity distribution improves test perplexity by $0.97\%$ over the no-pruning baseline, and keeping only the middle $30\%$ still improves by $$\text{PPL}(x,\{w_k\}_{k=1}^{N}) = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log p(w_i \mid x, w_1,\dots, w_{i-1})\right),$$0. The paper also reports that the method scales to $$\text{PPL}(x,\{w_k\}_{k=1}^{N}) = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log p(w_i \mid x, w_1,\dots, w_{i-1})\right),$$1B-parameter models, achieving $$\text{PPL}(x,\{w_k\}_{k=1}^{N}) = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log p(w_i \mid x, w_1,\dots, w_{i-1})\right),$$2 improvement over a same-size no-pruning baseline, and that perplexity outperforms EL2N and memorization at equal retention levels (Marion et al., 2023).

That work further shows that the quality of the reference model matters. A fully trained $$\text{PPL}(x,\{w_k\}_{k=1}^{N}) = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log p(w_i \mid x, w_1,\dots, w_{i-1})\right),$$3B reference model yields a $$\text{PPL}(x,\{w_k\}_{k=1}^{N}) = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log p(w_i \mid x, w_1,\dots, w_{i-1})\right),$$4 improvement over the best-performing setup that used a $$\text{PPL}(x,\{w_k\}_{k=1}^{N}) = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log p(w_i \mid x, w_1,\dots, w_{i-1})\right),$$5M reference model, and a $$\text{PPL}(x,\{w_k\}_{k=1}^{N}) = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log p(w_i \mid x, w_1,\dots, w_{i-1})\right),$$6M reference model trained on Wikipedia yields a $$\text{PPL}(x,\{w_k\}_{k=1}^{N}) = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log p(w_i \mid x, w_1,\dots, w_{i-1})\right),$$7 advantage over a $$\text{PPL}(x,\{w_k\}_{k=1}^{N}) = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log p(w_i \mid x, w_1,\dots, w_{i-1})\right),$$8M reference model trained on CommonCrawl in the compared setting (Marion et al., 2023). A reference model trained for about $$\text{PPL}(x,\{w_k\}_{k=1}^{N}) = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log p(w_i \mid x, w_1,\dots, w_{i-1})\right),$$9 of full reference training is reported to behave similarly to a fully trained one, which makes the approach more practical for large corpora.

"Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models" broadens this picture by showing that a $x^{(i)}$0M reference model can prune data effectively for $x^{(i)}$1B and $x^{(i)}$2B target models. On the Pile, the best rule is to retain the highest-perplexity $x^{(i)}$3 of examples; on Dolma, the best rule is to retain a medium-perplexity band containing $x^{(i)}$4 of examples around the median. The paper reports up to $x^{(i)}$5 average downstream improvement and up to a $x^{(i)}$6 reduction in pretraining steps to reach baseline performance (Ankner et al., 2024). It also shows that test-set perplexity on the original pretraining distribution can worsen while downstream task quality improves, so the usefulness of perplexity-based data pruning depends not only on the score itself but on corpus composition and evaluation target (Ankner et al., 2024).

Taken together, these studies establish a central principle of perplexity-guided data pruning: there is no universal "keep the lowest perplexity" rule. In the pretraining setting, the optimal retained region can be the middle band or the high-perplexity tail, depending on the corpus mixture, and a small reference model can be sufficient if the scoring rule is tuned to the corpus (Marion et al., 2023, Ankner et al., 2024).

3. Perplexity-guided pruning of model structure

For post-training model compression, the most explicit direct method is GISP. "From Local to Global: Revisiting Structured Pruning Paradigms for LLMs" argues that local layer-wise reconstruction is task-agnostic, whereas global structured pruning can use a model-level loss. For language modeling, that loss is the cross-entropy

$x^{(i)}$7

and the first-order importance of a weight is approximated by

$x^{(i)}$8

These element-wise scores are aggregated to attention heads and MLP channels, normalized within block types, ranked globally, and pruned iteratively rather than in one shot (Wang et al., 20 Oct 2025). The iterative schedule is central because one-shot global pruning shows "perplexity collapse" at high sparsity; on Llama2-7B, for example, one-shot global pruning reaches WikiText-2 perplexity $x^{(i)}$9 at $$\mathrm{NLL}_{x^{(i)}} = \frac{1}{|x^{(i)}|} \sum_{t_j \in x^{(i)}} -\log P(t_j \mid t_{<j}; \theta_{\mathrm{ref}}),$$0 sparsity, whereas iterative GISP reaches $$\mathrm{NLL}_{x^{(i)}} = \frac{1}{|x^{(i)}|} \sum_{t_j \in x^{(i)}} -\log P(t_j \mid t_{<j}; \theta_{\mathrm{ref}}),$$1 (Wang et al., 20 Oct 2025).

A second direct model-pruning formulation appears in "2SSP: A Two-Stage Framework for Structured Pruning of LLMs." Its first stage performs width pruning in FFNs using activation-magnitude importance, but its second stage removes whole attention submodules by iterative leave-one-module-out perplexity: $$\mathrm{NLL}_{x^{(i)}} = \frac{1}{|x^{(i)}|} \sum_{t_j \in x^{(i)}} -\log P(t_j \mid t_{<j}; \theta_{\mathrm{ref}}),$$2 This makes 2SSP only partially perplexity-guided, but the depth stage is explicitly so (Sandri et al., 29 Jan 2025). At $$\mathrm{NLL}_{x^{(i)}} = \frac{1}{|x^{(i)}|} \sum_{t_j \in x^{(i)}} -\log P(t_j \mid t_{<j}; \theta_{\mathrm{ref}}),$$3 sparsity on Llama-2 7B, the paper reports WikiText-2 perplexity $$\mathrm{NLL}_{x^{(i)}} = \frac{1}{|x^{(i)}|} \sum_{t_j \in x^{(i)}} -\log P(t_j \mid t_{<j}; \theta_{\mathrm{ref}}),$$4, compared with $$\mathrm{NLL}_{x^{(i)}} = \frac{1}{|x^{(i)}|} \sum_{t_j \in x^{(i)}} -\log P(t_j \mid t_{<j}; \theta_{\mathrm{ref}}),$$5 for SliceGPT, $$\mathrm{NLL}_{x^{(i)}} = \frac{1}{|x^{(i)}|} \sum_{t_j \in x^{(i)}} -\log P(t_j \mid t_{<j}; \theta_{\mathrm{ref}}),$$6 for EvoPress, and $$\mathrm{NLL}_{x^{(i)}} = \frac{1}{|x^{(i)}|} \sum_{t_j \in x^{(i)}} -\log P(t_j \mid t_{<j}; \theta_{\mathrm{ref}}),$$7 for BlockPruner (Sandri et al., 29 Jan 2025).

Historically, perplexity also served as the principal quality axis even when it was not the pruning objective. "Adaptive Pruning of Neural LLMs for Mobile Devices" studies structured QRNN pruning for mobile keyboards and evaluates operating points in a perplexity-efficiency tradeoff space, reporting an operating point with $$\mathrm{NLL}_{x^{(i)}} = \frac{1}{|x^{(i)}|} \sum_{t_j \in x^{(i)}} -\log P(t_j \mid t_{<j}; \theta_{\mathrm{ref}}),$$8 energy savings over the state of the art and only a $$\mathrm{NLL}_{x^{(i)}} = \frac{1}{|x^{(i)}|} \sum_{t_j \in x^{(i)}} -\log P(t_j \mid t_{<j}; \theta_{\mathrm{ref}}),$$9 relative increase in perplexity (Tang et al., 2018). This older use of perplexity as the deployment-side criterion prefigures much of the later LLM pruning literature.

4. Perplexity-guided pruning of reasoning traces and chain-of-thought

Perplexity-guided pruning has also been applied to reasoning traces rather than model parameters. "Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in LLMs" introduces SPIRIT, which treats one sentence as one reasoning step and removes the least important step at each iteration. In few-shot chain-of-thought refinement, the candidate step for deletion is chosen by

$$\mathrm{PPLX}_{x^{(i)}} = 2^{\mathrm{NLL}_{x^{(i)}}}.$$0

and in fine-tuning data refinement by

$$\mathrm{PPLX}_{x^{(i)}} = 2^{\mathrm{NLL}_{x^{(i)}}}.$$1

Thresholds $$\mathrm{PPLX}_{x^{(i)}} = 2^{\mathrm{NLL}_{x^{(i)}}}.$$2 and $$\mathrm{PPLX}_{x^{(i)}} = 2^{\mathrm{NLL}_{x^{(i)}}}.$$3 determine whether a step is deleted directly, merged into neighboring steps, or whether refinement should stop (Cui et al., 18 Feb 2025). The paper reports a better accuracy-efficiency tradeoff than random removal and than simply instructing the model to be concise (Cui et al., 18 Feb 2025).

A major critique of this paradigm appears in "Pruning the Unsurprising: Efficient Code Reasoning via First-Token Surprisal." That paper treats SPIRIT as a step-level perplexity-guided baseline and argues that full-step perplexity measures linguistic fluency rather than logical necessity in code reasoning. It replaces step-level perplexity with first-token surprisal after an anchor-guided coarse pruning stage, and reports on LiveCodeBench v4_v5 that SPIRIT reaches $$\mathrm{PPLX}_{x^{(i)}} = 2^{\mathrm{NLL}_{x^{(i)}}}.$$4 Pass@1 with $$\mathrm{PPLX}_{x^{(i)}} = 2^{\mathrm{NLL}_{x^{(i)}}}.$$5 tokens and $$\mathrm{PPLX}_{x^{(i)}} = 2^{\mathrm{NLL}_{x^{(i)}}}.$$6s latency, whereas ASAP reaches $$\mathrm{PPLX}_{x^{(i)}} = 2^{\mathrm{NLL}_{x^{(i)}}}.$$7 Pass@1 with $$\mathrm{PPLX}_{x^{(i)}} = 2^{\mathrm{NLL}_{x^{(i)}}}.$$8 tokens and $$\mathrm{PPLX}_{x^{(i)}} = 2^{\mathrm{NLL}_{x^{(i)}}}.$$9s latency (Zeng et al., 8 Aug 2025). The criticism is therefore not that token-level probability information is useless, but that average step perplexity may be the wrong aggregation for logic-heavy reasoning traces.

Inference-time branch pruning raises a related issue. "Inference-Time Chain-of-Thought Pruning with Latent Informativeness Signals" introduces KAPPA, which prunes partially generated reasoning branches using KL divergence, confidence, and entropy rather than perplexity. Perplexity appears only in the Full-BoN baseline, where the final answer is selected using a negative perplexity score. KAPPA instead scores alive branches by a trajectory-weighted combination of KL-based information change, confidence, and entropy, and reports up to approximately $L = -\frac{1}{N}\sum_{i=1}^{N}\log p(x_i|x_{<i}),$0 reduction in peak memory and approximately $L = -\frac{1}{N}\sum_{i=1}^{N}\log p(x_i|x_{<i}),$1 reduction in total token generation relative to BoN, with minimal impact on accuracy (Li et al., 1 Nov 2025). This marks a broader shift from perplexity-guided branch scoring toward richer distributional signals during decoding.

5. Misalignment, capability trade-offs, and failure modes

A recurring theme in the literature is that perplexity can be a useful pruning signal while still being an imperfect surrogate for the actual behavior one wishes to preserve. "Rethinking Pruning LLMs: Benefits and Pitfalls of Reconstruction Error Minimization" shows that aggressively minimizing calibration-set reconstruction error can lower reconstruction error by more than $L = -\frac{1}{N}\sum_{i=1}^{N}\log p(x_i|x_{<i}),$2 yet worsen language perplexity and downstream task performance because of calibration overfitting. The paper reports, for example, that on LLaMA-7B with SparseGPT, adding cross-block reconstruction lowers normalized error from $L = -\frac{1}{N}\sum_{i=1}^{N}\log p(x_i|x_{<i}),$3 to $L = -\frac{1}{N}\sum_{i=1}^{N}\log p(x_i|x_{<i}),$4 but worsens mean perplexity from $L = -\frac{1}{N}\sum_{i=1}^{N}\log p(x_i|x_{<i}),$5 to $L = -\frac{1}{N}\sum_{i=1}^{N}\log p(x_i|x_{<i}),$6 (Shin et al., 2024). This result does not reject perplexity-oriented pruning; rather, it shows that proxies beneath it can become misaligned unless calibration data generalize.

The same issue becomes sharper for reasoning models. "Think Before You Prune: Self-Reflective Structured Pruning for Reasoning LLMs" does not use perplexity explicitly at all, but it is directly relevant because it argues that pruning reasoning LLMs fails when calibration data and pruning objectives do not match the model’s decode-time reasoning distribution. RESP therefore replaces generic calibration with self-generated reasoning traces and uses a decode-only negative log-likelihood objective rather than full-sequence or prefill-dominated objectives (Wang et al., 1 Dec 2025). On Qwen3-8B at $L = -\frac{1}{N}\sum_{i=1}^{N}\log p(x_i|x_{<i}),$7 sparsity, RESP reaches $L = -\frac{1}{N}\sum_{i=1}^{N}\log p(x_i|x_{<i}),$8 accuracy on GSM8K and $L = -\frac{1}{N}\sum_{i=1}^{N}\log p(x_i|x_{<i}),$9 on MathQA, far above structured baselines, despite reporting no perplexity numbers (Wang et al., 1 Dec 2025). The implication is that generic perplexity guidance over mismatched text can preserve fluency while destroying multi-step reasoning.

"Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2" makes the limitation even more explicit. Under MAW-guided width pruning of GLU-MLP layers, WikiText and Lambada perplexities worsen monotonically as the expansion ratio decreases, and knowledge-heavy tasks such as MMLU and GSM8K degrade in the same direction. Yet instruction-following improves substantially and multi-step reasoning on MUSR remains robust, while TruthfulQA-MC2 improves as MMLU falls (Martra, 27 Dec 2025). The paper reports $2048$0 to $2048$1 IFEval gains and a robust inverse correlation $2048$2 between MMLU and TruthfulQA-MC2 in Llama-3B (Martra, 27 Dec 2025). Perplexity is therefore a good partial signal for preserving parametric knowledge and generic language modeling, but not a complete scalar objective for all desirable capabilities.

6. Beyond perplexity: probability-space analysis, safety evaluation, and emerging hybrids

Several papers explain why perplexity remains important for pruning even when it is not sufficient. "Demystifying When Pruning Works via Representation Hierarchies" decomposes model behavior into embedding space, logit space, and probability space, and shows that pruning-induced perturbations can remain small in embedding and logit space yet become much larger after softmax. The paper derives approximations such as

$2048$3

which directly connect pruning-induced logit perturbations to probability-space damage (He et al., 25 Mar 2026). Its empirical conclusion is that generation quality depends on probability-space preservation, so validation perplexity, token-level NLL increase, or KL divergence are more generation-relevant than hidden-state similarity alone (He et al., 25 Mar 2026).

At the same time, "Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression" argues that compression research often prioritizes preserving perplexity, but that perplexity alone misses degeneration harm, representational harm, dialect bias, and downstream task quality. The paper finds that quantization mostly preserves bias while pruning degrades quickly, that aggregate safety metrics can be confounded by generation collapse, and that dialect inequities remain under compression (Xu et al., 2024). In other words, perplexity can remain necessary for judging language-model fidelity while being insufficient for judging whether a pruned model is acceptable in deployment.

A notable hybrid use of perplexity appears outside standard pruning for compression. "LLM-Driven Data Pruning Enables Efficient Active Learning" inserts a perplexity-based first-stage filter before standard active-learning acquisition. Stage 1 uses KenLM $2048$4-gram perplexity to score every example in the unlabeled pool, stage 2 rescues high-quality items from the high-perplexity remainder using a quantized LLM, and a perplexity reweighting rule promotes underrepresented instances in later rounds (Azeemi et al., 2024). The paper reports up to $2048$5 reduction in end-to-end active-learning time (Azeemi et al., 2024). This demonstrates that perplexity-guided pruning has become a general design pattern for cheaply reducing search spaces before more expensive decision rules are applied.

Recent proxy-based compression methods push this logic further. ACE uses activation cosine similarity and activation variance as pruning surrogates and reports up to an $2048$6 reduction in perplexity and up to $2048$7 decrease in pruning time (2505.21987). TRIM reallocates row-wise sparsity using activation-quality metrics and reports that at $2048$8 sparsity it reduces perplexity by $2048$9 for Qwen2.5-14B and over $50\%$0 for OPT-13B compared to baseline methods (Beck et al., 22 May 2025). LoRP prunes whole transformer blocks by representation locality rather than perplexity, yet improves average perplexity over other one-shot depth-pruning baselines, especially on Qwen models with globally distributed redundancy (Yun et al., 27 May 2026). Agent-guided pruning uses perplexity as an iterative control signal rather than an inner-loop objective: checkpoint rollback is triggered when perplexity degradation exceeds a threshold, and the method reports $50\%$1 lower perplexity degradation and $50\%$2 better factual knowledge retention than structured baselines on Qwen3 models (Kodathala et al., 14 Jan 2026).

The field therefore treats perplexity-guided pruning less as a single algorithm than as a design space. Direct uses of perplexity remain prominent in data pruning, structural loss-based pruning, and stepwise chain-of-thought refinement (Marion et al., 2023, Wang et al., 20 Oct 2025, Cui et al., 18 Feb 2025). The surrounding literature shows, however, that the value of perplexity depends on the object being pruned, the distribution used for calibration, the capability one wishes to preserve, and whether probability-space fidelity is the true target. This suggests that modern pruning systems increasingly combine perplexity with task-specific objectives, richer uncertainty signals, or distribution-aware proxies rather than treating it as a universal scalar objective (Wang et al., 1 Dec 2025, Martra, 27 Dec 2025, He et al., 25 Mar 2026).