HFPrune: Entropy-Based MLP Pruning for LLMs

• HFPrune is a structured pruning method that evaluates neuron importance in Transformer MLP blocks through an entropy-based Taylor criterion.
• It prunes low-importance neurons layerwise, preserving the full output distribution rather than focusing solely on next-token prediction.
• Empirical results demonstrate that HFPrune, coupled with a short LoRA-based recovery stage, maintains or slightly improves zero-shot performance on LLaMA and Qwen models.

Searching arXiv for the primary HFPrune paper and closely related pruning references.

HFPrune is a structured pruning method for LLMs that targets hidden neurons in Transformer MLP blocks and defines their importance through the information entropy of the model’s own output distribution rather than the usual one-hot next-token cross-entropy. The method is presented in “High-Fidelity Pruning for LLMs” (Zhu et al., 9 Mar 2026). Its central premise is that pruning should preserve the original model’s global predictive distribution as faithfully as possible, because pretrained language-model behavior is encoded not only in the probability of the supervised next token but in the full vocabulary distribution. HFPrune operationalizes this premise with a first-order Taylor criterion computed on unlabeled calibration text, followed by structural removal of low-importance MLP neurons and a short LoRA-based recovery stage (Zhu et al., 9 Mar 2026).

1. Definition and problem formulation

HFPrune is designed for pruning LLMs whose deployment is constrained by computation and memory. The paper focuses specifically on pruning hidden neurons in the MLP modules of Transformer blocks rather than pruning attention heads, whole layers, or unstructured individual weights (Zhu et al., 9 Mar 2026). This makes HFPrune a structured neuron- or channel-pruning method applied to feed-forward blocks.

The method is motivated by a criticism of standard Taylor-based LLM pruning. In the usual formulation, if $\mathcal{L}$ is the pruning objective and neuron $h_i$ is ablated by setting it to zero, the loss change is approximated by the first-order Taylor term

$$\Delta \mathcal{L}_i \approx - \frac{\partial \mathcal{L}}{\partial h_i} h_i,$$

with importance score

$$\mathcal{I}(h_i) = \left| \frac{\partial \mathcal{L}}{\partial h_i} h_i \right|.$$

Prior work commonly instantiates $\mathcal{L}$ with one-hot next-token cross-entropy, which depends only on the probability assigned to the ground-truth next token $y$: $\mathcal{L}_{\mathrm{CE}}(x,y) = -\log p_y(x).$ HFPrune argues that this criterion is too narrow because it evaluates importance through a single coordinate of the full output distribution $P(x)=\{p_1(x),\dots,p_V(x)\}$, thereby encouraging preservation of target-token prediction rather than preservation of the model’s broader predictive behavior (Zhu et al., 9 Mar 2026).

The alternative proposed by HFPrune is to use the information entropy of the model’s own output distribution: $$\mathcal{C}_H(x) = - \sum_{j=1}^{V} p_j(x)\log_2 p_j(x).$$ This criterion is label-free, depends on all vocabulary probabilities, and does not require a separate teacher model. A plausible implication is that HFPrune can be interpreted as a lightweight surrogate for output-distribution fidelity rather than a direct distribution-matching method (Zhu et al., 9 Mar 2026).

2. Architectural scope and pruning mechanics

HFPrune prunes the MLP modules written in SwiGLU form as

$$\mathrm{MLP}(x)=\left(\mathrm{SiLU}(xW_{\mathrm{gate}})\odot (xW_{\mathrm{up}})\right)W_{\mathrm{down}},$$

where the hidden activation is

$h_i$0

The pruning units are the individual hidden dimensions $h_i$1 (Zhu et al., 9 Mar 2026).

Pruning is structural rather than mask-only. For each removed hidden neuron, HFPrune excises the corresponding rows in $h_i$2 and $h_i$3 and the corresponding columns in $h_i$4 (Zhu et al., 9 Mar 2026). The result is genuine width reduction of the MLP, which is important because the method is intended to produce deployable dense subnetworks rather than merely sparse masked models.

The paper states two reasons for focusing on MLPs. First, MLPs account for most parameters in modern LLMs. Second, pruning attention heads is treated as a coarser intervention with higher risk of catastrophic degradation (Zhu et al., 9 Mar 2026). The method does not claim to solve global architecture compression across all module types; instead, it deliberately restricts pruning to the feed-forward sublayers.

HFPrune also does not use a global ranking across all layers. Pruning is applied uniformly layer by layer. For each MLP layer with hidden width $h_i$5, the method prunes

$h_i$6

neurons with the lowest scores, where $h_i$7 is the MLP pruning ratio for that layer (Zhu et al., 9 Mar 2026). The procedure is therefore local to each layer: compute per-neuron scores, rank them ascending, and remove the lowest-scoring fraction.

This layerwise design distinguishes HFPrune from methods that allocate sparsity globally or solve an explicit cross-layer resource-allocation problem. The paper does not introduce score normalization beyond averaging over the calibration dataset, nor does it present a global sparsity solver or adaptive per-layer sensitivity allocation (Zhu et al., 9 Mar 2026).

3. Entropy-based Taylor criterion

The defining feature of HFPrune is its importance score. For each calibration input $h_i$8, the method computes the gradient of the entropy criterion with respect to hidden activations by backpropagation: $h_i$9 It then accumulates the absolute activation-gradient product for each neuron and averages over a calibration dataset $$\Delta \mathcal{L}_i \approx - \frac{\partial \mathcal{L}}{\partial h_i} h_i,$$0: $$\Delta \mathcal{L}_i \approx - \frac{\partial \mathcal{L}}{\partial h_i} h_i,$$1 The paper notes that the notation is slightly inconsistent and that the left-hand side should denote a final score $$\Delta \mathcal{L}_i \approx - \frac{\partial \mathcal{L}}{\partial h_i} h_i,$$2 rather than $$\Delta \mathcal{L}_i \approx - \frac{\partial \mathcal{L}}{\partial h_i} h_i,$$3, but the intended meaning is the dataset-averaged neuron importance (Zhu et al., 9 Mar 2026).

Under this criterion, a neuron is considered important if ablating it would induce a large first-order perturbation to the entropy of the output distribution. The authors interpret this as indicating a large effect on the fidelity of the model’s predictive capabilities in a global sense (Zhu et al., 9 Mar 2026).

The contrast with cross-entropy-based Taylor pruning is central. Under cross-entropy, a neuron appears important when it strongly affects the probability of the single supervised target token. Under the entropy objective, a neuron appears important when it perturbs the overall confidence structure across the vocabulary. The paper describes this as a more “holistic” criterion, because every token probability contributes to the pruning gradient (Zhu et al., 9 Mar 2026).

The method is also contrasted with self-distillation-based pruning, especially SDMPrune. The paper identifies two drawbacks of self-distillation in this context. It requires teacher-model supervision and therefore additional computation and memory, and if a self-distillation loss is used directly at initialization, the teacher and student are initially identical, so the distillation loss is zero and provides no gradient for the initial Taylor importance estimate (Zhu et al., 9 Mar 2026). HFPrune is proposed precisely to avoid both issues.

This suggests that HFPrune occupies an intermediate position between standard CE-Taylor pruning and explicit teacher-student distribution matching: broader than the former, cheaper than the latter.

4. Pipeline, calibration, and recovery training

The end-to-end HFPrune pipeline begins with a pretrained dense model $$\Delta \mathcal{L}_i \approx - \frac{\partial \mathcal{L}}{\partial h_i} h_i,$$4 and an unlabeled calibration dataset $$\Delta \mathcal{L}_i \approx - \frac{\partial \mathcal{L}}{\partial h_i} h_i,$$5. The paper uses 43,128 sequences randomly sampled from C4, each truncated or cropped to length 1024 following Wanda (Zhu et al., 9 Mar 2026).

For each MLP module, HFPrune initializes a score accumulator for its hidden neurons. For each calibration sample, it performs a forward pass to record hidden activations, computes the output distribution and its entropy, performs a backward pass to obtain gradients with respect to hidden activations, and accumulates

$$\Delta \mathcal{L}_i \approx - \frac{\partial \mathcal{L}}{\partial h_i} h_i,$$6

for every neuron (Zhu et al., 9 Mar 2026). After averaging over the calibration dataset, the method ranks neurons within each layer and structurally removes the lowest-scoring fraction.

The pruning stage is followed by recovery training. The paper states that all main comparison results include a brief fine-tuning stage unless otherwise stated. This recovery uses the LaMini-instruction dataset, runs for 2 epochs, and uses LoRA rather than full dense finetuning. The optimizer is AdamW with BF16 precision and a cosine learning-rate schedule (Zhu et al., 9 Mar 2026).

The appendix reports model-specific settings. Larger 7B models use batch size 512 on $$\Delta \mathcal{L}_i \approx - \frac{\partial \mathcal{L}}{\partial h_i} h_i,$$7 A6000 GPUs with learning rate $$\Delta \mathcal{L}_i \approx - \frac{\partial \mathcal{L}}{\partial h_i} h_i,$$8, while smaller models use batch size 256 on $$\Delta \mathcal{L}_i \approx - \frac{\partial \mathcal{L}}{\partial h_i} h_i,$$9 RTX 4090 GPUs with learning rate $$\mathcal{I}(h_i) = \left| \frac{\partial \mathcal{L}}{\partial h_i} h_i \right|.$$0. Weight decay is 0, $$\mathcal{I}(h_i) = \left| \frac{\partial \mathcal{L}}{\partial h_i} h_i \right|.$$1, $$\mathcal{I}(h_i) = \left| \frac{\partial \mathcal{L}}{\partial h_i} h_i \right|.$$2, and “Pack” is enabled in all listed settings (Zhu et al., 9 Mar 2026). The paper does not specify LoRA rank, alpha, or target modules in the provided description.

The computational profile is deliberately close to ordinary first-order Taylor pruning. HFPrune requires one forward pass and one backward pass per calibration sample, and the extra cost of computing entropy instead of cross-entropy is described as negligible relative to the full language-model pass (Zhu et al., 9 Mar 2026). Its main efficiency claim is relative to self-distillation approaches, not relative to CE-based Taylor pruning.

5. Empirical performance and output-fidelity evidence

HFPrune is evaluated on LLaMA and Qwen model families: LLaMA-2-7B, LLaMA-3.2-3.2B, LLaMA-3.2-1.2B, Qwen2.5-7B, Qwen2.5-1.5B, and Qwen3-1.7B (Zhu et al., 9 Mar 2026). The primary sparsity settings are 20% and 30% parameter/FLOP reduction. Evaluation uses zero-shot lm_eval on ten benchmarks: ARC-easy, ARC-challenge, BoolQ, Crows-Pairs, OpenBookQA, PIQA, Race, SocialIQA, TruthfulQA, and Winogrande (Zhu et al., 9 Mar 2026).

The main reported metric is average zero-shot accuracy across these benchmarks. On LLaMA-2-7B with post-pruning LoRA finetuning, the dense model averages 58.3. At 20% pruning, HFPrune reaches 59.0, compared with 58.2 for SDMPrune, and exceeds the dense model by 0.7 points. At 30% pruning, HFPrune reaches 56.3 versus 55.6 for SDMPrune (Zhu et al., 9 Mar 2026). This 20% LLaMA-2-7B result is the paper’s main headline because it combines compression with full recovery and slight improvement over the dense baseline after brief finetuning.

On LLaMA-3.2-3.2B, dense accuracy is 54.30; HFPrune obtains 54.07 at 20% pruning and 52.28 at 30%, compared with 53.37 and 51.03 for SDMPrune. On LLaMA-3.2-1.2B, dense is 51.47; HFPrune yields 50.77 and 48.59, compared with 48.94 and 46.40 for SDMPrune (Zhu et al., 9 Mar 2026).

The results on Qwen models are also consistently favorable. For Qwen2.5-7B, dense is 63.0; HFPrune scores 60.4 at 20% and 58.0 at 30%, compared with 59.7 and 55.3 for SDMPrune. For Qwen2.5-1.5B, dense is 56.3; HFPrune gets 54.6 and 51.7, compared with 51.1 and 47.4 for SDMPrune. For Qwen3-1.7B, dense is 54.4; HFPrune gets 54.3 at 20% and 51.7 at 30%, compared with 50.3 and 46.0 for SDMPrune (Zhu et al., 9 Mar 2026).

Two ablations are especially important for understanding the method rather than only the final tuned results. First, on LLaMA-2-7B without any post-pruning finetuning, the paper compares Taylor pruning using cross-entropy, self-distillation loss, and information entropy. At 20% pruning, average zero-shot accuracy is 52.6 for CE, 51.9 for SD, and 53.1 for IE; at 30%, it is 46.8 for CE, 45.2 for SD, and 47.3 for IE (Zhu et al., 9 Mar 2026). This supports the claim that the pruning masks themselves improve, not merely their downstream recoverability.

Second, the paper evaluates output-distribution fidelity directly on 5000 C4 prompts using Jensen–Shannon distance to the original output distribution and Top-15 Jaccard similarity of likely next tokens. At 20% pruning, CE gives JS distance 0.243 and Top-15 Jaccard 0.439, while IE gives 0.241 and 0.445. At 30%, CE gives 0.362 and 0.588, while IE gives 0.353 and 0.595 (Zhu et al., 9 Mar 2026). These gains are modest but systematic and are presented as the most direct evidence for the paper’s fidelity claim.

The paper also compares pruning different model parts and reports that pruning only MLP modules outperforms pruning both attention and MLP modules, both before and after tuning, with the gap widening at 30% pruning (Zhu et al., 9 Mar 2026). This reinforces the architectural restriction built into HFPrune.

6. Efficiency, relation to neighboring methods, and limitations

HFPrune is positioned against several structured pruning baselines, including LLM-Pruner, LoRAPrune, LoRAP, SDMPrune, and in smaller-model appendix tables, Compresso (Zhu et al., 9 Mar 2026). It is not directly benchmarked in the main tables against unstructured methods such as Wanda or SparseGPT, which the paper discusses but treats as a different class of pruning.

The efficiency comparison with SDMPrune is explicit. On 1000 C4 sequences of length 1024 using four A6000 GPUs, HFPrune on LLaMA2-7B takes 508.9s and 35.3GB peak memory, versus 1539.8s and 51.2GB for SDMPrune, which the paper summarizes as about $$\mathcal{I}(h_i) = \left| \frac{\partial \mathcal{L}}{\partial h_i} h_i \right|.$$3 faster and with 31% less memory (Zhu et al., 9 Mar 2026). Similar trends are reported for LLaMA3.2-1.2B and 3.2B.

Within the broader pruning literature, HFPrune belongs to first-order structured pruning rather than second-order curvature-based methods. For example, “Resource Efficient Neural Networks Using Hessian Based Pruning” (Chong et al., 2023) uses Hessian-trace-based channel saliency, whereas HFPrune retains the first-order activation-gradient product and changes the pruning objective from cross-entropy to entropy. This suggests that HFPrune’s novelty lies in supervision design rather than in changing the order of approximation. Relative to preference-conditioned resource-allocation frameworks such as HiPP-Prune for VLMs (Bai et al., 6 Mar 2026), HFPrune is much narrower: it does not optimize layer allocation with a learned policy and does not treat multi-objective trade-offs as first-class decision variables.

The paper also makes clear that entropy is only a surrogate for distribution preservation. It does not provide a theorem proving that minimizing first-order entropy change is the best proxy for fidelity, and it acknowledges that entropy is a scalar summary: two distributions can have similar entropy while differing substantially (Zhu et al., 9 Mar 2026). HFPrune should therefore be understood as an efficient proxy for preserving the shape and confidence structure of the output distribution, not as exact distribution matching.

Other limitations are similarly explicit. The method still depends on first-order Taylor approximation, which can become less accurate at aggressive pruning levels. Pruning is layerwise uniform rather than globally optimized. The strongest results include post-pruning LoRA finetuning, so best performance assumes some recovery budget. The evidence is also confined to the reported benchmark suite and model families (Zhu et al., 9 Mar 2026).

A further practical limitation is that the reported “20%” or “30%” pruning in the tables refers to total parameter/FLOP reduction, not directly to the hidden-neuron keep ratio within MLPs. The appendix notes, for example, that on LLaMA2-7B, “20%” parameter pruning corresponds to $$\mathcal{I}(h_i) = \left| \frac{\partial \mathcal{L}}{\partial h_i} h_i \right|.$$4, and “30%” corresponds to $$\mathcal{I}(h_i) = \left| \frac{\partial \mathcal{L}}{\partial h_i} h_i \right|.$$5 (Zhu et al., 9 Mar 2026). This is a notable reproduction detail because it means the actual MLP-width reduction is model-dependent.

Taken together, HFPrune is best characterized as an entropy-guided, first-order, structured MLP-neuron pruning method for LLMs that seeks to preserve output-distribution fidelity more effectively than standard CE-based Taylor pruning while remaining substantially cheaper than self-distillation-based alternatives (Zhu et al., 9 Mar 2026). Its technical distinctiveness lies less in how Taylor saliency is computed than in what scalar objective is differentiated to define importance.