Wanda Pruning for LLM Efficiency

• Wanda pruning is a one-shot weight pruning method that uses data-dependent activation norms to determine saliency and prune least critical weights.
• It combines absolute weight values with calibration activations to minimize output distortion, ensuring superior perplexity and feature preservation compared to magnitude pruning.
• Empirical results on models like LLaMA and OPT demonstrate that Wanda achieves high sparsity with minimal performance loss, though fairness and deployment challenges remain.

Wanda pruning is a post-training, one-shot weight pruning methodology originally developed for LLMs, designed to achieve high sparsity with minimal performance degradation by calibrating pruning decisions with data-dependent activation statistics. Wanda achieves superior perplexity, feature preservation, and empirical task performance relative to magnitude pruning, while maintaining extreme computational efficiency and simplicity. Subsequent research has provided rigorous theoretical grounding, practical enhancements, generalizations, and critical analyses of its broader impact on fairness and deployment.

1. Mathematical Foundation and Algorithmic Definition

Wanda pruning ranks individual weights according to a saliency score that explicitly combines absolute weight magnitudes with the typical norm of their corresponding input activations. For a linear layer with weight matrix $W \in \mathbb{R}^{m \times n}$ and a (calibration) activation matrix $X \in \mathbb{R}^{p \times n}$ (rows are samples or tokens, columns are input features), the Wanda saliency for $W_{ij}$ is defined as:

$S_{ij} = |W_{ij}| \cdot \| X_{(:,j)} \|_2$

where $X_{(:,j)}$ is the $j$-th column of calibration activations and $\|\cdot\|_2$ denotes the Euclidean norm. Weights with the smallest $S_{ij}$ are pruned globally or under a structured constraint (e.g., N:M patterns). Calibration typically uses 128–300k tokens, but Wanda is robust even with much smaller batches (Sun et al., 2023, Borobia et al., 26 Mar 2026).

Theoretical analyses have demonstrated that, for zero-mean, independent Gaussian inputs and a bias-free linear layer, Wanda achieves the minimizer of expected output distortion induced by the removal of single weights (Mecke et al., 28 Mar 2025, Yi et al., 31 Jan 2025). When the input distribution violates these assumptions (e.g., non-centered activations or layer biases), alternative metrics such as STADE yield better performance, but Wanda remains optimal for the majority of transformer layers that are preceded by normalization (Mecke et al., 28 Mar 2025).

2. Practical Implementation and Computational Efficiency

The practical workflow proceeds as follows: (i) accumulate input activations $X$ from a small calibration set using a single forward pass, (ii) compute $\| X_{(:,j)} \|_2$ for all features, (iii) calculate $X \in \mathbb{R}^{p \times n}$0 and select weights for pruning (either globally or by grouping, e.g., per-output row, necessary for specific LLMs (Sun et al., 2023, Hu et al., 16 Jan 2025)), and (iv) mask (zero) the lowest scoring weights:

$X \in \mathbb{R}^{p \times n}$8 Wanda operates with $X \in \mathbb{R}^{p \times n}$1 complexity per matrix and is typically 200–300× faster than second-order mask selection methods such as SparseGPT (Sun et al., 2023, Wagner et al., 16 Oct 2025). It requires only a single forward pass and does not impose any retraining requirement, though post-pruning reconstruction or local retraining can further close the performance gap (Wagner et al., 16 Oct 2025).

3. Empirical Performance and Model Quality

Benchmarking on Llama, OPT, and Gemma model families consistently demonstrates that Wanda achieves far superior perplexity and zero-shot accuracy compared to magnitude pruning and rivals more computationally intensive methods:

Model	Sparsity	Dense PPL	Wanda PPL	Magnitude PPL	SparseGPT PPL
LLaMA-7B	50%	5.68	7.26	17.29	7.22
LLaMA-65B	50%	3.56	4.57	5.90	4.57

On structured N:M patterns (2:4, 4:8), Wanda closely tracks SparseGPT and, given calibration with 128 samples, produces masks that result in nearly lossless performance up to moderate sparsity levels (Sun et al., 2023, Mecke et al., 28 Mar 2025, Wagner et al., 16 Oct 2025).

Beyond aggregate metrics, Wanda pruning preserves the geometric angles between Sparse Autoencoder (SAE) features up to 3.7× better than magnitude pruning, and feature survival rates for rare, specialized features are much higher. Notably, this geometric survival does not directly predict causal feature importance (Borobia et al., 26 Mar 2026).

4. Structured Pruning, Post-Pruning Reconstruction, and Extensions

Wanda is extensible to structured pruning under both per-layer and paired-row/column paradigms. In interlinked layers (e.g., MLP $X \in \mathbb{R}^{p \times n}$2, or Llama's gated FFN), Wanda scores inform which columns of $X \in \mathbb{R}^{p \times n}$3 to prune; corresponding rows of $X \in \mathbb{R}^{p \times n}$4 may be simultaneously removed with no extra forward distortion, thus preserving computational correctness and maximizing pruning efficacy (Hu et al., 16 Jan 2025).

Post-pruning restoration, either in closed form or via local block-level retraining, further recovers accuracy. For example, one-shot linear regression over retained units ($X \in \mathbb{R}^{p \times n}$5) restores performance more efficiently than iterative ADMM or global retraining (Hu et al., 16 Jan 2025). Local block-wise reconstruction, especially when performed at attention or MLP granularity, generally outperforms both fine-grained matrix-level approaches and full-model retraining in both accuracy and computational cost (Wagner et al., 16 Oct 2025).

5. Generalization, Theoretical Extensions, and Variants

The underlying minimization objective for Wanda can be extended to a symmetric input–output calibration framework, where both the input and output distributions are referenced in the pruning score. This yields a general scoring function $X \in \mathbb{R}^{p \times n}$6, which subsumes Wanda and its output-oriented dual as special cases. Relative-Importance (RIA) and its stochastic variants further augment score allocation based on normalized row and column statistics, extending Wanda's optimality to broader use cases (Yi et al., 31 Jan 2025). The STADE metric ($X \in \mathbb{R}^{p \times n}$7) captures cases where nonzero-mean activations or biases prevail and is empirically preferable in such layers (Mecke et al., 28 Mar 2025).

Wanda++ introduces a block-local (or regional) gradient enhancement, combining the fast original Wanda score with a per-block regional gradient norm. Subsequent local regional output mismatch minimization further reduces perplexity, with minimal runtime overhead and strong empirical advantages at structured (2:4) sparsity (Yang et al., 6 Mar 2025).

For multilingual LLMs, the M-Wanda extension explicitly incorporates cross-lingual activation variance and dynamic per-layer sparsity based on cross-lingual stability, yielding better retention of non-English capabilities with minimal added complexity (Choenni et al., 27 May 2025). Wanda is directly extensible to state-space model LLMs by identical score computation within SSM blocks and has been shown to outperform structured SSM-aware methods at moderate sparsity (Ghattas et al., 26 Feb 2025).

6. Limitations, Fairness, and Deployment Risks

Despite Wanda’s strong performance on perplexity and language modeling metrics, recent controlled evaluations have documented a marked amplification of social bias. Specifically, Wanda produces the highest Stereotype Reliance Score among tested methods—an 83.7% relative increase at 50% sparsity for Mistral-7B—and up to 59% of previously unbiased items develop stereotypical behaviors at 70% sparsity (Rath et al., 2 May 2026). As Wanda’s saliency emphasizes frequently activated weights, low-activation, alignment-relevant pathways (e.g., for epistemic humility or bias avoidance) are more likely to be pruned. Furthermore, unstructured pruning with Wanda does not yield actual storage or inference speed gains on edge hardware, reducing its practical impact for deployment on resource-constrained devices (Rath et al., 2 May 2026).

7. Usage Recommendations, Open Questions, and Ongoing Developments

Wanda is well-suited for rapid one-shot pruning to 40–50% sparsity for LLMs of all sizes where minimal engineering effort and low calibration cost are essential. Calibration sets of 100–128 samples drawn from in-domain data suffice; both global and per-output (row-wise) sparsity regimes are supported, with the latter observed to be uniquely beneficial for transformer LLMs (Sun et al., 2023).

Structured extensions, post-pruning reconstruction, block-local alternatives (Wanda++), and multilingual-aware variants (M-Wanda) are available for more specialized scenarios (Hu et al., 16 Jan 2025, Yang et al., 6 Mar 2025, Choenni et al., 27 May 2025). For layers with non-centered activations or biases, practitioners are advised to use STADE instead (Mecke et al., 28 Mar 2025). For deployment in fairness-critical contexts, bias-aware post-pruning evaluation is necessary due to Wanda’s documented capacity to induce representational harm undetectable by perplexity alone (Rath et al., 2 May 2026).

Outstanding theoretical questions include the precise mechanistic explanation for the observed per-output grouping advantage and the manner in which Wanda-style metrics may be adapted to on-the-fly sparse training or dynamic adaptation (Sun et al., 2023). Ongoing research continues to refine Wanda-based strategies to unify input–output calibration, integrate weight update–free fine-tuning, and generalize across LLM architectures and deployment constraints (Yi et al., 31 Jan 2025, Wagner et al., 16 Oct 2025).