SymWanda: Symmetric Pruning for LLMs

• SymWanda is a symmetric post-training pruning method for LLMs that defines a balanced reconstruction objective accounting for both input activations and output influence.
• It unifies existing heuristics such as Wanda, OWanda, and RIA by deriving a two-sided sensitivity score based on per-weight contributions from both sides of the model.
• Empirical results show that SymWanda and its variants improve robustness under high sparsity levels, offering efficient, training-free compression compared to traditional magnitude pruning.

Searching arXiv for SymWanda and the underlying pruning paper to ground the article in current sources. SymWanda is a symmetric post-training pruning method for LLMs situated within the broader study of communication-efficient compression. In the dissertation “Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization” (Yi, 10 Sep 2025), SymWanda appears as Chapter 6 on “symmetric post-training pruning” and is described as a framework that introduces a symmetric reconstruction objective for pruning pretrained dense transformer models without retraining. The chapter states that SymWanda simultaneously accounts for input activations and output influence of weights, and that Wanda and RIA arise as special cases of this symmetric formulation (Yi, 10 Sep 2025).

1. Position within post-training pruning

SymWanda is presented as a post-training compression method rather than as a federated learning algorithm. The chapter explicitly distinguishes it from earlier parts of the dissertation concerned with EF-BV, Scafflix, FedP3, and Cohort-Squeeze, noting that SymWanda is not an FL algorithm per se but a post-training compression method for LLMs (Yi, 10 Sep 2025). Its immediate context is unstructured pruning of pretrained transformer layers without gradient-based retraining.

The method is motivated by the empirical success of prior post-training pruning (PTP) methods such as SparseGPT, Wanda, and RIA. According to the chapter description, these methods work well in practice but lacked a clean theoretical formulation, and SymWanda is introduced to provide such a formulation through a symmetric objective that treats the left and right roles of a weight matrix in a balanced way (Yi, 10 Sep 2025).

The dissertation states that the SymWanda chapter is based on “Symmetric Pruning for LLMs” (Yi et al., 31 Jan 2025), attributed to Yi and Richtárik. This establishes SymWanda as both a concrete pruning method and a unifying theoretical framework for existing LLM pruning heuristics (Yi, 10 Sep 2025).

2. Symmetric reconstruction objective

For a single linear layer with input activations $\mathbf{X}\in\mathbb{R}^{a\times b}$ and weights $\mathbf{W}\in\mathbb{R}^{b\times c}$, the generic input-side reconstruction objective for post-training pruning is given as

$$f(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F^2,$$

under the sparsity constraint

$$\mathrm{Mem}(\widetilde{\mathbf{W}}) \;\le\; (1-\epsilon)\,\mathrm{Mem}(\mathbf{W}),$$

where $\epsilon\in[0,1)$ is the target sparsity ratio (Yi, 10 Sep 2025).

SymWanda replaces this one-sided formulation with a symmetric objective:

$$g(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F \;+\; \lVert (\widetilde{\mathbf{W}} - \mathbf{W}) \mathbf{Y} \rVert_F,$$

with the same sparsity constraint (Yi, 10 Sep 2025). In this expression, $\mathbf{Y}\in\mathbb{R}^{c\times d}$ is a calibration matrix intended to encode how columns of $\mathbf{W}$ influence downstream outputs.

The chapter emphasizes that the first term is Wanda-like, because it weights pruning by input activations, while the second term is its symmetric counterpart on the output side (Yi, 10 Sep 2025). A crucial detail is that the theoretical development uses non-squared Frobenius norms in the symmetric objective. This design makes the layer representation explicitly two-sided: pruning quality is measured not only by the change in $\mathbf{X}\widetilde{\mathbf{W}}$ but also by the change induced after right multiplication by $\mathbf{Y}$.

A plausible implication is that SymWanda recasts post-training pruning as a bidirectional sensitivity problem rather than as a purely activation-weighted approximation problem. The dissertation directly supports this interpretation by stating that the method “simultaneously accounts for input activations and output ‘influence’ of weights” (Yi, 10 Sep 2025).

3. Single-weight sensitivity and pruning score

The core pruning rule is derived by considering the removal of a single weight $\mathbf{W}\in\mathbb{R}^{b\times c}$0:

$\mathbf{W}\in\mathbb{R}^{b\times c}$1

Lemma 6.1, as summarized in the dissertation, states that the resulting increase in the symmetric objective is

$\mathbf{W}\in\mathbb{R}^{b\times c}$2

where $\mathbf{W}\in\mathbb{R}^{b\times c}$3 is column $\mathbf{W}\in\mathbb{R}^{b\times c}$4 of $\mathbf{W}\in\mathbb{R}^{b\times c}$5 and $\mathbf{W}\in\mathbb{R}^{b\times c}$6 is row $\mathbf{W}\in\mathbb{R}^{b\times c}$7 of $\mathbf{W}\in\mathbb{R}^{b\times c}$8 (Yi, 10 Sep 2025).

This yields the SymWanda pruning score

$\mathbf{W}\in\mathbb{R}^{b\times c}$9

Weights with small values of $$f(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F^2,$$0 are pruned first (Yi, 10 Sep 2025). The chapter identifies this as the core symmetric pruning criterion.

The significance of this expression is structural. Each scalar weight is evaluated at the intersection of an input-side factor and an output-side factor. The score is not just a function of local magnitude or a single activation statistic; it is the sum of left and right sensitivities, both modulated by $$f(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F^2,$$1. The dissertation explicitly characterizes this as a symmetric treatment of the layer’s role from both sides (Yi, 10 Sep 2025).

Algorithmically, the general pattern is stated clearly: compute per-element scores $$f(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F^2,$$2 using a chosen symmetric variant, then prune the smallest proportion according to the target sparsity (Yi, 10 Sep 2025). Although no full algorithm block is provided, the score-threshold-mask workflow is explicitly described.

4. Special cases and induced variants

A central contribution of SymWanda is its unification of prior pruning rules as special cases of the symmetric objective.

If $$f(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F^2,$$3, the score reduces to

$$f(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F^2,$$4

which the dissertation identifies exactly as the Wanda score (Yi, 10 Sep 2025). If $$f(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F^2,$$5, one obtains

$$f(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F^2,$$6

which is named OWanda, for “output Wanda” (Yi, 10 Sep 2025).

The chapter also studies a purely weight-based symmetric instantiation by setting

$$f(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F^2,$$7

In that case,

$$f(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F^2,$$8

so each weight is scaled by the sum of its row and column $$f(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F^2,$$9 norms (Yi, 10 Sep 2025). The text describes this as a symmetric treatment of row and column structure.

The relation to RIA is developed through Relative Importance (RI), defined as

$$\mathrm{Mem}(\widetilde{\mathbf{W}}) \;\le\; (1-\epsilon)\,\mathrm{Mem}(\mathbf{W}),$$0

Yi and Richtárik show, according to the dissertation summary, that RI can itself be realized as a SymWanda objective via specific synthetic choices of $$\mathrm{Mem}(\widetilde{\mathbf{W}}) \;\le\; (1-\epsilon)\,\mathrm{Mem}(\mathbf{W}),$$1 and $$\mathrm{Mem}(\widetilde{\mathbf{W}}) \;\le\; (1-\epsilon)\,\mathrm{Mem}(\mathbf{W}),$$2 (Yi, 10 Sep 2025). RIA then augments RI by an activation factor,

$$\mathrm{Mem}(\widetilde{\mathbf{W}}) \;\le\; (1-\epsilon)\,\mathrm{Mem}(\mathbf{W}),$$3

where $$\mathrm{Mem}(\widetilde{\mathbf{W}}) \;\le\; (1-\epsilon)\,\mathrm{Mem}(\mathbf{W}),$$4 is an activation matrix and $$\mathrm{Mem}(\widetilde{\mathbf{W}}) \;\le\; (1-\epsilon)\,\mathrm{Mem}(\mathbf{W}),$$5 controls the strength of activation weighting (Yi, 10 Sep 2025).

The chapter extends the framework further through general diagonal preconditioning, generalized $$\mathrm{Mem}(\widetilde{\mathbf{W}}) \;\le\; (1-\epsilon)\,\mathrm{Mem}(\mathbf{W}),$$6-norm symmetric scores, and stochastic RI. The generalized $$\mathrm{Mem}(\widetilde{\mathbf{W}}) \;\le\; (1-\epsilon)\,\mathrm{Mem}(\mathbf{W}),$$7 version is written as

$$\mathrm{Mem}(\widetilde{\mathbf{W}}) \;\le\; (1-\epsilon)\,\mathrm{Mem}(\mathbf{W}),$$8

and the appendix experiments reportedly show that $$\mathrm{Mem}(\widetilde{\mathbf{W}}) \;\le\; (1-\epsilon)\,\mathrm{Mem}(\mathbf{W}),$$9 is empirically best, aligning with RIA (Yi, 10 Sep 2025). The stochastic variant, StochRIA, samples subsets of row and column entries and defines

$\epsilon\in[0,1)$0

with reduced computational cost (Yi, 10 Sep 2025).

These constructions establish SymWanda not merely as a single heuristic but as a family of symmetric pruning rules parameterized by the choice of calibration operators on the left and right.

5. Post-training operation and training-free refinement

SymWanda is explicitly a post-training method. The dissertation defines post-training pruning in this setting as beginning with a fully pretrained dense LLM, gathering a small calibration set, computing activations and any auxiliary statistics, then computing pruning scores and zeroing weights without gradient-based retraining (Yi, 10 Sep 2025).

The calibration protocol reported in the experiments uses 128 sequences from C4, each with 2048 tokens (Yi, 10 Sep 2025). The chapter emphasizes that the main pruning stage is one-shot: there is no gradient-based fine-tuning after pruning, and calibration consists only of forward passes.

In addition to basic pruning, the chapter introduces a training-free fine-tuning procedure called $\epsilon\in[0,1)$1-DSnoT, described as Dynamic Sparse No-Training style mask adaptation without backpropagation (Yi, 10 Sep 2025). For a given row $\epsilon\in[0,1)$2, with reconstruction error $\epsilon\in[0,1)$3 and regularization parameters $\epsilon\in[0,1)$4, the relative-importance term is

$\epsilon\in[0,1)$5

The growth index is chosen as

$\epsilon\in[0,1)$6

and the pruning index as

$\epsilon\in[0,1)$7

where

$\epsilon\in[0,1)$8

This procedure grows one weight and prunes another while keeping the number of nonzeros fixed (Yi, 10 Sep 2025).

The chapter stresses that this still preserves the no-retraining property: the sparse values are not updated by SGD, and only the support changes through mask adaptation based on forward statistics and algebraic criteria (Yi, 10 Sep 2025).

6. Empirical behavior and reported results

The experiments reported in the dissertation use LLaMA2-7B, LLaMA2-13B, LLaMA3-8B, and OPT-1.3B, with evaluation by perplexity on WikiText-2 and zero-shot accuracy on BoolQ, RTE, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, and OpenBookQA (Yi, 10 Sep 2025). Unstructured sparsity levels of 50%, 60%, and 70% are studied, as are structured $\epsilon\in[0,1)$9 patterns 2:4 and 4:8 (Yi, 10 Sep 2025).

At dense baseline, the reported WikiText-2 perplexities are 5.47 for LLaMA2-7B, 4.88 for LLaMA2-13B, 6.14 for LLaMA3-8B, and 14.62 for OPT-1.3B (Yi, 10 Sep 2025). At 50% sparsity, magnitude pruning degrades sharply, with values including 16.03 for LLaMA2-7B, 205.44 for LLaMA3-8B, and 1712.39 for OPT-1.3B (Yi, 10 Sep 2025). Wanda improves substantially, giving 7.79, 6.28, 10.81, and 22.19 for LLaMA2-7B, LLaMA2-13B, LLaMA3-8B, and OPT-1.3B respectively (Yi, 10 Sep 2025). RIA further improves these results to 6.88, 5.95, 9.44, and 18.94 (Yi, 10 Sep 2025). StochRIA at $$g(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F \;+\; \lVert (\widetilde{\mathbf{W}} - \mathbf{W}) \mathbf{Y} \rVert_F,$$0 yields 6.91 $$g(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F \;+\; \lVert (\widetilde{\mathbf{W}} - \mathbf{W}) \mathbf{Y} \rVert_F,$$1 0.0032, 5.95 $$g(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F \;+\; \lVert (\widetilde{\mathbf{W}} - \mathbf{W}) \mathbf{Y} \rVert_F,$$2 0.0033, 9.46 $$g(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F \;+\; \lVert (\widetilde{\mathbf{W}} - \mathbf{W}) \mathbf{Y} \rVert_F,$$3 0.025, and 18.78 $$g(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F \;+\; \lVert (\widetilde{\mathbf{W}} - \mathbf{W}) \mathbf{Y} \rVert_F,$$4 0.05 across the same models (Yi, 10 Sep 2025).

The chapter interprets these results as showing that RIA and StochRIA significantly outperform Wanda and magnitude pruning, and that StochRIA matches or slightly improves over RIA in some cases while using far fewer samples for RI estimation (Yi, 10 Sep 2025). At structured 2:4 and 4:8 sparsity, the differences between RIA and StochRIA are described as small, with StochRIA sometimes improving over RIA on OPT-1.3B (Yi, 10 Sep 2025).

The activation exponent $$g(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F \;+\; \lVert (\widetilde{\mathbf{W}} - \mathbf{W}) \mathbf{Y} \rVert_F,$$5 is also ablated over $$g(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F \;+\; \lVert (\widetilde{\mathbf{W}} - \mathbf{W}) \mathbf{Y} \rVert_F,$$6. The dissertation states that $$g(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F \;+\; \lVert (\widetilde{\mathbf{W}} - \mathbf{W}) \mathbf{Y} \rVert_F,$$7, corresponding to no activation scaling, performs consistently worse, while $$g(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F \;+\; \lVert (\widetilde{\mathbf{W}} - \mathbf{W}) \mathbf{Y} \rVert_F,$$8 is often best and $$g(\widetilde{\mathbf{W}}) \;=\; \lVert \mathbf{X}(\widetilde{\mathbf{W}} - \mathbf{W}) \rVert_F \;+\; \lVert (\widetilde{\mathbf{W}} - \mathbf{W}) \mathbf{Y} \rVert_F,$$9 and $\mathbf{Y}\in\mathbb{R}^{c\times d}$0 degrade perplexity (Yi, 10 Sep 2025). This is presented as support for square-rooted activation norms.

At higher sparsities, the reported deterioration of Wanda is pronounced. At 60% sparsity, Wanda on LLaMA2-7B reaches 15.30 perplexity, whereas RIA and StochRIA remain in the range 10.39–10.62 (Yi, 10 Sep 2025). At 70% sparsity, Wanda can become catastrophic, with cited values of 214.93 on LLaMA2-7B and 412.9 on LLaMA3-8B, while RIA and StochRIA still produce usable, though degraded, perplexities (Yi, 10 Sep 2025). The dissertation explicitly concludes from these comparisons that symmetric RI/RIA variants are more robust at high sparsity than input-only Wanda or pure magnitude pruning.

For training-free refinement at 60% sparsity, the chapter reports that pure magnitude pruning gives 6900 perplexity on LLaMA2-7B and $\mathbf{Y}\in\mathbb{R}^{c\times d}$1 on LLaMA3-8B; DSnoT improves these to 4100 and $\mathbf{Y}\in\mathbb{R}^{c\times d}$2; and $\mathbf{Y}\in\mathbb{R}^{c\times d}$3-DSnoT further improves them to $\mathbf{Y}\in\mathbb{R}^{c\times d}$4 and $\mathbf{Y}\in\mathbb{R}^{c\times d}$5 respectively (Yi, 10 Sep 2025). For Wanda and RIA bases, the same refinement yields smaller but consistent improvements. Zero-shot classification over seven tasks at 60% sparsity is also said to improve in mean accuracy relative to DSnoT for both LLaMA2-7B and LLaMA3-8B (Yi, 10 Sep 2025).

7. Scope, limitations, and relation to distributed learning

The dissertation explicitly states that the SymWanda chapter does not define any federated learning or distributed protocol that uses SymWanda (Yi, 10 Sep 2025). Although the broader dissertation theme is communication efficiency in distributed and federated learning, SymWanda itself is presented purely as a centralized post-training pruning and training-free fine-tuning method.

The chapter description identifies several limitations or assumptions. Experiments are all on transformer LLMs, not on vision models or small models; the method is evaluated in centralized settings only; calibration quality depends on the representativeness of the C4 slices; and performance still degrades at very high sparsities, even if symmetric variants degrade less severely than Wanda (Yi, 10 Sep 2025). It is also stated that there are no formal convergence or generalization guarantees for the $\mathbf{Y}\in\mathbb{R}^{c\times d}$6-DSnoT mask-adaptation procedure (Yi, 10 Sep 2025).

Within the dissertation’s global framing, SymWanda can be viewed as a compression primitive that could in principle support communication reduction, but this is not formalized in the chapter. The text notes only that one might imagine it being plugged into federated pruning or sparse communication systems; no explicit protocol, mask-synchronization mechanism, or client-server exchange rule is provided (Yi, 10 Sep 2025). This suggests that SymWanda’s primary significance lies in post-training model compression theory and practice rather than in distributed optimization methodology.

Taken together, the dissertation presents SymWanda as a principled symmetric framework for LLM pruning: it derives an exact single-weight sensitivity under a two-sided reconstruction objective, recovers Wanda, OWanda, RI, RIA, generalized $\mathbf{Y}\in\mathbb{R}^{c\times d}$7 variants, and stochastic RI constructions as special cases, and shows empirically that symmetric variants are substantially more robust than magnitude pruning and often better than Wanda, especially in the 50–70% sparsity regime (Yi, 10 Sep 2025).