---
title: 'Pruning-at-Initialization: Techniques & Insights'
url: https://www.emergentmind.com/topics/pruning-at-initialization
type: topic
---

# Pruning-at-Initialization: Techniques & Insights

Searching arXiv for recent and foundational papers on pruning-at-initialization.
Pruning-at-initialization (PaI) denotes a family of sparsification procedures that select a sparse topology before training, or at the earliest stages of training, by applying a binary mask to randomly initialized parameters and then training only the surviving subnetwork. In the generic formulation, parameters $w \in \mathbb{R}^d$ are paired with a binary mask $m \in \{0,1\}^d$, yielding pruned weights $m \odot w$ under a sparsity constraint such as $\|m\|_0 \le k$ [2103.06460]. Relative to pruning-after-training, PaI is motivated by the possibility of eliminating dense pretraining, reducing training-time memory and compute, and discovering trainable sparse subnetworks directly at or near initialization; however, its empirical behavior is shaped by strong interactions among saliency criteria, signal propagation, architecture, and the degree to which the mask encodes information about the data [2103.06460][2402.01089].

## 1. Formalization and taxonomic structure

PaI is commonly expressed as the constrained optimization problem
$$
\min_{m \in \{0,1\}^d}\ \mathcal{L}(m \odot w_0;\ \mathcal{D})\quad \text{s.t.}\quad \|m\|_0 \le k,
$$
where $w_0$ is a random initialization and $\mathcal{D}$ is the training set [2103.06460]. A related notation uses a connectivity vector $c$ or $m$ to indicate which parameters remain trainable, with pruned weights written as $\tilde{W}=M\odot W$ and sparsity defined through the nonzero count of the mask [2103.06460]. In this framing, PaI differs from pruning-after-training because mask selection occurs at initialization or after only minimal warmup rather than after convergence.

A standard survey divides PaI into two major tracks: **sparse selection** and **sparse training** [2103.06460]. Sparse selection seeks a subnetwork at or near initialization, sometimes without further weight training, as in supermask-style methods. Sparse training instead fixes or updates a sparse mask and trains the resulting sparse model from scratch. The same survey also places dynamic sparse training methods such as DeepR, SET, DSR, SNFS, and RigL within the broader PaI landscape because they maintain sparsity throughout training even though connectivity can change over time [2103.06460].

This taxonomy matters because different PaI objectives are not interchangeable. Some methods approximate loss preservation at initialization, some attempt to preserve gradient flow, some are explicitly data-free and target balanced signal propagation, and some optimize for hardware-aware structure rather than weight-level sparsity [2103.06460][2203.02549]. A plausible implication is that “pruning-at-initialization” is best understood as a design space rather than a single algorithmic doctrine.

## 2. Saliency criteria and representative algorithms

The canonical single-shot data-dependent criterion is SNIP, which scores parameters by initialization-time connection sensitivity:
$$
s_i = \left| \frac{\partial \mathcal{L}(w)}{\partial w_i} \cdot w_i \right| \Big|_{w=w_0}.
$$
Operationally, SNIP uses one forward-backward pass on a small batch, ranks parameters globally or per layer, and keeps the top-$k$ entries [2103.06460]. A closely related formulation writes the score as
$$
s_i^{\mathrm{SNIP}} = \left| \frac{\partial L(m \odot w; x, y)}{\partial m_i} \right|\Bigg|_{m=\mathbf{1}}
\approx |g_i\,w_i|,
$$
with the implemented score reported as $z_i = |g_i \, w_i|$ at initialization [2107.01808].

GraSP extends this perspective to second-order gradient preservation. In the survey formulation, if $g=\nabla_w L(w_0)$ and $H=\nabla_w^2L(w_0)$, GraSP uses the approximation
$$
s_i \approx -\left((Hg)_i \cdot w_{0,i}\right),
$$
with efficient evaluation through Hessian-vector products [2103.06460]. The ablation paper on PaI randomization studies the absolute-value version, denoted GraspAbs, with score
$$
s_i^{\mathrm{GraSPAbs}} = |w_i\,h_i|,
$$
where $h = Hg$ in layerwise form [2107.01808].

SynFlow is the principal data-free alternative. It replaces task data by an all-ones input, forces weights positive, and computes a saliency based on a synthetic scalar objective:
$$
R = \sum_c \mathrm{logit}_c(\mathbf{1}; |w|), \qquad
s_i^{\mathrm{SynFlow}} = \left| \frac{\partial R}{\partial w_i} \cdot w_i \right|.
$$
The survey also gives the equivalent product form
$$
\mathcal{R} = \mathbf{1}^\top \left( \Pi_{l=1}^{L} |\mathbf{W}^{[l]}| \right) \mathbf{1},
$$
with iterative pruning used to avoid layer collapse [2107.01808][2103.06460].

Magnitude pruning at initialization retains the largest $|w_i|$ and is the simplest data-free baseline, but multiple sources emphasize that raw magnitude is usually weaker at initialization than after training [2103.06460][2107.01808]. More specialized proposals extend these core ideas. The abstract of “What to Prune and What Not to Prune at Initialization” reports two approaches: **K-starts**, which begins with $k$ random $p$-sparse matrices and selects the “fittest” one using weight magnitude, gradient accumulation, or both; and **Dissipating gradients**, which removes weights that remain within a fraction of their initial value during the first couple of epochs [2209.02201]. The same abstract states that combining dissipating gradients and kstarts “outperforms either methods and random dropout consistently” [2209.02201].

A more recent line learns PaI criteria from iterative pruning. “Learning effective pruning at initialization from iterative pruning” proposes **AutoSparse**, an end-to-end scorer trained from Iterative Rewind Pruning survival labels so that initialization features predict which parameters would survive strong iterative pruning [2408.14757]. This suggests a hybrid view in which iterative pruning acts as a teacher for one-shot PaI.

## 3. Signal propagation, initialization, and trainability

A recurrent theme in PaI is that sparsity selection cannot be separated from initialization dynamics. “A Signal Propagation Perspective for Pruning Neural Networks at Initialization” interprets connection sensitivity as a gradient-based quantity and argues that reliable saliency requires well-conditioned forward and backward propagation at initialization. For a feedforward network with Jacobian
$$
J^{0,K} = \frac{\partial x^K}{\partial x^0} = \prod_{l=1}^{K} D^l W^l,
$$
the paper identifies layerwise dynamical isometry as a sufficient condition under which gradient magnitudes remain comparable across layers and SNIP-style saliency is trustworthy [1906.06307]. It further proposes a data-free approximate isometry objective for the pruned network,
$$
\min_{W^l} \left\| \big( C^l \odot W^l \big)^\top \big( C^l \odot W^l \big) - I \right\|_F,
$$
to restore trainability after pruning [1906.06307].

A complementary mean-field treatment appears in “Robust Pruning at Initialization” [2002.08797]. There, sensitivity-based pruning is analyzed through the Edge of Chaos (EOC). The paper defines the ordered, chaotic, and EOC phases through $\chi = f'(1)$, where $c^{l+1}(x,x') = f(c^l(x,x'))$, and shows that sensitivity-based pruning is ill-conditioned away from EOC. Its central theorem states that in the ordered phase the expected critical sparsity before some layer is fully pruned obeys
$$
\mathbb{E}[s_{cr}] \le \frac{1}{L} \left(1 + \frac{\log(\kappa L N^2)}{\kappa}\right) + O\!\left(\frac{1}{\kappa^2 \sqrt{L N^2}}\right),
$$
with $\kappa = |\log \chi|/8$ [2002.08797]. The same work proposes a **Rescaling Trick** that restores EOC after sensitivity-based pruning by rescaling retained weights layerwise, and introduces **Stable ResNet** parameterization, in which residual branches are scaled by $1/\sqrt{L}$ so that saliency moments remain controlled at large depth [2002.08797].

These analyses clarify why PaI often fails at extreme sparsity even when winning tickets exist. The issue is not only which weights are removed, but whether the resulting sparse network preserves balanced Jacobians, non-collapsing layers, and viable gradient flow immediately after pruning.

## 4. Layerwise structure, randomization, and the shift from unstructured to structured PaI

An important empirical result is that many PaI methods are unexpectedly insensitive to the exact identities of retained weights. “Why is Pruning at Initialization Immune to Reinitializing and Shuffling?” studies Magnitude, SNIP, SynFlow, and GraspAbs under two perturbations: **Reinit**, which resamples surviving weights from the original initialization distribution, and **Layerwise Shuffling**, which permutes mask positions within each layer while preserving per-layer counts [2107.01808]. Across LeNet-300-100 on MNIST and ResNet-20 on CIFAR-10, the paper reports small Wasserstein distances between the distributions of unpruned weights under the original and perturbed masks, whereas random pruning yields much larger distances. For example, on LeNet-MNIST the average Wasserstein distance for SNIP is $0.0084$ under Reinit and $0.0083$ under Layerwise Shuffling, versus $0.0437$ for Random Pruning; on ResNet-CIFAR, SynFlow yields $0.0128$, $0.0146$, and $0.0489$, respectively [2107.01808]. The paper’s interpretation is that layerwise sparsity budgets and distributional properties dominate exact connection identity.

This observation underlies the structured PaI argument advanced by “Structured Pruning is All You Need for Pruning CNNs at Initialization” [2203.02549]. That work introduces **Synaptic Expectation** and proves a granularity-invariance theorem stating that the expected SynFlow score depends only on layerwise densities:
$$
\mathbb{E}_{M,W}[\mathcal{S}_{\mathrm{SF}}] = N\, C_{N+1} \prod_{l=1}^N \big(p_l\, C_l \cdot \mathbb{E}_{x \sim \mathcal{D}^l}[|x|]\big),
$$
so expected accuracy is independent of whether pruning is by weights, filters, or channels provided layerwise densities $p_l$ are the same [2203.02549]. Based on this, the paper proposes **PreCropping**, a channel-level structured PaI method that optimizes only layerwise densities under parameter and FLOP budgets. The parameter-only optimization has the closed-form KKT solution
$$
p_l = \min\Big(\frac{\mu}{\alpha_l},\, 1\Big),
$$
while the joint parameter-plus-FLOP formulation gives
$$
p_l = \min\Big(\frac{1}{\mu_1 \alpha_l + \mu_2 \beta_l},\, 1\Big),
$$
with $\alpha_l$ and $\beta_l$ denoting parameter and FLOP costs [2203.02549].

The same paper reports that on ImageNet, structured PaI can match or outperform SynFlow under matched budgets. For MobileNetV2, PreCrop improves over SynFlow by up to $+2.7\%$ top-1 at matched FLOPs/parameters; for EfficientNet-B0 it improves by $+0.9\%$ [2203.02549]. This suggests that, at least for CNNs, the central design variable may be layerwise density allocation rather than fine-grained unstructured masks.

Layer-level pruning extends this logic further. “When Layers Play the Lottery, all Tickets Win at Initialization” studies residual block removal rather than weight or filter pruning. For residual architectures with
$$
y_i = f_i(y_{i-1}) + y_{i-1},
$$
entire blocks can be removed while the identity path preserves shape compatibility [2301.10835]. On ResNet-32 for CIFAR-10, GraSP-based layer pruning at initialization produces winning tickets for all tested pruning densities, with gains such as $+0.40$ percentage points at $p=1$ removed block and $+0.72$ at $p=2$ [2301.10835].

## 5. Application-specific extensions and systems

PaI has increasingly been specialized to domains where training-time resource savings matter as much as model size. In 3D CNNs, **RANP** performs neuron pruning at initialization using loss sensitivity reweighted by resource costs [2010.02488]. For neuron $u$ in layer $l$, the default importance is the sum of absolute mask-gradient magnitudes over incoming channels,
$$
s^l_u = \sum_{v=1}^{N_{l-1}} |g^l_{uv}|,
$$
followed by layer-mean balancing and resource-aware reweighting
$$
\hat{s}^l_u
=
\left(1 + \lambda\, \frac{e^{-\tau_l}}{\sum_{k=1}^{K} e^{-\tau_k}}\right)\tilde{s}^l_u,
$$
where $\tau_l$ is a FLOP- or memory-based layer cost [2010.02488]. Reported reductions include roughly $50$–$95\%$ in FLOPs and $35$–$80\%$ in memory, with negligible loss in accuracy across 3D-UNet, MobileNetV2, and I3D tasks [2010.02488].

In federated learning, **FedPaI** uses PaI to fix sparse connectivity early and thereby reduce both communication and computation overhead. Its abstract reports support for both structured and unstructured pruning, personalized client-side pruning, sparsity-aware server-side aggregation, sparsity up to $98\%$ “without compromising the model accuracy compared to unpruned baselines,” and training acceleration by $6.4$ to $7.9$ times under non-IID settings [2504.00308]. The same abstract frames PaI as a natural fit for decentralized environments because sparsity patterns are fixed from the start [2504.00308].

For edge deployment, **Reconvene** combines unstructured PaI with structured postprocessing. It first applies unstructured PaI, then identifies convolution layers whose sparsity exceeds the model-wide average and converts those resilient sparse layers into smaller dense layers. Its layer test is
$$
\mathrm{PSE}(l)=
\begin{cases}
\text{sensitive} & \text{if } S_l < SAvg,\\
\text{resilient} & \text{otherwise},
\end{cases}
$$
with resilient layers assigned a reduced output-channel count $\lceil N_{\text{channels}} (1-S_l)\rceil$ [2404.16877]. The reported system creates pruned models “within seconds” and yields models up to $16.21\times$ smaller and $2\times$ faster while maintaining the same accuracy as an unstructured PaI counterpart [2404.16877].

PaI has also been adapted beyond classification. “Pruning Unrolled Networks (PUN) at Initialization for MRI Reconstruction Improves Generalization” prunes the denoiser component of unrolled MoDL architectures via direct optimization of a Bernoulli mask distribution at initialization:
$$
m^* = \underset{m \in \{0,1\}^d}{\arg \min} \sum_{i=1}^M \left\| \boldsymbol{F}_{\text{MoDL}}(x_i^{(0)}, y_i, \boldsymbol{\theta}_{\text{init}} \circ m) - x_i \right\|_2^2
\quad \text{s.t.} \quad \|m\|_0 \le s,
$$
relaxed through a Concrete/Gumbel-softmax mask parameterization [2412.18668]. On fastMRI knee, PUN-IT improves PSNR from $34.20$ to $34.69$ at $4\times$ acceleration and from $31.75$ to $32.72$ at $8\times$ acceleration relative to dense MoDL [2412.18668]. This suggests PaI can act as a robustness mechanism under distribution shift, not merely a compression tool.

## 6. Theory, limitations, and contemporary debates

The strongest recent criticism of PaI is information-theoretic. “No Free Prune: Information-Theoretic Barriers to Pruning at Initialization” argues that the effective capacity of a sparse network is not just the number of surviving parameters, but
$$
p_{\text{eff}} = \|W_{\text{final}}\|_0 + I(M; D),
$$
or, in the continuous setting,
$$
p_{\text{eff}} = I(\mathbf{m}^W; \mathcal{D}) + \mathbb{E}[\|\mathbf{m}\|_1] \log_2 \big(1 + 60 W J \epsilon^{-1}\big).
$$
The paper extends the Law of Robustness to sparse networks by replacing ordinary parameter count with $p_{\text{eff}}$, implying that robust interpolation of noisy data requires either many nonzero weights or a mask with substantial mutual information about the data [2402.01089]. Its central claim is that pruning during or after training naturally yields masks with higher $I(M;D)$ than masks chosen at initialization, which explains why lottery tickets can exist but be hard to find “fast” [2402.01089].

A related practical critique comes from rewinding results. “Stabilizing the Lottery Ticket Hypothesis” finds that on deeper networks and ImageNet-scale tasks, iterative magnitude pruning fails when weights are reset to iteration $0$, but succeeds when rewound to an early point such as epoch $4$–$10$ or the first $0.1\%$–$7\%$ of training [1903.01611]. For ResNet-50 on ImageNet, one-shot pruning with early rewind matches dense top-1 at $70\%$ pruned, and iterative pruning with rewind to epoch $6$ matches dense accuracy with only $20\%$ of weights remaining [1903.01611]. This suggests that the “initialization” relevant to sparse trainability may, in practice, be a narrow early-training window rather than the literal random draw at step zero.

At the same time, theory has also identified positive regimes. “Pruning Before Training May Improve Generalization, Provably” studies random PaI in overparameterized two-layer networks and proves that, if keep probability $p$ remains above a threshold $p_* = C_1 (\log d)/m$, gradient descent achieves low training loss and a generalization bound that improves as pruning increases:
$$
L_{\mathcal{D}}(\tilde{W}^{(T)}) \le O(K\epsilon) + \exp(-n^2/p).
$$
Below a smaller scale $p_{\text{bad}} = \Theta(1/(K m \log d))$, the network can still fit the training set by memorizing noise, with test performance no better than random guessing [2301.00335]. This result supports a nuanced view: moderate PaI can act as feature purification, whereas excessive PaI destroys signal pathways.

A different theoretical synthesis is offered by the path-based NTK analysis of “A Unified Paths Perspective for Pruning at Initialization” [2101.10552] and the graphon-limit framework of “Pruning at Initialisation through the lens of Graphon Limit” [2602.06675]. The former interprets SynFlow as equivalent to the $p=1$ path regularizer and defines a Path Kernel whose trace can be computed via
$$
\mathcal{R}_{\text{PK}} = \mathbbm{1}^{\top}\Big(\prod_{l=1}^{L}\theta_{l}^{2}\Big)\mathbbm{1},
$$
while the latter proves that under a Factorised Saliency Model, PaI masks converge to deterministic bipartite graphons
$$
W(u,v) := \mathbb{P}(\phi(u)\,Q_\psi(v)\,|\xi| > \tau),
$$
distinguishing homogeneous limits for random or magnitude pruning from heterogeneous, feature-selective limits for data-driven methods such as SNIP and GraSP [2101.10552][2602.06675]. These results suggest that PaI’s large-width behavior can be described either through path norms and NTK structure or through continuous graph-limit objects.

Taken together, the literature does not support a single verdict on PaI. It supports several narrower conclusions: PaI is computationally attractive; structured and resource-aware variants can be highly practical; moderate pruning can sometimes improve generalization; but at extreme sparsity, or on noisy high-dimensional tasks, initialization-time masks often lack the data dependence required to match pruning during or after training [2402.01089][2301.00335]. A plausible implication is that future progress will come less from one universal saliency rule than from task-specific combinations of initialization geometry, modest data dependence, and architecture-aware structure.

Source: https://www.emergentmind.com/topics/pruning-at-initialization