Pruning-at-Initialization: Techniques & Insights

• Pruning-at-Initialization is a sparsification method that applies binary masks to randomly initialized weights to select a sparse subnetwork prior to or early in training.
• It encompasses approaches like sparse selection and sparse training, where methods such as SNIP, GraSP, and SynFlow optimize gradient flow, signal propagation, and resource allocation.
• Research highlights that factors including saliency criteria, initialization dynamics, and structured mask design are critical for sustaining trainability and improving generalization at high sparsity levels.

Searching arXiv for recent and foundational papers on pruning-at-initialization. Pruning-at-initialization (PaI) denotes a family of sparsification procedures that select a sparse topology before training, or at the earliest stages of training, by applying a binary mask to randomly initialized parameters and then training only the surviving subnetwork. In the generic formulation, parameters $w \in \mathbb{R}^d$ are paired with a binary mask $m \in \{0,1\}^d$, yielding pruned weights $m \odot w$ under a sparsity constraint such as $\|m\|_0 \le k$ (Wang et al., 2021). Relative to pruning-after-training, PaI is motivated by the possibility of eliminating dense pretraining, reducing training-time memory and compute, and discovering trainable sparse subnetworks directly at or near initialization; however, its empirical behavior is shaped by strong interactions among saliency criteria, signal propagation, architecture, and the degree to which the mask encodes information about the data (Wang et al., 2021, Kumar et al., 2024).

1. Formalization and taxonomic structure

PaI is commonly expressed as the constrained optimization problem

$$\min_{m \in \{0,1\}^d}\ \mathcal{L}(m \odot w_0;\ \mathcal{D})\quad \text{s.t.}\quad \|m\|_0 \le k,$$

where $w_0$ is a random initialization and $\mathcal{D}$ is the training set (Wang et al., 2021). A related notation uses a connectivity vector $c$ or $m$ to indicate which parameters remain trainable, with pruned weights written as $\tilde{W}=M\odot W$ and sparsity defined through the nonzero count of the mask (Wang et al., 2021). In this framing, PaI differs from pruning-after-training because mask selection occurs at initialization or after only minimal warmup rather than after convergence.

A standard survey divides PaI into two major tracks: sparse selection and sparse training (Wang et al., 2021). Sparse selection seeks a subnetwork at or near initialization, sometimes without further weight training, as in supermask-style methods. Sparse training instead fixes or updates a sparse mask and trains the resulting sparse model from scratch. The same survey also places dynamic sparse training methods such as DeepR, SET, DSR, SNFS, and RigL within the broader PaI landscape because they maintain sparsity throughout training even though connectivity can change over time (Wang et al., 2021).

This taxonomy matters because different PaI objectives are not interchangeable. Some methods approximate loss preservation at initialization, some attempt to preserve gradient flow, some are explicitly data-free and target balanced signal propagation, and some optimize for hardware-aware structure rather than weight-level sparsity (Wang et al., 2021, Cai et al., 2022). A plausible implication is that “pruning-at-initialization” is best understood as a design space rather than a single algorithmic doctrine.

2. Saliency criteria and representative algorithms

The canonical single-shot data-dependent criterion is SNIP, which scores parameters by initialization-time connection sensitivity:

$m \in \{0,1\}^d$0

Operationally, SNIP uses one forward-backward pass on a small batch, ranks parameters globally or per layer, and keeps the top-$m \in \{0,1\}^d$1 entries (Wang et al., 2021). A closely related formulation writes the score as

$m \in \{0,1\}^d$2

with the implemented score reported as $m \in \{0,1\}^d$3 at initialization (Singh et al., 2021).

GraSP extends this perspective to second-order gradient preservation. In the survey formulation, if $m \in \{0,1\}^d$4 and $m \in \{0,1\}^d$5, GraSP uses the approximation

$m \in \{0,1\}^d$6

with efficient evaluation through Hessian-vector products (Wang et al., 2021). The ablation paper on PaI randomization studies the absolute-value version, denoted GraspAbs, with score

$m \in \{0,1\}^d$7

where $m \in \{0,1\}^d$8 in layerwise form (Singh et al., 2021).

SynFlow is the principal data-free alternative. It replaces task data by an all-ones input, forces weights positive, and computes a saliency based on a synthetic scalar objective:

$m \in \{0,1\}^d$9

The survey also gives the equivalent product form

$m \odot w$0

with iterative pruning used to avoid layer collapse (Singh et al., 2021, Wang et al., 2021).

Magnitude pruning at initialization retains the largest $m \odot w$1 and is the simplest data-free baseline, but multiple sources emphasize that raw magnitude is usually weaker at initialization than after training (Wang et al., 2021, Singh et al., 2021). More specialized proposals extend these core ideas. The abstract of “What to Prune and What Not to Prune at Initialization” reports two approaches: K-starts, which begins with $m \odot w$2 random $m \odot w$3-sparse matrices and selects the “fittest” one using weight magnitude, gradient accumulation, or both; and Dissipating gradients, which removes weights that remain within a fraction of their initial value during the first couple of epochs (Haroon, 2022). The same abstract states that combining dissipating gradients and kstarts “outperforms either methods and random dropout consistently” (Haroon, 2022).

A more recent line learns PaI criteria from iterative pruning. “Learning effective pruning at initialization from iterative pruning” proposes AutoSparse, an end-to-end scorer trained from Iterative Rewind Pruning survival labels so that initialization features predict which parameters would survive strong iterative pruning (Liu et al., 2024). This suggests a hybrid view in which iterative pruning acts as a teacher for one-shot PaI.

3. Signal propagation, initialization, and trainability

A recurrent theme in PaI is that sparsity selection cannot be separated from initialization dynamics. “A Signal Propagation Perspective for Pruning Neural Networks at Initialization” interprets connection sensitivity as a gradient-based quantity and argues that reliable saliency requires well-conditioned forward and backward propagation at initialization. For a feedforward network with Jacobian

$m \odot w$4

the paper identifies layerwise dynamical isometry as a sufficient condition under which gradient magnitudes remain comparable across layers and SNIP-style saliency is trustworthy (Lee et al., 2019). It further proposes a data-free approximate isometry objective for the pruned network,

$m \odot w$5

to restore trainability after pruning (Lee et al., 2019).

A complementary mean-field treatment appears in “Robust Pruning at Initialization” (Hayou et al., 2020). There, sensitivity-based pruning is analyzed through the Edge of Chaos (EOC). The paper defines the ordered, chaotic, and EOC phases through $m \odot w$6, where $m \odot w$7, and shows that sensitivity-based pruning is ill-conditioned away from EOC. Its central theorem states that in the ordered phase the expected critical sparsity before some layer is fully pruned obeys

$m \odot w$8

with $m \odot w$9 (Hayou et al., 2020). The same work proposes a Rescaling Trick that restores EOC after sensitivity-based pruning by rescaling retained weights layerwise, and introduces Stable ResNet parameterization, in which residual branches are scaled by $\|m\|_0 \le k$0 so that saliency moments remain controlled at large depth (Hayou et al., 2020).

These analyses clarify why PaI often fails at extreme sparsity even when winning tickets exist. The issue is not only which weights are removed, but whether the resulting sparse network preserves balanced Jacobians, non-collapsing layers, and viable gradient flow immediately after pruning.

4. Layerwise structure, randomization, and the shift from unstructured to structured PaI

An important empirical result is that many PaI methods are unexpectedly insensitive to the exact identities of retained weights. “Why is Pruning at Initialization Immune to Reinitializing and Shuffling?” studies Magnitude, SNIP, SynFlow, and GraspAbs under two perturbations: Reinit, which resamples surviving weights from the original initialization distribution, and Layerwise Shuffling, which permutes mask positions within each layer while preserving per-layer counts (Singh et al., 2021). Across LeNet-300-100 on MNIST and ResNet-20 on CIFAR-10, the paper reports small Wasserstein distances between the distributions of unpruned weights under the original and perturbed masks, whereas random pruning yields much larger distances. For example, on LeNet-MNIST the average Wasserstein distance for SNIP is $\|m\|_0 \le k$1 under Reinit and $\|m\|_0 \le k$2 under Layerwise Shuffling, versus $\|m\|_0 \le k$3 for Random Pruning; on ResNet-CIFAR, SynFlow yields $\|m\|_0 \le k$4, $\|m\|_0 \le k$5, and $\|m\|_0 \le k$6, respectively (Singh et al., 2021). The paper’s interpretation is that layerwise sparsity budgets and distributional properties dominate exact connection identity.

This observation underlies the structured PaI argument advanced by “Structured Pruning is All You Need for Pruning CNNs at Initialization” (Cai et al., 2022). That work introduces Synaptic Expectation and proves a granularity-invariance theorem stating that the expected SynFlow score depends only on layerwise densities:

$\|m\|_0 \le k$7

so expected accuracy is independent of whether pruning is by weights, filters, or channels provided layerwise densities $\|m\|_0 \le k$8 are the same (Cai et al., 2022). Based on this, the paper proposes PreCropping, a channel-level structured PaI method that optimizes only layerwise densities under parameter and FLOP budgets. The parameter-only optimization has the closed-form KKT solution

$\|m\|_0 \le k$9

while the joint parameter-plus-FLOP formulation gives

$$\min_{m \in \{0,1\}^d}\ \mathcal{L}(m \odot w_0;\ \mathcal{D})\quad \text{s.t.}\quad \|m\|_0 \le k,$$0

with $$\min_{m \in \{0,1\}^d}\ \mathcal{L}(m \odot w_0;\ \mathcal{D})\quad \text{s.t.}\quad \|m\|_0 \le k,$$1 and $$\min_{m \in \{0,1\}^d}\ \mathcal{L}(m \odot w_0;\ \mathcal{D})\quad \text{s.t.}\quad \|m\|_0 \le k,$$2 denoting parameter and FLOP costs (Cai et al., 2022).

The same paper reports that on ImageNet, structured PaI can match or outperform SynFlow under matched budgets. For MobileNetV2, PreCrop improves over SynFlow by up to $$\min_{m \in \{0,1\}^d}\ \mathcal{L}(m \odot w_0;\ \mathcal{D})\quad \text{s.t.}\quad \|m\|_0 \le k,$$3 top-1 at matched FLOPs/parameters; for EfficientNet-B0 it improves by $$\min_{m \in \{0,1\}^d}\ \mathcal{L}(m \odot w_0;\ \mathcal{D})\quad \text{s.t.}\quad \|m\|_0 \le k,$$4 (Cai et al., 2022). This suggests that, at least for CNNs, the central design variable may be layerwise density allocation rather than fine-grained unstructured masks.

Layer-level pruning extends this logic further. “When Layers Play the Lottery, all Tickets Win at Initialization” studies residual block removal rather than weight or filter pruning. For residual architectures with

$$\min_{m \in \{0,1\}^d}\ \mathcal{L}(m \odot w_0;\ \mathcal{D})\quad \text{s.t.}\quad \|m\|_0 \le k,$$5

entire blocks can be removed while the identity path preserves shape compatibility (Jordao et al., 2023). On ResNet-32 for CIFAR-10, GraSP-based layer pruning at initialization produces winning tickets for all tested pruning densities, with gains such as $$\min_{m \in \{0,1\}^d}\ \mathcal{L}(m \odot w_0;\ \mathcal{D})\quad \text{s.t.}\quad \|m\|_0 \le k,$$6 percentage points at $$\min_{m \in \{0,1\}^d}\ \mathcal{L}(m \odot w_0;\ \mathcal{D})\quad \text{s.t.}\quad \|m\|_0 \le k,$$7 removed block and $$\min_{m \in \{0,1\}^d}\ \mathcal{L}(m \odot w_0;\ \mathcal{D})\quad \text{s.t.}\quad \|m\|_0 \le k,$$8 at $$\min_{m \in \{0,1\}^d}\ \mathcal{L}(m \odot w_0;\ \mathcal{D})\quad \text{s.t.}\quad \|m\|_0 \le k,$$9 (Jordao et al., 2023).

5. Application-specific extensions and systems

PaI has increasingly been specialized to domains where training-time resource savings matter as much as model size. In 3D CNNs, RANP performs neuron pruning at initialization using loss sensitivity reweighted by resource costs (Xu et al., 2020). For neuron $w_0$0 in layer $w_0$1, the default importance is the sum of absolute mask-gradient magnitudes over incoming channels,

$w_0$2

followed by layer-mean balancing and resource-aware reweighting

$w_0$3

where $w_0$4 is a FLOP- or memory-based layer cost (Xu et al., 2020). Reported reductions include roughly $w_0$5–$w_0$6 in FLOPs and $w_0$7–$w_0$8 in memory, with negligible loss in accuracy across 3D-UNet, MobileNetV2, and I3D tasks (Xu et al., 2020).

In federated learning, FedPaI uses PaI to fix sparse connectivity early and thereby reduce both communication and computation overhead. Its abstract reports support for both structured and unstructured pruning, personalized client-side pruning, sparsity-aware server-side aggregation, sparsity up to $w_0$9 “without compromising the model accuracy compared to unpruned baselines,” and training acceleration by $\mathcal{D}$0 to $\mathcal{D}$1 times under non-IID settings (Wang et al., 1 Apr 2025). The same abstract frames PaI as a natural fit for decentralized environments because sparsity patterns are fixed from the start (Wang et al., 1 Apr 2025).

For edge deployment, Reconvene combines unstructured PaI with structured postprocessing. It first applies unstructured PaI, then identifies convolution layers whose sparsity exceeds the model-wide average and converts those resilient sparse layers into smaller dense layers. Its layer test is

$\mathcal{D}$2

with resilient layers assigned a reduced output-channel count $\mathcal{D}$3 (Eccles et al., 2024). The reported system creates pruned models “within seconds” and yields models up to $\mathcal{D}$4 smaller and $\mathcal{D}$5 faster while maintaining the same accuracy as an unstructured PaI counterpart (Eccles et al., 2024).

PaI has also been adapted beyond classification. “Pruning Unrolled Networks (PUN) at Initialization for MRI Reconstruction Improves Generalization” prunes the denoiser component of unrolled MoDL architectures via direct optimization of a Bernoulli mask distribution at initialization:

$\mathcal{D}$6

relaxed through a Concrete/Gumbel-softmax mask parameterization (Liang et al., 2024). On fastMRI knee, PUN-IT improves PSNR from $\mathcal{D}$7 to $\mathcal{D}$8 at $\mathcal{D}$9 acceleration and from $c$0 to $c$1 at $c$2 acceleration relative to dense MoDL (Liang et al., 2024). This suggests PaI can act as a robustness mechanism under distribution shift, not merely a compression tool.

6. Theory, limitations, and contemporary debates

The strongest recent criticism of PaI is information-theoretic. “No Free Prune: Information-Theoretic Barriers to Pruning at Initialization” argues that the effective capacity of a sparse network is not just the number of surviving parameters, but

$c$3

or, in the continuous setting,

$c$4

The paper extends the Law of Robustness to sparse networks by replacing ordinary parameter count with $c$5, implying that robust interpolation of noisy data requires either many nonzero weights or a mask with substantial mutual information about the data (Kumar et al., 2024). Its central claim is that pruning during or after training naturally yields masks with higher $c$6 than masks chosen at initialization, which explains why lottery tickets can exist but be hard to find “fast” (Kumar et al., 2024).

A related practical critique comes from rewinding results. “Stabilizing the Lottery Ticket Hypothesis” finds that on deeper networks and ImageNet-scale tasks, iterative magnitude pruning fails when weights are reset to iteration $c$7, but succeeds when rewound to an early point such as epoch $c$8–$c$9 or the first $m$0–$m$1 of training (Frankle et al., 2019). For ResNet-50 on ImageNet, one-shot pruning with early rewind matches dense top-1 at $m$2 pruned, and iterative pruning with rewind to epoch $m$3 matches dense accuracy with only $m$4 of weights remaining (Frankle et al., 2019). This suggests that the “initialization” relevant to sparse trainability may, in practice, be a narrow early-training window rather than the literal random draw at step zero.

At the same time, theory has also identified positive regimes. “Pruning Before Training May Improve Generalization, Provably” studies random PaI in overparameterized two-layer networks and proves that, if keep probability $m$5 remains above a threshold $m$6, gradient descent achieves low training loss and a generalization bound that improves as pruning increases:

$m$7

Below a smaller scale $m$8, the network can still fit the training set by memorizing noise, with test performance no better than random guessing (Yang et al., 2023). This result supports a nuanced view: moderate PaI can act as feature purification, whereas excessive PaI destroys signal pathways.

A different theoretical synthesis is offered by the path-based NTK analysis of “A Unified Paths Perspective for Pruning at Initialization” (Gebhart et al., 2021) and the graphon-limit framework of “Pruning at Initialisation through the lens of Graphon Limit” (Pham et al., 6 Feb 2026). The former interprets SynFlow as equivalent to the $m$9 path regularizer and defines a Path Kernel whose trace can be computed via

$\tilde{W}=M\odot W$0

while the latter proves that under a Factorised Saliency Model, PaI masks converge to deterministic bipartite graphons

$\tilde{W}=M\odot W$1

distinguishing homogeneous limits for random or magnitude pruning from heterogeneous, feature-selective limits for data-driven methods such as SNIP and GraSP (Gebhart et al., 2021, Pham et al., 6 Feb 2026). These results suggest that PaI’s large-width behavior can be described either through path norms and NTK structure or through continuous graph-limit objects.

Taken together, the literature does not support a single verdict on PaI. It supports several narrower conclusions: PaI is computationally attractive; structured and resource-aware variants can be highly practical; moderate pruning can sometimes improve generalization; but at extreme sparsity, or on noisy high-dimensional tasks, initialization-time masks often lack the data dependence required to match pruning during or after training (Kumar et al., 2024, Yang et al., 2023). A plausible implication is that future progress will come less from one universal saliency rule than from task-specific combinations of initialization geometry, modest data dependence, and architecture-aware structure.