---
title: Pruning Before Training (PBT) in Neural Networks
url: https://www.emergentmind.com/topics/pruning-before-training-pbt
type: topic
---

# Pruning Before Training (PBT) in Neural Networks

Pruning Before Training (PBT) describes a class of neural network compression strategies in which pruning—the removal of weights, channels, blocks, or other substructures—is integrated directly into the network design phase, often before or during the main phase of supervised training. While conventional pruning protocols typically excise parameters after a baseline model is trained, PBT leverages search and constraint optimization to select efficient sub-architectures early in the pipeline. PBT approaches are often, but not exclusively, intertwined with Neural Architecture Search (NAS), yielding direct exploration of the accuracy/efficiency Pareto frontier for compact, hardware-conscious deployment.

## 1. Conceptual Landscape and Taxonomy

PBT encompasses a spectrum of methodologies unified by the notion that pruning decisions are not delayed until post hoc fine-tuning, but rather made as an implicit or explicit outcome of the model construction, parameter initialization, or supernet training regime. Distinct subcategories include:

- **NAS-integrated PBT**: Architecture search explicitly includes per-layer sparsity, channel width, layer depth, or operator selection as first-class variables, with pruning embedded in or co-optimized with search [1905.13543, 2003.12563, 2206.01198].
- **Distribution- and proxy-driven PBT**: Progressive sampling, probabilistic gating, or gradient-based proxy objectives identify low-utility structures for early removal [1905.13543, 2602.02891, 2003.13683].
- **Weight-sharing and supernet-based PBT**: Single-shot or two-stage supernet training, where subnetworks (with varying levels of pruning) are sampled and trained jointly, so only the most promising subnets are retained for downstream deployment [2405.02267, 2501.08504, 2405.18377].
- **Differentiable and gradient-driven PBT**: Latent variable or gating schemes (e.g., Gumbel-Softmax, ℓ₁-proximal optimization) directly encode pruning as part of the optimization landscape [2003.13683, 2206.01198].

This conception departs from classical “post-training” magnitude or sensitivity-based pruning, where weights are trimmed based on activity measured after full model convergence.

## 2. Formal Problem Statement and Search Space Construction

The formal objective of PBT is typically a bilevel or multi-objective optimization over architectural and pruning variables, subject to resource budgets:

$${\min_{\mathcal{A}, \theta} \; L_{\text{train}}(f(x;\theta, \mathcal{A})) \;\;\text{subject to}\;\; \text{FLOPs}(\mathcal{A}) \leq B, \;\; \text{Params}(\mathcal{A}) \leq S_{max}}$$

Here, $\mathcal{A}$ parameterizes the architectural/pruning choices—such as binary masks, per-layer channel widths, block retention masks, or latent vectors controlling hypernetworks—and $\theta$ the network weights. In the NAS context, $\mathcal{A}$ may combine operator selection per edge in a cell graph [1905.13543, 1906.07528], as well as layerwise/channelwise sparsity [2206.01198, 2102.00390].

Table 1 summarizes representative PBT-related search spaces:

| Paper                | Architecture Pruning Variables              | Resource Constraints         |
|----------------------|--------------------------------------------|-----------------------------|
| DDPNAS [1905.13543]  | Discrete op selection per edge in DAG      | FLOPs, latency              |
| TAS [1905.09717]     | Per-layer channel/depth multinomials       | Expected FLOPs penalty      |
| PaS [2206.01198]     | Binary gates per channel                   | Target MACs (hard/soft)     |
| DHP [2003.13683]     | Latent vectors $\{z^l\}$, input/output     | Target FLOPs                |
| Joint-DetNAS [2105.12971] | Stage/block width/depth, channel masks     | GFLOPs                      |

Methods differ in whether the architectural variables are discrete, continuous, probabilistic, differentiable, or updated by evolutionary, RL, or gradient-based strategies.

## 3. Principal Algorithms and Optimization Strategies

Several optimization approaches are prevalent in PBT:

**a) Progressive distribution pruning**: Methods such as DDPNAS [1905.13543] maintain a joint categorical distribution over architectural/pruning choices, prune low-probability options after each round, and update probabilities using performance-based utilities with momentum. The search proceeds until only one candidate per slot remains; a final exhaustive scan yields the optimal subnetwork under constraint.

**b) Prune-and-replace and morph-based search**: PR-DARTS [1906.07528] alternates between (i) pruning poor operators from the candidate pool on each cell edge and (ii) introducing new operations via network morphisms, followed by retraining.

**c) Differentiable pruning via proxies/latent variables**: DHP [2003.13683] utilizes hypernetworks parameterized by sparse latent vectors $\{z^l\}$, optimized jointly with network weights under an $\ell_1$ penalty. Proximal gradient steps induce channel-level sparsity. PaS [2206.01198] employs gating vectors updated through straight-through estimators (STE) with hard or soft resource constraints.

**d) Supernet and sandwich-training**: LLaMA-NAS [2405.18377], SuperSAM [2501.08504], and WS-NAS [2405.02267] train a supernetwork embedding all possible pruned configurations, leveraging weight-sharing. The NAS search (evolutionary, random, or Bayesian) identifies Pareto-optimal subnetworks post hoc, many of which were never explicitly trained as standalone models.

**e) Reinforcement learning or Bayesian meta-modeling**: NPAS [2012.00596] employs meta-controllers (Q-learning, BO) that select filter type, per-layer pruning scheme, and pruning rate. Real-world latency is measured per candidate to align search with deployment constraints.

**f) Zero-shot functional proxies**: TraceNAS [2602.02891] introduces a gradient trace correlation proxy: for each subnetwork, similarity to the pre-trained model’s loss landscape is measured via blockwise Pearson correlation of low-rank gradient traces. This allows candidate evaluation without full fine-tuning.

## 4. Practical Pipelines and Implementation Patterns

The practical instantiations of PBT frequently exhibit the following steps:

1. **Supernet or broad model preparation**: Initialize a large, overparameterized model or construct a supernet with architectural elasticity.
2. **Search/optimization loop**:
   - Sample or evaluate candidate subnetworks (via pruning variables)
   - Score via proxy (validation, predicted, or zero-shot metric)
   - Update pruning/architectural variables (distribution shift, evolution, gradient steps)
   - Apply hard pruning/removal after scheduled steps or as dictated by criterion
3. **(Optional) Final selection & post-processing**: Extract optimal or Pareto-subnetworks; often includes final fine-tuning or continued pretraining with frozen pruning decisions.
4. **Deployment-oriented steps**: Structural re-parameterization (PaS, DHP), code generation for hardware (NPAS), or program autotuning (SuperSAM).

A high-level algorithmic outline (DDPNAS [1905.13543]):

```python
# Initialization
theta = uniform_distribution()   # architecture/pruning variable
while any edge e has >1 candidate:
    for T epochs:
        # Sample architectures and validate
        sample and evaluate candidates
    # Update theta by softmax utility
    theta = update(theta, performance)
    # Prune lowest-probability element per edge
    prune_worst_candidates()
# Output final architecture
```

## 5. Empirical Results and Comparative Insights

PBT approaches consistently demonstrate that integrating pruning into the search or training pipeline yields superior accuracy–efficiency trade-offs:

- **Model quality at fixed cost**: On ImageNet, AACP [2102.00390] achieves 42% FLOP reduction in ResNet-50 with minimal ($0.18\%$) drop in top-1 accuracy, strictly outperforming contemporaneous NAS-based pruning and magnitude-based methods.
- **Search efficiency**: DDPNAS discovers ImageNet-mobile architectures in $1.8$ GPU-hours [1905.13543]; DA-NAS attains $2\times$ speedup on “Shuffle+Mobile” search versus prior baselines [2003.12563].
- **LLM and Transformer scale**: WS-NAS [2405.02267] finds $2\times$ compressed BERT/RoBERTa variants with less than $1$ point accuracy loss, outperforming fixed-head/quantized baselines. TraceNAS [2602.02891] matches or exceeds training-aware pruning at one-tenth GPU cost, realizing $4\times$ throughput gains after search.
- **Detection, segmentation, generative tasks**: Joint-DetNAS [2105.12971] delivers $+5$ mAP at fixed FLOPs over “prune-only” pipelines. DHP and PaS demonstrate compressive gains on classification, SISR, denoising, and instance segmentation with maintained or improved accuracy [2003.13683, 2206.01198].

Comparison of pruning schemes, NAS-only, and joint approaches reveals that hybrid NAS+prune frameworks strictly dominate simple post-training pruning on the accuracy–cost frontier [2105.12971].

## 6. Practical Considerations, Limitations, and Extensions

Key practical insights for adopting PBT include:

- **Constraint handling**: IDE (AACP) and PaS demonstrate that hard FLOPs/parameter budgets can be enforced through vector quantization/projection [2102.00390, 2206.01198]; Lagrangian relaxations can also be used for soft constraints.
- **Batchnorm and representation recalibration**: PBT pipelines often include explicit BN recalibration to improve subnetwork validity post-pruning [2102.00390].
- **Search complexity**: Search spaces are exponentially large in depth or per-layer width (noted as a limitation in [1905.09717, 2405.18377]); per-layer step vectors, or compressed representations (AACP, TAS), address tractability.
- **Proxy reliability**: The fidelity of analytic or zero-shot proxies (L₁-norm, gradient correlation) is domain dependent; some methods may fail in regimes with uncooperative filter dynamics [2102.00390, 2602.02891].
- **Hardware and deployment**: Integrated compiler- or autotuner-based code generation, as in NPAS [2012.00596] or SuperSAM [2501.08504], is necessary to translate theoretical compressions into real-world speedups.

Open challenges include improved proxies for non-pretrained settings, direct latency optimization, and generalization to multimodal or more complex task structures.

## 7. Significance and Emerging Directions

The PBT paradigm demonstrably advances the accuracy/efficiency envelope for compressed and compact neural networks by fusing pruning and architecture selection. Direct layer/channel/block selection within the training or NAS loop enables the discovery of non-uniform allocations that outperform uniform or magnitude-only counterparts. Recent momentum towards zero-shot and differentiable proxies (TraceNAS, DHP), NAS-based pruning for LLMs and foundation models (SuperSAM, LLaMA-NAS), and hardware/compiler-aligned search (NPAS) signals a convergence of model search, pruning, and deployment-aware optimization.

By framing pruning as a component of the architecture definition itself, PBT eliminates the dichotomy between network compression and design, yielding algorithms and pipelines that are more adaptive, efficient, and better matched to target resource budgets across a diversity of neural network workloads [1905.13543, 2012.00596, 2206.01198, 2602.02891, 2405.02267, 2405.18377, 2501.08504].

Source: https://www.emergentmind.com/topics/pruning-before-training-pbt