Greedy Algorithms with Pruning

• Greedy algorithm with pruning is an iterative scheme that makes local optimal decisions while discarding unpromising candidates to keep the search tractable.
• It is applied in areas such as symbolic regression, transformer compression, CNN filter removal, and submodular maximization to approximate global solutions efficiently.
• The method balances forward selection and backward elimination using adaptive criteria and dynamic recomputation to enforce structural invariants and guarantee performance.

Searching arXiv for relevant papers on greedy algorithms with pruning across symbolic regression, transformer compression, submodular optimization, sparse recovery, and theoretical guarantees. A greedy algorithm with pruning is an iterative optimization scheme in which each step makes a locally optimal addition or deletion while simultaneously eliminating candidates, paths, parameters, or ground-set elements judged to be unpromising by an explicit criterion. In the literature, this pattern appears in symbolic regression, transformer compression, CNN filter pruning, boosted decision-tree training, sparse signal recovery, and submodular maximization. The greedy move can be formulated as removing the single edge whose deletion causes the smallest increase in a data-fit loss, pruning the lowest-scoring attention head, selecting the layer whose removal leaves the largest remaining validation score, or adding the neuron that most reduces loss; the pruning move can be a beam restriction, a path cutoff, an interval-based discard rule, or a ground-set reduction step. Across these settings, pruning is used to keep the search tractable, but the resulting solution is generally an approximation rather than a global optimum (Wu et al., 2024, Peer et al., 2021, Chen et al., 8 May 2026, Nath et al., 2024).

1. Core algorithmic pattern

The common structure is a two-part loop. First, a greedy criterion ranks immediate actions by a local objective such as current validation performance, marginal gain, residual reduction, or loss increase. Second, a pruning rule removes options that are already dominated, exceed a threshold, have nonpositive marginal contribution, or fall outside a fixed beam. This coupling of local choice with candidate elimination is what distinguishes greedy-with-pruning procedures from plain greedy selection.

The literature also distinguishes backward elimination from forward selection. Backward methods start from a dense model or full candidate set and iteratively remove components; PruneSymNet removes linear connections, Greedy-Gnorm removes attention heads, Greedy-layer Pruning removes transformer layers, and FP-Backward removes CNN filters (Wu et al., 2024, Guo et al., 4 Feb 2026, Peer et al., 2021, Purohit et al., 2024). Forward methods start from an empty or sparse structure and add components; the Forward Selection algorithm for subnetworks adds the neuron that most reduces loss, and orthogonal matching pursuit-based filter pruning grows the kept filter set one filter at a time (Ye et al., 2020, Purohit et al., 2024). A third pattern is adaptive candidate pruning, in which exact greedy optimality is preserved by ruling out candidates with certified lower and upper bounds, as in boosted decision-tree training (Aziz et al., 2018).

Algorithm	Object	Greedy / pruning rule
PruneSymNet (Wu et al., 2024)	ordinary edges	remove the single edge whose deletion causes the smallest increase in the data-fit term of $L(W)$
Greedy-Gnorm (Guo et al., 4 Feb 2026)	attention heads	prune the head with smallest $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$
GLP (Peer et al., 2021)	transformer layers	remove the layer maximizing $M(L\setminus(R_{n-1}\cup\{\ell\}),T)$
TMP (Lee et al., 2014)	search-tree paths	prune a path when its approximated cost exceeds $\epsilon$
Adaptive-Pruning Boost (Aziz et al., 2018)	stumps / features	discard a stump once $L_m^i > U_{m'}^j$
GreedyWithPruning (Chen et al., 8 May 2026)	active-set elements	remove $x$ if $\Delta_f(x \mid A' \setminus \{x\}) \le 0$

This suggests that “pruning” is not a single operation but a family of admissibility tests. In some cases the test is heuristic but loss-aware, in others it is derived from exact interval bounds, and in still others it is coupled to a structural invariant, such as positivity of all nonempty subsets of the current active set (Chen et al., 8 May 2026).

2. Symbolic regression and model-structure pruning

In symbolic regression, PruneSymNet embeds a combinatorial expression space into a differentiable network whose activation function consists of common elementary functions and operators. Each subnetwork corresponds to an expression, and a “minimalist” subnetwork is obtained by pruning extra linear connections one by one, starting from the output node and proceeding backward through layers. Training uses

$$L(W)=\frac{1}{n}\sum_{i=1}^n \bigl[y_i-\mathrm{net}_W(X_i)\bigr]^2 + \|W\|_{0.5},$$

with $\|W\|_{0.5}=\sum_j |W_j|^{0.5}$, while pruning evaluates only the data-fit term because the regularizer is constant once training stops. The greedy rule preserves the edge with the least loss in each pruning, and beam search of width $B$ is added because greedy algorithm often can not get the optimal solution. Two additional heuristics are reported: in the first two layers, top-$S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$0 selection may be replaced by a softmax over $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$1, and if at least $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$2 of beam members remain identical to their parents, the SGD learning-rate is bumped and the network is retrained briefly before the next prune (Wu et al., 2024).

Transformer pruning instantiates the same pattern at different structural granularities. Greedy-Gnorm prunes attention heads by dynamically recalculating head importance after each pruning step. For head $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$3, the score is the element-wise product of the $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$4-norms of its Q/K/V gradient blocks,

$S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$5

estimated from a hold-out validation set for the current partially pruned model $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$6. Heads with smaller scores are pruned next. The central design choice is recomputation: static rankings are treated as stale once the model has changed (Guo et al., 4 Feb 2026).

Greedy-layer Pruning applies greedy deletion at the layer level. Given a pre-trained Transformer encoder of depth $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$7, the formal objective is to choose $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$8, $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$9, maximizing the downstream performance measure $M(L\setminus(R_{n-1}\cup\{\ell\}),T)$0. Exhaustive search over $M(L\setminus(R_{n-1}\cup\{\ell\}),T)$1 candidates is replaced by removing exactly one further layer whose removal gives the largest remaining score. Because the method computes a nested removal sequence $M(L\setminus(R_{n-1}\cup\{\ell\}),T)$2, model size can be adapted dynamically to a desired performance/speedup tradeoff without additional pre-training phases (Peer et al., 2021).

CNN whole-network filter pruning extends the greedy-with-pruning pattern hierarchically. At the lower level, FP-OMP performs forward greedy selection of kept filters from a sparse-approximation objective, while FP-Backward starts with all filters and removes one at a time using a closed-form error criterion $M(L\setminus(R_{n-1}\cup\{\ell\}),T)$3. At the higher level, HBGS greedily selects the next layer to prune by layer-wise relative reconstruction error, whereas HBGTS uses final classification error as the global pruning criterion. The explicit separation between filter-selection and layer-selection turns pruning fractions into adaptive outputs rather than fixed per-layer inputs (Purohit et al., 2024).

A distinct variant is Greedy Adversarial Pruning, which removes the most important model parameters based on the gradient on adversarial inputs rather than the least important parameters for clean accuracy. The importance score is

$M(L\setminus(R_{n-1}\cup\{\ell\}),T)$4

and global thresholding produces the mask $M(L\setminus(R_{n-1}\cup\{\ell\}),T)$5. Here the greedy objective is not compression alone but resistance to transfer attacks from the uncompressed model (Weiss et al., 2022).

3. Search trees, exact split selection, and ground-set reduction

In sparse signal recovery, Lee et al.’s Matching Pursuit with Tree Pruning (TMP) addresses the standard weakness of single-candidate greedy pursuit: myopic support decisions. TMP first performs pre-selection to restrict attention to a small subset $M(L\setminus(R_{n-1}\cup\{\ell\}),T)$6, then grows a search tree of depth $M(L\setminus(R_{n-1}\cup\{\ell\}),T)$7. For each partial path, it computes a causal residual, appends a rough noncausal completion, evaluates the residual norm of the resulting full candidate, and prunes paths whose approximated cost exceeds a running threshold $M(L\setminus(R_{n-1}\cup\{\ell\}),T)$8. The effect is to explore multiple promising candidates via the tree search while curbing combinatorial blow-up (Lee et al., 2014).

Adaptive-Pruning Boost uses pruning to compute exact greedy-optimal decision stumps more efficiently. After evaluating a stump $M(L\setminus(R_{n-1}\cup\{\ell\}),T)$9 on the first $\epsilon$0 heaviest examples, it maintains exact intervals

$\epsilon$1

for the true weighted error $\epsilon$2. If $\epsilon$3, stump $\epsilon$4 cannot be optimal and is discarded. The algorithm then behaves like a best-arm identification procedure, alternately refining the current best arm and its strongest competitor until all others are certifiably inferior. The pruning is therefore not heuristic: it preserves exact greedy optimality while reducing feature–example assessments (Aziz et al., 2018).

For submodular maximization, pruning can occur before or during greedy selection. Submodular Sparsification builds a directed submodularity graph with edge weights

$\epsilon$5

samples a probe set $\epsilon$6 of size $\epsilon$7, computes $\epsilon$8, and prunes a $\epsilon$9 fraction of nodes at each step. The downstream greedy algorithm is then run on a reduced set $L_m^i > U_{m'}^j$0 (Zhou et al., 2016).

QuickPrune addresses billion-scale ground sets for knapsack-constrained maximization of a monotone, $L_m^i > U_{m'}^j$1-submodular function. In QuickPrune-Single, each element $L_m^i > U_{m'}^j$2 is processed once, kept only if $L_m^i > U_{m'}^j$3 exceeds a threshold proportional to $L_m^i > U_{m'}^j$4, and old snapshots $L_m^i > U_{m'}^j$5 are occasionally forgotten when $L_m^i > U_{m'}^j$6 has grown enough. QuickPrune-Multi runs several such instances in parallel for budgets $L_m^i > U_{m'}^j$7 and returns the union. The pruned universe is then fed to a standard greedy maximizer (Nath et al., 2024).

In “Curvature Beyond Positivity,” pruning is interleaved directly with greedy selection. The algorithm picks an element of maximal marginal gain $L_m^i > U_{m'}^j$8; if the gain is non-positive it stops, and after each addition it repeatedly removes any $L_m^i > U_{m'}^j$9 with $x$0. This pruning loop is part of the approximation proof for arbitrary submodular functions, including negative-valued objectives (Chen et al., 8 May 2026).

4. Guarantees, rates, and complexity

Greedy-with-pruning methods range from exact algorithms with aggressive screening to approximate algorithms with explicit structural guarantees. In Adaptive-Pruning Boost, the interval rule yields exact greedy-optimal decision trees while visiting many fewer feature–example pairs; the subroutine at a tree node never visits any feature–example pair more than twice (Aziz et al., 2018). In contrast, Greedy-Gnorm is explicitly described as a greedy approximation to an NP-hard subset-selection problem, although dynamic recomputation is reported to close much of the gap between greedy and global optimality (Guo et al., 4 Feb 2026).

Several papers provide explicit complexity reductions. PruneSymNet with beam width $x$1 performs up to $x$2 forward passes per operator node, for total cost roughly $x$3, and memory $x$4; in practice $x$5 and $x$6 are small, with $x$7 and $x$8 (Wu et al., 2024). Greedy-Gnorm has overall cost $x$9, with score aggregation $\Delta_f(x \mid A' \setminus \{x\}) \le 0$0 per iteration and termination after at most $\Delta_f(x \mid A' \setminus \{x\}) \le 0$1 iterations (Guo et al., 4 Feb 2026). Greedy-layer Pruning reduces exhaustive $\Delta_f(x \mid A' \setminus \{x\}) \le 0$2 search to at most $\Delta_f(x \mid A' \setminus \{x\}) \le 0$3 fine-tuning runs, with worst-case $\Delta_f(x \mid A' \setminus \{x\}) \le 0$4; for a 12-layer BERT pruned by 6 layers, this becomes $\Delta_f(x \mid A' \setminus \{x\}) \le 0$5 fine-tunes rather than $\Delta_f(x \mid A' \setminus \{x\}) \le 0$6 (Peer et al., 2021). FP-Backward has overall cost approximately $\Delta_f(x \mid A' \setminus \{x\}) \le 0$7 and is reported as $\Delta_f(x \mid A' \setminus \{x\}) \le 0$8–$\Delta_f(x \mid A' \setminus \{x\}) \le 0$9 orders of magnitude faster than repeated OMP calls (Purohit et al., 2024). Submodular Sparsification has total time $$L(W)=\frac{1}{n}\sum_{i=1}^n \bigl[y_i-\mathrm{net}_W(X_i)\bigr]^2 + \|W\|_{0.5},$$0 after the final greedy stage is included (Zhou et al., 2016).

The strongest formal rate results come from subnetwork selection and submodular analysis. Ye et al. prove that the discrepancy between a pruned network and the original decays at an exponentially fast rate with respect to the size of the pruned network, $$L(W)=\frac{1}{n}\sum_{i=1}^n \bigl[y_i-\mathrm{net}_W(X_i)\bigr]^2 + \|W\|_{0.5},$$1, so to achieve discrepancy at most $$L(W)=\frac{1}{n}\sum_{i=1}^n \bigl[y_i-\mathrm{net}_W(X_i)\bigr]^2 + \|W\|_{0.5},$$2 it is enough to keep $$L(W)=\frac{1}{n}\sum_{i=1}^n \bigl[y_i-\mathrm{net}_W(X_i)\bigr]^2 + \|W\|_{0.5},$$3 neurons under boundedness and Lipschitz assumptions (Ye et al., 2020). The Forward Selection framework proves $$L(W)=\frac{1}{n}\sum_{i=1}^n \bigl[y_i-\mathrm{net}_W(X_i)\bigr]^2 + \|W\|_{0.5},$$4 in general and $$L(W)=\frac{1}{n}\sum_{i=1}^n \bigl[y_i-\mathrm{net}_W(X_i)\bigr]^2 + \|W\|_{0.5},$$5 under an over-parameterization assumption, and it is explicitly contrasted with backward elimination, for which no $$L(W)=\frac{1}{n}\sum_{i=1}^n \bigl[y_i-\mathrm{net}_W(X_i)\bigr]^2 + \|W\|_{0.5},$$6 or $$L(W)=\frac{1}{n}\sum_{i=1}^n \bigl[y_i-\mathrm{net}_W(X_i)\bigr]^2 + \|W\|_{0.5},$$7 bound holds in general (Ye et al., 2020). For arbitrary submodular functions, GreedyWithPruning returns an active set $$L(W)=\frac{1}{n}\sum_{i=1}^n \bigl[y_i-\mathrm{net}_W(X_i)\bigr]^2 + \|W\|_{0.5},$$8 satisfying

$$L(W)=\frac{1}{n}\sum_{i=1}^n \bigl[y_i-\mathrm{net}_W(X_i)\bigr]^2 + \|W\|_{0.5},$$9

with $\|W\|_{0.5}=\sum_j |W_j|^{0.5}$0; when $\|W\|_{0.5}=\sum_j |W_j|^{0.5}$1 is monotone, this recovers the classical $\|W\|_{0.5}=\sum_j |W_j|^{0.5}$2 guarantee (Chen et al., 8 May 2026). QuickPrune supplies complementary guarantees on retained optimal value and pruned-set size, including

$\|W\|_{0.5}=\sum_j |W_j|^{0.5}$3

for every budget $\|W\|_{0.5}=\sum_j |W_j|^{0.5}$4, and a pruned universe of size

$\|W\|_{0.5}=\sum_j |W_j|^{0.5}$5

(Nath et al., 2024).

5. Empirical behavior across application domains

In symbolic regression, PruneSymNet achieved the lowest test-set MSE on six public datasets relative to EQL, GP, and DSR, and by “voting” it won in $\|W\|_{0.5}=\sum_j |W_j|^{0.5}$6–$\|W\|_{0.5}=\sum_j |W_j|^{0.5}$7 of cases across datasets. It also recovered exact symbolic ground-truth expressions more often than the baselines; on the Livermore set it found $\|W\|_{0.5}=\sum_j |W_j|^{0.5}$8 of $\|W\|_{0.5}=\sum_j |W_j|^{0.5}$9 expressions exactly, whereas the best baseline found $B$0. Ablations show that removing gradient-descent fine-tuning before pruning or replacing greedy pruning by random-edge deletion sharply degrades performance, and beam-search augmented pruning tends to find a low-loss subnetwork faster than pure greedy while avoiding occasional large-loss pruning decisions (Wu et al., 2024).

In transformer compression, Greedy-Gnorm reports stable accuracy under substantial head removal. When pruning down to about $B$1 of original heads, the reported before/after accuracies are $B$2 for BERT, $B$3 for ALBERT, $B$4 for RoBERTa, and $B$5 for XLM-RoBERTa. The paper further states that the BERT accuracy curve degrades smoothly even past $B$6 removal and that Greedy-Gnorm outperforms attention entropy, its inverse variants, and random pruning (Guo et al., 4 Feb 2026).

Greedy-layer Pruning reports a consistent speed/accuracy trade-off on GLUE. For $B$7, the baseline is $B$8 GLUE at $B$9 ms CPU and $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$00 ms GPU. $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$01 gives $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$02 GLUE, $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$03M parameters, $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$04 ms CPU, $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$05 ms GPU, and $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$06 speedup, outperforming $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$07 at the same depth and parameter count and remaining competitive with DistilBERT. On the two-layer exhaustive comparison, GLP$S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$08 exactly recovers Opt$S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$09 or is within $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$10 points in $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$11 tasks for BERT and $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$12 for RoBERTa (Peer et al., 2021).

Hierarchical CNN pruning reports similarly strong compression regimes. On CIFAR-10, HBGTS gives $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$13 accuracy on VGG16 with $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$14 parameter reduction and $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$15 FLOPs reduction, and $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$16 on ResNet18 with $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$17 parameter reduction and $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$18 FLOPs reduction. For ResNext101, the method reduces RAM requirement from $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$19 GB to $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$20 GB and achieves a $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$21 reduction in FLOPS without losing accuracy on CIFAR-10 (Purohit et al., 2024).

For exact split selection in boosted trees, Adaptive-Pruning Boost uses $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$22–$S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$23 fewer example assessments than Quick Boost, with CPU-time gains up to $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$24 on w4a and satimage and $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$25–$S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$26 on a6a and MNIST; the method is described as near-optimal relative to both a weight-order lower bound and a stronger exact lower bound (Aziz et al., 2018). In large-scale submodular optimization, QuickPrune discards $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$27–$S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$28 of the ground set, retains more than $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$29 of the original value after greedy selection on $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$30, yields $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$31–$S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$32 end-to-end speedups, and uses $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$33–$S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$34 fewer oracle calls than Submodular Sparsification for the same quality (Nath et al., 2024).

6. Limitations, variants, and recurrent misconceptions

A recurrent misconception is that greedy pruning is a single backward-elimination heuristic. The literature is more heterogeneous. Forward Selection starts from an empty network and greedily adds neurons; the associated theory shows that sufficiently large pre-trained networks contain small subnetworks with lower loss than networks directly trained with gradient descent, and the same convex-hull argument extends to randomly weighted networks (Ye et al., 2020). By contrast, several practical pruning systems are backward elimination procedures, and some, such as PruneSymNet, state explicitly that greedy algorithm often can not get the optimal solution (Wu et al., 2024).

A second misconception is that pruning scores are static. Greedy-Gnorm is designed around the opposite claim: pruning any head causes nonlocal changes in Q/K/V gradients across all layers, so scores computed once on the unpruned model become stale. Dynamic recomputation is therefore not a minor implementation detail but the central distinction from attention-entropy ranking and static Gnorm variants (Guo et al., 4 Feb 2026).

A third misconception is that pruning necessarily improves robustness in a broad sense. Greedy Adversarial Pruning is more specific: it improves resistance to transfer attacks from the uncompressed counterpart, but it does not harden models against PGD or other first-order attacks run on themselves, and adversarial accuracy on self-generated perturbations remains low (Weiss et al., 2022).

In submodular optimization, pruning is sometimes treated only as a preprocessing acceleration. The arbitrary-curvature framework shows a stronger role: pruning can be embedded inside greedy itself, and after each pruning stage every nonempty subset of the active set has positive value while $S(n)=G_{Q(n)} \circ G_{K(n)} \circ G_{V(n)}$35 is monotone on that active set. This suggests that pruning can function not only as candidate reduction but also as an invariant-enforcing mechanism in the approximation analysis (Chen et al., 8 May 2026).

Taken together, these results support a precise interpretation. Greedy algorithms with pruning are most effective when a large search space contains many redundant or low-impact alternatives, when local re-evaluation is materially cheaper than global search, and when the pruning rule is tied either to exact certificates, as in interval-based split selection, or to task-relevant surrogates such as current loss, residual norm, or curvature. A plausible implication is that their success depends less on “greediness” in isolation than on how aggressively and how reliably the pruning stage reshapes the feasible search trajectory.