Pruning Interaction Paradox in Models

• Pruning Interaction Paradox is defined as the conflicting outcome where pruning appears beneficial in reducing parameters yet destabilizes generative tasks and sensitive subspaces.
• Empirical studies show that while pruning preserves robust internal representations like embeddings and logits, it can severely disrupt probability distributions and subgroup reliability.
• Researchers face challenges in recovering optimal sparse subnetworks as overparameterization and task-specific hierarchies cause mismatches between theoretical gains and practical pruning outcomes.

Searching arXiv for papers on pruning interactions across LLMs, retrieval, fairness, and compression.

The Pruning Interaction Paradox denotes a recurrent pattern across model compression research in which pruning appears benign, beneficial, or even theoretically elegant under one evaluation lens, yet becomes harmful, unstable, or misleading under another. Across recent work, the paradox does not refer to a single mechanism but to a family of pruning-induced mismatches: between non-generative and generative language tasks, between aggregate accuracy and example-level reliability, between sparsity existence and sparsity recoverability, between interaction efficiency and interaction fidelity, and between nominal compression gains and subgroup- or phase-specific failure modes. In this sense, pruning is not merely a reduction in parameter count or token count; it is an intervention whose effects depend on which representational space, task subspace, data slice, or dynamical regime is being probed (He et al., 25 Mar 2026, Tropeano et al., 27 Mar 2025, Zhang et al., 2024, Zong et al., 17 Apr 2025, Pan et al., 12 Mar 2026).

1. Representation-level paradox in LLMs

In decoder-only LLMs, the pruning interaction paradox is formalized as the discrepancy that the same pruned model may remain strong on retrieval and multiple-choice benchmarks while failing catastrophically on generative tasks such as chain-of-thought, code generation, and open-ended question answering (He et al., 25 Mar 2026). The central explanation is a representation hierarchy consisting of embedding space, logit space, and probability space. The paper "Demystifying When Pruning Works via Representation Hierarchies" decomposes computation into hidden states $h$, logits $z$, and probabilities $p$, and shows that pruning induces relatively small perturbations in embeddings and logits, but that these perturbations are amplified by softmax and then compounded over autoregressive time steps (He et al., 25 Mar 2026).

The representational pipeline is written as

$$x = \mathcal{T}(T), \quad e = E[x], \quad h^{(l)} = f^{(l)}(h^{(l-1)}), \quad z = W h^{(L)}, \quad p = \text{softmax}(z/T).$$

Pruning is analyzed as perturbations $h \mapsto h+\Delta h$, $z \mapsto z+\Delta z$, and $p \mapsto p+\Delta p$, with cosine similarity and KL divergence used to quantify deviation (He et al., 25 Mar 2026). The crucial empirical observation is that cosine similarity remains high in embedding and logit spaces, but probability-space similarity degrades strongly, especially across generation steps. This implies that pruning leaves many representation subspaces nearly intact while destabilizing the output distribution actually used in generation.

The task dependence follows directly from this hierarchy. Retrieval uses embedding-space cosine similarity,

$S(q,d) = \text{CosineSim}(h_q, h_d),$

and multiple-choice prediction only depends on a tiny subset of output tokens,

$$y = \arg\max_{j \in \mathcal{C}} p(j \mid x), \qquad |\mathcal{C}| \ll |V|,$$

so both remain relatively robust under pruning (He et al., 25 Mar 2026). Generation, by contrast, depends on the full vocabulary distribution over many autoregressive steps. A small perturbation in logits can therefore produce a large perturbation in the token distribution, which then alters future context and recursively magnifies downstream errors.

The paper gives concrete examples. On Mistral-7B-Instruct, dropping 8 attention layers changes the multi-choice average from $69.3$ to $z$0, while the generative average falls from $z$1 to $z$2. Dropping 8 MLP layers changes the multi-choice average from $z$3 to $z$4, but the generative average collapses from $z$5 to $z$6. On E5-Mistral retrieval, dropping 8 attention layers changes the average retrieval score from $z$7 to $z$8, and dropping 8 MLP layers yields $z$9, indicating only mild degradation despite substantial parameter removal (He et al., 25 Mar 2026). These results define the paradox numerically: the same pruning operation is nearly harmless on some tasks and destructive on others.

Theoretical approximations reinforce this interpretation. For logits, directional change is governed by the orthogonal perturbation,

$p$0

which remains small when the LM head attenuates orthogonal error (He et al., 25 Mar 2026). In probability space, however,

$p$1

with $p$2, and

$p$3

so the relevant quantity is not merely perturbation norm but variance along probability-relevant directions (He et al., 25 Mar 2026). This establishes softmax as a variance amplifier and autoregression as a temporal compounding mechanism.

A plausible implication is that the paradox is not specific to pruning as such, but to any compression or perturbation that is benign in internal representation spaces yet unstable in probability space under closed-loop decoding. The same paper explicitly notes similar behavior for quantization, although typically with smaller perturbations because weights are approximated rather than removed (He et al., 25 Mar 2026).

2. Example-level paradox: averages conceal concentrated damage

A second form of the pruning interaction paradox arises when pruning appears acceptable under mean accuracy or F1 but disproportionately harms a small subset of examples that are harder, more influential, and more semantically complex (Tropeano et al., 27 Mar 2025). The paper "As easy as PIE: understanding when pruning causes LLMs to disagree" defines Pruned Identified Exemplars (PIEs) as examples where pruned and unpruned models disagree in a stable, seed-aggregated sense (Tropeano et al., 27 Mar 2025).

For single-label tasks, let $p$4 be pruned model instances and $p$5 be unpruned model instances. With majority predictions $p$6 and $p$7, an example $p$8 is a PIE if

$p$9

For multi-label tasks, the same idea is applied to sets of labels, with any disagreement sufficing to define a PIE (Tropeano et al., 27 Mar 2025). This converts pruning impact from a model-level statistic into an example-level disagreement structure.

The experiments span BERT-base and BiLSTM across IMDB, SNLI, Reuters, and AAPD, with local unstructured weight pruning under magnitude, impact, and random scoring, combined with pruning at initialization or iterative pruning with fine-tuning or weight rewinding (Tropeano et al., 27 Mar 2025). The aggregate picture is initially unremarkable: up to $$x = \mathcal{T}(T), \quad e = E[x], \quad h^{(l)} = f^{(l)}(h^{(l-1)}), \quad z = W h^{(L)}, \quad p = \text{softmax}(z/T).$$0 pruning often yields tolerable drops in accuracy or F1. Yet the PIE analysis reveals that the degradation is highly concentrated.

The paper reports that PIE fractions grow with sparsity. For BERT with IIBP-FT, the PIE fraction on SNLI grows from $$x = \mathcal{T}(T), \quad e = E[x], \quad h^{(l)} = f^{(l)}(h^{(l-1)}), \quad z = W h^{(L)}, \quad p = \text{softmax}(z/T).$$1 at $$x = \mathcal{T}(T), \quad e = E[x], \quad h^{(l)} = f^{(l)}(h^{(l-1)}), \quad z = W h^{(L)}, \quad p = \text{softmax}(z/T).$$2 pruning to $$x = \mathcal{T}(T), \quad e = E[x], \quad h^{(l)} = f^{(l)}(h^{(l-1)}), \quad z = W h^{(L)}, \quad p = \text{softmax}(z/T).$$3 at $$x = \mathcal{T}(T), \quad e = E[x], \quad h^{(l)} = f^{(l)}(h^{(l-1)}), \quad z = W h^{(L)}, \quad p = \text{softmax}(z/T).$$4 pruning; on Reuters from $$x = \mathcal{T}(T), \quad e = E[x], \quad h^{(l)} = f^{(l)}(h^{(l-1)}), \quad z = W h^{(L)}, \quad p = \text{softmax}(z/T).$$5 to $$x = \mathcal{T}(T), \quad e = E[x], \quad h^{(l)} = f^{(l)}(h^{(l-1)}), \quad z = W h^{(L)}, \quad p = \text{softmax}(z/T).$$6; and on AAPD from $$x = \mathcal{T}(T), \quad e = E[x], \quad h^{(l)} = f^{(l)}(h^{(l-1)}), \quad z = W h^{(L)}, \quad p = \text{softmax}(z/T).$$7 to $$x = \mathcal{T}(T), \quad e = E[x], \quad h^{(l)} = f^{(l)}(h^{(l-1)}), \quad z = W h^{(L)}, \quad p = \text{softmax}(z/T).$$8 (Tropeano et al., 27 Mar 2025). The paradox is sharper than a mere rise in disagreement counts, because accuracy on PIEs collapses much faster than accuracy on the full test set. On SNLI, overall accuracy may only drop modestly under pruning, while accuracy on PIEs declines drastically (Tropeano et al., 27 Mar 2025). Mean metrics therefore smooth away concentrated harm.

The paper connects PIEs to EL2N, defined as

$$x = \mathcal{T}(T), \quad e = E[x], \quad h^{(l)} = f^{(l)}(h^{(l-1)}), \quad z = W h^{(L)}, \quad p = \text{softmax}(z/T).$$9

where the expectation is over training steps after the first epoch (Tropeano et al., 27 Mar 2025). High-EL2N examples are more influential for shaping the decision boundary. PIEs are heavily concentrated in the highest-EL2N bins: for BERT, up to $h \mapsto h+\Delta h$0 of the most influential training examples are PIEs; for BiLSTM, the concentration reaches up to about $h \mapsto h+\Delta h$1 (Tropeano et al., 27 Mar 2025). Thus pruning does not merely hurt random edge cases; it disproportionately affects exactly the examples that matter most for generalization.

The linguistic profile of PIEs is also systematic. Using readability metrics, difficult-word counts, and token length, the paper finds that PIEs are longer, more lexically complex, and more difficult by multiple readability indices. The ratio of difficult words reaches up to $h \mapsto h+\Delta h$2, and PIEs are up to $h \mapsto h+\Delta h$3 longer on IMDB and up to $h \mapsto h+\Delta h$4 longer on Reuters (Tropeano et al., 27 Mar 2025). PIEs are distributed across many classes rather than being confined to rare labels, so the phenomenon is not reducible to a simple long-tail class-imbalance explanation.

This suggests that pruning can preserve the dominant shortcuts used on easy examples while eroding the representational resources needed for semantically nuanced, influential examples. That interpretation is consistent with the paper’s conceptual account of pruning as a form of capacity reduction that disproportionately harms the parts of feature space requiring subtle distinctions (Tropeano et al., 27 Mar 2025).

3. Algorithmic paradox: sparse subnetworks exist but pruning fails to recover them

A third sense of the pruning interaction paradox concerns the gap between existence and recoverability of sparse subnetworks (Zhang et al., 2024). The paper "Sparsest Models Elude Pruning: An Exposé of Pruning's Current Capabilities" constructs a controlled setting in which extremely sparse, high-performing subnetworks are known to exist, yet state-of-the-art pruning algorithms systematically fail to find them (Zhang et al., 2024).

The study uses the Cubist Spiral, a synthetic binary classification task in $h \mapsto h+\Delta h$5, and 4-layer ReLU MLPs. Ideal sparse models are defined as solutions to

$h \mapsto h+\Delta h$6

with accuracy targets such as $h \mapsto h+\Delta h$7 and $h \mapsto h+\Delta h$8 (Zhang et al., 2024). A combinatorial search over structured and unstructured masks yields benchmark sparse networks that no tested pruning method can match.

At width $h \mapsto h+\Delta h$9, the search finds a $z \mapsto z+\Delta z$0+ solution with $z \mapsto z+\Delta z$1 nonzeros and a $z \mapsto z+\Delta z$2 solution with $z \mapsto z+\Delta z$3 nonzeros. A second search using the best discovered initialization finds an even sparser $z \mapsto z+\Delta z$4 model with $z \mapsto z+\Delta z$5 nonzeros, and a $z \mapsto z+\Delta z$6 model with $z \mapsto z+\Delta z$7 nonzeros (Zhang et al., 2024). By contrast, pruning methods such as GMP, LTH, SynFlow, Iter-SNIP, FORCE, and RigL require substantially more nonzeros to achieve similar accuracy. For the $z \mapsto z+\Delta z$8 threshold, LTH needs $z \mapsto z+\Delta z$9 nonzeros, RigL $p \mapsto p+\Delta p$0, and SynFlow $p \mapsto p+\Delta p$1. For $p \mapsto p+\Delta p$2, GMP needs about $p \mapsto p+\Delta p$3, Iter-SNIP $p \mapsto p+\Delta p$4, and RigL around $p \mapsto p+\Delta p$5 (Zhang et al., 2024).

The paradox sharpens under overparameterization. One might expect wider networks to make sparse subnetwork recovery easier because they contain more candidate subnetworks. The experiments show the opposite: as width increases from $p \mapsto p+\Delta p$6 to $p \mapsto p+\Delta p$7, pruning performance at high sparsity often collapses (Zhang et al., 2024). The proposed explanation is structural. Unstructured pruning tends to induce disconnected paths: nonzero weights attached to neurons with no nonzero incoming or outgoing path to the output. Such weights contribute to nominal sparsity but not to effective function.

The paper proves a theorem showing that for sufficiently wide networks with fixed numbers of nonzeros per layer, a random sparse mask has probability tending to $p \mapsto p+\Delta p$8 of containing no connected input-output path as width $p \mapsto p+\Delta p$9 (Zhang et al., 2024). In simplified form, if hidden width exceeds a function of per-layer nonzeros, then misalignment across adjacent layers makes connected paths vanishingly unlikely. This formalizes why overparameterization can increase the entropy of bad sparse masks faster than pruning heuristics can navigate toward the good ones.

Equally important, pruning still fails even when given the optimal width and initialization from combinatorial search. Starting from the initialization of the benchmark $S(q,d) = \text{CosineSim}(h_q, h_d),$0 solution, pruning methods do not recover a mask as sparse and accurate as the combinatorial optimum (Zhang et al., 2024). This indicates that current pruning algorithms are trapped not only by representational limitations but by mask-selection dynamics and local optimization path dependence.

A plausible implication is that the pruning interaction paradox in this regime is fundamentally algorithmic: overparameterization guarantees the presence of strong sparse subnetworks, but local pruning heuristics are poorly aligned with the combinatorial structure required to realize them.

4. Interaction-level paradox in late-interaction retrieval

In late-interaction retrieval, the pruning interaction paradox concerns the fact that dense token-level interaction appears necessary for retrieval fidelity, yet a large fraction of those interactions can be pruned with minimal loss if pruning is done at the right granularity (Pony et al., 2 Feb 2026, Zong et al., 17 Apr 2025).

For ColBERT-style models, a query $S(q,d) = \text{CosineSim}(h_q, h_d),$1 and document token set $S(q,d) = \text{CosineSim}(h_q, h_d),$2 are scored by

$S(q,d) = \text{CosineSim}(h_q, h_d),$3

$S(q,d) = \text{CosineSim}(h_q, h_d),$4

This implies an $S(q,d) = \text{CosineSim}(h_q, h_d),$5 matrix of MaxSim interactions across $S(q,d) = \text{CosineSim}(h_q, h_d),$6 candidate documents and $S(q,d) = \text{CosineSim}(h_q, h_d),$7 query tokens, with exact reranking requiring all entries (Pony et al., 2 Feb 2026). The paradox is that late-interaction models derive their effectiveness from this dense matching, but most of the matrix may be redundant for identifying the Top-$S(q,d) = \text{CosineSim}(h_q, h_d),$8 ranking.

The paper "Col-Bandit: Zero-Shot Query-Time Pruning for Late-Interaction Retrieval" recasts reranking as a finite-population Top-$S(q,d) = \text{CosineSim}(h_q, h_d),$9 identification problem (Pony et al., 2 Feb 2026). It reveals only a subset $$y = \arg\max_{j \in \mathcal{C}} p(j \mid x), \qquad |\mathcal{C}| \ll |V|,$$0 of MaxSim entries and defines coverage

$$y = \arg\max_{j \in \mathcal{C}} p(j \mid x), \qquad |\mathcal{C}| \ll |V|,$$1

Each document is treated as an arm with unknown mean token score, and uncertainty-aware lower and upper confidence bounds are maintained over its total score (Pony et al., 2 Feb 2026). The algorithm stops when the weakest current winner has lower confidence bound at least as large as the strongest current loser’s upper confidence bound. This allows query-time pruning of the interaction matrix itself, rather than offline pruning of all document or query tokens.

Empirically, Col-Bandit reaches $$y = \arg\max_{j \in \mathcal{C}} p(j \mid x), \qquad |\mathcal{C}| \ll |V|,$$2 Overlap@1 on BEIR at about $$y = \arg\max_{j \in \mathcal{C}} p(j \mid x), \qquad |\mathcal{C}| \ll |V|,$$3 coverage with Jina-ColBERT-v2, and $$y = \arg\max_{j \in \mathcal{C}} p(j \mid x), \qquad |\mathcal{C}| \ll |V|,$$4 Overlap@5 at about $$y = \arg\max_{j \in \mathcal{C}} p(j \mid x), \qquad |\mathcal{C}| \ll |V|,$$5 coverage, corresponding to multi-fold reductions in MaxSim FLOPs (Pony et al., 2 Feb 2026). On multimodal REAL-MM-RAG, it reaches $$y = \arg\max_{j \in \mathcal{C}} p(j \mid x), \qquad |\mathcal{C}| \ll |V|,$$6 Overlap@1 at roughly $$y = \arg\max_{j \in \mathcal{C}} p(j \mid x), \qquad |\mathcal{C}| \ll |V|,$$7 coverage and $$y = \arg\max_{j \in \mathcal{C}} p(j \mid x), \qquad |\mathcal{C}| \ll |V|,$$8 Overlap@5 at about $$y = \arg\max_{j \in \mathcal{C}} p(j \mid x), \qquad |\mathcal{C}| \ll |V|,$$9 coverage (Pony et al., 2 Feb 2026). This demonstrates that dense interaction is computationally redundant, but that the redundancy is query-dependent and must be pruned adaptively.

A complementary perspective appears in "Towards Lossless Token Pruning in Late-Interaction Retrieval Models" (Zong et al., 17 Apr 2025). There the goal is not query-time interaction pruning but document-token pruning without score change. The paper modifies ColBERT scoring to

$69.3$0

and introduces a dominance criterion under which a document token $69.3$1 can be pruned if, for every query vector $69.3$2, either $69.3$3 or some retained token $69.3$4 has strictly larger dot product (Zong et al., 17 Apr 2025). This yields a strong notion of lossless pruning: removing dominated tokens does not change the score for any query vector. Dominance checking is reduced to a linear-programming feasibility test, and regularizers such as nuclear norm, token-similarity, and $69.3$5 penalties are used to make document token sets more prunable (Zong et al., 17 Apr 2025).

The paper reports that with appropriate training and LP-based or norm-based pruning, ColBERT performance can be preserved while using only about $69.3$6 of tokens, with small in-domain and modest out-of-domain effectiveness drops (Zong et al., 17 Apr 2025). This suggests that what appears to be an interaction-preserving requirement may in practice contain a large redundancy budget—provided pruning respects the max-interaction geometry rather than using generic heuristics.

Taken together, these two papers show two sides of the same paradox. Dense interaction is both essential and highly redundant: essential because naive coarse approximations hurt ranking, redundant because many token-level interactions never alter the Top-$69.3$7 decision or score-maximizing configuration (Pony et al., 2 Feb 2026, Zong et al., 17 Apr 2025).

5. Pruning as task-, group-, and structure-dependent intervention

The pruning interaction paradox also appears outside language modeling as a mismatch between global utility and localized harm. In fairness-aware image classification, pruning can preserve headline accuracy while worsening subgroup disparities (Paganini, 2020). In gene regulatory network inference, pruning can simplify inferred graphs while deleting biologically meaningful redundant motifs (Saint-Antoine et al., 2019). In embodied VLA systems, token pruning can accelerate inference while eliminating visually sparse but structurally critical interaction regions (Cheng et al., 24 Mar 2026).

The paper "Prune Responsibly" analyzes over 120,906 pruned image classification models across LeNet, AlexNet, VGG11, and ResNet18, and finds that pruning exacerbates per-class performance disparities (Paganini, 2020). It defines class accuracy $69.3$8, class imbalance $69.3$9, and class complexity via average image entropy $z$00, and fits an OLS model showing that underrepresented classes and more complex classes are more severely affected by pruning (Paganini, 2020). Fairness is then quantified by metrics such as the max–min class-accuracy gap,

$z$01

This exposes another paradox: accuracy-efficiency trade-offs that look acceptable in aggregate may be dominated once fairness is treated as a third objective (Paganini, 2020).

In "Evaluating Pruning Methods in Gene Network Inference", the paradox is structural rather than statistical (Saint-Antoine et al., 2019). ARACNE and Phixer use DPI-inspired pruning to remove redundant edges in inferred gene interaction networks. ARACNE prunes $z$02 if

$z$03

and Phixer applies an analogous rule with the $z$04-mixing coefficient (Saint-Antoine et al., 2019). While such pruning is intended to eliminate indirect regulation, the paper shows that it often lowers AUROC and, in five of six cases, lowers AUPR, plausibly because it removes true direct edges in feed-forward loops, where direct and indirect regulation coexist (Saint-Antoine et al., 2019). The simplifying assumption that redundancy is expendable conflicts with the biological reality that redundant motifs are functional.

In "VLA-IAP: Training-Free Visual Token Pruning via Interaction Alignment for Vision-Language-Action Models", the paradox is that visually sparse regions such as contact edges, handles, and support surfaces may appear unimportant under semantic saliency, yet are critical for stable physical interaction (Cheng et al., 24 Mar 2026). The method introduces an interaction-aligned pruning scheme combining semantic priors, motion priors, and a geometric edge prior, with semantic-motion IoU used to switch between conservative and aggressive pruning modes (Cheng et al., 24 Mar 2026). It reports $z$05 success rate with $z$06 speedup on LIBERO, and up to $z$07 speedup while maintaining performance comparable to the unpruned backbone (Cheng et al., 24 Mar 2026). The empirical claim is that interaction-aware pruning can resolve the efficiency–stability tension by preserving structural anchors during uncertain early phases and pruning more aggressively only after interaction is locked.

These cases share a common structure. Pruning is rarely uniformly harmful or uniformly benign; rather, it redistributes model fidelity across tasks, subgroups, motifs, or temporal phases. The paradox emerges when the evaluation protocol does not align with the slice of behavior that pruning perturbs most strongly.

6. Pruning, generalization, and phase behavior

A further dimension of the pruning interaction paradox concerns learning dynamics themselves. Pruning can improve generalization in ways not explained by simple model-size reduction, and can induce sharp phase-like behavior under joint training-time and inference-time pruning (Jin et al., 2022, Pan et al., 12 Mar 2026).

The paper "Pruning's Effect on Generalization Through the Lens of Training and Regularization" studies IMP with learning-rate rewinding and argues that pruning combines two effects: better training at some sparsities and additional regularization at others (Jin et al., 2022). On clean data, pruned models can achieve lower training loss than dense baselines, especially on high-EL2N examples, while also improving test error. Extended dense training with the same cyclic learning-rate schedule largely reproduces these gains, showing that part of pruning’s benefit is really an optimization effect rather than a sparsity effect (Jin et al., 2022). On noisy-label datasets, however, pruning improves test error precisely by fitting some training examples worse than the dense model, especially high-EL2N noisy examples; here sparsity acts as regularization and extended dense training is insufficient (Jin et al., 2022).

This yields a two-regime resolution of the paradox. In one regime, pruning helps because the pruning procedure implicitly changes optimization dynamics; in the other, it helps because capacity reduction prevents memorization of harmful examples (Jin et al., 2022). The contradiction with modern overparameterization theory is therefore only apparent: pruning is not a pure size reduction, but a compound intervention on both optimization and effective capacity.

A more formal phase-theoretic treatment appears in "Pruning-induced phases in fully-connected neural networks: the eumentia, the dementia, and the amentia" (Pan et al., 12 Mar 2026). Here dropout is interpreted as random neuron pruning, independently controlled at training time and evaluation time via rates $z$08 and $z$09. The key observable is the scaling of test cross-entropy with training-set size $z$10,

$z$11

which defines three phases: eumentia with $z$12, dementia with $z$13, and amentia with $z$14 (Pan et al., 12 Mar 2026). In eumentia, the network learns and retains learning; in dementia, it learns under low training-time dropout but fails under heavy inference-time dropout; in amentia, heavy training-time dropout prevents learning in the first place.

The eumentia–dementia boundary shows crossing of loss curves for different $z$15 and finite-size scaling consistent with a BKT-like transition. Using the ansatz

$z$16

the paper obtains stable collapses with $z$17–$z$18 and interprets the eumentia phase as QLRO-like, analogous to neural scaling laws (Pan et al., 12 Mar 2026). This introduces yet another form of pruning interaction: moderate training-time pruning can make the model robust to inference-time pruning, while excessive pruning destroys trainability altogether.

A plausible implication is that many practical reports of pruning being “helpful,” “harmless,” or “catastrophic” are observations from different regions of an implicit phase diagram, whether or not that phase structure is made explicit.

7. Conceptual synthesis

Across these literatures, the pruning interaction paradox is best understood not as a single contradiction but as a general warning against projection from one evaluation space to another. The paradox arises whenever pruning is assessed in a space where it is weakly expressed, and then deployed in a space where its effects are amplified.

The representational version shows that stable embeddings and logits do not imply stable probability distributions under autoregressive decoding (He et al., 25 Mar 2026). The example-level version shows that modest mean accuracy changes do not imply uniform reliability across influential examples (Tropeano et al., 27 Mar 2025). The algorithmic version shows that existence of strong sparse subnetworks does not imply that local pruning heuristics can recover them (Zhang et al., 2024). The interaction-retrieval version shows that dense matching is both indispensable and massively redundant, depending on whether pruning respects the score geometry (Pony et al., 2 Feb 2026, Zong et al., 17 Apr 2025). The fairness and biological-network versions show that pruning can shift error toward underrepresented groups or motif-rich structures while preserving global metrics (Paganini, 2020, Saint-Antoine et al., 2019). The generalization and phase-transition versions show that pruning changes optimization dynamics, regularization, and even learning phase itself, not merely model size (Jin et al., 2022, Pan et al., 12 Mar 2026).

A useful unifying interpretation is that pruning interacts with hierarchies: representational hierarchies, data hierarchies, mask-structure hierarchies, interaction hierarchies, and phase hierarchies. In each case, there exist relatively stable subspaces and highly sensitive subspaces. Non-generative language tasks live largely in robust embedding/logit or categorical-token subspaces, whereas generation depends on fragile probability trajectories (He et al., 25 Mar 2026). Average benchmark metrics are dominated by easy examples, while PIEs occupy a concentrated but high-importance slice of the data distribution (Tropeano et al., 27 Mar 2025). Dense late-interaction retrieval contains many interactions that are irrelevant for Top-$z$19 identification, but a small decision-boundary subset remains crucial (Pony et al., 2 Feb 2026). Biological network pruning mistakes redundant-looking structure for dispensable structure (Saint-Antoine et al., 2019).

This suggests that the practical lesson is not that pruning “works” or “fails,” but that pruning must be evaluated in the subspace where it is intended to operate. When the task depends on stable embeddings, small option sets, or certifiable interaction redundancy, pruning can be highly effective. When the task depends on full distributions, influential examples, structural motifs, or fragile temporal feedback loops, the same pruning may become destructive.

In that sense, the pruning interaction paradox is a boundary object linking compression, robustness, evaluation, and scientific interpretation. It names the recurring failure of compression claims that are valid in one slice of model behavior but invalid when transferred to another.