Determine the cause of progressive pruning degradation

Investigate whether irreversible mask allocation, omitted residual terms, or other interactions cause the observed degradation of progressive QK-Wanda pruning relative to one-shot pruning, and determine whether other adaptive pruning algorithms can avoid this degradation.

Background

The paper evaluates progressive QK-Wanda pruning, in which query and key scores are recomputed after successive portions of the pruning mask are applied. The authors explain that these refreshed scalar scores account for the local squared change from the next deletion but omit residual cross terms against the accumulated reconstruction error, while simultaneous deletions also introduce interactions between deleted weights and between the query and key projections.

In the reported experiments, additional progressive-pruning steps generally failed to improve perplexity and often worsened it under separate matrix-wide budgets. The authors explicitly state that the experiments do not determine whether this behavior results from irreversible mask allocation, omission of residual terms, or other interactions, nor whether alternative adaptive algorithms could perform better.