Incorporate interactions among jointly removed dimensions

Develop covariance-aware scoring or greedy joint selection for PARSER that captures pairwise interactions among multiple hidden dimensions removed during residual sparsification, including error reinforcement and cancellation effects that are ignored by PARSER’s per-dimension importance scores.

Background

PARSER estimates the output importance of each hidden dimension independently and removes multiple low-scoring dimensions in a single selection step. The resulting objective accounts for each dimension’s individual output error but omits pairwise cross terms between removed dimensions, even though those terms can cause errors to reinforce or cancel one another.

The paper explains that estimating all pairwise interactions would make one-shot selection substantially more expensive. It therefore uses a diagonal approximation, while identifying covariance-aware scoring and greedy joint selection as concrete unresolved alternatives for more accurately modeling the combined effect of removed dimensions.

References

Our experiments suggest that the diagonal score is already effective in practice, but incorporating such interactions through covariance-aware scoring or greedy joint selection remains future work.

— Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMs  (2609.00575 - Jung et al., 1 Sep 2026) in Section Limitations, subsection “Per-dimension importance”