Develop a theory of middle-layer vulnerability and principled sparsity allocation

Develop a formal theoretical account of how internal representations evolve with depth, explain why middle layers exhibit heightened sensitivity to pruning, and use that account to derive principled rather than heuristic layer-wise sparsity allocations.

Background

The experiments find that middle layers are more vulnerable to pruning than early or late layers, while the proposed layer-wise sparsity schedule is presented only as a preliminary heuristic. The paper hypothesizes that accumulated upstream perturbations propagated through residual connections contribute to this pattern, but does not provide a formal explanation.

The unresolved problem is therefore twofold: characterize the depth-dependent evolution of internal representations sufficiently to explain middle-layer sensitivity, and translate that characterization into a theoretically justified sparsity-allocation rule rather than an empirically motivated schedule.

References

In Theorem~5, the authors provide an estimate for the accuracy of the resulting network approximation in terms of the fidelity of the retrained network. We hypothesize that a similar result holds in the deep-network case, which we leave for future research.

— Layerwise Decoupling for Stable Structured Sparsification of Fully Connected Layers  (2609.21126 - Kulick et al., 17 Sep 2026) in Section 3, immediately following Theorem 3.3 (the theorem labeled \ref{thm:deep_equiv})

The normalization preserves the function only for a positively homogeneous activation, which OPT provides through ReLU; most recent LLMs use GELU or gated activations, for which the algorithm can still be run but neither the equivalence of Section~\ref{sec:theory} nor the invariance of Section~\ref{sec:exp-robust} applies, and whether the robustness carries over is untested.

— Layerwise Decoupling for Stable Structured Sparsification of Fully Connected Layers  (2609.21126 - Kulick et al., 17 Sep 2026) in Section 6.6, Scope and limitations

Finally, while the middle-layer sensitivity finding is empirically robust, our framework does not yet explain it theoretically. A formal account of how the structure of internal representations evolves with depth, and how this motivates principled rather than heuristic sparsity allocation, remains an open problem.

— When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs  (2608.25941 - Gupte et al., 26 Aug 2026) in Section 5, “Limitations and Future Work” (Section \ref{sec:limitations})

One possible account is that removing a contiguous sub-block creates one residual-stream discontinuity whereas dispersed removal creates two. This explanation is untested; causal activation analysis and a larger factorial candidate set are needed.

— X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation  (2609.11412 - Zhang et al., 10 Sep 2026) in Appendix, Section 5, “Layer-Interaction Details” (label sec:interaction_detail)

A finer $k$-grid (e.g.\ $k \in {0,1,2,3,4,5,6,8}$) would reduce the granularity-induced noise in the fraction metric and is left as future work.

— The Residual Stream's Effective Depth  (2609.31098 - Gahtan et al., 25 Sep 2026) in Appendix S19, “Pruning Boundary Checks,” subsection “Reading”