Predicting LLM Compression Degradation from Spectral Statistics

Abstract: Matrix-level low-rank compression is a promising way to reduce the cost of LLMs, but running compression and evaluating the resulting models on language tasks can be prohibitively expensive. Can compression-induced degradation be predicted before committing to this compute? We systematically analyze the Qwen3 and Gemma3 model families across four representative low-rank compression methods: vanilla SVD, two ASVD variants, and SVD-LLM. We find that stable rank and information density, measured in bits per parameter, dominate performance degradation. The interaction term $γ\cdot \barρ_s$, defined as compression ratio times stable rank, is a robust predictor of accuracy degradation, achieving leave-one-out cross-validation Pearson correlations of $0.890$ for attention layers and $0.839$ for MLP layers. We provide theoretical intuition for why this predictor succeeds by connecting it to standard SVD truncation bounds and error composition mechanisms in transformer layers. These findings enable a predict-then-compress workflow: compute $γ\cdot \barρ_s$ from weights, estimate degradation, and invest compute only in desirable configurations.

• The paper demonstrates that the joint effect of compression ratio and stable rank robustly predicts LLM accuracy degradation.
• It introduces a regression framework that uses symbolic regression to derive compact, interpretable formulas for forecasting performance loss.
• It highlights the architectural impact, showing that attention layers degrade more predictably than MLP layers under low-rank compression.

Predictive Spectral Statistics for LLM Compression Degradation

Motivation and Problem Formulation

Low-rank matrix decompositions are a principal mechanism for reducing parameter counts and computational footprint in LLMs. The process of compressing and evaluating different configurations is computationally prohibitive: practitioners contend with a costly compress-evaluate-adjust loop, lacking an efficient, principled methodology to forecast accuracy degradation given a compression specification. The central research question addressed is: Can model performance loss due to low-rank compression be predicted solely from pre-compression spectral statistics of weight matrices and configuration parameters—prior to committing expensive compute?

Overview of Approach

The study systematically analyzes Qwen3 and Gemma3 model families, applying four representative SVD-based compression approaches: vanilla SVD, two variants of activation-aware SVD (ASVD), and SVD-LLM. The predictive framework correlates spectral quantities (specifically stable rank and effective rank) and information density with downstream model accuracy and perplexity. A catalog of 42 interpretable formula templates, augmented by formulas discovered via symbolic regression, forms the basis of regression-based predictive modeling. The Pearson correlation of predicted degradation with actual accuracy loss is assessed using leave-one-out cross-validation.

Spectral Statistics and Compressibility

Stable rank $\rho_s(W) = \|W\|_F^2/\|W\|_2^2$ and effective rank $\rho_{\text{eff}}(W) = \exp(H(W))$ are employed as continuous, noise-robust measures of intrinsic matrix dimensionality; aggregation across layers is parameter-weighted to yield a single model-level predictor $\bar{\rho}_s$. The compression ratio $\gamma$ (retained parameter fraction), information density $\mathcal{B}$ (bits-per-parameter, estimated via minimum description length on weight distributions), and interaction terms (notably $\gamma \cdot \bar{\rho}_s$) are tested for their explanatory power.

Empirical Findings: Dominant Predictors

The principal empirical result is that the product $\gamma \cdot \bar{\rho}_s$ robustly predicts accuracy degradation across families, compression methods, and tasks, with leave-one-out Pearson correlation $0.89$ for attention layers and $0.84$ for MLP layers. Layer-specific predictors consistently outperform method-specific predictors, indicating architectural operation type is more decisive than algorithmic nuances. Spectral properties and information density dominate variable importance rankings; log-ratio, bits-to-compression, and inverse-density formula structures are repeatedly rediscovered by symbolic regression.

Perplexity degradation is less predictable in aggregate ($r=0.093$), but improves markedly with layer-specific calibration ($\rho_{\text{eff}}(W) = \exp(H(W))$0 for SVD-LLM on MLP). Perplexity-accuracy coupling depends on benchmark scoring protocols: sequence-scored tasks (HellaSwag, PIQA) yield high correlation ($\rho_{\text{eff}}(W) = \exp(H(W))$1); single-token scoring and AFLITE adversarial filtering (BoolQ, WinoGrande) disrupt the relationship.

Theoretical Analysis: Linear Algebraic Bounds

The paper delivers theoretical justification for the dominance of spectral rank predictors. The Frobenius norm error bound after SVD truncation reveals intrinsic resistance to compression is governed by spectral rank:

$\rho_{\text{eff}}(W) = \exp(H(W))$2

The compression impact scales jointly with $\rho_{\text{eff}}(W) = \exp(H(W))$3 (compression aggressiveness) and $\rho_{\text{eff}}(W) = \exp(H(W))$4 (dimensionality resistance). The model-level aggregation of this effect is justified by the parameter-weighted sum heuristic.

Architectural analysis explains the predictability gap between attention and MLP layers: attention error propagates via tensor contractions, composed through spectral norms—yielding robust, data-agnostic scaling laws. MLP errors (SwiGLU) propagate through Hadamard products involving activation correlations, introducing input-dependent, less predictable degradation. Synthetic experiments confirm rates: attention layers degrade $\rho_{\text{eff}}(W) = \exp(H(W))$5 more rapidly than MLP layers under aggressive compression.

Implications for Compression Practice

These findings enable a predict-then-compress workflow: compute compression ratio and stable rank from model weights, use concise formulaic scaling laws (primarily $\rho_{\text{eff}}(W) = \exp(H(W))$6) to estimate accuracy loss, and restrict expensive evaluation to configurations predicted to exhibit tolerable degradation. This increases the efficiency of compression research by drastically reducing wasted compute cycles.

The theoretical framework suggests the applicability of spectral scaling laws to other forms of compression (quantization, pruning) but cautions that architectural operation type—specifically, layer interaction modality—needs to be calibrated. For transformers, spectral-rank-based predictors are foundational.

Numerical Validation and Symbolic Regression

Formula discovery via symbolic regression independently identifies bits-to-compression ratio, log-ratio normalization, and inverse-density scaling as high-performing predictors, further validating the generality of the spectral-statistics-driven approach. Nonlinear, task-dependent functional forms are evident, but parsimony (compactness of formulas) maximizes generalization.

Limitations and Future Directions

The study is limited to Qwen3 and Gemma3 architectures; broader applicability requires systematic cross-family validation. Bootstrap confidence intervals and baseline comparisons with trivial functional forms would strengthen statistical claims. Extension of predictive scaling laws to quantization, pruning, and non-transformer architectures remains an open avenue.

Further theoretical work is warranted to develop closed-form error propagation bounds for nonlinear operations and to elucidate the interplay between compression, information density, and data complexity (entropy).

Conclusion

The interaction between compression ratio and stable rank is the dominant, robust predictor of LLM performance degradation under low-rank compression. Layer operation type, particularly the distinction between attention contractions and MLP Hadamard products, accounts for predictability differences and should be considered when devising compression strategies. These results lay the foundation for principled, efficient model compression through spectral-statistics-driven prediction, obviating expensive trial-and-error evaluation loops and informing future scaling law research for LLMs.

---

Reference: "Predicting LLM Compression Degradation from Spectral Statistics" (2604.18085)