Low-loss preparation of existing checkpoints for low-rank compression

Develop a method for preparing an existing conventional-ell_2-weight-decay neural-network checkpoint for low-rank compression without causing a substantial increase in model loss.

Background

The paper introduces spectral weight decay, which replaces conventional squared-Frobenius weight decay with a decoupled nuclear-norm update that reduces the effective rank of neural-network weight matrices. The main experiments apply this regularization during pretraining, where it improves subsequent SVD-based compression at matched validation loss.

The authors investigate whether the same rank reduction can be induced during a short fine-tuning stage starting from an existing checkpoint trained with standard ell_2 weight decay. Their experiments on a 124M-parameter LLaMA model show that fine-tuning requires much larger spectral regularization coefficients and incurs substantially more loss at a given effective rank than applying spectral weight decay during pretraining. Thus, obtaining substantial rank reduction from an already trained checkpoint while preserving model quality remains unresolved.