Create and explore imaginative 3D worlds with our new product Dreamfield (opens in a new tab)

The Surprising Effectiveness of Deleting Weights in LLM Post-Training

This lightning talk explores a counterintuitive finding: you can adapt large language models by learning which weights to delete rather than which new values to learn. Deletion Tuning freezes pretrained parameters and learns only binary masks that zero out fewer than 0.05% of a model's weights, yet matches full fine-tuning performance across reinforcement learning, supervised fine-tuning, and distillation while storing adapters thousands of times smaller and avoiding catastrophic forgetting.
Script
What if the key to adapting a language model isn't learning new parameter values, but simply deciding which existing weights to delete? The authors demonstrate that zeroing fewer than 0.05 percent of a model's weights can match the performance of full fine-tuning.
Deletion Tuning freezes every pretrained weight and introduces trainable binary masks exclusively on the gate projection matrices in each transformer block. During training, the method learns which weights to retain and which to set exactly to zero, writing no new nonzero values anywhere in the model.
The compression is dramatic. On Llama 8 billion trained with reinforcement learning, the final mask stores only 8.4 megabytes while achieving 83.55 percent accuracy on grade-school math, slightly exceeding full fine-tuning's 83.24 percent. The full checkpoint is 30,633 megabytes, yielding a compression ratio exceeding 3,600 to 1.
The retention result is the most surprising finding. Deletion Tuning maintains forward KL below 0.05 and loses almost no accuracy on six prior tasks, while full fine-tuning drifts to KL values of 1.5 and drops up to 3.1 points. Even random sparse fine-tuning at comparable sparsity forgets more, suggesting that the constraint matters: setting selected values to zero, not learning arbitrary deltas.
Why does deletion approximate full fine-tuning? The learned masks produce updates with stable rank between 66 and 93, intermediate between LoRA's near-rank-one constraint and random sparse updates. Their spectrum drift stays within 1.4 times that of full fine-tuning, and they exhibit the same sub-random overlap with principal pretrained weights that characterizes effective reinforcement learning updates.
The method leaves open whether its low policy drift translates to sequential multi-task learning, whether deletion works equally well on attention projections or non-gated architectures, and why zeroing is preferable to learning small signed updates at the same coordinates. Still, the core result stands: under the right conditions, you can teach a language model new capabilities by learning only which pretrained knowledge to remove. Explore the full paper and create your own research videos at emergentmind.com.