SlimQwen: Compressing Giant Mixture-of-Experts Models Without Losing Their Edge
This presentation explores a comprehensive study on compressing large Mixture-of-Experts language models through structured pruning and knowledge distillation during pretraining. We examine how the authors achieve 4x compression while retaining 86.5% of teacher performance, uncovering critical insights about progressive pruning schedules, hybrid distillation objectives, and expert merging strategies that enable efficient deployment of massive models on constrained hardware.Script
Mixture-of-Experts models can have 80 billion parameters, but what if you could compress them down to 23 billion and keep almost all their intelligence intact? The SlimQwen study demonstrates exactly that, achieving 4x compression while recovering 86.5% of the original model's performance.
The authors discovered that pruning a pretrained model and continuing training beats building a small model from scratch by a stunning margin. Under identical compute budgets, the pruned initialization achieves up to 11.79 points higher performance on benchmarks like MMLU, because it inherits crucial weight structure and inductive biases that random initialization must learn painfully from zero.
When compressing the expert layers, one surprising finding emerged: after extensive pretraining, different one-shot expert pruning methods perform similarly. But the partial-preservation merging strategy, where half of each expert is kept before merging, consistently wins by preventing the remaining experts from becoming too homogeneous and losing their specialized knowledge diversity.
The research challenges conventional wisdom by showing that pure knowledge distillation is not enough. Hybridizing distillation with standard language modeling loss produces superior results, and extending distillation to predict multiple future tokens simultaneously, rather than just the next one, enhances both accuracy and speculative decoding efficiency with measurably higher token acceptance rates.
Progressive pruning emerges as the critical insight: gradually removing capacity in multiple stages, with training between each stage, dramatically outperforms one-shot compression. A depth-first progressive schedule improved MMLU from 75.86 to 77.39, because the gradual approach allows smoother optimization and prevents catastrophic forgetting that happens when you cut too much at once.
These compression strategies deliver real deployment advantages: faster training, faster inference, reduced memory, and models that fit on single devices instead of requiring massive clusters. You can explore the full SlimQwen methodology and create your own video explanations of compression research at EmergentMind.com, where cutting-edge model efficiency meets accessible learning.
