Reliable early prediction of the compact skill bank

Determine whether training statistics collected during an early stage of skill-guided distillation can reliably predict the compact skill bank that would be selected using statistics from the full training run.

Background

SGUID normally selects skills using training statistics gathered over a full 200-step distillation run with the candidate skill bank. The paper evaluates whether the selection process can be made less expensive by using only statistics from the first 25 training steps.

The early-training proxy performs comparably to full-run selection for Qwen3-1.7B but is substantially less reliable for Qwen3-4B, where it trails the full-run selection by 1.7 mean avg@12 points. The authors therefore leave open whether early training information can consistently identify the skills that remain useful throughout distillation.

References

These results suggest that early training statistics contain useful information, but that reliably predicting the compact skill bank from a much earlier training stage remains an open direction for future work.

— Which Skill to Distill? SGUID: Selecting a Compact Skill Bank for Model-Skill Co-Evolution  (2610.12367 - Liu et al., 8 Oct 2026) in Section 5, “Ablation Studies and Discussion”