Self-Improvement Without Labels: On-Policy Self-Training
This presentation explores a data-free approach to improving language and vision-language models through On-Policy Self-Training (OPST). Starting only with coarse topic descriptors like 'Abstract Algebra,' models generate their own questions, reasoning traces, and answers, then use these as privileged context for an exponential moving average teacher that guides an on-policy student. The method achieves substantial gains—including a 10.2-point MMLU improvement on small Gemma models—by exploiting latent capabilities already encoded in model parameters, without requiring curated demonstrations or externally verified answers.Script
What if a language model could teach itself by generating its own homework, grading its own answers, and repeating the cycle without a single human label? That is the promise of On-Policy Self-Training, a data-free method that transforms coarse topic descriptors into measurable capability gains.
The process begins with nothing more than topic names. From these descriptors, the model generates questions, reasoning traces, and candidate answers. Two filters then remove unreliable samples: pseudo-verification tests whether answers are reproducible after paraphrasing, and a consensus gate discards problems the student already solves consistently, keeping only the frontier where learning can occur.
The learning signal comes from asymmetric information. The student sees only the question and samples its own response. The teacher, an exponentially smoothed copy of the student, also sees the generated rationale and candidate answer as privileged context. It uses this extra information to complete reasoning steps and provide token-level guidance that the student refines through on-policy distillation.
On Gemma models, OPST produces double-digit gains. MMLU improves by 10.2 points, HellaSwag by 7.5 points, and the method consistently outperforms supervised fine-tuning and reinforcement learning trained on identical synthetic data. Remarkably, OPST recovers 88 percent of the improvement achieved by fully supervised on-policy distillation, which relies on curated questions and verified answers.
The same idea extends to vision. The model generates image-grounded questions and uses scene captions or detected object locations as privileged teacher context. On VQAv2, this visual variant improves accuracy by 6.9 points and reduces object hallucination, demonstrating that self-improvement is not confined to text.
On-Policy Self-Training reveals that models already contain knowledge they struggle to express. By treating synthetic reasoning as a hint rather than gospel, the method amplifies latent capabilities without external labels. Yet it cannot conjure facts absent from parameters, and pseudo-verification ensures only consistency, not correctness. To explore how these ideas might scale and where they break, visit EmergentMind.com, where you can create your own videos and dive deeper into the frontier of self-improving AI.
