Create a Video View Paper

Training Foundation Models with a Single Algorithm

This presentation examines a rethinking of how we train foundation models. Instead of treating pretraining, instruction tuning, and reinforcement learning as separate stages requiring different algorithms, this work shows they can be unified under a single weighted likelihood objective. The key insight is that what changes across training is not the learning rule itself, but the mixture of data sources, their weights, and their timing. We explore the theoretical foundation of this approach, its practical implementation, and experimental evidence showing that carefully overlapping training phases can improve accuracy, generalization, retention, and later adaptability compared to traditional staged pipelines.

Script

Foundation models are typically trained in distinct stages: pretraining on raw text, then instruction tuning, then reinforcement learning. This paper from researchers at several institutions asks a provocative question: what if these stages are not fundamentally different algorithms, but simply different mixtures of data flowing through the same learning procedure?

The authors show that maximum-likelihood reinforcement learning can be expressed as fitting the model to its own success-conditioned distribution. This means pretraining, supervised tuning, and certain RL objectives all reduce to weighted likelihood on an evolving mixture of sources. The algorithm stays fixed; only the mixture weights and data timing change.

In controlled reasoning experiments with a 124 million parameter model, unified training outperformed staged baselines on both final accuracy and generalization to harder puzzles. The strongest configurations mixed reasoning traces into late pretraining and continued mixing them during reinforcement learning, reusing the corpus multiple times across the trajectory.

The real-world experiments with a 756 million parameter model revealed nuanced capability trade-offs. Increasing the proportion of heterogeneous data during mathematical reinforcement learning preserved general knowledge and improved coverage, but sometimes reduced coding pass at 1 accuracy. Retention cannot be captured by a single number; the mixture reshapes the distribution of what the model remembers and forgets.

Perhaps the most striking result concerns continual learning. After mathematical training, models underwent identical coding reinforcement learning. The model trained with 5 percent unified mixing reached 24.96 percent HumanEval pass at 1, compared to 22.94 percent for the staged baseline. Unified training produced an initialization less specialized to the preceding task and more adaptable to new objectives.

This work reframes the training design problem as one of data composition and timing across the full trajectory, rather than stage boundaries. The experiments reveal that mixture optimization matters, but they do not yet provide a predictive theory of optimal schedules. To explore the full design space and create your own video summaries of cutting-edge research like this, visit EmergentMind.com.