Training Foundation Models with a Single Algorithm

Abstract: Foundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning (SFT), and reinforcement learning (RL), with changes in both data and learning algorithms. We show that this algorithmic separation is not fundamental. By deriving suitable RL objectives as maximum-likelihood fitting to model-generated target distributions, we place all of these stages under a single likelihood-based training algorithm, unified training. This perspective turns the foundation-model training lifecycle into a problem of choosing which data to train on, in what proportions, and when, with conventional staged training as a special case. We instantiate this framework in a unified training system that mixes static data and RL generations within a single trainer. Across controlled and real-world experiments, unified training improves the accuracy-compute tradeoff and task coverage while preserving broader capabilities, and provides a stronger starting point for continual learning. This work lays the groundwork for a unified science of foundation-model training, connecting mathematical principles, training infrastructure, and the development of model capabilities.

• Training Foundation Models with a Single Algorithm presents a unified approach that combines pretraining, reinforcement learning (RL), and supervised fine-tuning into a single weighted maximum-likelihood training process.
• The empirical study evaluates unified training with two models demonstrates that overlapping training distributions improves final capability and data usage efficiency.
• Integrating midtraining data into pretraining enhances model capability retention and subsequent adaptation to new tasks, outperforming staged training at large and small compute budgets.

Central thesis and scope

The paper argues that the conventional separation between pretraining, midtraining, SFT, and RL is algorithmic rather than fundamental. Its central proposal, unified training, expresses these phases as a single weighted maximum-likelihood procedure over an evolving mixture of static examples and model-generated responses (2610.40564). The learning rule remains fixed; only the source distributions, mixture weights, and timing of data exposure change. Under this formulation, staged training is recovered as the degenerate case in which one source receives all probability mass during each stage.

The claim is not that all RL algorithms are literally equivalent to supervised learning. Rather, the paper identifies a class of policy-gradient objectives whose expected updates can be represented as likelihood fitting to a reward-conditioned target distribution. This distinction is important: the framework accommodates MaxRL, REINFORCE, and TailRL under explicit assumptions, but does not automatically encompass PPO or GRPO with arbitrary advantage normalization, clipping, or KL constraints.

The empirical study evaluates whether overlapping training distributions improve final capability, compute efficiency, retention, and subsequent adaptation. It uses two settings: a controlled reasoning-puzzle benchmark with a 124M-parameter model, and a heterogeneous mathematical-RL experiment with a 756.8M-parameter model. The results consistently show that data composition and timing matter jointly with the RL objective. They do not establish a universally optimal mixture.

Likelihood formulation of RL

For static data, the formulation is standard. A source is defined by a prompt distribution and a conditional completion distribution, and training minimizes the negative log-likelihood of sampled prompt–completion pairs. Pretraining, midtraining, and SFT differ primarily in their data distributions and token masks.

The key theoretical step is the likelihood interpretation of MaxRL. For binary rewards, let the model’s success probability on prompt $x$ be $p_\theta(x)$. The policy gradient of the log success probability can be written as an expectation over the model distribution conditioned on successful completions:

$$\nabla_\theta \log p_\theta(x) = \mathbb{E}_{y\sim \pi_\theta^+(\cdot\mid x)} \left[ \nabla_\theta \log \pi_\theta(y\mid x) \right].$$

Consequently, an RL update can be implemented by fitting the current model to its own success-conditioned distribution $\pi_{\theta_t}^+$, with the target held fixed during optimization. Static data and successful RL generations can therefore be combined in one objective:

$$\mathcal{L}(\theta;q_t) = -\mathbb{E}_{(x,y)\sim q_t} \left[ \log \pi_\theta(y\mid x) \right],$$

where $q_t$ is a weighted mixture of source distributions.

This interpretation also clarifies the role of different RL objectives. MaxRL uses the original prompt distribution and conditions completions on success. REINFORCE uses the same binary success-conditioned completion distribution but reweights prompts by their success probability. For continuous rewards, TailRL constructs a mixture of distributions conditioned on exceeding reward thresholds. These are different target and prompt distributions, even though they share the same likelihood trainer.

The framework permits generations from a policy $\nu$ distinct from the policy being optimized. In that case, the trainer fits $\nu^+$ rather than the current-policy target. This enables asynchronous generation and optimization, but the resulting update is generally off-policy and is not equal to the current policy’s MaxRL gradient. The distinction is especially consequential because the implementation uses stale generations: in the real-world experiments, inspected generator policies lagged the trainer by approximately 12–44 optimizer updates.

Unified implementation

The implementation represents every training example as a token sequence with nonnegative per-token weights. Pretraining and document-continuation data receive unit weights. SFT masks prompt tokens and trains only on response tokens. RL examples assign nonzero weights to successful responses and zero weights to unsuccessful responses. The trainer applies one token-weighted cross-entropy objective to the resulting batch.

For a prompt with $N$ sampled completions, the binary-reward implementation estimates the pass rate $\hat p$ and assigns each successful response weight $p_\theta(x)$0. This yields an empirical success-conditioned target. If no sampled completion is correct, the prompt contributes no positive RL target, although unsuccessful response tokens remain part of the normalization denominator.

The system separates generation and optimization asynchronously. A generator samples responses and appends them to a shared buffer containing both RL generations and static data. The trainer samples from this buffer, updates the model, and periodically publishes new weights. This architecture is operationally important: the unified method is not merely a reparameterization of a synchronized training loop, but a common infrastructure for heterogeneous data streams.

In the controlled experiments, the shared loss uses a token mixture directly. In an alternative baseline, RL and midtraining losses are normalized separately and combined with a fixed coefficient. The paper finds that the shared normalization is competitive and often achieves the best aggregate result, but it is not uniformly superior. Separately normalized objectives frequently perform better on the primary Extended reasoning task, and their optimal coefficient varies across initializations and data mixtures. Thus, unified loss aggregation reduces a tuning dimension but does not dominate every manually weighted alternative.

Controlled reasoning experiments

Experimental design

The controlled setting isolates transfer from demonstrated reasoning operations to novel compositions. A 124M-parameter model is pretrained on 20B tokens and receives a 510M-token corpus of oracle-generated reasoning traces. The corpus contains puzzles with six to nine people and demonstrates individual reasoning operations, while RL introduces compositions of these operations that are absent from midtraining.

The Base setting evaluates puzzles with eight to nine people. The Extended setting introduces ten- and eleven-person puzzles during RL. Extended generalization evaluates twelve-person puzzles absent from both midtraining and RL. RL uses 6,000 updates for Base and 12,000 for Extended, with 512 sampled responses per update and 16 responses per prompt. Performance is reported as pass@1 and pass@16, averaged over two RL seeds.

Unified configurations mix midtraining traces into the final 8,147 pretraining updates and optionally continue mixing them into RL. The study varies pretraining exposure from approximately 133M to 2.14B midtraining tokens and adds between 4 and 64 midtraining examples per RL update. Staged baselines use one, two, or four passes over the same corpus, at two learning rates.

Performance, generalization, and reuse

The best unified configurations outperform the selected staged baseline on final Base and Extended performance and also improve Extended generalization. The result is significant because the generalization test requires larger puzzles than those seen during either midtraining or RL. The paper therefore reports an advantage that cannot be attributed solely to memorizing the RL prompt distribution.

The strongest benefit arises from exposing the model to midtraining data during the latter portion of pretraining. The two best configurations reuse the fixed midtraining corpus for approximately two and four passes during pretraining. Midtraining data mixed during RL also helps, particularly in Extended and Extended generalization, although its effect is less monotonic in Base.

Figure 1: Unified training combines static data and RL generations under one weighted likelihood loss and attains a stronger accuracy–coverage frontier than the tested staged configurations.

The result contradicts a simple interpretation of midtraining as a one-time preparatory phase. In the staged sweep, increasing the number of midtraining passes beyond the best configuration reduces final RL accuracy. Under unified training, repeated exposure can improve subsequent RL, provided that it is distributed across the training trajectory rather than concentrated into a separate phase.

Compute allocation

The compute-matched analysis includes pretraining, midtraining, RL generation, and RL optimization. This is essential because unified configurations process additional static examples and therefore cannot be compared solely by equal RL update counts.

Staged training is compute-optimal for accuracy at some intermediate budgets and has a stronger coverage frontier in parts of the in-distribution evaluation. However, unified training reaches higher accuracy at the smallest and largest tested budgets and is generally superior on Extended generalization. The paper’s strongest compute claim is therefore conditional rather than universal: unified training is more compute-efficient in several important regimes, especially at larger budgets and under distribution shift, but staged training remains competitive at intermediate in-distribution budgets.

Figure 2: Accuracy and coverage as functions of cumulative training compute, with unified and staged configurations compared across Base and Extended reasoning tasks.

The implication is that the best training organization depends on the available compute budget and evaluation objective. Selecting a single final checkpoint from a single recipe obscures this distinction; the paper instead constructs empirical frontiers over both configuration and RL duration.

Retention during initialization

Mixing midtraining data into pretraining preserves general capabilities substantially better than a separate midtraining phase. Across unified initializations, held-out pretraining loss remains close to the pretraining-only model: the reported mean loss ranges from 2.537 to 2.540 compared with 2.537 for the reference. The mean score over five general benchmarks remains within 0.52 percentage points of the pretraining-only model.

By contrast, every staged initialization increases held-out pretraining loss and lowers the aggregate benchmark score. The staged configuration selected for the best final RL accuracy is not the one with the best retention. This separation between immediate reasoning performance, retention, and downstream RL performance is one of the paper’s more important empirical observations.

Figure 3: Unified mixing preserves pretraining loss and general benchmark performance more effectively than the evaluated staged midtraining configurations.

The result does not imply uniform preservation across all capabilities. Staged midtraining improves TruthfulQA in the reported benchmark set while reducing MMLU, ARC-Easy, HellaSwag, and WinoGrande. The retention advantage is therefore an aggregate statement about the chosen metrics, not a claim that unified training dominates on every individual capability.

Real-world heterogeneous midtraining

Data mixture and capability trade-offs

The second experimental suite starts from a 756.8M-parameter model pretrained on 354.5B tokens and midtrained on 17.73B Dolmino tokens. Mathematical RL uses 11,977 filtered prompts. The additional 240M-token corpus contains web text, mathematics, code, question answering, and reasoning traces. Unified runs mix this corpus into mathematical RL at constant proportions of 0.1%, 0.2%, 0.5%, 1%, 2%, or 5%; the staged baseline consumes it before RL.

Mathematical pass@1 remains remarkably stable across mixture proportions. The reported Math pass@1 values range from 24.27% at 0.5% mixing to 24.80% at 5%, while the staged reference reaches 24.17% under the corresponding averaged benchmark protocol. The more pronounced effect concerns coverage and retention. At 5% mixing, mathematical pass@32 reaches approximately 69.0%, compared with 68.6% for the zero-mixing control and 68.4% for the staged baseline. Held-out corpus loss improves monotonically with mixture proportion, reaching a change of $p_\theta(x)$1 nats/token at 5%.

The trade-off is coding accuracy. Coding pass@1 declines as more heterogeneous midtraining data is retained during mathematical RL, although coding coverage improves relative to the RL-only control at every nonzero proportion. Thus, the same mixture can reduce one-shot coding reliability while increasing the probability that at least one of many samples is correct.

Figure 4: Increasing the midtraining proportion improves retention and often coverage, while producing task-specific accuracy trade-offs.

These results reject the idea that retention can be characterized by a single scalar objective. The mixture changes the distribution of failure modes: it can improve coverage and general knowledge while degrading pass@1 on another specialized task.

Scheduling exposure

The paper next fixes the average configured mixture at 0.5% and changes its temporal schedule. Constant mixing, full annealing, linear growth, and half annealing produce similar mathematical performance. Half annealing, which starts at 2% and reaches zero halfway through RL, produces the highest reported Coding pass@1, 18.85%, compared with 16.51% for constant 0.5% mixing and 17.76% for full annealing. It also reaches Coding pass@32 of 46.93%, versus 46.38% for constant mixing.

The improvement comes with weaker retention: the held-out loss change under half annealing is $p_\theta(x)$2 nats/token, compared with $p_\theta(x)$3 for constant mixing. The timing of data exposure therefore matters even when the average configured proportion is held fixed.

Figure 5: Data schedules with the same average configured proportion yield different trade-offs among coding accuracy, coverage, and held-out corpus loss.

The implication is direct: mixture optimization must include temporal allocation, not merely total data volume. Early exposure appears more favorable for later specialized accuracy, whereas continued exposure better preserves the midtraining distribution. The experiments do not determine whether this pattern is due to representation formation, optimization interference, or the interaction between static examples and the evolving RL policy.

RL objective choice

MaxRL and REINFORCE produce different capability profiles under the same unified data mixture. At 2% constant mixing, REINFORCE obtains Math pass@1 of 24.81%, compared with 23.99% for MaxRL. MaxRL obtains higher Math pass@32, 69.61% versus 67.70%. REINFORCE also gives lower held-out corpus loss, $p_\theta(x)$4 versus $p_\theta(x)$5 nats/token, and higher SciQ and ARC-Challenge accuracy.

This comparison is methodologically important because it shows that the mixture alone does not determine the outcome. The RL target distribution and effective prompt weighting remain consequential. Unified training supplies a common implementation, but it does not collapse MaxRL and REINFORCE into the same algorithm.

Continual learning

The strongest downstream result concerns subsequent coding RL. After mathematical training, all models receive the same coding RL procedure without additional midtraining data. The staged mathematical model reaches 22.94% HumanEval+ pass@1 and 29.81% MBPP+ pass@1 after 4,000 coding updates. The model previously trained with 5% unified mixing reaches 24.96% and 32.69%, respectively. Its pass@16 scores are 46.95% on HumanEval+ and 58.47% on MBPP+, compared with 45.12% and 52.12% for the staged model.

Figure 6: Unified mathematical training produces stronger subsequent coding-RL performance than staged mathematical training under an identical coding-RL procedure.

The same pattern appears in a WebInstruct continuation experiment. Unified models initially have lower accuracy but surpass the staged model after further RL on several overall and free-form metrics. The result supports a specific interpretation: unified training can produce an initialization that is less specialized to the preceding task and consequently more amenable to later adaptation. It does not establish that unified training universally improves continual learning, since the advantage varies by mixture proportion, answer type, and coverage metric.

Limitations and open questions

The empirical scope is limited. Most controlled experiments use one RL task at a time, small or medium model sizes, fixed corpora, and binary verifiable rewards. The real-world study uses a 0.76B-parameter model and compares equal optimizer-update budgets rather than fully compute-matched training. The paper therefore does not establish scaling laws that predict the optimal mixture as model size, corpus size, or RL budget changes.

The controlled pretraining mixtures also change batch size without retuning the learning rate or batch-size-dependent optimization parameters. The authors explicitly defer this interaction. In addition, unified training often uses stale generations, and the exact age distribution of consumed samples is not logged. Some comparisons therefore conflate the effect of data mixing with asynchronous execution, baseline subtraction, and implementation differences.

The theoretical framework has a defined scope. Its likelihood equivalence applies to fixed generation distributions and suitable policy-gradient objectives. PPO and GRPO mechanisms involving empirical advantages, clipping, policy ratios, or KL penalties are not automatically represented by the same fixed-target likelihood objective. Continuous rewards require additional estimators, and finite-sample behavior can differ materially from population-gradient identities.

Finally, the experiments do not identify a generally optimal schedule. They reveal trade-offs among pass@1, pass@k, held-out loss, general knowledge, and downstream adaptation, but no single mixture dominates all metrics. Joint multi-task training and longer sequences of continual-learning stages remain untested.

Conclusion

"Training Foundation Models with a Single Algorithm" (2610.40564) presents a coherent mathematical and systems-level account of foundation-model training in which static data, demonstrations, and reward-selected generations are handled by a shared weighted likelihood trainer. Its empirical evidence indicates that overlapping data sources can improve accuracy–compute trade-offs, generalization, retention, and later adaptation relative to the evaluated staged baselines.

The main contribution is not the claim that stages disappear operationally. Rather, it is the redefinition of the training-design problem: algorithmic continuity permits systematic study of data composition, reuse, timing, and compute allocation across the full trajectory. The evidence supports unified training as a competitive and flexible alternative to staged pipelines, while leaving the predictive theory of mixture schedules, optimization adaptation, and multi-task continual learning unresolved.