When One Model Isn't Enough: Scaling with Ensembles and Smart Hyperparameters
This presentation examines a critical challenge in training language models when unique training data runs out. The authors demonstrate that existing data-constrained scaling laws can predict worse performance with more compute—a pathology they trace to two missing pieces: hyperparameter transfer rules that adapt weight decay and learning rate across model size and repetition count, and ensemble size as an explicit compute-allocation variable. By fitting empirical power laws for hyperparameters and extending scaling laws to include independent ensemble members, they achieve monotonic improvements and reduce perplexity by 14% on a fixed 100 million token budget.Script
Some data-constrained scaling laws predict a paradox: give them more compute, and validation loss actually gets worse. The authors trace this pathology to two missing ingredients in how we allocate training resources when unique tokens run out.
Once you have exhausted your unique training tokens, additional compute can go into larger models, more epochs over the same data, or independent ensemble members. Existing prescriptions ignore ensembling entirely and hold hyperparameters fixed, even though optimal weight decay changes dramatically with repetition.
The authors fit empirical power laws that transfer effective learning rate and weight decay across dataset size, model size, and epoch count. Weight decay scales with the number of epochs raised to the 0.66 power, providing the strong regularization that repeated-data training demands.
The proposed scaling law adds an explicit ensemble term that depends on model size, unique tokens, epochs, and the number of members M. The functional form is motivated by variance reduction but learns its exponents from data, capturing how ensemble benefits change across training regimes.
At lower compute budgets, the prescription enlarges model size and epoch count. Beyond roughly 50,000 times the dataset size raised to the 1.9 power in floating-point operations, the optimal allocation shifts: model size and epochs stabilize, and additional compute flows into independent ensemble members.
On 100 million tokens, the combined method improves validation loss by more than 0.15 nats and reduces perplexity by 14 percent, while maintaining monotonic gains as compute increases. To explore these scaling laws and create your own research videos, visit EmergentMind.com.
