Unified Neural Network Scaling Laws and Scale-time Equivalence (2409.05782v1)

Abstract: As neural networks continue to grow in size but datasets might not, it is vital to understand how much performance improvement can be expected: is it more important to scale network size or data volume? Thus, neural network scaling laws, which characterize how test error varies with network size and data volume, have become increasingly important. However, existing scaling laws are often applicable only in limited regimes and often do not incorporate or predict well-known phenomena such as double descent. Here, we present a novel theoretical characterization of how three factors -- model size, training time, and data volume -- interact to determine the performance of deep neural networks. We first establish a theoretical and empirical equivalence between scaling the size of a neural network and increasing its training time proportionally. Scale-time equivalence challenges the current practice, wherein large models are trained for small durations, and suggests that smaller models trained over extended periods could match their efficacy. It also leads to a novel method for predicting the performance of large-scale networks from small-scale networks trained for extended epochs, and vice versa. We next combine scale-time equivalence with a linear model analysis of double descent to obtain a unified theoretical scaling law, which we confirm with experiments across vision benchmarks and network architectures. These laws explain several previously unexplained phenomena: reduced data requirements for generalization in larger models, heightened sensitivity to label noise in overparameterized models, and instances where increasing model scale does not necessarily enhance performance. Our findings hold significant implications for the practical deployment of neural networks, offering a more accessible and efficient path to training and fine-tuning large models.

• The paper presents a unified theoretical framework integrating model size, training time, and data volume to predict neural network performance and explain phenomena like double descent.
• A core finding is "scale-time equivalence," demonstrating that increasing model size is functionally equivalent to proportionally extending training duration, suggesting smaller models trained longer can match larger ones.
• The unified law incorporates insights into generalization efficiency (larger models need less data), susceptibility to noise in overparameterized models, and the non-monotonic nature of performance scaling.

The paper "Unified Neural Network Scaling Laws and Scale-time Equivalence" presents a novel theoretical framework that integrates three crucial factors—model size, training time, and data volume—to predict the performance of deep neural networks. Recent research highlights neural network scaling laws that describe how test error decreases as model size and dataset volume increase. However, these existing laws often fail to encapsulate all determinants of performance, particularly the role of training time and its interaction with model scaling, leaving phenomena like double descent unexplained.

The authors propose a theoretical and empirical proof that scaling model size is functionally equivalent to increasing training time proportionally, a concept termed 'scale-time equivalence'. This assertion challenges the prevalent practice of training large models for limited durations and suggests that smaller models trained over extended periods can achieve comparable efficacy. This relationship is substantiated through both a simplified linear model and empirical validations using CNN and MLP architectures on standard vision benchmarks.

The paper proceeds to unify this scale-time equivalence with double descent theory by offering a comprehensive scaling law that elucidates various empirical observations:

1. Scale-time Equivalence: The paper demonstrates that increasing the size of a neural network proportionally equates to extending its training duration. This is supported by their analysis of a random subspace model and further confirmed via experiments with benchmarks like MNIST and CIFAR-10.
2. Generalization Efficiency: Larger models require less data to generalize. Empirical results show that the interpolation threshold, where model performance sharply improves with data volume, is lower for larger models.
3. Noise Impact: The paper reveals that overparameterized models become more susceptible to label noise, which leads to test errors growing even with model scale increases.
4. Non-improving Performance with Scale: Increasing the model size doesn't consistently yield improvements, contradicting common scaling laws which suggest a monotonic decrease in error with larger models.

These insights have potentially significant implications for the deployment and scaling of neural networks, especially in the context of training LLMs. If smaller models trained extensively can rival larger, computationally expensive models, this could revolutionize the accessibility of training and fine-tuning large models.

The presented unified scaling law attempts to incorporate these dynamics, providing a new understanding and forecasting tool for optimizing the combination of model size, data volume, and training time to achieve efficient learning in neural networks. This theoretical advancement fosters a more nuanced perspective on traditional scaling laws, expanding them beyond their conventional limits by incorporating double descent characteristics w.r.t. time, model scale, and data quantity.