Generalizing Scaling Laws for Dense and Sparse Large Language Models

Abstract: Over the past few years, the size of LLMs has grown exponentially, as has the computational cost to train these large models. This rapid growth has motivated researchers to develop new techniques aimed at enhancing the efficiency of the training process. Despite these advancements, optimally predicting the model size or allocating optimal resources remains a challenge. Several efforts have addressed the challenge by proposing different scaling laws, but almost all of them are architecture-specific (dense or sparse). In this work we revisit existing scaling laws and propose a generalized scaling law to provide a unified framework that is applicable to both dense and sparse LLMs. We evaluate and compare our proposed scaling law with existing scaling laws to demonstrate its effectiveness.