Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks (2401.02731v4)

Abstract: LLMs have demonstrated considerable proficiency in general NLP tasks. Instruction tuning, a successful paradigm, enhances the ability of LLMs to follow natural language instructions and exhibit robust generalization across general tasks. However, these models often encounter performance limitations across multiple tasks due to constrained model capacity. Expanding this capacity during the instruction tuning phase poses significant challenges. To address this issue, we introduce parameter-efficient sparsity crafting (PESC), which crafts dense models into sparse models using the mixture-of-experts (MoE) architecture. PESC integrates adapters into the MoE layers of sparse models, differentiating experts without altering the individual weights within these layers. This method significantly reduces computational costs and GPU memory requirements, facilitating model capacity expansion through a minimal parameter increase when guaranteeing the quality of approximation in function space compared to original sparse upcycling. Our empirical evaluation demonstrates the effectiveness of the PESC method. Using PESC during instruction tuning, our best sparse model outperforms other sparse and dense models and exhibits superior general capabilities compared to GPT-3.5. Our code is available at https://github.com/wuhy68/Parameter-Efficient-MoE.

Background

LLMs are widely recognized for their exceptional performance in NLP tasks, enhanced by a training method known as instruction tuning. Despite their capabilities, they often encounter performance ceilings when processing multiple tasks due to fixed model capacities. Traditional methods to scale up these models can be resource-intensive and impractical. A novel approach is imperative for enabling LLMs to transcend these bounds without incurring prohibitive costs.

Introducing Parameter-Efficient Sparsity Crafting

The paper presents Parameter-Efficient Sparsity Crafting (PESC), bridging the gap between the need to scale and resource limitations. PESC transitions a dense LLM into a sparse model by employing a Mixture-of-Experts (MoE) framework. It does so by incorporating adapters — small tunable modules — into the MoE layers, effectively allowing the model to expand its processing capacity without the need to modify existing weights. This methodology not only promises computational efficiency but also introduces a minimal set of new parameters, thus avoiding the intensive resource demands typically associated with model scaling.

PESC's Empirical Validation

Empirically, the sparse models developed using the PESC method, referred to as Camelidae, have outperformed all open-source sparse models in testing. Moreover, they exhibit broader general capabilities than the well-known GPT-3.5 in various standard benchmarks. The research meticulously documents the methods of implementing PESC and compares the performance of Camelidae against several other benchmarks, showcasing its superior efficacy in general tasks.

Advantages and Future Potential

PESC stands out by elegantly addressing the perennial conflict of scaling model capacity with manageable increases in parameters and computational demand. The documented results indicate that the models created with PESC could significantly influence the future of model tuning. They hint at the possibility of crafting LLMs that not only understand and generate human-like language but do so with an underlying architecture that is both powerful and efficient.

Overall, the PESC approach marks a notable advance in the domain of LLM fine-tuning. It yields models like Camelidae that could potentially set new standards in NLP tasks while maintaining manageable computational and resource overhead.