Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild (2410.05357v2)

Abstract: As LLMs excel across tasks and specialized domains, scaling LLMs based on existing models has garnered significant attention, which faces the challenge of decreasing performance when combining disparate models. Various techniques have been proposed for the aggregation of pre-trained LLMs, including model merging, Mixture-of-Experts, and stacking. Despite their merits, a comprehensive comparison and synergistic application of them to a diverse model zoo is yet to be adequately addressed. In light of this research gap, this paper introduces Model-GLUE, a holistic LLM scaling guideline. First, our work starts with a benchmarking of existing LLM scaling techniques, especially selective merging, and variants of mixture. Utilizing the insights from the benchmark results, we formulate an optimal strategy for the selection and aggregation of a heterogeneous model zoo characterizing different architectures and initialization.Our methodology involves the clustering of mergeable models and optimal merging strategy selection, and the integration of clusters through a model mixture. Finally, evidenced by our experiments on a diverse Llama-2-based model zoo, Model-GLUE shows an average performance enhancement of 5.61%, achieved without additional training. Codes are available at: https://github.com/Model-GLUE/Model-GLUE.

• The paper introduces a Model-GLUE methodology that benchmarks and integrates diverse pre-trained LLMs for efficient scaling.
• It employs model clustering, filtering, and selective merging to optimize aggregation from heterogeneous model zoos.
• Experiments on Llama-2 models reveal an average performance boost of 5.61%, highlighting cost-efficient improvements in reasoning, mathematics, and coding.

Overview of Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild

The paper introduces Model-GLUE, a comprehensive guideline aimed at democratizing LLM scaling. It addresses the challenges faced in aggregating pre-trained LLMs from diverse model zoos by providing a clear comparison of techniques such as model merging and Mixture-of-Experts (MoE). With the increasing availability of open-sourced LLMs, the need for efficient model scaling strategies is crucial to mitigate computational costs and harness previous advancements.

Key Contributions

• Benchmarking Existing Techniques: The paper begins by benchmarking existing LLM scaling techniques, particularly focusing on selective merging and various mixture methods. This approach helps in understanding the current landscape of LLM scaling.
• Comprehensive Strategy for Model Zoo Aggregation: Utilizing benchmark insights, the paper formulates a strategy to efficiently select and aggregate models from a heterogeneous model zoo. This involves clustering mergeable models and selecting optimal merging strategies, followed by integrating these clusters through a model mixture approach.
• Model-GLUE Methodology: The introduced Model-GLUE methodology consists of several steps:
  • Model Clustering is performed based on architecture and weight similarity.
  • Model Filtering and Searching help eliminate detrimental candidates for merging.
  • Model Merging is conducted within each cluster.
  • Model Level Mixture integrates merged models across clusters.
• Performance Enhancements: Experiments conducted using a diverse Llama-2-based model zoo demonstrated an average performance enhancement of 5.61% without requiring additional training. The approach not only improved performance in general reasoning tasks but also in specific domains like mathematics and coding.

Implications and Future Directions

The implications of this research extend across both theoretical and practical realms. By providing a detailed benchmarking and combination strategy, this work facilitates a deeper understanding of how disparate LLMs can be effectively unified. Practically, the Model-GLUE guideline can be instrumental for researchers and practitioners looking to scale LLMs without incurring the computational overhead associated with training new, larger models from scratch.

The research also opens avenues for future developments where model stacking and more sophisticated model communication methods could be integrated with existing strategies to further enhance scalability. Investigating the permutation symmetry in neural networks and optimizing router designs for MoE could yield even more efficient ways to aggregate knowledge from different sources.

Overall, the paper offers a significant contribution towards efficient and economical LLM scaling, highlighting the potential of leveraging existing models' collective advancements rather than starting from scratch.