Composition of Experts: A Modular Compound AI System Leveraging Large Language Models (2412.01868v1)

Abstract: LLMs have achieved remarkable advancements, but their monolithic nature presents challenges in terms of scalability, cost, and customization. This paper introduces the Composition of Experts (CoE), a modular compound AI system leveraging multiple expert LLMs. CoE leverages a router to dynamically select the most appropriate expert for a given input, enabling efficient utilization of resources and improved performance. We formulate the general problem of training a CoE and discuss inherent complexities associated with it. We propose a two-step routing approach to address these complexities that first uses a router to classify the input into distinct categories followed by a category-to-expert mapping to obtain desired experts. CoE offers a flexible and cost-effective solution to build compound AI systems. Our empirical evaluation demonstrates the effectiveness of CoE in achieving superior performance with reduced computational overhead. Given that CoE comprises of many expert LLMs it has unique system requirements for cost-effective serving. We present an efficient implementation of CoE leveraging SambaNova SN40L RDUs unique three-tiered memory architecture. CoEs obtained using open weight LLMs Qwen/Qwen2-7B-Instruct, google/gemma-2-9b-it, google/gemma-2-27b-it, meta-llama/Llama-3.1-70B-Instruct and Qwen/Qwen2-72B-Instruct achieve a score of $59.4$ with merely $31$ billion average active parameters on Arena-Hard and a score of $9.06$ with $54$ billion average active parameters on MT-Bench.

• The paper presents Composition of Experts (CoE), a modular system design using multiple expert LLMs and a router to improve efficiency and adaptability over monolithic models.
• Empirical evaluation demonstrates that CoE achieves comparable or superior performance on benchmarks while requiring significantly fewer active parameters and computational resources.
• The CoE method offers a practical and cost-effective approach for deploying advanced AI solutions, potentially democratizing access to high-performance LLM capabilities.

Composition of Experts: A Modular AI System Leveraging LLMs

The paper presents an innovative approach titled Composition of Experts (CoE), which proposes a modular system design to address the inefficiencies associated with monolithic LLMs. By employing multiple expert LLMs, coupled with a sophisticated routing system, CoE offers a resource-efficient and customizable alternative to traditional LLM deployments.

The primary challenge CoE attempts to address is the inherent rigidity and resource intensiveness of current monolithic LLMs. Models like GPT-4, despite their high competence, are often criticized for their massive size and the consequent computational and financial costs. Furthermore, adapting such models for specialized tasks is both complex and costly. The CoE method navigates these hurdles by allowing integration of various specialized or domain-specific expert models instead of solely relying on one large model for all tasks.

System Architecture and Approach

The CoE system architecture involves a router that dynamically guides input to the most appropriate expert model. This is achieved through a two-step process: a category router classifies input into predefined categories, and a category-to-expert mapping designates the suitable expert model. This architecture thereby divides the decision-making process into granular steps, thus offering flexibility and modularity.

Key advantages of this system include:

• Scalability and Modularity: The CoE system allows for the easy addition or removal of expert models, thereby offering a flexible system that can evolve with changing requirements or improvements in model development.
• Efficiency: By selecting only the necessary expert models for each input, CoE minimizes computational overhead, ensuring resources are used optimally.
• Interpretability and Control: This modular approach grants system designers more oversight and flexibility in configuration, allowing for precise tuning to meet specific application needs.

Empirical Evaluation

The authors present empirical evidence showing that CoE considerably enhances performance while requiring fewer computational resources than traditional monolithic models. Through leveraging open weight LLMs for the expert models and an efficient implementation using SambaNova SN40L's unique memory architecture, CoE achieves notable results on benchmarks such as Arena-Hard and MT-Bench. The flexible modular approach allows CoE to achieve superior scores with significantly fewer active parameters, demonstrating the efficiency of the expert model selection.

Implications and Future Directions

The flexibility and reduced resource demands of the CoE method position it as a highly practical approach for enterprises looking to deploy AI solutions efficiently. This innovation opens pathways to democratizing access to advanced AI by providing cost-effective solutions without sacrificing performance quality. As LLM technology continues to evolve, the capability to seamlessly incorporate and optimize new models will become increasingly crucial.

In terms of theoretical implications, the CoE raises interesting questions about the balance between model size and task specialization. The successful implementation of a modular architecture using category-to-expert mapping may influence future research on optimizing modular LLM architectures and routing algorithms.

Future research could explore even more nuanced routing mechanisms, leveraging advanced machine learning techniques to further refine input-expert allocation. Additionally, the CoE model could be tested across a broader range of applications to validate its robustness and adaptability to varied tasks and languages.

Overall, the Composition of Experts model represents a significant stride in enhancing the efficiency and adaptability of LLM deployments, offering a feasible solution to many challenges currently faced by large-scale AI systems.