Optimizing Model Selection for Compound AI Systems (2502.14815v1)

Abstract: Compound AI systems that combine multiple LLM calls, such as self-refine and multi-agent-debate, achieve strong performance on many AI tasks. We address a core question in optimizing compound systems: for each LLM call or module in the system, how should one decide which LLM to use? We show that these LLM choices have a large effect on quality, but the search space is exponential. We propose LLMselector, an efficient framework for model selection in compound systems, which leverages two key empirical insights: (i) end-to-end performance is often monotonic in how well each module performs, with all other modules held fixed, and (ii) per-module performance can be estimated accurately by an LLM. Building upon these insights, LLMselector iteratively selects one module and allocates to it the model with the highest module-wise performance, as estimated by an LLM, until no further gain is possible. LLMselector is applicable to any compound system with a bounded number of modules, and its number of API calls scales linearly with the number of modules, achieving high-quality model allocation both empirically and theoretically. Experiments with popular compound systems such as multi-agent debate and self-refine using LLMs such as GPT-4o, Claude 3.5 Sonnet and Gemini 1.5 show that LLMselector confers 5%-70% accuracy gains compared to using the same LLM for all modules.

• The paper introduces LLMSelector, a framework that optimizes model allocation in compound AI systems via module-specific selection.
• It reveals that overall system performance is closely tied to improvements in individual modules through diagnostic LLM evaluations.
• Empirical results indicate performance increases from 5% to 70% over uniform model setups, ensuring significant cost and efficiency gains.

An Overview of Optimizing Model Selection for Compound AI Systems

The paper "Optimizing Model Selection for Compound AI Systems" addresses a critical aspect of compound AI systems: the selection of LLMs for various modules within these systems. Compound AI systems, which integrate multiple LLM calls to solve complex tasks, exhibit performance that is heavily influenced by the selection of models used in each module. The authors propose an effective framework, LLMselector, to address this challenge amidst an exponentially large search space.

Compound AI systems often employ techniques like self-refinement and multi-agent debate to improve task performance compared to single-model approaches. These systems divide tasks into simpler, manageable subtasks, each handled by different LLMs. Despite advancements, current optimizations largely focus on prompt engineering and module interactions, often using a uniform LLM across modules. This overlooks the potential performance enhancements gained by tailoring LLM selection to each specific module.

The paper makes notable assertions regarding model selection in static (fixed-number-of-module) compound systems. First, the system’s overall performance can be monotonic in relation to the performance of individual modules. Thus, optimizing each module will likely lead to enhanced overall performance. Secondly, the authors demonstrate that LLM performance for each module can be accurately estimated using another LLM as a diagnoser.

LLMselector capitalizes on these insights, iteratively optimizing model allocation in a compound system to maximize performance relative to a budget constraint. The process involves using an LLM diagnoser to assess module performance and iteratively adjusting model allocations. The framework claims a linear scaling relative to the number of modules in terms of LLM API calls, achieving both empirical and theoretical efficacy.

The experimental validation, involving a diverse set of systems and LLMs such as GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5, illustrates the efficacy of LLMselector. Results indicated substantial accuracy gains, ranging from 5% to 70% over configurations employing a singular LLM for all modules. Moreover, LLMselector outperforms advanced prompt optimization techniques, highlighting the criticality of model selection in compound systems.

The implications of this paper are profound for both practical applications and theoretical developments. Practically, LLMselector provides a scalable solution for optimizing complex systems, suggesting significant cost and performance efficiencies. Theoretically, it opens discussions on the modular benefits of diversified AI systems, encouraging further research into the specific characteristics that allow LLMs to excel in distinct module roles.

Future research directions could explore dynamic adjustments within compound systems as model capabilities and datasets evolve, further refining the selection strategies. Exploring collaboration between LLMs with different strengths, potentially enhanced by real-time model diagnostics, could drive these systems towards more human-like problem-solving abilities.

In conclusion, the paper contributes significantly to the optimization of compound AI systems by providing a systematic approach to model selection, backed by robust empirical validation. It emphasizes the nuanced capabilities of LLMs beyond prompt engineering, paving the way for more sophisticated and efficient AI applications.