Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning (2310.03094v3)

Abstract: LLMs such as GPT-4 have exhibited remarkable performance in a variety of tasks, but this strong performance often comes with the high expense of using paid API services. In this paper, we are motivated to study building an LLM cascade to save the cost of using LLMs, particularly for performing reasoning (e.g., mathematical, causal) tasks. Our cascade pipeline follows the intuition that simpler questions can be addressed by a weaker but more affordable LLM, whereas only the challenging questions necessitate the stronger and more expensive LLM. To realize this decision-making, we consider the "answer consistency" of the weaker LLM as a signal of the question difficulty and propose several methods for the answer sampling and consistency checking, including one leveraging a mixture of two thought representations (i.e., Chain-of-Thought and Program-of-Thought). Through experiments on six reasoning benchmark datasets, with GPT-3.5-turbo and GPT-4 being the weaker and stronger LLMs, respectively, we demonstrate that our proposed LLM cascades can achieve performance comparable to using solely the stronger LLM but require only 40% of its cost.

LLM Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning

The paper "LLM Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning" presents a strategic approach to harness the capabilities of LLMs for reasoning tasks, focusing on achieving comparable performance at reduced computational costs. The work addresses the challenge posed by the expense of utilizing top-tier LLMs like GPT-4, which can be significantly higher compared to weaker variants such as GPT-3.5-turbo. This cost differential motivates the exploration of a cascading model where questions are selectively routed based on difficulty, thereby optimizing both performance and expenditure.

Key Contributions and Methodologies

• LLM Cascade Framework: The proposed framework employs a two-step routing process, wherein questions are initially handled by a weaker LLM. Subsequent routing to a stronger LLM depends on the perceived difficulty of the question, gauged through "answer consistency." This approach suggests that simpler questions can be reliably answered by the weaker LLM, while more complex ones are escalated only when necessary.
• Mixture of Thought Representations (MoT): The paper introduces a novel method leveraging thought representations, specifically Chain-of-Thought (CoT) and Program-of-Thought (PoT). By generating answers using both methods, the cascade can effectively measure consistency across diverse reasoning paths, emulating expert perspectives. This diversity in intermediate representations aids the cascade decision-making by providing robust signals regarding question difficulty.
• Answer Consistency Mechanism: Two practical methods are proposed for embedding answer consistency into the cascade decision-making process:
  • Vote-based Decision-making: Utilizes multiple answer samples from different prompt styles to compute an agreement score. This score determines whether the weaker LLM's answer should be accepted based on a predefined threshold.
  • Verification-based Decision-making: Compares the most consistent answers derived from different thought representations or demonstration sets, verifying if they match to ascertain reliability.

Experimental Results

The paper evaluates the cascade framework on six reasoning datasets encompassing mathematical, symbolic, and causal tasks. Key findings include:

• Cost Efficiency: The cascade methods yielded accuracies comparable to solely using the stronger LLM (GPT-4), while realizing substantial cost savings—approximately 40% of the cost in some scenarios.
• Effectiveness of MoT: Employing a combination of CoT and PoT prompts significantly enhanced the precision of distinguishing between easy and hard questions, thereby streamlining the routing process. These variants outperformed both individual thought representations, illustrating the power of diverse perspectives in reasoning consistency.

Implications and Future Directions

The research offers several critical implications:

• Practical Cost Reduction: For organizations utilizing LLMs intensively, the cascade can be a pivotal cost-saving mechanism, enabling the utilization of leading-edge reasoning capabilities without incurring hefty costs.
• Theoretical Advancements: Joint efforts integrating different reasoning frameworks like CoT and PoT can extend applicability across various domains, enriching AI decision-making models.
• Scalability and Efficiency in AI Systems: The cascade model exemplifies potential for scaling AI systems by optimizing human-like delegation of tasks using a tiered approach.

Future work could explore the application of these methods beyond reasoning tasks, exploring factual and open-domain challenges. Additionally, enhancements in prompt engineering and alignment of task demonstrations promise further efficacy in deploying low-cost yet potent AI solutions.

In conclusion, the paper provides a comprehensive analysis of how intelligently designed LLM cascades can achieve high levels of efficiency, making it a substantial step towards democratizing access to powerful AI technologies.