MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2409.12147v1)

Abstract: LLMs' (LLM) reasoning can be improved using test-time aggregation strategies, i.e., generating multiple samples and voting among generated samples. While these improve performance, they often reach a saturation point. Refinement offers an alternative by using LLM-generated feedback to improve solution quality. However, refinement introduces 3 key challenges: (1) Excessive refinement: Uniformly refining all instances can over-correct and reduce the overall performance. (2) Inability to localize and address errors: LLMs have a limited ability to self-correct and struggle to identify and correct their own mistakes. (3) Insufficient refinement: Deciding how many iterations of refinement are needed is non-trivial, and stopping too soon could leave errors unaddressed. To tackle these issues, we propose MAgICoRe, which avoids excessive refinement by categorizing problem difficulty as easy or hard, solving easy problems with coarse-grained aggregation and hard ones with fine-grained and iterative multi-agent refinement. To improve error localization, we incorporate external step-wise reward model (RM) scores. Moreover, to ensure effective refinement, we employ a multi-agent loop with three agents: Solver, Reviewer (which generates targeted feedback based on step-wise RM scores), and the Refiner (which incorporates feedback). To ensure sufficient refinement, we re-evaluate updated solutions, iteratively initiating further rounds of refinement. We evaluate MAgICoRe on Llama-3-8B and GPT-3.5 and show its effectiveness across 5 math datasets. Even one iteration of MAgICoRe beats Self-Consistency by 3.4%, Best-of-k by 3.2%, and Self-Refine by 4.0% while using less than half the samples. Unlike iterative refinement with baselines, MAgICoRe continues to improve with more iterations. Finally, our ablations highlight the importance of MAgICoRe's RMs and multi-agent communication.

Formic Overview of MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning

The paper "MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning" by Chen et al. presents a novel approach to enhance the reasoning capabilities of LLMs by leveraging a multi-agent framework that selectively applies iterative refinement to difficult problems. The method aims to address the limitations associated with traditional test-time aggregation strategies that often reach performance saturation, consequently wasting computation without significantly improving solution quality.

Key Contributions

The paper's central contribution is the MAgICoRe framework, which embodies a Multi-Agent Iteration for Coarse-to-fine Refinement methodology. This framework distinguishes itself with several notable strategies:

1. Selective Refinement Based on Difficulty:
   • The framework categorizes problems into easy and hard instances. Easy problems are tackled using standard aggregation methods such as Weighted Self-Consistency (WSC), whereas hard problems undergo fine-grained iterative refinement. The classification is determined by evaluating solution quality and confidence using external Reward Models (RMs).
2. External Reward Models for Error Localization:
   • To enhance error localization, the method employs step-wise scoring via Process Reward Models (PRMs). This localized scoring informs the Reviewer agent, which generates targeted feedback to improve specific parts of the reasoning chain.
3. Multi-Agent System for Refinement:
   • The framework comprises three distinct agents: Solver, Reviewer, and Refiner. The Solver generates initial solutions, the Reviewer provides targeted feedback based on PRM scores, and the Refiner uses this feedback to improve the solutions iteratively. This multi-agent system promotes effective communication and refinement, especially for challenging problems.
4. Iterative Refinement Process:
   • The refinement is not a one-shot process; it is re-evaluated iteratively. This iterative approach continues until a predefined stopping criterion is met, ensuring sufficient and effective refinement processes.

Numerical Results

The authors evaluate MAgICoRe on two LLMs—Llama-3-8B and GPT-3.5—across five math reasoning datasets: GSM8K, SVAMP, MATH, MMLU, and SAT. The results demonstrate consistent improvements achieved by MAgICoRe over both refinement and aggregation baselines:

• Performance Gains:
  • On Llama-3-8B, one iteration of MAgICoRe outperforms Self-Consistency by 3.4\% and Best-of-$k$ by 3.2\%, despite using fewer samples. This significant improvement reflects the efficiency and effectiveness of MAgICoRe in handling difficult problems.
  • The GPT-3.5-Turbo model exhibits a similar trend, with MAgICoRe outperforming the baselines by 3.5% to 8.0% on different datasets.
• Efficiency:
  • MAgICoRe demonstrates sample efficiency by selectively refining only hard problems. This targeted approach ensures that computational resources are used judiciously, achieving better results with fewer iterations compared to uniform refinement strategies.

Implications and Future Research

The implications of MAgICoRe are manifold:

• Practical Applications:
  • In practical applications, MAgICoRe's ability to allocate computational resources efficiently makes it suitable for scenarios where computational cost is a constraint. Its iterative refinement ensures robust solutions to complex reasoning problems, potentially benefiting fields such as automated theorem proving and complex decision-making systems.
• Theoretical Insights:
  • From a theoretical perspective, the paper underscores the importance of external scoring models in enhancing LLM performance. It challenges the conventional reliance on LLMs for self-verification and highlights the benefits of integrating external feedback loops for error correction.

Speculations on Future Developments

In the evolving landscape of AI research, the principles introduced by MAgICoRe could inspire several future developments:

• Enhanced Multi-Agent Communication:
  • Future work could explore more sophisticated communication protocols between agents, possibly leveraging mechanisms like meta-learning to optimize interaction strategies dynamically.
• Adaptive Iteration Control:
  • Advances in adaptive control algorithms might refine the iterative process even further, enabling the framework to autonomously determine the optimal number of refinement iterations based on real-time feedback quality.
• Expansion to Other Domains:
  • Adapting the MAgICoRe methodology to other domains beyond mathematical reasoning, such as natural language understanding, scientific discovery, and technical problem-solving, could reveal its broader applicability and versatility.

In conclusion, the MAgICoRe framework introduced by Chen et al. represents a significant stride in enhancing LLM reasoning capabilities through adaptive, fine-grained refinement strategies. By addressing key challenges in existing methods and offering clear performance improvements, it sets a strong foundation for future research aimed at refining and expanding the capabilities of AI-driven reasoning systems.