Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains

Abstract: LLMs have achieved remarkable performance in recent years but are fundamentally limited by the underlying training data. To improve models beyond the training data, recent works have explored how LLMs can be used to generate synthetic data for autonomous self-improvement. However, successive steps of self-improvement can reach a point of diminishing returns. In this work, we propose a complementary approach towards self-improvement where finetuning is applied to a multiagent society of LLMs. A group of LLMs, all starting from the same base model, are independently specialized by updating each one using data generated through multiagent interactions among the models. By training each model on independent sets of data, we illustrate how this approach enables specialization across models and diversification over the set of models. As a result, our overall system is able to preserve diverse reasoning chains and autonomously improve over many more rounds of fine-tuning than single-agent self-improvement methods. We quantitatively illustrate the efficacy of the approach across a wide suite of reasoning tasks.

• The paper introduces a multiagent finetuning framework that utilizes collaborative debate among specialized language models to generate refined reasoning outputs.
• It employs generation and critic agents to iteratively fine-tune models, yielding significant performance gains on datasets like MATH and GSM.
• The method reduces reliance on static datasets by enabling self-generated, diversified data for continuous model improvement.

Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains

Introduction

The paper "Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains" (2501.05707) introduces a novel methodology aimed at enhancing the performance of LLMs. It addresses the limitations imposed by the static nature of the training data traditionally used in LLM development, proposing a dynamic multiagent framework that leverages self-generated data for iterative refinement. This multiagent system enables specialization and diversification beyond what is achievable with single-agent finetuning, offering promising advances in reasoning task performance.

Methodology: Multiagent Finetuning

The proposed method operates on a multiagent society of LLMs derived from the same base model. Each agent is independently fine-tuned using data that emerges from multiagent interactions. This data-centric approach allows for the specialized tuning of models, fostering diverse reasoning chains that enhance task performance and enable self-improvement over successive rounds of finetuning.

Multiagent Debate Framework: The core mechanism involves a debate-based interaction among multiple agents. Initially, each agent offers a response to a given query, after which responses are exchanged, debated, and iteratively refined through majority consensus. The aim is to converge on the most accurate outputs, thus generating a robust finetuning dataset.

Figure 1: Overview of Multiagent Finetuning, showcasing the debate and majority voting-based data creation process followed by specialized model finetuning.

Specialization of Agents: Two distinct roles are defined within the agent society: generation agents, which produce initial problem-solving attempts, and critic agents, tasked with evaluating and refining these solutions. This division encourages intricate feedback loops that benefit subsequent iterations of response generation and optimization.

Experimental Results

The methodology's efficacy is demonstrated through extensive testing across diverse reasoning tasks, including arithmetic and complex mathematical problem datasets like MATH and GSM. Results reflect significant performance uplifts compared to previously existing methods, particularly in environments necessitating advanced reasoning and problem-solving strategies.

Figure 2: Multiagent finetuning significantly boosts reasoning performance over multiple rounds, as shown with the MATH dataset.

The paper reports that multitiered finetuning iterations continue to yield performance gains, contrasting sharply with the plateauing effects observed in mono-agent configurations. This iteration-based improvement is quantitatively validated, showcasing robustness across various model architectures, including both open-source and proprietary LLMs like GPT-3.5.

Implications and Future Directions

The introduction of multiagent finetuning heralds a shift towards more autonomous and self-sustaining model training paradigms, potentially alleviating the dependency on large, static datasets. By facilitating continual learning through synthetic data generation and agent specialization, this approach could redefine efficiency and scalability standards within the field.

Practical Applications: The framework's ability to generalize to entirely new datasets highlights its utility in real-world applications, offering LLMs a mechanism to adapt to evolving data landscapes without extensive retraining from external sources.

Figure 3: Demonstrates the iterative finetuning improvements across MATH problem levels, highlighting the model's adaptability to varying difficulty levels.

Future Research: Exploration into integrating this methodology with other finetuning techniques, such as human-in-the-loop systems, presents an intriguing avenue for future enhancements. Additionally, leveraging this multiagent strategy could optimize various other machine learning domains, potentially extending beyond natural language processing.

Conclusion

Overall, the paper introduces a robust advancement in LLM training methodologies, emphasizing multiagent interactions as a pathway to achieving more intelligent and adaptable AI systems. The demonstrated improvements in both performance and diversity underscore the significant potential of this approach in overcoming the innate limitations of traditional model finetuning techniques.