Democratizing Reasoning Ability: Tailored Learning from Large Language Model (2310.13332v1)

Abstract: LLMs exhibit impressive emergent abilities in natural language processing, but their democratization is hindered due to huge computation requirements and closed-source nature. Recent research on advancing open-source smaller LMs by distilling knowledge from black-box LLMs has obtained promising results in the instruction-following ability. However, the reasoning ability which is more challenging to foster, is relatively rarely explored. In this paper, we propose a tailored learning approach to distill such reasoning ability to smaller LMs to facilitate the democratization of the exclusive reasoning ability. In contrast to merely employing LLM as a data annotator, we exploit the potential of LLM as a reasoning teacher by building an interactive multi-round learning paradigm. This paradigm enables the student to expose its deficiencies to the black-box teacher who then can provide customized training data in return. Further, to exploit the reasoning potential of the smaller LM, we propose self-reflection learning to motivate the student to learn from self-made mistakes. The learning from self-reflection and LLM are all tailored to the student's learning status, thanks to the seamless integration with the multi-round learning paradigm. Comprehensive experiments and analysis on mathematical and commonsense reasoning tasks demonstrate the effectiveness of our method. The code will be available at https://github.com/Raibows/Learn-to-Reason.

Overview of "Democratizing Reasoning Ability: Tailored Learning from LLM"

This paper introduces a methodology to endow smaller LLMs (LMs) with reasoning capabilities that are characteristic of larger, often restrictive LLMs. Reasoning, an elusive and computationally intensive feature, remains a significant hurdle in the democratization of LLMs, largely due to their closed-source disposition and high computational requirements. The authors propose a novel approach to impart reasoning skills to more accessible, smaller LMs, a process they term "tailored learning."

Methodology

The tailored learning methodology distinguishes itself by utilizing a multi-round interactive learning paradigm, where the student (the smaller LM) interacts continuously with a teacher (the LLM). This paradigm emphasizes a dynamic feedback loop:

1. Initial Setup and Data Generation:
   • The smaller LM starts with undertaking a preliminary 'exam' to identify areas of weakness by generating responses to a set of tasks. Incorrect reasoning paths (wrong answers) are analyzed to provide structured feedback.
   • Unlike static data annotation, this approach incorporates a design where the LLM acts not merely as an annotator, but as an interactive teacher that adjusts its feedback based on the student's learning status.
2. Iterative Feedback and Learning:
   • The student model provides this preliminary output as feedback to the LLM, which then crafts tailored rationale annotations meant to address the identified shortcomings. This feedback guides the LLM to create more suitable training data.
   • The learning process further employs self-reflection where the student LM learns to discern between right and wrong reasoning paths by revisiting its errors, a strategy inspired by human learning processes.
3. Round-by-Round Improvement:
   • This iterative process is reiterated over several rounds, continuously refining the student's reasoning prowess. The dual focus on receiving customized feedback and engaging in self-reflection underlines the process's robustness.

Experimental Evaluations

The efficacy of this tailored learning paradigm is tested across various reasoning tasks, including mathematical word problem solving (GSM8K, MultiArith) and commonsense reasoning tasks (CSQA, StrategyQA). These tasks allow for an effective assessment of the enhanced reasoning capabilities imparted by the proposed method.

• Numerical Outcomes: Results show significant improvements in accuracy rates of the smaller LMs after employing tailored learning, notably surpassing previously reported methodologies that focus solely on knowledge distillation or static CoT (Chain-of-Thought) prompting methods.
• Comparative Analysis with Existing Approaches: The method notably outperforms precedent techniques by narrowing the performance gap between smaller LMs and their large counterparts, in some cases achieving parity in commonsense tasks.

Implications and Future Directions

The implications of this research extend to both theoretical and practical domains. By enhancing the reasoning capabilities of smaller, more computationally economical models, this methodology promotes broader accessibility and utilization of advanced AI technologies across varied applications.

Future trajectories can include:

• Exploring more complex domains and reasoning tasks to further test scalability.
• Applying this model in distributed systems or mobile applications where resource constraints are a critical factor.
• Enhancing the robustness of interactive distillation techniques, possibly integrating advanced automatic evaluation metrics for rationale quality.

In conclusion, by effectively democratizing reasoning capabilities through structured interactions and self-reflective learning paradigms, this research significantly contributes to making sophisticated AI tools available for wider, more inclusive use. It lays an important foundation for future work aimed at augmenting AI accessibility without sacrificing capability.