SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correction

Abstract: LLMs like GPT-4, DeepSeek-R1, and ReasonFlux have shown significant improvements in various reasoning tasks. However, smaller LLMs still struggle with complex mathematical reasoning because they fail to effectively identify and correct reasoning errors. Recent reflection-based methods aim to address these issues by enabling self-reflection and self-correction, but they still face challenges in independently detecting errors in their reasoning steps. To overcome these limitations, we propose SuperCorrect, a novel two-stage framework that uses a large teacher model to supervise and correct both the reasoning and reflection processes of a smaller student model. In the first stage, we extract hierarchical high-level and detailed thought templates from the teacher model to guide the student model in eliciting more fine-grained reasoning thoughts. In the second stage, we introduce cross-model collaborative direct preference optimization (DPO) to enhance the self-correction abilities of the student model by following the teacher's correction traces during training. This cross-model DPO approach teaches the student model to effectively locate and resolve erroneous thoughts with error-driven insights from the teacher model, breaking the bottleneck of its thoughts and acquiring new skills and knowledge to tackle challenging problems. Extensive experiments consistently demonstrate our superiority over previous methods. Notably, our SuperCorrect-7B model significantly surpasses powerful DeepSeekMath-7B by 7.8%/5.3% and Qwen2.5-Math-7B by 15.1%/6.3% on MATH/GSM8K benchmarks, achieving new SOTA performance among all 7B models. Code: https://github.com/YangLing0818/SuperCorrect-LLM

• The paper presents a two-stage framework where a large teacher model guides a smaller student model to improve error correction in complex mathematical reasoning tasks.
• Hierarchical thought templates extracted from the teacher model enable detailed reasoning, surpassing conventional methods like CoT and BoT.
• Experimental results show SuperCorrect-7B achieves state-of-the-art performance with up to a 15.1% improvement on GSM8K benchmarks.

Essay on "SuperCorrect: Supervising and Correcting LLMs with Error-Driven Insights"

The paper "SuperCorrect: Supervising and Correcting LLMs with Error-Driven Insights" addresses the challenges smaller LLMs face when handling complex mathematical reasoning tasks. Despite advancements in LLMs, smaller models like Llama-3-8B and DeepSeekMath-Base continue to exhibit limitations in this domain. The authors propose a two-stage framework, SuperCorrect, that introduces a large teacher model to guide and refine the reasoning and reflection processes of a smaller student model.

Key Contributions

1. Two-Stage Framework:
   • The framework employs a large teacher model to supervise the smaller student model in reasoning tasks. It integrates thought templates and collaborative optimization techniques to significantly improve the student model's self-correction capabilities.
2. Hierarchical Thought Templates:
   • In the first stage, hierarchical high-level and detailed thought templates are extracted from the teacher model. These templates guide the student model to develop finer reasoning processes. This approach surpasses traditional methods like CoT and BoT, providing deeper insights necessary for error correction.
3. Cross-Model Collaborative Direct Preference Optimization (DPO):
   • The second stage introduces cross-model collaborative DPO. This innovative optimization framework leverages the teacher's correction traces to enhance the student model's self-correction abilities. It allows the student model to break through its reasoning bottlenecks and acquire new problem-solving skills.

Experimental Results

The experimentation confirms SuperCorrect’s superiority over existing methods. Notably, the SuperCorrect-7B model outperforms the DeepSeekMath-7B by 7.8%/5.3% and the Qwen2.5-Math-7B by 15.1%/6.3% on MATH and GSM8K benchmarks, respectively. These results establish new state-of-the-art performance among all 7B models.

Implications and Future Directions

SuperCorrect presents both theoretical and practical implications. Theoretically, it advances the understanding of error correction in LLMs by utilizing external model supervision. Practically, it provides a method for improving the performance of smaller models, which are often more accessible due to lower computational requirements.

Looking forward, the framework could be extended to explore:

• Generalizations to larger models,
• Applications across diverse reasoning tasks beyond mathematics,
• Further optimization of cross-model corrective interactions.

Conclusion

SuperCorrect catalyzes advancements in error-driven insights for LLMs, marking significant progress in their ability to handle mathematically intensive reasoning tasks. It delineates a strategic pathway for employing larger models as educators for their smaller counterparts, thus enhancing the efficacy of LLMs in an efficient and scalable manner.