Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought (2404.03414v1)

Published 4 Apr 2024 in cs.CL and cs.AI
Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought

Abstract: We introduce a novel framework, LM-Guided CoT, that leverages a lightweight (i.e., <1B) LLM (LM) for guiding a black-box large (i.e., >10B) LM in reasoning tasks. Specifically, the lightweight LM first generates a rationale for each input instance. The Frozen large LM is then prompted to predict a task output based on the rationale generated by the lightweight LM. Our approach is resource-efficient in the sense that it only requires training the lightweight LM. We optimize the model through 1) knowledge distillation and 2) reinforcement learning from rationale-oriented and task-oriented reward signals. We assess our method with multi-hop extractive question answering (QA) benchmarks, HotpotQA, and 2WikiMultiHopQA. Experimental results show that our approach outperforms all baselines regarding answer prediction accuracy. We also find that reinforcement learning helps the model to produce higher-quality rationales with improved QA performance.

PDF HTML Abstract

LM-Guided Chain-of-Thought: Enhancing Reasoning in LLMs with Smaller Counterparts

Introduction

The research presents an innovative approach to improving reasoning in LLMs (LMs) through the assistance of smaller, lightweight LMs. This strategy, termed LM-Guided Chain-of-Thought (CoT), involves a two-step process where a small LM generates a rationale for a given input, which then guides a larger, black-box LM to predict the task output. Unlike traditional methods that require extensive resources for training large models, this framework emphasizes efficiency by only necessitating the training of the smaller LM. Knowledge distillation and reinforcement learning are employed to fine-tune the model based on rationale-oriented and task-oriented rewards, demonstrating superior performance in multi-hop extractive question answering tasks.

Methodology

The framework comprises two main components:

  1. Rationale Distillation: Involves generating a rationale for each input instance using a small LM. This step utilizes knowledge distillation from a larger LM to bootstrap the smaller model's reasoning capabilities, focusing on generating high-quality rationales.
  2. Rationale Refinement: Builds upon the distilled model, applying reinforcement learning to optimize it based on a set of predefined rationale-specific aspects. These aspects include relevance, factuality, and coherence among others. The reinforcement learning component is designed to refine the quality of rationales further, thereby enhancing the overall reasoning process.

The experimental setup employs FLAN-T5 models of different sizes to serve the roles of small and large LMs, utilizing HotpotQA and 2WikiMultiHopQA datasets for evaluation. The method's success is attributed to its novel split between rationale generation and answer prediction, allowing for focused optimization of each part.

Results

The experimental results indicate several key findings:

  • The LM-Guided CoT method surpasses baseline performances across multiple metrics, including answer prediction accuracy. This success showcases the efficacy of the proposed two-step reasoning process, especially when complemented with self-consistency decoding strategies.
  • The application of reinforcement learning contributes to a noticeable improvement in the quality of generated rationales and, subsequently, question answering performance.
  • The method reveals that selecting the highest-quality rationales does not unequivocally lead to better task outcomes, suggesting a complex relationship between rationale quality and effective reasoning in LMs.

Implications and Future Work

This research opens up new avenues for enhancing reasoning in LMs, particularly by leveraging smaller models to guide larger counterparts. The LM-Guided CoT framework proposes a resource-efficient method to improve large LM's reasoning capabilities without the need for extensive retraining. The findings underscore the significance of rationale quality in complex reasoning tasks and hint at potential strategies for further optimization. Future research directions could explore the generalizability of this approach to other forms of reasoning tasks, the integration of more sophisticated reinforcement learning techniques, and the exploration of other model combinations to enhance reasoning across diverse domains.

Limitations and Ethical Considerations

The paper acknowledges limitations, including its focus on FLAN-T5 models and multi-hop QA tasks, which may not fully represent the framework's applicability across different settings and tasks. Future work is encouraged to test the framework's robustness and versatility across various reasoning demands and model architectures. Ethically, while the generated rationales were monitored for factual and respectful content within this paper, deploying such models in real-world scenarios necessitates rigorous scrutiny to prevent the dissemination of inaccurate or harmful outputs.

Acknowledgments

The paper concludes by thanking reviewers for their constructive feedback, fostering an environment conducive to significant advancements in AI research.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (26)
  1. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.
  2. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166.
  3. Roscoe: A suite of metrics for scoring step-by-step reasoning. In The Eleventh International Conference on Learning Representations.
  4. Are machine rationales (not) useful to humans? measuring and improving human utility of free-text rationales. arXiv preprint arXiv:2305.07095.
  5. Maieutic prompting: Logically consistent reasoning with recursive explanations. arXiv preprint arXiv:2205.11822.
  6. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406.
  7. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702.
  8. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843–3857.
  9. Symbolic chain-of-thought distillation: Small models can also" think" step-by-step. arXiv preprint arXiv:2306.14050.
  10. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634.
  11. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688.
  12. Sci-cot: Leveraging large language models for enhanced knowledge distillation in small models for scientific qa. arXiv preprint arXiv:2308.04679.
  13. Receval: Evaluating reasoning chains via correctness and informativeness. arXiv preprint arXiv:2304.10703.
  14. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
  15. Distilling reasoning capabilities into smaller language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7059–7073.
  16. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021.
  17. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv preprint arXiv:2305.04388.
  18. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint arXiv:2212.10001.
  19. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048.
  20. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
  21. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.
  22. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  23. The unreliability of explanations in few-shot prompting for textual reasoning. Advances in neural information processing systems, 35:30378–30392.
  24. Verify-and-edit: A knowledge-enhanced chain-of-thought framework. arXiv preprint arXiv:2305.03268.
  25. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps.
  26. HotpotQA: A dataset for diverse, explainable multi-hop question answering.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Jooyoung Lee (48 papers)
  2. Fan Yang (877 papers)
  3. Thanh Tran (52 papers)
  4. Qian Hu (37 papers)
  5. Emre Barut (3 papers)
  6. Kai-Wei Chang (292 papers)
  7. Chengwei Su (5 papers)
Citations (6)
View on Semantic Scholar
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube