Orca 2: Teaching Small Language Models How to Reason (2311.11045v2)

Abstract: Orca 1 learns from rich signals, such as explanation traces, allowing it to outperform conventional instruction-tuned models on benchmarks like BigBench Hard and AGIEval. In Orca 2, we continue exploring how improved training signals can enhance smaller LMs' reasoning abilities. Research on training small LMs has often relied on imitation learning to replicate the output of more capable models. We contend that excessive emphasis on imitation may restrict the potential of smaller models. We seek to teach small LMs to employ different solution strategies for different tasks, potentially different from the one used by the larger model. For example, while larger models might provide a direct answer to a complex task, smaller models may not have the same capacity. In Orca 2, we teach the model various reasoning techniques (step-by-step, recall then generate, recall-reason-generate, direct answer, etc.). More crucially, we aim to help the model learn to determine the most effective solution strategy for each task. We evaluate Orca 2 using a comprehensive set of 15 diverse benchmarks (corresponding to approximately 100 tasks and over 36,000 unique prompts). Orca 2 significantly surpasses models of similar size and attains performance levels similar or better to those of models 5-10x larger, as assessed on complex tasks that test advanced reasoning abilities in zero-shot settings. make Orca 2 weights publicly available at aka.ms/orca-lm to support research on the development, evaluation, and alignment of smaller LMs

• The paper demonstrates that small LMs trained with multi-strategy reasoning can perform on par with models five to ten times larger.
• It introduces a progressive learning approach combining zero-shot and few-shot phases to enhance diverse reasoning tasks such as arithmetic, grammar correction, and logic deduction.
• Empirical evaluations across 15 benchmarks show that Orca 2 not only meets but often exceeds the performance of larger models while addressing key safety and alignment challenges.

Evaluative Overview of "Orca 2: Teaching Small LLMs How to Reason"

The paper entitled "Orca 2: Teaching Small LLMs How to Reason" explores methodologies to enhance reasoning capabilities in relatively smaller LLMs (LMs), challenging the prevailing reliance on imitation learning. Traditional imitation learning often restricts the potential of smaller models by merely replicating the output of larger, more capable models. Instead, Orca 2 endeavors to instruct small LMs to identify and employ distinct solution strategies tailored to various tasks, including step-by-step reasoning, recall followed by generation, and direct answering.

Methodological Advancements

To achieve these objectives, Orca 2 employs an expertly crafted dataset sourced predominantly from the FLAN v2 collection, coupled with a meticulous technique labeled as "Prompt Erasing." This approach ensures that the smaller model assimilates multiple reasoning strategies without visibility into the prompts that originally elicited these strategies from the larger models. Moreover, Orca 2 navigates through a set of carefully interwoven training phases comprising zero-shot and few-shot datasets, enriched with both systematic instruction prompts and templates.

Orca 2's training comprises a progressive learning approach. The model starts with a basic LLaMA-2 setup, then successively integrates a blend of existing models and novel synthetic data to evolve its reasoning capacities across a broad spectrum of tasks, from arithmetic and grammar correction to more complex logical deductions.

Empirical Evaluation

Orca 2 demonstrates its prowess through rigorous testing against fifteen benchmarks spanning approximately 100 tasks. Notably, Orca 2 matches or surpasses the performance of models five to ten times larger within complex reasoning tasks in zero-shot settings. The benchmarks include AGIEval, BigBench Hard (BBH), and GSM8K, among others, where Orca 2 consistently excels. Its performance is further validated against various dimensions of LLMing tasks like language understanding (MMLU), common sense reasoning (HellaSwag), and safety and truthfulness (through datasets such as ToxiGen and TruthfulQA).

Numerical Results and Model Comparisons

The Orca 2 variants, particularly the 13B model with and without a cautious reasoning system message, frequently outperform other models of a similar parameter count, such as the LLaMA-2-Chat (13B). Impressively, they also hold their own against larger-scale models, including WizardLM and even occasionally ChatGPT. This convergence in performance while leveraging fewer computational resources underscores the potential of innovative training methodologies and strategic data synthesis.

Implications and Future Prospects

The Orca 2 framework illustrates the promise of integrating structured ethical guidelines in LM training, addressing hallucination rates critically while fostering safer model behavior. While the models have not undergone RLHF (Reinforcement Learning from Human Feedback) training explicitly, the refined synthetic datasets and the Prompt Erasing technique equip Orca 2 with enhanced grounding and alignment capabilities.

Though a significant step forward, the paper acknowledges residual challenges in addressing content harms, comprehensive safety, and the effects of inherent data biases from base model pre-training. Future research is anticipated to further refine model safety, explore more nuanced hallucination mitigation strategies, and potentially leverage reinforcement learning to amplify model reliability.

In conclusion, "Orca 2: Teaching Small LLMs How to Reason" establishes a compelling foundation for ongoing advancements in developing efficient, ethically aligned, and contextually aware small LLMs, thereby expanding their utility across diverse applications while adhering to rigorous performance standards.