Warm Up Before You Train: Unlocking General Reasoning in Resource-Constrained Settings (2505.13718v2)

Abstract: Designing effective reasoning-capable LLMs typically requires training using Reinforcement Learning with Verifiable Rewards (RLVR) or distillation with carefully curated Long Chain of Thoughts (CoT), both of which depend heavily on extensive training data. This creates a major challenge when the amount of quality training data is scarce. We propose a sample-efficient, two-stage training strategy to develop reasoning LLMs under limited supervision. In the first stage, we "warm up" the model by distilling Long CoTs from a toy domain, namely, Knights & Knaves (K&K) logic puzzles to acquire general reasoning skills. In the second stage, we apply RLVR to the warmed-up model using a limited set of target-domain examples. Our experiments demonstrate that this two-phase approach offers several benefits: $(i)$ the warmup phase alone facilitates generalized reasoning, leading to performance improvements across a range of tasks, including MATH, HumanEval$^{+}$, and MMLU-Pro; $(ii)$ When both the base model and the warmed-up model are RLVR trained on the same small dataset ($\leq100$ examples), the warmed-up model consistently outperforms the base model; $(iii)$ Warming up before RLVR training allows a model to maintain cross-domain generalizability even after training on a specific domain; $(iv)$ Introducing warmup in the pipeline improves not only accuracy but also overall sample efficiency during RLVR training. The results in this paper highlight the promise of warmup for building robust reasoning LLMs in data-scarce environments.

Overview of "Warm Up Before You Train: Unlocking General Reasoning in Resource-Constrained Settings"

LLMs with reasoning capabilities are pivotal in addressing complex problems requiring multi-step cognitive abilities. The paper "Warm Up Before You Train: Unlocking General Reasoning in Resource-Constrained Settings" presents an innovative approach to training reasoning-capable LLMs efficiently when quality training data is limited. Traditionally, developing such models involves Reinforcement Learning with Verifiable Rewards (RLVR) or distillation using curated Long Chain of Thoughts (CoT), which demand substantial domain-specific data. The authors propose a two-stage sample-efficient method that includes an initial warmup phase followed by domain-specific adaptation, significantly improving reasoning abilities and overall performance in data-scarce environments.

Two-Stage Training Framework

1. Warmup Phase: The authors introduce a novel warmup phase wherein the model is exposed to generalized reasoning behaviors using Long CoT distillation from a simplified logic puzzle domain, specifically Knights and Knaves (K{content}K) puzzles. The core objective of this phase is to instill generalizable reasoning strategies in the model without requiring extensive domain-specific knowledge. The choice of K{content}K puzzles was strategic, as these puzzles necessitate boolean logic reasoning, allowing the model to develop reasoning skills applicable across various domains.
2. Target-Domain Adaptation: Following the warmup phase, the model undergoes RLVR training using a limited number of domain-specific examples. The aim is to specialize the model's reasoning strategies rapidly under minimal supervision.

Key Findings

• The warmup phase enables significant improvements across various benchmarks with reasoning-intensive tasks. For instance, the warmed-up Qwen2.5-3B model showed marked performance gains, such as a 10.2% increase on the MATH dataset.
• The warmed-up models consistently outperformed their counterparts trained solely on domain-specific data, even when using a small dataset (≤100 examples), underscoring enhanced sample efficiency and stronger performance.
• The warmed-up model maintained cross-domain generalizability post-RLVR training, which is often compromised when models are directly trained in a specific domain, advocating the benefit of initial warmup in retaining broader reasoning abilities.

Implications and Future Direction

The two-phase approach suggests practical application in environments where acquiring vast quantities of quality domain-specific data is infeasible. The potential to rapidly adapt warmed-up models to different domains can significantly reduce computational costs and facilitate faster deployment of reasoning LLMs across varied applications. The findings also highlight the role of general reasoning patterns as transferable learning elements that can be effectively woven into the fabric of LLM training paradigms.

Further research could explore alternative simplified environments for warmup phase development, particularly those that strike a balance between task complexity and domain generalization potential. Evaluating scalability to larger models and broader application domains could yield deeper insights into the viability and adaptability of warmup strategies.

In conclusion, the paper demonstrates a promising methodology for enhancing reasoning capabilities in LLMs, offering substantial improvements in performance and generalization, while addressing constraints posed by limited training data.