Quantify Trait Amplification vs New Learning in RLMT Gains

Determine the relative contributions of (i) amplification of pre-existing traits in the base language model and (ii) acquisition of new traits during the supervised fine-tuning warm-start and during Reinforcement Learning with Model-rewarded Thinking (RLMT) to the observed performance improvements in RLMT-trained language models. Ascertain how much each stage—SFT warm-start and RLMT—contributes to these gains to inform the design of improved post-training pipelines.

Background

The paper introduces Reinforcement Learning with Model-rewarded Thinking (RLMT), which combines long chain-of-thought generation (as in RLVR) with online reinforcement learning against a reward model (as in RLHF) to improve general-purpose chat and open-ended tasks. Across multiple backbones (Llama-3.1-8B and Qwen-2.5-7B) and algorithms (DPO, PPO, GRPO), RLMT consistently outperforms standard RLHF on chat and creative writing benchmarks.

Despite these gains, the authors highlight uncertainty about the mechanism behind the improvements: whether RLMT primarily amplifies reasoning and planning traits already present in the pretrained models, or whether it induces learning of new capabilities during the SFT warm-start and/or the RL training stages. Clarifying this decomposition would guide the design of more effective post-training pipelines and the allocation of effort across SFT versus RL.