Stage-specific effects prior to RL in Olmo-3–Think on CoT controllability

Determine the specific effects of supervised finetuning and direct preference optimization stages, prior to reinforcement learning, on chain-of-thought controllability in Olmo-3–Think models by isolating each stage’s causal contribution to the observed declines.

Background

The authors evaluate CoT controllability across stages of the Olmo-3–Think post-training pipeline (SFT, DPO, RL) and observe substantial changes, including a sharp drop after SFT for the 32B model and partial recovery following DPO and RL.

They note that while RL pressure reduces controllability overall, the specific impacts attributable to pre-RL stages remain unresolved and are left for future investigation.

References

Order effects in sequential post-training are well covered, we make no novelty claim, and with $n$=1 per ordering we cannot attribute what we see to order as a cause.

— Off-Target Effects of Response-Style Alignment in a Korean 27B Language Model  (2609.11291 - Han, 10 Sep 2026) in Section 5, Section 7 (RQ4: composition and order)

However, the specific impact of training stages prior to RL remains inconclusive, which we leave for future work to explore.

— Reasoning Models Struggle to Control their Chains of Thought  (2603.05706 - Yueh-Han et al., 5 Mar 2026) in Appendix, Figure “olmo_think_stages” caption