Explaining drop-and-rise dynamics during RL after SFT initialization

Investigate the causes of the observed drop-and-rise behavior in sequence length and tool-call counts during reinforcement learning when initializing from supervised fine-tuning, and establish whether it results from unlearning unsuccessful SFT behaviors before stabilizing and exploring new strategies.

Background

The training curves show an initial decrease followed by an increase in sequence length and tool usage during RL. The authors hypothesize that this may reflect an unlearning phase followed by exploration and stabilization, as observed in other domains.

A rigorous analysis would help diagnose and control training dynamics, potentially leading to better training schedules or regularization strategies for deep research agents.