Mechanism behind RL-induced increases in training prompt likelihood

Determine the specific mechanism by which Proximal Policy Optimization (PPO) post-training in the Open-Reasoner-Zero model increases the likelihood of its reinforcement learning training prompts relative to the Qwen 2.5 base model, thereby contributing to the memorization/regurgitation of training prompts despite the RL objective not explicitly optimizing sequence likelihoods.

Background

The paper measures the likelihood of PPO training prompts under the Qwen 2.5 base model and the Open-Reasoner-Zero post-trained model and finds that many prompts have higher likelihood after RL post-training. This behavior is unexpected because, unlike supervised fine-tuning, PPO does not directly optimize sequence likelihood.

Understanding the mechanism that raises prompt likelihood after RL post-training would clarify how alignment methods interact with memorization and could inform techniques to mitigate unintended regurgitation in RL-trained assistants.